Towards Fair Video Summarization
Authors: Anshuman Chhabra
Kartik Patwari
Chandana Kuntala
Sristi
Deepak Kumar Sharma
Prasant Mohapatra
Venue: Accepted by TMLR ()

Abstract: Automated video summarization is a vision task that aims to generate concise summaries of lengthy videos. Recent advancements in deep learning have led to highly performant video summarization models; however, there has been a lack of attention given to fairness and unbiased representation in the generated summaries. To bridge this gap, we introduce and analytically define the fair video summarization problem, and demonstrate its connections to the well-established problem of fair clustering. To facilitate fair model development, we also introduce the FairVidSum dataset, which is similar in design to state-of-the-art video summarization datasets such as TVSum and SumMe, but also includes annotations for sensitive attributes and individuals alongside frame importance scores. Finally, we propose the SumBal metric for quantifying the fairness of an outputted video summary. We conduct extensive experiments to benchmark the fairness of various state-of-the-art video summarization models. Our results highlight the need for better models that balance accuracy and fairness to ensure equitable representation and inclusion in summaries. For completeness, we also provide a novel fair-only baseline, FVS-LP, to showcase the fairness-utility gap models can improve upon.

Kartik Patwari Published in Transactions on Machine Learning Research (12/2023)
Towards Fair Video Summarization
Anshuman Chhabra
chhabro@hadaavis. edu
University of California, Davis
kpadwari@ucdavvis. edu
University of California, Davis
Chandana Kuntala"
chandanakunlakall @amaaal
Indira Gandhi Delhi Technical University for Women
Sristi'
sristi010@@@mail.coo
Indira Gandhi Delhi Technical University for Women
Dcepak Kumar Sharma
dk.sharma1 1tltarrmmn1922yyyhhoooooo com
Indira Gandhi Delhi Technical University for Women
Prasant Mohapatra
pmotapatra &rsf. edu
University of South Florida, Tampa
Reviewed OpenReview: https://
https://openrevieu/ openreview. net/ forum? id= Uy6MR
Abstract
Automated video summarization is a vision task that aims to generate concise summaries of lengthy videos. Recent advancements in deep learning have led to highly performant video summarization models; however, there has been lack of attention given to fairness and unbiased representation in the generated summarics. To bridge this gap, we introduce and analytically define the fair video summarization problem, and demonstrate its connections to the well-established problem of fair clustering. To facilitate fair model development, we also introduce the VidSun dataset, which is similar in design to state-of-the-art video summarization datasets such as TVSum and SumMe, but also includes annotations for sen- sitive attributes and individuals alongside frame importance scores. Finally, propose the SumBal metric for quantifying the fairness of outputted video sunmary. We conduct extensive experiments to benchmark the fairness of various state-of-the-ane video summa- rization models. Our results highlight the need for better models that balance accuracy and fairness to ensure equitable representation and inclusion in summarics. For completenoss, we also provide a novel fair-only baseline, FVS-LP, to showcase the fairness-utility gap models can improve upon.
1 Introduction
With the rapid growth of video content on the internet, there is an increasing need to automatically summarize lengthy videos to provide users with a condensed version that contains the most salient information. This has led to the machine learning (MI.) vision task of automated video summarization. which entails generating short, representative summary video (comprised of key-frames) of a longer input video that showcases its main content and events. In recent years, deep learning (DL) based models have achieved the state-of-the-art (SOTA) in video summarization by leveraging powerful feature representations and lcarning complex relationships between vidco frames (Apostolidis et al. 2021a). Furthermore, the vidco summarization lask itself is employed in several downstream practical applications, such as surveillance
"Equal Contribution.

---

Published in Transactions on Machine Learning Research (12/2023)
(Senthil Murugan et al. 2018: Thomas et al. 2017]. video retrieval (Gong & Liu 2003 Peng & Ngo 2006):
among others. Advancements in video summarization can then directly impact and improve performance on these downstream video analysis tasks.
The ML/DL community has also recently pivoted to studying model fairness as models can exhibit harmful biases against minority groups and individuals [Mehrabi et al. (021]. These issues of unfairness have been evidenced in many high-impact applications as well Thus, with the growing use of video summarization in numerous applications, it is extremely important to ensure that these automated methods are fair and unbiased, both at the individual-Letee (Berk et al.) 2017) and at the group-level (Dwork et al., 2012) (such as with regards to sensitive attributes like ethnicity and sex). However, no work has been undertaken in fair video summarization, while significant progress has been made in developing fair models for other tasks/fielde in ML/DL [Chhabra et al. 2021a Mehrabi et al. 2021
To bridge this gap, we propose and analytically de- fine the fuir video summarization problem, to allow for the development of fair methods at the individual- and group-level. Our fairness definition is conceptualized similar to the well-studied problem of representation-
Video #1
Video 24
Video
based fair clustering (Chierichetti et al.) 2017 Chhabra et al.) 2021ap. Another hindrance to fairness evalua- tion in summarization models stems from a lack of any video summarization datasets containing individuals, and appropriate annotations reflecting their sensitive attributes (such as sex and ethnicity). Current bench- mark datasets used to train and evaluate video summa- rization models are the TVSum (Song et al., 2015) and SumMe (Gygli et al., 2014) datasets which do not primarily contain human subjects and lack information regarding any protected groups or sensitive attributes. To this end, we propose the Fuir VidSum dataset containing multiple individuals spanning diverse settings such as interviews, podcasts, and panel discus- sions. Unlike the other datasets, we provide manual annotations for sensitive attributes (fairness) as well as frame importance scores (utility). Furthermore, we also propose novel metrics to evaluate (un) (un)ffireess in current SOTA supervised and unsupervised video summarization models and benchmark them. Finally, for completeness we also propose a novel unsupervised method for fair video summarization named FVS-LP, which is a linear program (Schrijver. 1998) based simple baseline that only optimizes for fairness. Finally, we would also like to emphasize that while video summarization has been studied extensively over the past few decades, our work primarily concerned with more recent learning- based vidco summarization approaches (proposed by the vision community) as they are highly performant (Apostolidis et al. 2021a). For more details on classical video summarization studied by the multimedia community, please refer to (Truong & Venkatesh 2007: for a survey of existing methods. In summary, through this work, we make the following contributions:
Vidco 63
Video
Video #15
Video #21
Video
Figure 1:
Video samples from Fair VidSum.
We provide an analytical definition for the fair video summarization problem to allow for the development of accurate, fair, and unbiased video summarization models that can help ensure equal representation and inclusion in video content. Frames from few randomly sampled FairVidSum videos are shown in Figure
We propose the SumBal fairness metric to evaluate model fairness, which is derived from the Balance Chierichetti et al. 2017) metric proposed to measure fairness in unsupervised learning.
We introduce the VidSum benchmark datasct, designed similarly to existing SOTA vidco summa-
rization benchmarks TVSum (Song et al. 2015) and SumMe (Gygli et al. 2014). which contains annotated individual and group-level fairness information.
Using Fair VidSum and SumBal, we benchmark numerous SOTA supervised and unsupervised models for unfairness. We find that since most models do not optimize for fairness, they can be highly unfair, prompting the need for newer methods that can balance both accuracy and fairness.
Notable examples include Microsoft's Tay chatbot that became racist and homophobic after training on user data online Neff 2016), and the COMPAS toal which recommended that black individuals were more likely to reaffend compared other ethnicities, despite statistical differences between the individuals themselves (Angwin et al. 2016).
2

---

Published in Transactions on Machine Learning Research (12/2023)
For completeness, we also propose FVS-LP: novel baseline for insupervised fair video summarization that solely optimizes for fairness, to empirically demonstrate the fairness-utility gap. Based on FVS-LP, we showcase a simple sampling strategy that can control the fairness- utilliy tradeoff as well.
2
Related Work
Classical Video Summarization. Video summarization has been studied extensively by the multimedia community over the past few decades (Truong & Venkateshi 2007). However, these approaches are not learning-based (that is, they do not employ ML/DL models to undertake the video summarization task), and tend to be less performant than learning-base based approaches. Moreover, classical approaches often utilize multiple modalities for summarization, whereas learning- baaeid approaches often consider only visual infor- mation contained in video frames. For instance, in Ma et al. summarization method for fusing visual, audio, and linguistic information contained in the video. Other seminal classical summarization approaches include Ngo et al. (2005) where the authors use temporal graph modeling methods; Taskiran et al. (2006) where speech transcripts are used instead of visual/ /audio infor- mation; Chen et al. (2009) which proposes a concept-entity method for summarization; and Yu et al. (2003) which utilizes the annotators' browsing patterns (logs) an alternative method for future summarization, among others. Note that in this work, we focus on recent learning- approaches developed by the vision community (Apostolidis et al. (221aa) since they achieve stale-of-the-ar performance on current benchmarks (TVSum and SumMe), which we discuss in more detail below.
(2002), the authors propose a simple video
Learning- Based Video Summarization. Video summarization approaches can be categorized (A (Aposto-
lidis al. 2 as eit supervised (Trame-level importance scores are used in training) (Zhang al. 2016:
2019
Huang & Wang 2019 Fu et al.
2019
Lebron Casas & Koblents
2019 or unsupervised (only visual
frame information is used during training) (Mahasseni et al.
2017
Yuan et al.
2019 He et al. 2019 Yaliniz
& Ikizler-Cinbis 2021 Apostolidis et al. 2019 Zhou et al.
2018a
with regards
to the learning setting, and
as unimodal (only visual information is used for training)
Zhang et al.
2016 He et al., 2019 Yuan et al.
2019
Fu et al.
2019
or multimodal (other video metadata is also utilized) (Otani et al. 2017 Wei et al,
2018 Zhou et al., 2018b Yuan et al., 2017 Song et al., 2016) with regards to the input data 2018 Lei et al.
type. Unsupervised unimodol approaches model the application scenarios for video summarization better as annotated frame importance scores and additional video metadata (such as transcripts) are generally hard to obtain (Apostolidis et al. 2021a). For video sunmarization, the SOTA supervised approaches constitute DSNet (Zhu et al.) (020), PGL-SUM (Apostolidis et al., 2021b), among others and the SOTA unsuper- vised approaches constitute CA-SUM (Apostolidis et al. 2022), AC-SUM-GAN (Apostolidis et al., (0200a), : SUM-GAN-AAE (Apostolidis et al. 2020b), SUM-GAN-SI (Apostolidis ct al. 2019). All these models are benchmarked on the TVSum (Song el al. 2015) and SumMe (Gygli el al. 2014 datasels.
Fairness in Machine Learning and Summarization. While video summarization has not yet been studied from the purview of fairness, fair models have been developed for various ML tasks and problem settings (Mehrabi et al. 2021 Chhabra et al. 2021a). These include supervised learning (Agarwal et al.
2018 Zafar et al. 2017). unsupervised learning (Chierichetti et al.] 2017: Chhabra et al. 2022 Kleindessner
et al. 2019b), recommendation systems (Rastegarpanah et al. 2019 Pitoura et al. 2022), active learning (Anahideh et al.) 2022 Shen et al. 2022). outlier detection (Song et al.) 2021 Davidson & Ravi 2020 among others. Works have also investigated the interplay between fairness and other desirable model
behaviors, such as robustness (Chhabra et al., 20233 20216). Further, fairness has also been studied for data summarization such for k-center based summarization (Kleineessner et al. 2019a Chiplunkar et al, 2020 Angelidakis et al. 2022) and text summarization (Shandilya et al., 2018 Keswani & Celis, 2021] However, these approaches are not general- they are highly specific to the learning algorithm being used (for e.g. k-center (Gonzalez, 1985)) and the fairness definitions employed are not consistent with our goal. As such, these are not applicable for fair video summarization. Moreover, fairness can be enforced at the pre-processing (before training), in-processing (modifying the model), or post-processing (after training) stage of the learning pipeline (Mehrabi et al., (201)). As will be clear in subsequent sections, the proposed FVS-LP baseline belongs to the in-processing category, as it enforces fairness constraints during training.
3

---

Published in Transactions on Machine Learning Research (12/2023)
3 Problem Statement and Preliminaries
In this section, we first describe the standard vidco summarization problem and discuss protocols for eval- uating utility of trained models. Then, we introduce the fair video summarization problem we provide an analytical formulation, along with evaluation metrics and motivating use-cases, Note that we only consider unimodal video summarization models in this paper as these are more commonly used in the context of deep learning [Apostolidis al. 2021a
3.1 The Video Summarization Problem
Unsupervised Video Summarization. Let a vidco V consist of n frames X [21.22g 22. Fnl where 2, E Rd. These are sampled at some frequency (usually frames per second (Apostolidis al. 2021al) from Vand hence, is generally large. Here, d is the dimension of the feature descriptor of the frame (for example, this could represent features extracted per frame using a ResNet (He et al.] 20166). ) An unsupervised video summarization model can then generally be denoted as Munsup that takes in as imput a summary length requirement l & n and the original video frame set X, and outputs a set of key frames constituting the video summary as S = = S. X. That is, MMnnuu(X, k) = S, where 5 E Rexd The summary length budget generally set to be 15% of the original video length, that is, k/n 0.15.
Supervised Video Summarization. While unsupervised variants are better suited for video summa- rization (Apostolidis et al. 2021a) since they model the application scenarios in more realistic manner (hunan-level annotations are hard to obtain), supervised models are employed as well. A supervised model also takes in as input Y = {u 1 where 0 < yı S 1 is an importance score given by a human annotator for a corresponding frame 2; E x Annotations are only obtained for a small subset of frames m since n can be quite large. Thus, for a supervised model, we can obtain a summary as "MPP( k. Y) = S where |S] = k.
Evaluating Modcls. Trained video summarization models are evaluated based on the agreement of the generated summary for a video with its ground truth summary obtained using the annotated importance scores provided by given user. Note that obtaining summaries from the importance scores Y is also an optimization problem since we have budget & for the length of the summary. Usually, the 0/1 knapsack [Martello et al. 1999) problem is used to obtain user summaries in this manner (Zhang et al., 2016 Thus, if we have u users who annolated video V, we will have summaries available denoted as O O ,...... corresponding to each user. The given model generates a summary SV for a particular video V. We can then obtain the precision and recall between cach оУ and SV, denotod as p and ry, respectively. To evaluate models, we then calculate the average pairwise Fy-measure averaged over all user summaries as:
(1+ xpX X (52 +
FV
Usually, B is set to 1 (Song et al. 2015). so we compute the average pairwise FV -measure for a given video V. These values are then averaged over all videos V the test set, and overall Fi-measure calculated. Further, note that for the supervised setting, vidcos that are used for model training cannot be used in the evaluation /test set. Hence, cross validation is generally undertaken (Apostolidis et al., 2021a to create 80% (train) 20% (test) splits. Although this issue of train-test splits does not arise for unsupervised models, for consistency, we follow the same protocol for evaluation of all models.
3.2 The Fair Video Summarization Problem
Problem Statement. We now define the fair video summarization problem for video V. Here, along with X, Y, and k, we are also given (fairness) information regarding individuals or protected groups as H = {HL H2 ,..., Hd) where HJ E {0, 13 and Hi = implies that individual/group j is present in frame i. Conversoly, Hj = 0 implics that individual/grag /group j is absent in frame i. Note that unlike importance scores these are not subjective decisions, so we have discrete labels indicating dhiiiidua//grou presence
Importance score annotations generally obtained between 1 (least important) and (most. important) and then nor- malized to lie between 0 and 1.
4

---

Published in Transactions on Machine Learning Research (12/2023)
in frames. Note that this abstraction using HD is very flexible, and can allow for the development of fair models that optimize for individual fairness or group-fairness. For individual fairness this constitutes the idea that all persons in the video should be represented in approximately the same proportions in the generated summary as they appear in the entire vidco. For group fairness, this could constitute different groups being represented in the same proportions the summary frames as their proportions the overall video frames. For example, for ethnicity as the sensitive attribute, this would necessitate proportional representation for each ethnicity in summary frames compared to total video frames. This is the very notion of disporate impact [Kleinberg et al. 2018) and ensures that no protected group or individua be adversely affected as result of predictive algorithm.
A fair video summarization model M then also takes in as input H and generates summary S for video V as M (X, k, H) = S. Along with optimal utility performance, the model must ensure that the proportion of appearance of entities represented by H are as close as possible to their overall proportions the video V. The supervised fair variant can also be defined similarly. As is evident by our dcfinition, in this work we only consider optimizing one lype of H al time (that, is, sex consisting of male/female appearances in frames). However, as our dataset has information regarding multiple groups, this can be studied in future work.
Motivating Examples. Consider a platform such as YouTube (Covington et al. 2016 For simplicity, consider a set of news/podcast videos on the platform that have one male and one female host. Summaries for these videos are generated on the platform as the user browses the homepage. Here too, if a standard summarization model is used, there is no guarantee that the outputted video summary will respect the appearance proportions of the male/femall hosts in the original video. In fact, even if the original video has equal appearance proportions for both male/female hosts, the model might skew these proportions heavily in the generated summary. A fair video summarization model instead would ensure that both male and female hosts appear in roughly the same amounts as in the original video, leading to fair representation.
Consider another example of a video surveillance application that utilizes video summarization models in the backend, such as in (Senthil Murugan et al. 2018 Thomas et al. 2017], being used by law enforcement with multiple persons appearing in the video. If a standard video summarization model is used, the generated summary footage might have certain individuals appearing for large segments of the summary and might not reflect their actual proportion of appearance in the overall video. As result, this might lead to falsified description of the original footage. On the other hand, if fair video summarization model is used, the individuals would appear in the summary in the same proportions as in the original video footage, and result in a more fair overview. The same arguments can be made with regards to people from different ethnicities or gender appearing in the summary footage and preventing discrimination and bias at the group-level.
Evaluating Unfairness. Now that we have described the fair summarization problem, it is important to propose metrics for evaluating the discrepancies in fairness. Our basic goal is to measure whether or not each entity constitating H follows the same proportions in the summary as they do in the original video. To do so, we propose the SumBal metric, which is a modified version of the Balance fairness metric generally employed in fair unsupervised lcarning tasks (Bera et al.. 2019
Chicrichetti et al. 2017):
(S,
min min { [RSS X, Hº), HRER
R(S, X, Ho)
(2)
n 11
H
where R(S,X, H9)
n ByCS
k
Here, R(S,X, H9) is the ratio of the proportion of appearances of group/individuall g in the overall video to the generated summary S. We take the minimum between R(S, XSS HD) and X, H9) to account for both under-representation and over-representation cases. Finally, SumBal returns the minimum over all groups/inder iiniiiidai and hence SumBal E [0, 1]. We can take a simple example where we have a video with two individuals, A and B. Person A appears in 20% of the video frames and Person appears in 50% of the frames, with 30% frames having no individuals. Now, assume that we generate a summary using model which has the following proportions- Person A appears in 40% of the summary frames (over-ropresentation)
For brevity, times we use the term protected groups to also refer to the set of individuals, but make it clear from context.
5

---

Published in Transactions on Machine Learning Research (12/2023)
and Person B appears in 30% of the summary frames (under- reneerrrrppesnntattin))) Then the SumBal term for Person A would be calculated as: min 0.4/0.2) = 0.5 and for Person B would be calculated as min 0.3/0.5) = 0.6. Since we take the minimum over all groups/ /nddividiaalls to calculate SumBal for the video, we get min 0.6} = 0.5 and the violating individual (with lowest fairness) is Person A.
3.3 Relating Fair Video Summarization to Fair Clustering
Given a dataset where each sample belongs to some protected group, the unsupervised task of group-level fair clustering involves partitioning samples in the dataset into k clusters according to some utility objective, while ensuring that each cluster has the same proportion of samples from each protected group as in the original dataset (Chhabra et al. 2021a Chierichetti et al. 2017 Chhabra & Mohapatra 2022: Bera et al. 2019). This is thematically similar to our notion of fairness in video summarization video frames selected in the summary should have high utility, while ensuring that each protected group is represented in the summary output in the same proportions as in the original video.
There are also significant differences between these two problems: for instance, in fair clustering the entire dataset is selected, and the number of selections correspond to & clusters where proportional fairness needs to be ensured for each cluster. However, for fair video summarization, part of the set of samples (that is, frames) are absent in the output, and & corresponds to subset of the dataset itself over which proportional fairness constraints need to be ensured. Moreover, in fair clustering, each sample is assumed to have some protected group membership, whercas video frames might have zero protected group appearances (for c.g. in the case when no individuals are present in a frame) changing the landscape of the problem considerably. Another issue stems from the value of & itself- in clustering, & is generally not very large whereas in video summarization can be order of magnitudes larger as typically set to be 15% of all video frames. is well-known that for such large k; there often clustering breakdown in cluster quality and computational efficiency (Fränti & Sieranoja] 2019; Pelleg al., 2000). These challenges make it non-trivial to utilize fair clustering for fair video summarization directly without considerable modifications. However, clustering based approaches are often employed for data summarization (Ahmed 2019), and we believe future work can exploit these connections to propose improved methods for fair video summarization as well.
4
Benchmarking Fairness Using FairVidSum
4.1 Curating Videos and Annotation Details
Collecting Vidcos. Our goal is to select vidcos that fcature multiple individuals in diverse settings that allow us to amno- tate and account for fairness information. Moreover, we wanted Fair VidSum be similar to TVSum and SumMe so that exist- ing video summarization models can utilize it in a plug-and-play manner. Similar to TVSum (Song et al. 2015), we collect videos from YouTuub (Covington et al. 2016). We use the search terms "panel discussions", "podcasts", "interviews", "debates", "news", "discussions", and combinations of these keywords. We utilized these categories mainly because our primary requirement was to collect videos with human subjects from diverse backgrounds, which these categories guaranteed. Moreover, we wanted cer- tain number of individuals (at least 2 3) in each video, which is also generally the case for these categorics. Moreover, we restrict our vidcos to ones with a Creative Commons license, that lie between 1-4 minules, and those that contain more than single shot. Using this strategy we obtain 22 videos. While the SumMe dataset (Gygli et al. 2014) has no videos that meet this criteria, TVSum has set of few vidcos (such as in the "documentaries" category) that we can use. In this manner, we also add another 12 videos from TVSum Fair VidSum and annotate them for fairness information. Thus, Fair VidSum currently has total of 34 videos, in line with current video sunmarization
Length (seconds)
400
300
200
100
10 12
16
18
Video
Figure 2: Video length distribution.

---

Published in Transactions on Machine Learning Research (12/2023)
datasets (SumMe has 25 videos and TVSum has 50 videos). We show the length distribution of these videos
in Figure
2
Video 3
Video 13
Video
Video
1000
3000
4000
I
2000
Frame
500
2000
2000
1000
Person
(a) Distribution of unique entities siaities/iidviiduulls appearing across subset of videos.
Video I
Video
Video 6
Video 19
3000
10000
2000
2000
2000
5000
Female
Male
Feraae
Female
Male
Male
Female
(b) Distribution of sez sensitive attribute across subset of videos.
Video
Video
Video 16
Video 22
110000
2000
1
f White
I
5000
I
Frame
5000
2500
0
Black White LatinX
Black
Asian
Asian
White M. Eastem
Ethnicity
(c) Distribution of ethnicily sensitive attributes across subset of videos.
Figure 3: Violin plots showcasing the distribution of individuals and protected groups sensitive attributes across randomly sampled vidcos.
Annotating Videos with Importance Scores. We follow much of the same procedure as used in TVSum (Song et al. 2015). We employ 10 annotators who consist of individuals from diverse fields in either graduate or post-graduate study. Annotators are first required to watch videos on mute in a single setting to ensure that the annotation scores are only based on visual information (Song et al. 2015). Similarly, to alleviate chronological bias (Song al.) 2015), frames are shuffled randomly. Next, obtain scores annotators are shown uniformly sampled frames 1/2 frames per second. Each annotator annotates every video and is roquired to label the provided frames with score between (least important) to (most important) to be included in the summary. This task excludes the 12 TVSum additions as those already possess annotation scores. In this manner, we obtain 15400 responses total over all videos. Note that the number of annotators employed for this purpose also satisfactory, as our annotation consistency analysis will later show.
A Annotating Videos with Fairness Information. Other than these subjective annotations for impor- tance scores, part of our dataset requires objective annotations for individuals and their sensitive attribute information. To do so we employ 4 annotators who collectively annotate all 34 videos with this information. Note that compared to annotation scores which generally obtained for subset of frames, fairness information needs to be collected for the entire video to calculate unfairness (such using the SumBal metric). Thus, here, we annotate over 168120 frames total with information regarding different individuals appearing in frames and their sensitive attributes with respect to sex and ethnicity. For ser we annotate as Male/ Female and for ethnicity we annotate for White, Black, Middle Eastern, Asian, and Hispanic.
4.2 Distribution of Individuals and Sensitive Attributes Across Videos
We aim to analyze the distribution individuals appearing across videos. For this purpose, we randomly sample vidcos out of 34, and plot the distributions of individuals as well as the distributions of sez and ethnicity protected groups in those vidcos as a function of their video frames using violin plots. These are visualized in Figure The distributions for all the remaining videos are shown in Appendix A due to space

---

Published in Transactions on Machine Learning Research (12/2023)
constraints. It is evident that both the number well frame-level distribution of individuals and protected groups varies widely across videos. This demonstrates one of the challenges associated with developing fair summarization models, as they need to be able to account for fairness in many diverse settings.
We also analyze group-level information for each video as a function the annotation trend. Here, we can visualize the mean importance score for a video as a function of the frame indices, while also denoting sensitive attribute information for the framess We demonstrate this for Video 19 and set the protected group in Figure 4a and for Video 16 with ethnicity as the protected group in Figure 4b] It can be seen that group-level information varies widely, and there is little correlation between importance scores and group-level information that would allow existing models to be fair.
1.0
Meen Inpertanee Scores
0.8
0.2
0.0
Female
Male
Male Female
Moan Importance Scares
0.6
1.66
0.2
0.0
(b)
Figure 4: Mean importance (annotated) scores for (a) Video 19 with protected group labels for ser and (b)
for Video 16 with protected group labels for ethnicity.
4.3 Annotator Consistency
We now cover another aspect of our dataset- the annotations, and their consistency. Annotator consistency with respect to vidoo summarization usually measured using the Cronbach's alpha (CA) (Cronbach) 1951) A higher CA value indicates more consistency among annotations. For VidSum, the CA value 0.995. This is much higher than both SumMe (CA=0.74) and TVSum (CA=0.81). We posit that this is because 1) FiiSSidSum has a lot of intra-video homogeneity (a byproduct of our initial search terms) compared to both TVSum and SumMe that have more diversity in video categories, and 2) the visual complexity of certain TVSum/Summe videos over FairVidSmm results more variance in annolation (for e.g. TVSum has some challenging videos in categories such as Dog Show and Flash Mob Gathering with rapid scene changes).
Annotator consistency can also be observed qualitatively for a given video. We can visualize this as heatmap with as individual annotators, columns respective video frames, and each cell thus representing the
8

---

Published in Transactions on Machine Learning Research (12/2023)
annotator's importance score for that frame. We show this in Figure 5 for Video 19. It can be seen that for
most frames, annotators agree on similar importance scores.
4.4 Discussion on Limitations
Tt is important to note that number of possible improvements can be made to FairVidSum by analyzing and alleviating some of its current limitations.
2 2 8 7
For instance, a current limitation of the dataset is the set of categorics that
Annotator ID
the videos are sourced from ("panel discussions", 'interviews', 'debates',
0.6
'news", among others). Videos from these categories often have repetitive frames and at least few consistent individuals appear throughout the video. While these restricted source domain videos are useful for studying the fair video summarization problem as first step, there are many other video categorics that also might require fairness enforcement, such as those with large number of individuals (> 1000) appearing in them for a very short du- ration time (such as concert recordings, comedy shows, or large lectures). The challenges with enforcing fairness such setting are manifold: an- notating fairness information due the large number of individuals would be non-trivial task and the large growth in 9 (the number of groups/individie /idiiiidvid would possibly lead to resulting in marginal SumBal scores. For the latter,
0.4
0.2
40
60
Frame Index
Figure 5: Annotator consis-
tency matrix for Video 19.
small overall proportions for each group/indiviin since SumBal scores might tend to zero with such large scale, new fairness evaluation metrics would need to be proposed as well. Moreover, incorporating videos from other source categories, such as those with rapid scene changes (for example sports sequences or movic montages), might lead to other unforseen challenges as well, both data curation/ auration/annotation /anooaaiio and in fairness enforcement. We defer the study and analysis of such problems future work.
5
The FVS-LP Fair-Only Baseline
5.1
FVS-LP
We present baseline for fair video summarization the Fair Video Summarization Linear Program (FVS-LP) which is simple lincar program (Schrijver 1998) approach that only optimizes for fairness and selects frames such that the group proportions in the selected summary are as close as possible to the group proportions the overall video. Note that this contribution is analogous to having a constant predictor in fair classification (Mehrabi et al.] 2021]- as predicts a constant will always achieve maximum fairness, but low atility/accuraae However, in fair video summarization, even such a simple fair-only bascline is not conceptually straightforward as a constant,/annee random predictor, and hence, we propose FVS-LP to showcase the fairness-utility tradeoff gap.
Let Om and 1m denote an m length vector of all zeros and all ones, respectively. We have given video V and its set of frames X, along with the set of group memberships H. First, we transform H to matrix form for formulating the LP. Let G {0, 1]axg be derived from H such that each row vector G, E {0, 139, i E [n] represents a frame and each of its entries are either 0 for absence or for presence of group in the frame. Let S I be the optimization variable where each entry of x E R. indicates a frame is selected in the summary, then the LP can be written Equation 3
minimize
0, x
s.t.
g x k
g,
(3)
1 x = k
< x 1
Note that since we are only optimizing for fairness, we do not care about utility and our optimization objective can simply be vector of all zeros. Now, is evident, the first constraint simply ensures that the sum of the
9

---

Published in Transactions on Machine Learning Research (12/2023)
Table 1: Comparison of SOTA video summarization approaches on Fair VidSum. The utility and fairness averages are calculated across all five splits. The violating groups that achieve the minimum fairness SumBal scores are also presented. Results on FVS-LP (our fairness baseline) along with Random and Human baselines are also provided. Blue/red indicates righest/lowestt performance.
Average
Sumaaal (Sex)
Sumiaa (Atheirctt)
SumBal (Indisidaal)
Model
Type
Messuue
Average
Min
Vidlating
Average
Min
Violeting
Average
Min
Violating
Randlmm
14,32
0.9457
0.8814
Mule (Vil. +:0)
0.9468
0.0070
Anion [Vid. +33)
0.8747
0.9670
Parase: 15 (Vil. #-33
Human
68.91
0,4805
0,0000
Pemale (Vid. 4300
0.5505
0.00000
Asian [Vid. +255
0,2773
0,0000
Person 2 (Vid. 418)
Unsmpervtzed
62.78
0.5201
0,0000
Famale (Vid. 430)
0.5468
0.0000
Asion [Vid. 424)
0.2441
0.0000
Person (Vid. 45)
AC-SUM-GAN
Uneuperriied
64.33
0.5176
0.0000
Remale (Vid. #300
0.5455
0.0000
Asion [Vid. +24)
0.2616
0.0000
Person (V9.. #8)
SUM GAN GAN-AAE AAB
Uamupevvaed
63.81
0.5222
0.1322
Pemate (Vid. 4226
0.5063
0.0000
Asian (Vid. +24
0.2730
0.0001
Person 4 (Vid. 4-)
SUM-GAN-ST.
Unempervieed
64.92
0.5254
0.0000
Female (Vid. (0300)
0.5661
0.0000
Asion [Vid.
0.2550
0.0000
Person (VHt. (88)
SUM-IND
Ussupevised
50.57
0,0077
0,0000
Remale (Vid. *224
0.5889
0.0000
Asian [Vid.
+224
0.2541
11000
Person (VSL. 4-5
DSNet
Supervised
63.09
0,5558
0,0000
Female (Vid. #9300
0.5478
0 0.0000
Asion (Vid.
#224
0,2706
0.0000
Person 7 (Vid. #26)
VASNet
Supervied
0.4622
0.0000
Rande (Vid. *25)
0.5291
0.00000
Asiun (Vid.
+24]
0.2536
0.0000
Punoo (VGL. #8)
PGL
Supervised
63.75
0,4804
0.1042
Famale (Vid. 434)
0.0000
Asion [Vid. 4224
0.2575
Person (Vid. 45)
IVS LP (Ser)
Unmpervtaed
15.89
0.9987
0.9960
Female (Vid. #-4
0.7411
0.0000
Hepanic (V5d.48)
0.3062
0.0000
Person (Vid. #5)
FVS-LP (Kilooictg)
Unsuperiied
13.46
0.0942
0.0000
Ramule (Vid. #25)
0,9080
0.9822
Asion (Vid. +33)
0.2727
0.0000
Prrwon 6 (Vil. #25)
LV (Jndiriduel)
Uampervised
14.13
0,9556
0,6289
Male (Vsd. 428)
0.9471
0.6539
Whste (Vid. #119
0.9932
0.9704
Person 6 (Vid. #25)
selected samples' group memberships is equal to l times the group proportions for the overall video. The second coistraint. ensures that the number of selected samples must be exactly k. After solving the above LP for x, we can obtain the indices of summary frames selected from the set of frames X by rounding the solution, as I = {i round(x.) = 1}. Then, we can get the summary S of video V as S = (X; Vi E 1}.
5.2 Mixing Sampling Strategy for Controlling Fairness- Utiilty Tradeoff
Since FVS-LP can be utilized to obtain fair summaries and current SOTA video summarization models can output summaries that have high utility, we propose simple sampling strategy that mixes frames together from both of these to control the fairness-accuracy tradeoff.
We start with two distinct summaries: Sace, generated from an existing model that optimizes for accuracy, and Stair generated using the FVS-LP baseline for fairness. To produce a summary that harmoniously blends both these objectives, we introduce the mixing strategy as follows. A mixing ratio, denoted as X, determines the proportion of frames from the fairness-optimized summary Star that will be integrated into the cccuracy-optimized summary Sace. Specifically, for a given X, we randomly select À |Sacel frames from Staie- These selected frames are then incorporated into Sace by randomly substituting an equal number of included frames. Frames can then be sorted using timestamps. This procedure also ensures that the merged summary maintains the original summary length. Through this mixing sampling strategy, the resultant summary strikes a balance between the characteristics of accuracy and fairness.
6 Results
We now present results for benchmarking SOTA supervised and unsupervised models on Fair VidSum. We utilize the following unsupervised models: CA-SUM (Apostolidis et al.) 2022), AC-SUM-GAN (Aposto-
lidis et al. 2020a), SUM-GAN-AAR (Apostolidis et al. 2020b)), SUM-GAN-ST. (Apostolidis et al. 2019
SUM-IND Yaliniz & Ikizler- Cinbis, 2021) and the following supervised models: DSNct (Zhu et al. 2020
VASNet (Fajtl et al., 2 PGL-SUM (Apostolidis al.] 2021b). Moreover, we also provide baseline re- sults for a randomly generated summary (Random) and summary generated using the knapsack algorithm on the average human annotated importance scores (Human). Finally, we also present results for FVS-LP while optimizing for each protected group type (individual, sex, and ethnicity). For each model/baseline. we provide the group members that achieve the minimum fairness values as well. Code, reprodueibility, and miscellaneous dataset details are provided in Appendix D
Training and Evaluation. We follow the standard evaluation procedure in existing video summarization literature, which involves randomly splitting the entire dataset into multiple parts or splits, typically 5,
10

---

Published in Transactions on Machine Learning Research (12/2023)
Table 2: Comparison of SOTA video summarization model on Fair VidSum for evaluation Split #1.
Aveeage
Semmal (Ses)
SumBal (Bhhnicity)
SumBal (Jedroidual)
Model
Type
F; Mensure
Average
Min
Violating
Average
Min
Violating
Average
Min
Violatnn
Random
14.74
0.9473
0:8818
Made (Vid. *26)
0.97
0.9415
Whithe (Vid. #30
0,5039
0.7477
Person (Va1. #10)
Human
67.08
0,6500
0.0000
Femole (Vid. #25)
0.6717
0.0000
Asion (Vid. 425)
0.3080
0.0000
Person (V5d. 425)
CA-SUM
Unsupervised
11
0.5104
Female (Vid. #34
0.6746
0,4292
Asion (Vid. #9)
0.3012
0.0000
Person (V5d. 825)
AC-SUM-AAN
Unsuperr Sad
63.99
0.4481
0.00000
Femule (Vil. #22)
0.6467
0.0000
Asion (Vil. +25
0.3261
0.0000
Pesson (VK1. #25)
SUM-CAN-LAE
Unsupervise leed
63.44
0.4887
0.13
Female (Vál. #*a6
0.0021
0.4667
Asicu (Vil. +9)
0.2802
0.000
Person (Va), +26)
SUM GAN SL
Unsupeevizod
64.77
0.5256
0,2156
Male (Vid. 4226
0.7056
0.4567
Asien (Vid. 4.9
0.2792
0.0000
Person (V5d. 426)
SUMLIND
Unsuperrised
49.47
0.5415
0.3958
Male (Vid. +26)
0.6641
0.4533
Anizo (Vid. +9)
0.3018
0.000
Person (VG). +25)
TINNat
Supervied
0.4452
0.19
Pamale (Vid. #331)
0.8085
0.2742
Awion (Vid.. +25)
0.3040
0.0000
Person (VG.. (35)
VASNet
Supervised
06.14
0.3386
0.0000
Pemale (Vid. #225
0,5886
0.0000
Asion (Vié. 425)
0.2500
0.0000
Person (Vid. 425)
PGL-SUM
Supervised
65.18
0.4302
0.101
Female (Vvt. #344)
0.5880
0.2631
Avion (Vid. #34)
0.2701
0.0000
Person (V9). 825)
TVS-LP (8ex)
Uneepervised
17:00
0.9990
0.9975
Female (Vid. #25)
0.6357
0.6359
White (Vid. #919)
0.4417
0.0000
Person 3 (Vvd. (19)
TVS-LP (Riimiiiit)
Unsepervised
12.13
0.5067
0.0000
Pemale (Vid, #25)
0.9983
0.9983
Anion (Vid. #25)
0.3:86
0.0000
Person (V5), #10)
FVS-LP (Judivsbuad)
Ussupervsodd
0,3502
Male (Vid. +24)
0.836M
Wldte (Vid. #119
0.9930
0.9704
Person (Vk). #25)
each split subjected to an 80:20 train/test partitioning (Apostolidis et al.) 2022 2020a b Zhu et al.; 2020;
Fajtl et al.] 2019 Apostolidis et al., 2021b Kanafani et al,, 2021). The models are trained on the training set of given split and subsequently evaluated the corresponding test set within the same split. A detailed breakdown of the video distribution for all 5 train/test splits are present in the Appendix [ The Fi-measure evaluates the similarity between a model predicted summary and a user-defined summary by assessing their overlap. The F1 scores are calculated for each individual video and then averaged over the entirety of a given split which are then averaged across all 5 splits. SumBal is also evaluated per vidco, and the same procedure is followed to obtain averages.
Details on Model Training. We downsample videos to 1/2 frames per second as our video frames are often repetitive. Following prior work, we utilize GoogleNet (Szegedy et al., 2015) (trained on ImageNet) to extract frame features from the pool5 layer, which outputs a dimensionality of 1024. When training the various models, we adhere to their original procedures, and generally employ default settings and hyperparameters. Any altcrations or adjustments to the default training parameters are detailed in Appendix To ensure a fair comparison, we employ the same splits for baaiiing/tettnngg across all models.
Fair VidSum Benchmarking Results. Since we have 5 evaluation splits, we present average results over all splits in Table [ We also present results for Split #1 in Table 2 and for the remaining evaluation splits in Appendix due to space constraints. As can be seen in Tables D and 2 both unsupervised and supervised SOTA models tend to achieve high utility performance computed in terms of the F1-measure (> 60) averaged over all videos and all splits 4 However, these models have low fairness performance, with minimum SunBal scores for all three group types: sex, elhmicily, and individuals most often tend to be 0 and generally < 0.5. Interestingly, the group members that achieve the lowest fairness values across all splits and videos tend to consistently be Female for ser as the protected group, Asian for ethnicity as the protected group; and Person for individual fairness. We also present average SumBal scores which are higher at times, but have very large variance showcasing that models are not inherently optimizing for fairness. The human annotated summary also fares similarly to the SOTA models, as it is only annotated for performance. Moreover, the randomly generated summary has very low utility performance scores- typically with Fi-meauure values less than 15 which follows the fact that summary frames are picked completely at random. However, the random summary has high fairness scores. We hypothesize that this is the case because by picking frames uniformly at random, the probability that cach group member is picked according to their proportions is uniform in expectation. As a result, random frame selection leads to improved fairness.
Furthermore, we provide results for three versions of FVS-LP, each instantiated to optimize one type of protected group/sensitive altribute. For each of these, FVS-LP achieves the highest fairness performance across all models and baselines for the group it is optimizing for. However, it does not lead to good utility performance, which is to be expected as it is only directly optimizing for fairness. This implics that while
4This utility performance in line with SOTA results for TVSum and SumMe; refer to (ong et al. 2015 A Apostolidis al.
2021a) for details.
11

---

Published in Transactions on Machine Learning Research (12/2023)
Ethnicity
Sex
Individual
60-
F1-Meesure
50
50
50
40
40
30
30
30
20
20
20
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
1.0
1.0
0.8
0.8
SumBal (Min)
0.6
0.6
0.6
0.4
0.2
0.2
0.0
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Mixing Ratio
CA-SUM
SUM-GAN-AAE
SUM-IND DSNET
VASNET
AC-SUM-GAN
SUM-GAN-SL
PGL-SUM
Figure 6: Results for the mixing sampling strategy averaged over all splits. The top row demonstrates the effect of increasing mixing ratio (A) on utility (Fi Measure) whereas the bottom row showcases the effect on fairness (SumBal). Each column represents mixing done for particular protected group (ethnic- ity seexiiniiiddul) X is varied in increments of 0.1 from 0 to 1.
there is a gap in fairness that can be optimized for, optimizing for both fairness and performance is non-trivial task. For future work, methods that jointly optimize both fairness and utility can thus be proposed. Note that the trends between the average performance and Split #1 are very similar, and this is also the case for the other evaluation splits (Appendix Cp. Generally, we observe that supervised models tend to exhibit lower average SumBal values. This trend might. be a direct consequence of these models' learning process, which strives to closely align with human or ground truth summaries that, as previously mentioned, are solely optimized for utility. This observation further underscores the importance of incorporating fairness evaluation and learning criterion in the model design and training process. Another crucial insight from our benchmarking analysis is the distinct difficulty in upholding individual fairness. This is clearly evident by the consistently lowest average SumBal values (compared with sex and ethnicily) and predominant minimum values of zero. A SumBal value of zero essentially indicates that group individual, though present in the original video, has been completely excluded from the generated summary.
Mixing Strategy Results to Showcase the Pairness-Utility Tradeoff. We now presents results for the mixing sampling strategy based on FVS-LP described in Section 5.2 Results averaged over all splits are shown in Figure and for each of the individual splits in Appendix C.2 In Figure 0 each column indicates mixing undertaken for particular protected group: ethnicity, scx, and individuals. Each SOTA model whose original utility summary the fair summary (FVS-LP for the iculir protected group) is mixed with is shown as an individual line. The top row of the figure showcases utility results (Fi Measure) and the bottom row denotes fairness (minimun value of SumBal obtained). It can be seen that as the mixing ratio A is increased from 0 to 1, in increments of 0.1, the utility starts to decrease and fairness increases. Thus, X can be used as utiiess-utillity tradeoff for balancing fairness and utility. Results for the individual splits in Appendix C.2 exhibit similar trends.
Discussion Results. Video summarization models designed to extract the most representative and salient content from videos, emphasizing features like motion, distinct objects, and long-range temporal sequences. Current models use mechanisms such as self-attention (Vaswani et al. 2017), GANs (Goodlelow let al. 2020), and LSTMs (Hochreiter & Schnidhuber 1997 to often drive this raction process, seeking to replicate human-like summaries that focus on pivotal moments and high-saliencey content.
We believe that the sole focus on saliency results in the fairness issues exhibited by these models. Models may prioritize certain segments or individuals that align with their understanding of importance, which is
12

---

Published in Transactions on Machine Learning Research (12/2023)
often derived from visual and temporal cues. For instance, high-contrast motion-intensive sequences might overshadow static, less visually distinctive moments, regardless of the duration frequency of appearance of entities within those moments. Furthermore, most, if not all, of the individual components (GANs, LSTMs, transformers, etc.) used by models have been shown to exhibit bias and unfairness in prior rescarch (refer to [Kenfack et al. 2021 Sun et al., 2021 Qiang et al. 2023 for more details).
Moreover, human annotations, which serve as ground truths for models, might themselves be biased. Human annotators may not be focused representative fairness, but only capturing what they perceive as the most noteworthy moments, causing models to adopt similar biases. This is evidenced by the slightly lower average SumBal scores for fully supervised models which largely rely on ground truth annotations.
To summarize: fairness aims for proportional representation of entities or sensitive altributes while utility focuses on capturing the most informative and representative moments guided by visual and temporal cues, Thus, we believe that models optimized for accuracy may inadvertently neglect frequently occurring yet less visually impactful segments and vice versa.
7 Conclusion
In this paper, we propose the fair video summarization problem which has thematic connections to the problem of fair clustering. We introduce the Fair VidSum dataset, which consists of videos with annotations of frame-level importance scores, appearance of individuals across frames, as well as information regarding their sensitive attributes such as sex and ethnicity. We also propose the SumBal metric, which measures the disparity in fairness of the generated summary with regards to the original video. Through Fair VidSum and SumBall we benchmark a number of existing SOTA video summarization models and find that these generate highly unfair summaries as they do not directly optimize for fairness. Finally, we also propose the FVS-LP method which is a linear programming baseline optimized only for fairness, analogous to constant predictor in fair classification.
Our paper constitutes the first work on fair video summarization, and hence, impacts the communily in numerous w It is important to ensure that learning algorithms account for equitable representation. Through this work, we seek to bridge this gap for the task vidco summarization. Furthermore, there are multiple avenues for future work. Better fair video summarization models can be developed using Fair VidSSu, which improve upon both accuracy and fairness. Pre- and post-processing fair approaches can also be proposed. Methods that optimize for multiple protected groups jointly can also be investigated. Novel fairness definitions can also be proposed, similar to the trend observed in the fair clustering community CChaaba et al. 2021a).
References
Alekh Agarwal, Alina Beygelzime, Miroslav Dudík, John Langford, and Hanna Wallach. A reductions approach to fair classification. In International Conference on Machine Learning, pp. 60-69. PMLR, 2018.
Mohiuddin Ahmed. Data summarization: survey. Knowledge and Information Systems, 58(2):249-22 273, 2019.
Hadis Anahideh, Abolfazl Asudeh, and Saravanan Thirumuruganathan. Fair active learning. Expert Systems with Applications, 199:116981, 2022.
Haris Angelidakis, Adam Kurpisz, Leon Sering, and Rico Zenklusen. Fair and fast k-center clustering for data summarization. In International Conference on Machine Learning, pp. 669-702. PMLR, 2022.
Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. Machine bias. In Ethics of Dota and Analytics, pp. 254 264. Auerbach Publications, 2016.
Evlampios Apostolidis, Alexandros 1 Metsai, Eleni Adamantidou, Vasileios Mezaris, and loannis Patras. A stepwise, label-based approach for improving the adversarial training in unsupervised video summarization.
5 Additional information broader impact and ethics is provided Appendix F
13

---

Published in Transactions on Machine Learning Research (12/2023)
In Proceedings of the 1st International Workshop AI for Smart TV Content Production, Access and Delivery, pp. 17-25, 2019.
Evlampios Apostolidis, Eleni Adamantidou, Alexandros I Metsai, Vasileios Mezaris, and loannis Patras. Ac-sum-gan: Connecting actor-critic and generative adversarial networks for unsupervised video summa- rization. IEEE Transactions on Circuits and Systems for Video Technology, 31(8):3278 3292, 2020a.
Evlampios Apostolidis, Eleni Adamantidou, Alexandros I Mctsai, Vasilcios Mczaris, and Ioannis Patras. Unsupervised video summarization ion via altention-driven adversarial learning. In MulliMedia Modeling: 26th International Conference, MMM 2020, Daejeon, South Korea, January 5 8, 2020, Proceedings, Part I 26, pp. 492-504. Springer, 2020b.
Evlampios Apostolidis, Eleni Adamantidou, Alexandros 1 Metsai, Vasileios Mezaris, and loannis Patras. Video summarization using deep neural networks: A survey. Proceedings of the IEEE, (09(11):1838-186) 2021a.
Evlampios Apostolidis, Georgios Balaouras, Vasileios Mezaris, and Ioannis Patras. Combining global and local attention with positional encoding for video summarization. In 2021 IEEE International Symposium on Multimedia (ISM), pp. 226-234. IEEE, 2021b.
Evlampios Apostolidis, Georgios Balaouras, Vasileios Mezaris, and loannis Patras. Summarizing videos using concentrated attention and considering the uniqueness and diversity of the video frames. In Proceedings of the 2022 International Conference on Multimedia Retrieval, pp. 407-415, 2022.
Suman Bera, Deeparnab Chakrabarty, Nicolas Flores, and Maryam Negahbani. Fair algorithms for clustering. Advances in Newral Information Processing Systems, 32, 2019.
Richard Berk, Hoda Heidari, Shahin Jabbari, Matthew Joseph, Michael Kearns, Jamie Morgenstern, Seth Neel, and Aaron Roth. A convex framework for fair regression. arXiv preprint orXiv:17060 02000, 2017.
Bo-Wei Chen, Jia-Ching Wang, and Jhing-Pa Fa. Wang. A novel video summarization based mining the story-structure and semantic relations among concept entities. IEEE Transactions on Multimedia, 11(2): 295-312, 2009.
Anshuman Chhabra and Prasant Mohapatra. Fair algorithms for hierarchical agglomerative clustering. In 2022 21st IEEE International Conference Machine Learning and Applications (ICMLA), pp. 206-211. IEEE, 2022.
Anshuman Chhabra, Karina Masalkovaité, and Prasant Mohapatra. An overview fairness in clustering. IEEE Access, 9:130698 130720,, 2021a.
Anshuman Chhabra, Adish Singla, and Prasant Mohapatra. Fairness degrading adversarial attacks against clustering algorithms. arXiv preprint arXv:22110.10000 2021b.
Anshuman Chhabra, Adish Singla, and Prasant Mohapatra. Fair clustering using antidote data. In Algo- rithmicc Fairness through the Lens of Causality and Robustness Workshop, pp. 19 39. PMLR, 2022.
Anshuman Chhabra, Peizhao Li, Prasant Mohapatra, and Hongfu Liu. Robust fair clustering: A novel fairness attack and defense framework. In The Fleventh International Conference on Learwing Represen- tations, 2023.
Flavio Chierichotti, Ravi Kumar, Silvio Lattanzi, and Sergei Vassilvitskii. Fair clustering through fairlets. Advances in Newral Information Processing Systems, 30, 2017.
Ashish Chiplunkar, Sagar Kalc, and Sivaramakrishnan Natarajan Ramamoorthy. How to solve fair k-center in massive data models. In International Conference on Machine Learning, pp.
PMLR, 2020.
Paul Covington, Jay Adams, and Emre Sargin. Deep neural networks for youtube recommendations. Proceedings of the 10th ACM Conference Recommender Systems, pp. 191-198, 2016.
14

---

Published in Transactions on Machine Learning Research (12/2023)
Lee Cronbach. Coefficient alpha and the internal structure of tests. Psychometrika, 16(3):297-33) 1951.
lan Davidson and Selvan Suntiha Ravi. A framework for determining the fairness of outlier detection. ECAI 2020, pp. 2465 2472. IOS Press, 2020.
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Proceedings of the 3rd Innovalions in Theoretical Computer Science Conference, pp. 214- 226, 2012.
Jiri Fajtl, Hajar Sadeghi Sokch, Vasileios Argyriou, Dorothy Monekosso, and Paolo Remagnino. Summarizing vidcos with attention. In Computer Asiv 2018 Workshops: 14th Asian Conference on Computer Vision, Perth, Australia, December 2 2- 2018, Revised Selected Papers 14, pp. 39 54. Springer, 2019.
Pasi Fränti and Sami Sieranoja. How much k-means be improved by using better initialization and repeats? Pallern Recognilion, 93:95 112, 2019.
Tsu-Jui Fu, Shao-Heng Tai, and Hwann-Tzong Chen. Attentive and adversarial learning for video summa- rization. 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1579 1587. IEEE, 2019.
Yihong Gong and Xin Liu. Video summarization and retricval using singular value decomposition. Multimedia Systems, 92):115 168, 2003.
Teofilo F Gonzalez. Clustering to minimize the maximum intercluster distance. Theoretical Computer Science, 38:293-306, 1985.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Couvvile, and Yoshua Bengio. Generative adversarial networks. Commumications of the ACM, 63(11):
139-144, 2020.
Michael Gygli, Helmut Grabner, Hayko Riemenschneider, and Luc Van Gool. Creating summaries from user videos. In Computer Vision ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VII 13, pp. 505-520. Springer, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pallern Recognilion, pp. 770-778 2016.
Xufeng He, Yang Hua, Tao Song, Zongpu Zhang, Zhengui Xue, Ruhui Ma, Neil Robertson, and Haibing Guan. Unsupervised video summarization with attentive conditional generative adversarial networks. In Proceedings of the 27th ACM International Conference on Multimedia, pp. 2296-23020 2019.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Newral Computation, 9(8):1735 ((8):1735-1780, 1997.
Cheng Huang and Hongmei Wang. novel key-frames selection framework for comprchensive vidco sum- marization. IEEE Transactions on Cirewils and Systems for Video Techmology, 30(2):577 589, 2019.
Hussain Kanafani, Junaid Ahmed Ghauri, Sherzod Hakimov, and Ralph Ewerth. Unsupervised vidco sum- marization via multi-source features. In Proceedings of the 2021 International Conference on Multimedia Retrieval, pp. 466-470, 2021.
Patrik Joslin Kenfack, Daniil Dmitrievich Arapov, Rasheed Hussain, SM Ahsan Kazmi, and Adil Khan. On the fairness of generative adversarial networks (gans). In 2021 International Conference" Nonlinearity, Information and Robotics ( pp. 1-7. IEEE, 2021.
Vijay Keswani and L Elisa Celis. Dialect diversity in text summarization twitter. In Proceedings of the Web Conference 2021, pp. 3802-3814, 2021.
Jon Kleinberg, Jens Ludwig, Sendhil Mullainathan, and Cass R. Sunstein. Discrimination the age of algorithms. Journal of Legal Analysis, 10:113 174, 2018.
15

---

Published in Transactions on Machine Learning Research (12/2023)
Matthäus Kleindessner, Pranjal Awasthi, and Jamie Morgenstern. Fair k-center clustering for data summa- rization. In International Conference on Machine Learning, pp.
PMLR, 2019a.
Matthäus Kleindessner, Samira Samadi, Pranjal Awasthi, and Jamie Morgenstern. Guarantees for spectral clustering with fairness constraints. In International Conference on Machine Learning, pp. 1458-3467. PMLR, 2019b.
Luis Lebron Casas and Eugenia Koblents. Video summarization with LSTM and deep attention models. In MultiMedia Modeling: 25th International Conference, MMM 2019, Thessoloniki, Greece, January 8- 8-11, 2019, Proceedings, Part II 25, pp- 67-79. Springer, 2019.
Jie Lei, Qiao Luan, Xinhui Song, Xiao Liu, Dapeng Tao, and Mingli Song. Action parsing-driven video summarization based on reinforcement learning. IEEE Transactions on Cireuits and Systems for Video Technology, (7())2112 22122226277 2018.
Yu-Fei Ma, Lie Lu, Hong-Jiang Zhang, and Mingjing Li. A user attention model for video summarization. Proceedings of the tenth ACM International Conference on Multimedia, pp. 533 333-542, 2002.
Behrooz Mahasseni, Michael Lam, and Sinisa Todorovic. Unsupervised video summarization with adversarial LSTM networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition,
pp. 202-2112 2017.
Silvano Martello, David Pisinger, and Paolo Toth. Dynamic programming and strong bounds for the 0-1 knapsack problem. Management Science, 45(3):414-P5) 424, 1999.
Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR), 54(6):11335, 2021.
Gina Neff. Talking to bots: Symbiotic agency and the case of Tay. International Journal of Communication, 2016.
Chong- Wah Ngo, Yu-Fei Ma, and Hong-Jiang Zhang. Video summarization and scene detection by graph modeling. IEEE Transactions on Circuits and Systems for Video Technology, 15(2):296 305, 2005.
Mayu Otani, Yuta Nakashima, Esa Rahtu, Janne Heikkilä, and Naokazu Yokoya, Video summarization using deep semantic features. In Computer Vision-ACCV 2016: 19th Asian Conference on Computer Vision, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part V 13, pp. 361-377. Springer, 2017.
Dan Pelleg, Andrew W Moore, et al. X-means: Extending k-means with efficient estimation of the number of clusters. In International Conference Machine Learning, volume 1, pp. 727-734, 2000.
Yuxin Peng and Chong- Wah Ngo. Clip-based similarity measure for query-dependent clip retrieval and video summarization. IEEE Transactions on Circuits and Systems for Video Technology, 16(5):612-6)) 627, 2006.
Evaggelia Pitoura, Kostas Stefanidis, and Georgia Koutrika. Fairness in rankings and recommendations: an overview. The VLDB Journal, pp. 1-28, 2022.
Yao Qiang, Chengyin Li, Prashant Khanduri, and Dongxiao Zhu. Fairness-awaro vision transformer via debiased self-attention. Xrxio preprint ar 2rx::23001.38000 2023.
Bashir Rastegarpanah, Krishna P Gummadi, and Mark Crovella. Fighting fire with fire: Using antidote data to improve polarization and fairness of recommender systems. In Proceedings of the 12th ACM International Conference on Web Search and Data Mining, pp. 231-239, 2019.
Alexander Schrijver. Theory of Linear and Integer Programming. John Wiley & Sons, 1998.
A Senthil Murugan, K Suganya Devi, A Sivaranjani, and P Srinivasan. A study on various methods used for video summarization and moving object detection for video surveilancce applications. Multimedia Tools and Applications, 77(18):23273 23290, 2018.
16

---

Published in Transactions on Machine Learning Research (12/2023)
Amurag Shandilya, Kripabandhu Ghosh, and Saptarshi Ghosh. Fairness of extractive text summarization. In Companion Proceedings of the The Web Conference 2018, pp. 97-98, 2018.
Jie Shen, Nan Cui, and Jing Wang. Metric-fair active learning. In International Conference on Machine Learning, pp. 19809-19826. PMLR, 2022.
Hanyu Song, Peizhao Li, and Hongfu Liu. Deep clustering based fair outlier detection. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 1481-1489, 2021.
Xinhui Song, Ke Chen, Jie Lei, Li Sun, Zhiyuan Wang, Lei Xie, and Mingli Song. Category driven deep recurrent neural network for video summarization. In 2016 IEEE International Conference on Multimedia & Eapo Workshops (TCMEW), pp. 1-6. TEFF, 2016.
Yale Song, Jordi Vallmitjana, Amanda Stent, and Alejandro Jaimes. Tvsum: Summarizing web videos using titles. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5179 5187, 2015.
Bing Sun, Jun Sun, Ting Dai, and Lijun Zhang. Probabilistic verification of neural networks against group fairness. In Formal Melhods: 24th International Symposium, FM 2021, Virtual Event, November 20-20 26, 2021, Proceedings 24, pp. 83 102. Springer, 2021.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE Conference on Compuler Vision and Pullern Recognilion, pp. 1 9, 2015.
Cüneyt M Taskiran, Zygmunt Pizlo, Arnon Amir, Dulce Ponceleon, and Edward J Delp. Automated video program summarization using speech transcripts. TEEE Transactions on Mullimedia, 8(4):775-799) 2006.
Sinnu Susan Thomas, Sumana Gupta, and Venkatesh K Subramanian. Smart surveillance based on video summarization. In 2017 IEFE Region 10 Symposium (TENSYMP), pp. 1-5. IEEE, 2017.
Ba Tu Truong and Svetha Venkatesh. Video abstraction: A systematic review and classification. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 3(1):3ess, 2007.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 30, 2017.
Huawei Wei, Bingbing Ni, Yichao Yan, Huanyu Yu, Xiaokang Yang, and Chen Yao. Video summarization via semantic attended networks. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 32, 2018.
Gokhan Yaliniz and Nazli Ikizler- Cinbis. Using independently recurrent networks for reinforcement learning based unsupervised video summarization. Mullimedia Tools and Applications, 80:17827-178411 2021.
Bin Yu, Wing Ma, Klara Nahrstedt, and Hong-Jiang Zhang. Video summarization based on user log enhanced link analysis. In Proceedings of the 11th ACM International Conference Multimedio, pp. 382 391, 2003.
Li Yuan, Francis EH Tay, Ping Li, Li Zhou, and Jiashi Feng. Cyclc-sum: Cyclo-consistent adversarial LSTM networks for unsupervised video summarization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 9143-9151 2019.
Yitian Yuan, Tao Mei, Peng Cui, and Wenwu Zhu. Video summarization by learning deep side semantic embedding. IEEE Transactions on Circuits and Systems for Video Technology, 29)):226--237, 2017.
Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rogriguez, and Krishna P Gummadi. Fairness con- straints: Mechanisms for fair classification. In Artificial Intelligence and Statistics, pp. 962-970. PMLR, 2017.
17

---

Published in Transactions on Machine Learning Research (12/2023)
Ke Zhang, Wei-Lum Chao, Fei Sha, and Kristen Grauman. Video summarization with long short-term memory. In Computer AosionEECVV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11 11-14, 2016, Proceedings, Part VII 14, pp. 766 782. Springer, 2016.
Yujia Zhang, Michael Kampffneyer, Xiaoguang Zhao, and Min Tan. Dtr-gan: Dilated temporal relational adversarial network for video summarization. In Proceedings of the ACM Turing Celebration Conference- China, pp. 1 6, 2019.
Kaiyang Zhou, Yu Qiao, and Tao Xiang. Deep reinforcement learning for unsupervised video summarization with reiressiyyrrreeennaaaiveeesss reward. In Proceedings of the AAA Conference Arlificial Intelligence, volume 32, 2018a.
Kaiyang Zhou, Tao Xiang, and Andrea Cavallaro. Video summarisation by classification with deep rein- forcement learning. arXiv preprint arXiv:1807.00055 2018b.
Wencheng Zhu, Jiwen Lu, Jiahao Li, and Jie Zhou. DSNet: A flexible detect-to-suumariie network for video summarization. IEEE Transactions on Image Processing, 30:948 962, 2020.
18

---

Published in Transactions on Machine Learning Research (12/2023)
Appendix
A
Additional Violin Plots for Remaining Videos
We provide remaining violin plots for all videos in Fair VidSum to visualize the distribution of unique in- dividuals (Figure 7 sex sensitive attribute (Figure 8 and ethnicily sensitive attribute (Figure g These distributions highlight the importance and difficulty of summarizing videos while ensuring fairness. The individual plots, in particular, showcase the most challenging scenarios, as they contain videos (such as Vid. #24, #25, #30,, #33) with numerous individuals appearing very limited framees Consequently, any missing individuals would result in an unfair summary and SumBal score of zero. The plots emphasize the need capture and represent proportionality and fairness in the video summaries.
Video
Video 2
Video
Video
Video 6
1000
3000
2000
4000
10000
2000
500
1000
2000
5000
I
I
1000
Video
Video 8
Video
Video
Video 11
4000
2000
10000
3000
2000
2000
5000
1000
2000
1000
2
5
2
1
2
Video 12
Video 16
Video 17
Video 18
Video 19
2000
3000
4000
3000
3000
2000
2000
1000
2000
Frame Index
1000
1000
0
0
1
2
2
2
Video 20
Video 21
Video 22
Video 23
Video 24
4000
6000
3000
2000
4000
4000
2000
2000
1000
2000
2000
1000
0
Video 25
Video 26
Video 27
Video
Video 29
6000
4000
3000
4000
4000
2000
2000
1000
2000
2000
1000
2
3
2
2
3
2
Video 30
Video 31
Video 32
Video 33
Video 34
4000
10000
15000
4000
1
4000
10000
2000
5000
2000
2000
5000
0
2
3
2
4
10
13
1
4
5
Person ID
Figure 7: Violin plots showcasing the distribution of unique Videos #3, #13, #14, #15 are present in main paper.
appearing across all vidcos.
19

---

Published in Transactions on Machine Learning Research (12/2023)
Video 2
Video
Video
Video
Video
Video
1000
4000
1000
4000
10000
2000
500
500
2000
5000
1000
Video 10
Video 11
Video
Video
Video
Video
2000
2000
3000
4000
2000
1000
1000
2000
2000
1000
M
M
M
F
Video 16
Video 17
Video 18
Video 20
Video 21
Video 22
Frame Index
3000
4000
3000
4000
6000
2000
2000
2000
4000
2000
2000
1000
1000
M
M
M
Video 23
Video 24
Video 25
Video
Video
Video
3000
6000
4000
3000
4000
2000
4000
2000
2000
2000
1000
1000
Video 29
Video 30
Video 31
Video
Video
Video
4000
4000
10000
15000
4000
10000
2000
2000
5000
0
M
M
M
Sex
Figure 8: Violin plots showcasing the distribution see sensitive attribute across all videos. Here M denotes Male, and F denotes Female. Videos #1, #4, #6, #19 are present in main paper.
B Additional Details Regarding Model Training and Evaluation
B.1 Evaluation Splits
In accordance with previous research methodologies, we have created five splits, chosen randomly. These splits follow 80:20 partitioning for training and testing, respectively. With 34 videos total, each split comprises 28 videos in the training set and 6 in the testing set. We provide the test sets for every split used in our benchmarks below. It is important to note that any videos not included in the test sets for each split would naturally be part of the corrosponding training sets. The splits are as follows: Split #1 Test set: Vid. #9, Vid. #11, Vid. #19, Vid. #25, Vid. #26, Vid. #34; Split #2 Test set: Vid. #2, Vid. #8, Vid. #20, Vid. #24, Vid. #29, Vid. #32; Split #3 Test sel: Vid. #8, Vid. #18, Vid. #20, Vid. #24, Vid. #28, Vid. #33; Split #4 Test set: Vid. #4, Vid. #8, Vid. #17, Vid. #18, Vid. #29, Vid. #30; Split #5 Test set: Vid. #3, Vid. #15, Vid. #16, Vid. #20, Vid. #25, Vid. #28.
B.2 Models and Training Details
We follow the original training procedures established for all benchmarked models and adjustments to specific hyperparameters are described below. Any parameters not. explicitly mentioned were maintained at the default values specified in the original code or paper for each model. For DSNet, we use the anchor-fres model.
All adjustments made to parameters to train models Fair VidSum:
20

---

Published in Transactions on Machine Learning Research (12/2023)
1
Video
Video
Video
Video
Video
3000
1000
1000
4000
10000
2000
500
500
5000
Video 7
Video 10
Video 11
Video
Video
Video
2000
2000
3000
4000
1000
1000
2000
1000
0-
BL
WH
WH
VHH Video 19
WH
WH Video 21
Video 15
Video 17
Video 18
Video 20
Frame Index
4000
4000
3000
2000
4000
2000
2000
2000
2000
1000
1000
WH
WH
AS
WH
WH Video
WH
BL
WH
Video 23
Video 24
Video
Video
Video
3000
6000
4000
3000
4000
2000
4000
2000
2000
2000
1000
1000
Video 29
Video 30
Video 31
Video 32
Video
Video
4000
4000
10000
15000
4000
10000
2000
2000
5000
0
WH
WH Ethnicity
WM
BL WH
A5
AS
Figure 9: Violin plots showcasing the distribution ethnicity sensitive attribute across all videos. Here WH denotes White, BL denotes Black, and AS denotes Asian. Videos #8, #9, # #16, #22 are present main paper.
AC-SUM-GAN: rww.uarrzzaaiiiiiioooaaaaaaa factor 5.0, clip = 1.0, Aicion_state_o.
CA-SUM: block_size= 1e-6, factor 5.0
size 60, init_gain 1.0, n_epochs 200, clip 1.0, ir 1e-4, 12_ r2_req
PGL-SUM: clip 1.0, 1r le-4, 12 re_req le-4
SUM-GAN-AAE, clip = 1.0, hidden hidden_size = 512, regularization factor -..0, 1r 1o-5
SUM-GAN-SI: clip = 1.0, hidden_sitd size 512, regularizaaionnfactee =500
C Results on Remaining Evaluation Splits
C.1 Benchmarking Results
Here, we present results for the remaining evaluation splits: Split #2, Split #3, Split #4, and Split #5 Tables and D respectively. The results show similar trends the averages table and Split #1 table presented in the main text.
21

---

Published in Transactions on Machine Learning Research (12/2023)
Table 3: Comparison of SOTA video sunmarization models on Fair VidSum for evaluation Split #2.
Average
SumBall [Ser]
Sunffll (Ailsiinll))
SumBal (Nudénidina))
Model
Type
F. Mesoure
Average
Min
Viobating
Average
Min
Violsting
Average
Min
Vicialig
Random
0.0555
Pomete (Vid. #29)
0.9190
Asion [Vid. 424)
0,5506
0.7417
(Vid. #24)
Human
65,00
0.4651
0.0000
Fimute (Val. +26)
0.4768
0.0000
Aninn [Vid.
+243
0.2909
0.0000
Permon (VSL. #24)
CA-SUM
Usoupersssee sed
60.77
0.6449
0.0000
Pamate (Val. +26)
0.4971
0.0000
Anina (Vid.
+24)
0.1819
0.0000
Proson 4 (Vs). #6)
AC SUM GAN
Uamperrzzed
62.41
0,5062
0,0000
Pemate (Vud. +26)
0.4577
0.0000
Asion (Vid.
424)
0.2634
0.0000
Person (Vid. 424)
AR
Uvempervised
61.49
0:5213
0.0000
Plomate (Vil. #26)
0.4558
0.0000
Anion [Vid.
#24)
0.1835
0.0000
Person ( (Vv. (8)
SUMI-GAN-SL
Usmppervaed
0.4008
0.0000
Pemale (Vid. +24)
0.4329
0.00011
Asion [Vid.
424)
0.1714
0,0000
Person (Vid. (8)
SUM-IND
UssuperSaed
0.0805
0.0000
Pemate (Vid. +24)
0,0000
0.0000
Asion [Vid.
#244
0.3570
0.0000
Person J (VSd. 42)
DSNNet
Supervised
62.45
0.5661
0.0000
Pemele (Vid. #24)
0.5032
0.0000
Asion [Vid.
#24)
0.2888
0.0000
Person ( (V5d. (88
VASNet
Supervise
0.4796
0.00000
Pamale (Vall +24)
0.4676
0.000
Asion (Vid.
#24)
0.1963
0.0000
Proson 4 (VGJ. (8)
PGL-SUM SUM
Supervised
61.04
0.4627
0.0000
Pemale (Vid. 424)
0.4300
0.0000
Asion (Vid. 424)
0.1881
0.0000
Person 4 (Vid. 48)
TVS LP (Sez)
Camperssed
15,30
0.9085
0.9068
Male (Vid, #24)
0,7003
0.0000
Hispanic (Vid.
0.4183
0.0000
Person I (Vid. 42)
FVS-LP (Ktlaiidlg)
Usoupers sed
0,8000
0.4721
Fammtte (Vil. +29)
0,9981
0.9934
Anioa (Vid. +24)
0.3795
0.0000
Person (Vi). +2)
FVS (Jndisiduel)
Uamperszedd
12:41
0:9714
0.8350
Maic (Vid. #2)
0.5350
Disck (Vid. 42]
0.9963
0.9905
Person 2 (Vid. 42)
Table 4: Comparison of SOTA video summarization models on Fair VidSum for evaluation Split #3.
Average Messure
Sum (Sex)
SunDall (Ethnicstt)
SamlBal (Jedividual)
Maddd
Type
Average
Min
Violating
Arseage
Mil
Violating
Arenuge
Mil
Violating
Razdoon
14.96
0,9477
0.8758
Femaic (Vid. #24)
0,8844
0.0670
Auias (Vid. 443)
0.9614
0.0670
Person 19 (Vid. (333)
Human
67.61
0.4784
0.0000
Femaii (Vis. #24)
0.4518
0.0000
Avioa (Vv). (24)
0.2007
0.0000
Peraim 8 (Vid. # 118)
CA-SUM
Unsupervise
38.76
0.5547
Femaic (Vád.
0.3859
0.0000
Asias (V3d. #24)
0.1901
Person (Vid. 483
AC-SUM-GAN
Uneupervised
61.68
0.6236
Femaic #24)
0.4125
Arias (Vsd, g24)
0.1000
(Vid. #8)
SUM-GAN-AAE
Unsqperiie
61.63
0.5830
0.0000
Fimanc (Vál.
#24)
0.4291
0.0000
Abiis (V&l. +24)
0.2779
0.0000
Parsom J [Vid. #24)
SUM GAN SL
Unsupervised
62.77
0.8211
0.0000
Femaie (V5d, #24)
0.4122
0.0000
Anios (Vid. #24)
0.2787
0.0000
Person 2 (Vid. 424)
SUM-IND
Unoqerried
44.79
0.5061
0.0000
Armanic (Vil. #24)
0.4214
0.0000
Aviso (Vil. (+44
0.1481
0.0000
Person (Vil. #55)
DSNet
Supeevised
60.16
0,5448
0.0000
Femaic (Vid. 426)
0.4364
0.0000
Aasias (Vid. 424)
0.2004
0.0000
Person (Vid. 438)
VASNet
Sigarissll
60.95
0.5616
0.0000
Femair (Vir). #24)
0.4661
0.0000
Asios (Vv). (24)
0.2668
0.0000
Peraim: [Vid. #21)
PGL SUM
Suprevised
29.15
0,5117
0.0000
Femaic (Vid) #24)
0.4086
0.0000
Asias (V5d. #24)
0.1965
Person (Vid.
EVS-LP (Sex)
Unsupervsed
17.03
0,9990
0.9908
Male (Vid. #24)
0.5242
0.0000
Hispasic (Vid. *8)
0.0639
0.0000
Person (Vid. 445)
FVS-IP /iiiiiiity
Uneppervisd
16.06
0.7394
0.0000
Male (Vid. #53)
0.9951
0.0822
Aviss (VV), 833)
0.1387
0.0000
Person (Vid. #8)
FVS-PP (Andiciiual)
Unsupersied
16.08
0,9671
0.0289
Male (Vid. #28)
0.5078
0.6420
White (Vid. #25)
0.9928
0.9731
Person 11 (Vid. (33)
C.2 Mixing Strategy Results
Here we presont results for the mixing sampling stratogy described in Scction 5.2 for cach of the individual splits as Figures 10 (Split #1), D (Split. #2), 12 (Split #3), 13 (Split #4), and 14 (Split #5). The mixing ratio A is varied from 0 to 1 in increments of 0.1.
D Code, Reproducibility, and Miscellaneous Details
D.1 Github Repository
The Github repository contains all the code needed for reproducing experiments, and also hosts the Fair Vid- Sum dataset. It is located here: http:////gituub...ooo/////aaaaat
D.2
Environment Specifications
We use Python 3.8.16 and Anaconda to install all required libraries to run all models. The Anaconda environment yaml file is provided in our repository. The experiments were conducted on Ubuntu 20.04 using NVIDIA GeForce RTX 3070 GPUs (CUDA version 11.1).
D.3 Code Details
The training codes utilized in our experiments were directly obtained from the official github repositories of the respective models, all implemented in PyTorch (v1.12.1). We condense codes into less files for ease
22

---

Published in Transactions on Machine Learning Research (12/2023)
Table 5: Comparison of SOTA video summarization models on Fair VidSum for evaluation Split #4.
Average
SumBall [Ser]
Sunfflll (Ailsiisll))
SumBal (Iudeoidinl))
Model
Type
F. Mesoure
Average
Min
Violating
Average
Min
Violsting
Average
Min
Vicialng
Random
0.9321
0.8814
Male (Vid, #500)
0.5043
White (Vid. #18)
0,5586
0.7253
(Vid. 4:30
Human
76.51
0.3735
0.0000
Fimutf (Val. +30)
0.5885
0.0772
Whate (Vil. #:30)
0.2322
0.0000
Permon (VSL. #18)
CA-SUM
Uroupersised sed
71,89
0.4395
0.0000
Pamate (Val. +30)
0.6222
0.0750
White (Vid.
#330
0.2301
0.0000
Proson 4 (Vil. #6)
AC SUM GAN
Uampervzzdd
74.02
0.4258
0.0000
Pemale (Vid. +-00)
0.6222
0.2741
Whdte (Vid. 4-00)
0.2026
0.0000
Person 4 (Vid. 48)
SUM-GAN-AAR
Uvempersised
72.40
0.5096
0.0000
Plomafe (Vil. #:30)
0.0094
0.4450
White (Vid. #22)
0.3625
0.0000
Person ( (Vv. #8)
SUM--GAN-SL
Uamppervaed
73.42
0.4658
0.0000
Pemale (Vid. +-30)
0.0388
0.4450
(Vid. #22)
0.2615
0,0000
Person (Vid. (8)
SUM-IND
UssuperSaed
001.35
0,5692
0.0000
Maae (Vid. (/20))
0.6363
0.3419
Whdte (Vid.
0.2001
0.0000
Person 4 (VSd. 48)
DSNNe
Supervised
71.53
0.5743
0.0000
Pemele (Vid. #300
0.6981
0.4450
White (Vid. #220)
0.2412
0.0000
Person 5 (V5). (8)
VASNet
Supervise
60.15
0.4644
0.00000
Pamate (Vall +800)
0.64770
0.2719
White (Vád. ++00
0.2065
0.0000
Proson 4 (VGJ. (8)
PGL-SUM SUM
Supervised
70.03
0.5058
0,0000
Pemale (Vid. 4-300
0.7343
0.4450
Whate (Vid. #28)
0.3027
0.0000
Person 4 (Vid. 48)
TVS LP (Sez)
Compertsed
16.89
0.9081
0.9060
Female (Vid. #44)
0,5231
0.0000
Hispanic (Vid. 45)
0.3:69
0.0000
Person 5 (Vid. 440
FVS-LP (Ktloiidlg)
Usoupers sed
0.0000
Faualle +-4)
0,9994
0.9987
(Vid. #188
0.2257
0.0000
Person (Vil. (-1
FVS (Jndisiduel)
Uampersised
0:9782
0.8895
(Vid. #17)
0.9621
(Vid. #20)
0.9949
0.9520
(Vid. 430)
Table 6: Comparison of SOTA video summarization models on Fair VidSum for evaluation Split #5.
Avecage
SummBl (Sez)
SumBal (Ahhnicity)
SumBal (Judiutdual)
Model
Type
Ft Measure
Anozage
Violating
Averuge
Min
Violating
Avecuge
Min
Violoting
Random
13.96
0,9375
0,8992
Male (Vid. 43)
0.9242
0.8092
Whitee (Vid. 43)
0,8732
0.7020
Person (Vid. 425)
Human
0.5157
Pimale (Vid. #25)
0.3615
0.0000
Anion (Vid. +25)
0.3257
0.0000
(V9). (35)
CA-SUM
Unseperriaed
30.31
0.5967
0.0384
Female (Vid. #3)
0.5542
0.0084
Asico (Vid. (3)
0.2044
0.0000
Person (V9). (25)
GC-
Unsupeevsood
38.56
0.5545
B.0130
Pemale (Vsd. (43)
0.5890
0.0000
Asien (Vid. +3)
0.3158
0.0000
Person (VK). #26)
SUML-GAN-AAE
Ussupeevised
G03001
0,505
0.0101
Pemale (Vad. (-3)
0.54m0
0.0401
Aaico (Vid. +0)
0.3304
0,000
Person (Vld. 425)
SUM-GAN-SL
Unsuperrised
50
0.5346
0.0384
Female (Vsd,
0.5815
0.0084
Asion (Vid. #3)
0.2863
0.0000
Person (V5), $25)
SUM-NND
Unsuper Sed
47.81
0.5411
0103300
Pemale [Vkl.
0.5287
Asion (Vid. +43
0.2622
0.0000
Person (Vk). #25)
DSN
Supercised
06
0.5506
0.0401
Pemale [Vid. +K3)
0.5288
0.0111
Asico (Vid. +3)
0.2883
0.000
Pesson (Vk). #25)
VASNet
Supervised
62.90
0.6838
0.0000
Female (Vid. #25)
0.5005
0.0000
Asien (Vid. 425)
0.2807
0.0000
Person (V5d, 425)
PGL-SUM
Supervied
63.37
0.4828
0.0000
Female (Vid. #25)
0.5274
0.0000
Awion (Vid. +25)
0.3304
0.0000
Person (Va), (25)
FVS-LP (Sex)
Unsuperriisd
12.42
0.9988
0.9975
Femade (Vid. #25)
0.8192
0.5633
Alock (Vid. +20)
0,2504
0.000
Piraan 3 (Vid. #3)
FVS-LP (Rilmiinly)
Unsupersieed
9.528
0.7632
Pemale (Vil. #25)
0.9983
0.9974
Blork (Vid. #28)
0.3062
0.0000
Peraso (Vid. (3)
IVS (Jadielbaa))
Unsupervisod
9.512
0.3367
0.6280
Made (Vid. 428)
0.9406
0.6499
White (Vid. #25]
0.9894
0.9704
Person (Vid. 425)
Ethnicity
Sex
Individual
60
60
FL-Measure
50
50
50
40
40
40
30
30
30
20
20
20
10
0.0
0.5
1.0
1.0
1.0
SumBal (Min)
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
0.0
0.0
0.5
0.5
Mixing Ratio
CA-SUM
SUM-GAN-AAE SUM-GAN-SL
SUM-IND DSNET
VASNET PGL-SUM
AC-SUM-GAN
Figure 10: Results for the mixing sampling strategy for Split #1.
of running. We use our dataset splits (randomly generated) evaluate and train all models (also provided in repository). We selected the trained models that. achieved the highest Fj scores per split (which was
23

---

Published in Transactions on Machine Learning Research (12/2023)
Ethnicity
Sex
Individual
60
50
50
50
40
40
40
30
30
20
20
10
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
1.0
1.0
1.0
0.8
0.6
0.6
0.6
SumBal
0.4
0.2
0.2
0.2
0.0
0.0
0.0
0.0
0.5
1.0
0.0
0.5
0.0
0.5
Mixing
CA-SUM
SUM-GAN-AAF SUM-GAN-SL
SUM-IND DSNET
VASNET
AC-SUM-GAN
PGL-SUM
Figure 11: Results for the mixing sampling strategy for Split #2.
Ethnicity
Individual
60
60
60
50
50
40
30
30
20
20
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
1.0
1.0
1.0
0.8
0.8
0.8
SumBal (Min)
0.6
0.6
0.4
0.4
0.4
0.2
0.0
0.0
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Mixing Ratio
CA-SUM
SUM-CAN-AAE SUM-GAN-SL
SUM-IND DSNET
VASNET
AC-SUM-GAN
PGL-SUM
Figure 12: Results for the mixing sampling strategy for Split #3.
standard procedure in all models' codes). Since all methods use a common evaluation code, which involves the Knapsack algorithm to generate summaries based on frame importance scores and the Fi score evaluation, created a unified evaluation script for reporting the average Fi scores. This script takes the predicted summarics (post-Knapsack, from models and the ground truth user summarics (from annotations) from the hō dataset as input. We also developed a unified script for evaluating SumBals which takes as inputs the predicted summaries and user summaries, similar to the F1 score evaluation. The SumBal evaluation also requires the fairness labels, which are numpy binarics provided for individuals/entities in the frames for each video. Please follow the README on our github or instructions our webpage for detailed instructions how train and evaluate all models.
24

---

Published in Transactions on Machine Learning Research (12/2023)
Ethnicity
Sex
Individual
60
60
F1-Measure
40
20
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
1.0
1.0
1.0
SumBal (Min)
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
0.0
0.5
0.0
0.5
0.5
Mixing
CA-SUM
SUM-GAN-AAF SUM-GAN-SL
SUM-IND DSNET
VASNET
AC-SUM-GAN
PGL-SUM
Figure 13: Results for the mixing sampling strategy for Split #4.
Ethnicity
Sex
Individual
60
60
60
50
50
40
40
30
30
20
20
20
10
10
10
0.0
0.5
0.0
0.5
0.0
0.5
1.0
1.0
1.0
0.8
0.8
0.8
SumBal (Min)
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.0
0.0
0.0
0.0
0.5
0.0
0.5
0.0
0.5
Mixing Ratio
CA-SUM
SUM-CAN-AAE SUM-GAN-SL
SUM-IND DSNET
AC-SUM-GAN
PGL-SUM
Figure 14: Results for the mixing sampling strategy for Split 45.
D.4
Videos in FairVidSum
Fair VidSum currently consists of the following YouTube vidcos:
Video #1:
Vidco #2:
Video #3:
ht http://www..ootouo.cccc//aaatggp
Video #4:
channel=MSNBC
25

---

Published in Transactions on Machine Learning Research (12/2023)
Video #5:
http:///ww.pptpccoococcc/cot/tta Video #6: httpa//www....a..a.a.a.a...oaaa.
Video #7:
Video #8:
Video #9:
ht http:///ww.a..oo....o/.///a/taaa
Video #10:
Vidco #11:
Video #12:
Video #13:
Video #14: http:///wg..tp..oo..../ct//aaaaa
Video #15:
attps://ww..ww http///wwwwppapooooa/////a/ea/ap
Video #16:
Video #17:
Video #18:
attps://www.youe http:///www.oootbeccccccccaootat Video #19:
https://www.pee http:://www..pooouuoc.cc//////rp
Video #20:
Vidco #21: http::/www...ooooteccc////asssss
Video #22: https://www.yout http://www..ooouteecccc////aaaas
Video #23:
ht https://www.n sttps://ww.youuube..oo/o// /atpruu.....ommaaaaa//a333713110
Vidco #24:
Video #25:
Video #26: ht https://www.
Video #27:
https:
Video #28:
Video #29:
Video #30:
https: http://www.yottooocccocaaaaaasaa
Video #31:
https://www.yon httpes//wwwy..ouuuuuooooo///dddd
26

---

Published in Transactions on Machine Learning Research (12/2023)
Video #32:
fttps://wwwr http///wwwwwuuub.eccoo www.yytu..o///aaaaoooo wtwpttbbw......aaccc..../o....o.
Video #33:
Video #34:
D.5 Dataset Collection
We performed the frame labeling process using the Kili labeling platform 66 the labeling task was instructed to read the video title and watch the video on mute first. For annotation purposes, each user received frames from 22 videos, which were downsampled to 1/2 frames per second. The users were asked to assign an importance score ranging from 1 (least important) to (most important) to each frame. To prevent any polential bias, frames from every video were shuffed before being presented the users for annotation.
Each participating in
D.6 Dataset Details
All the models included in the benchmark provide the datasets (SumMe and TVSum) in structured h5 format, which can be accessed either through their respective github repositories or download links, and also provide their splits.json files used for training and evaluating models. In a similar manner, we have included the FairVidSum dataset in the fvs.h5 file, along with the corresponding fvs_ rrs_splits.joon file used for our benchmarks. Both files are already provided the repository, and we also provide the dropbox download links them.
We also provide fair patr_ny___a ppy__ata// which contains all fairness labels and data required for SumBal evaluations and generating summaries using FVS-LP. fair npy data/ contains three subfolders, sex/ eth/ and ind/ which further contain corresponding fairness labels for each video in our dataset in numpy binary (.upy) format. There a corresponding .npy file associated with cach vidco (named aadoo_<vid___ sii__<viddmmmppppp mmm_>mppp)) These mpy files are structured as mumpy arrays, with rows representing the frame index, and columns representing the protected groups. For each group, the value of 0 the numpy array indicates that the corresponding protected group is not present that frame, while a value of indicates its presence.
The TVSum videos (Vid. #23 #34) were solely labeled for fairness. The h5 dataset was taken directly from and we simply extracted
the benchmarking model repositories (ecev16 dataset dataset_tvsum __suuttsmmmmmm googleoooooooooo the chosen 12 videos. Hence, the TVSumm videos adhere to their downsampling rate (2 frames per second), user annotations, frame features, etc. We append the TVSumm 12 video h5 dataset to our fvs.h5 (containing videos) to create a standard video summarization dataset of 34 videos.
D.7
Dataset License
The Fair VidSSm dataset is released under CC-BY-SA license. Please refer to https://crettivveccmmooo org/ stg//iicnsees/yysa//..00 for license details.
D.8 Videos for Dataset Extension
We plan to add the following videos to the next version of the dataset:
1. httee://rrrr.stttuu.ccoo//sssssr
2. ottee://wrri.ggatuu.oooooooorrar
3. httpss/wwwwyyooobeeeccooaaaaaaap
4. https:/ http::/www....aaaaaooooo//oaaata
5. htthge//hhaayyooabeeeeeoo/////tp
6. bttp:s///mrrrgquuuuuaoooo/oootbr
http:://hsll--ecchlllgyy.cmm
27

---

Published in Transactions on Machine Learning Research (12/2023)
7. sttes////wkkyyouruooooooo/ooooot
8. ottes://wwr.yoouuuuocccoooooosts
9. sttee///ftr..gouuuuoocoooo//ooot
10. httpss//wwa..gouuuuuocccoooooost
11. httpa://wwa...auuuuecccoooo/sosr
12. ottp:///wwa..oonnooooooo///o//sr
13. ottp:://wwwn.ggnouuooooo////ooor
14. https:/wwww..oottuoooooo/////o/t
15. http://wwww.guttuuu.cooo/otttttt
16. httpp://wwwa.ppuuuucccoo///ooo/2
17. httppp///wwaaaaouuuuccoo//////a2
18. htt.gp//wwwaappuuuucccooo/oaoo02
19. http:://wwaa.ooauuuccccc//////a2
20. httpgp//wwannpooouuoocoo/ssaaass
21. https//wwwrr.ooataoocoo///////22
22. http://wwww.gouuuuu.coo/////2222
23. http://wwwarppouuuoccco/////t111
24. httpss//wwrr.pooouucccco////aaa2
25. http:://wwa..ppuuu..cooo//tttst2
26. httpp://wwa..goouueeec.oo/aataas
E
Ethics Statement
Our paper studied the important and novel problem of fair video summarization. As individuals and groups (based sensitive attributes) can be directly impacted by automated video summarization, it is important to recognize the importance of ethical considerations in the development and deployment of such systems. Without sensitive attribute information available for video summarization, it is not possible for methods being developed to adhere to the principles of fairness, accountability, and transparcncy as part of thoir research goals. Therefore, our work primarily aims to bridge this gap; and address potential biases and discriminatory effects in the video summarization task. However, it is important also advocate for the responsible use of fair video summarization, and ensure that the provided data and benchmarks are only used to enhance the fairness of existing methods propose novel fair variants to current models.
F Possible Directions for Future Work
There are multiple dircctions for possible future work:
F.1 Relaxing Knowledge of Protected Groups
Future work can be proposed that relaxes the assumption on knowledge of protected group annotations, either partially or fully, or can even aim to predict them and then ensure fairness. All these problem settings give rise diverse of non-trivial problem formulations. For instance, methods that assume probabilistic group memberships (such when a classifier is used to predict the group) would differ significantly from methods that assume partial knowledge of certain groups.
F.2 Generalized Methods for Different Definitions of Fairness
Currently, vidco summarization methods differ significantly from each other- some employ LSTMs, others employ GANs, etc. However, for future research on fair video summarization, we believe a more generalized and unified direction for methods can studied. For example, approaches can be propased that take an existing video summarization model as input and make fair for any general fairness definition provided at run- nunttime. Moreover, new fairness definitions for fair video summarization specific to certain applications, can be motivated.
28

---

Published in Transactions on Machine Learning Research (12/2023)
F.3 Pre-process rre-processing, In- nn-prrccesingg and Post-processing Approaches for Fairness
Fairness methods can be pre-processing (transform the input data space to ensure fairness), in-processing (fair model variants), or post-processing (modifying the output summaries to make them fair) based. All three modalitics are important to study for future work, as different vidco summarization pipelines have different data pipeline requirements. Furthermore, is important to understand how each type of method differs, and whether some are better at ensuring fairness than others.
29


Reviews:
[{"review_id": "jvwSCQWDap", "reviewer": "Reviewer_8FYH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The target issues of the paper are meaningful and worth exploring. This submission gives a valuable implementation of such an idea and presents good results.\n2. The paper is generally well-written, clearly structured, and quite easy to follow. \n\nWeaknesses:\n1.  The motivation is not so clear. In Section 3.2, the authors claim that all persons in the video should be represented in approximately the same proportions in the generated summary as they appear in the entire video.  What if some people are not important for the video? For example, although the main character's frames are less than sub-character, we still focus on the main character. In this situation, fairness is a conflict of importance.\n2. The number of videos in the dataset should be more.\n3. The author does not provide suggestions for further improvements of methods in their dataset.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a fair video summarization task. They build a FairVidSum dataset and SumBal metric for quantifying fairness. They also provide a fair-only baseline called FVS-LP.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See Weaknesses.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PJYgKG3zWz", "reviewer": "Reviewer_EwqW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main strengths of the paper are proposing a new dataset and evaluation metric for studying fairness. There is a lot of analysis on the dataset and the distribution of people, attributes, etc. in the dataset. There is good discussion on the limitations of the dataset.\n\nThe main weakness is the evaluation metric is pretty simple. It is just calculating the percent of time an entity is in the summary vs. the source material. This will provide some insight, but is also limited to only datasets where these annotations are available, and any biases in those datasets. \n\nThe evaluations are done by training the methods on the new dataset. Have you considered evaluating pretrained models directly on this dataset without any further training?\n\nThere's not much analysis or insight in the results. Why are some methods better than others?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a dataset and evaluation metric for fairness in video summarization. They explore many existing video summarization methods using these components. The paper proposes evaluating fairness of a summarization by measuring the proportion of the summary each entity takes up compared to the proportion of the video each entity takes up. The dataset is collected similar to others, but annotated by humans counting each entity.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "It would be helpful to add more insights in the results section and more experiments studying different aspects of the models, what makes them fair or unfair, etc.", "broader_impact_concerns": "It is discussed in the paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Dn2Vyq4RPt", "reviewer": "Reviewer_5vEw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper very well written. The problem, related work and the contributions as well explained and presented clearly.\n\nIt seems that the video summarization problem as defined in the literature and in this work is not really concerned with the \"video\" itself, but what it tries to do is to select a subset from the inputs. Firstly, the audio of the video is removed. Secondly, the frames are randomly selected at a low frame-rate without attention to the content. Finally, during the annotation process the frames of the video are shuffled. \nI think that selecting a frame for video summarization should be concerned with the surrounding frames and the audio specially in \"people talking\" setting, it is important to know what is being said and decision cannot be made solely based on the visual features of a single frame.\n\nFurthermore the problem of unsupervised fair summarization seems unrealistic as it assumes that the ground truth group memberships `H` are available. In a realistic setting, given a new test video, `H` has to be predicted but this was not studied in this work.\n\nThe proposed baseline seems too rudimentary. There is no hyper-parameter to control for a trade-off between the utility and fairness.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper as is titled takes a step towards fair video summarization. They collect a dataset, propose a metric and an unsupervised video summarization baseline.\nVideo summarization in this work (and other related words) is defined as the task of selecting a sub-set of the video frames from the whole video from sparsely sampled frames with a low frame-rate. In this task no audio is used. Video summarization is usually evaluated using the F-measure.\nThis work introduces the FairVidSum dataset. The dataset consists of 22 new videos from Youtube and 12 videos from the TVSum dataset. The nature of the videos are \"people talking\" like podcasts, interviews, etc.\nIn this work a fair summarizer would select frames from the video that respects and contains the same proportion of protected groups (such as an ethnicity) as it is present in the input video itself.\nThe proposed evaluation metric SumBal is measuring the most violated proportion from a set of protected groups in the video summarization output.\nThe proposed baseline is an supervised approach for video summarization that given the proportions of the protected groups in the input video and their presence in each frame, is able to output a summarization that maximally satisfies the fairness goal using a linear program without any other utility or accuracy in its objective.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Improve the baseline to incorporate utility/accuracy of the summarization alongside fairness.\n- Increase the size of the datset.\n- Treatment of the video should be as a video not as a collection of images sampled from a video after removing the audio.", "broader_impact_concerns": "I do not have any concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "K3ciwcWHNZ", "reviewer": "Reviewer_ZZ3b", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n+ Overall, the paper is well-written and easy to follow.\n+ The authors clearly define the new task, fair video summarization, and provide all for the development of fair video summarization models, including the new evaluation metric and the benchmark dataset.\n+ The experimental results clearly show that existing video summarization models do not account for fairness, at least according to the evaluation metric, SumBal.\n\nWeaknesses\n- The new benchmark, FairVidSum, contains videos of a few selected categories such as news, interviews and panel discussion so that each video features a very few individuals in a restricted setting (at most 16 in Video #33). The paper lacks the discussion about the fairness in summarizing videos that feature much more individuals and diverse settings, such as concert or large lecture videos featuring thousands of people.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work is the first to address the concept of fairness in video summarization. The authors provide an analytical definition of fair video summarization and a novel metric called SumBal to measure the fairness in video summaries. To benchmark existing video summarization models, the authors newly collect the dataset, FairVidSum, which is designed similarly to existing datasets, TVSum and SumMe, but also with individual- and group-level fairness annotations. The experiments conducted on FairVidSum clearly showcase the fairness-utility gap between existing video summarization models and the proposed baseline, FVS-LP, that are optimized only for utility and fairness, respectively.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Please add the discussion about the fairness in summarizing videos of different domains and settings that the collected benchmark FairVidSum does not cover.\n- Please fix the mismatch in descriptions of the frame sampling rate for important score annotation: the sampling rate is stated as 2 fps in Section 4.1 while 0.5 fps in Appendix D.5.", "broader_impact_concerns": "The authors adequately addressed the ethical implications of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: These manuscripts underwent review by four experts, with three providing the final recommendations. The cumulative scores present a diverse perspective, with two experts advocating for acceptance, and one reviewer proposing rejection.

Most reviewers acknowledged the novelty of the fair video summarization problem, praised the quality of the gathered dataset, and commended the paper's writing. Reviewer 5vEw identified certain weaknesses in the proposed baseline, the nature of the addressed task, and the dataset size. These comments are generally insightful, and the Associate Editor (AE) values the feedback from Reviewer 5vEw.

Video summarization has been extensively explored in both the Multimedia and vision communities, with various methods developed. This paper focuses on addressing the fairness issue within recent learning-based approaches. The authors have made commendable contributions in this regard. According to the TMLR evaluation criteria, the authors' claims align well with the presented evidence. However, the AE notes the abundance of existing literature on video summarization, suggesting that the authors should discuss the concerns raised by Reviewer 5vEw—specifically regarding baselines and datasets—in the related work and method sections. Consequently, the AE recommends acceptance with minor revisions.
Decision: Accept (poster)