Global Prototype Encoding for Incremental Video Highlights Detection
Authors: Sen Pei
Shixiong Xu
Ye Yuan
Jiashi Feng
Xiaohui Shen
Xiaojie Jin
Venue: Submitted to ICLR 2023 (2023)
Keywords: Video highlights detection
Incremental learning
Newly released dataset

Abstract: Video highlights detection (VHD) is an active research field in computer vision, aiming to locate the most user-appealing clips given raw video inputs. However, most VHD methods are based on the closed world assumption, \emph{i.e.}, a fixed number of highlight categories is defined in advance and all training data are available beforehand. Consequently, existing methods have poor scalability with respect to increasing highlight domains and training data. To address above issues, we propose a novel video highlight detection method named \textbf{G}lobal \textbf{P}rototype \textbf{E}ncoding (GPE) to learn incrementally for adapting to new domains via parameterized prototypes. To facilitate this new research direction, we collect a finely annotated dataset termed \emph{LiveFood}, including over 5,100 live gourmet videos that consist of four domains: \emph{cooking}, \emph{eating}, \emph{ingredients} and \emph{presentation}. To the best of our knowledge, this is the first work to explore video highlight detection in the incremental learning setting, opening up new land to apply VHD for practical scenarios where both the concerned highlight domains and training data increase over time. We demonstrate the effectiveness of GPE through extensive experiments. Notably, GPE surpasses popular domain-incremental learning methods on \emph{LiveFood}, achieving significant mAP improvements on all domains. The code and dataset will be made publicly available.

Under review as a conference paper at ICLR 2023
GLOBAL PROTOTYPE ENCODING FOR INCREMENTAL VIDEO HIGHLIGHTS DETECTION
Anonymous authors Paper under double-blind review
ABSTRACT
Video highlights detection (VHD) an active research field in computer vision, aiming to locate the most user-appealing clips given raw video inputs. However, most VHD methods are based on the closed world assumption, i.e,, a fixed num- ber of highlight categories is defined in advance and all training data are available beforehand. Consequently, existing methods have poor scalability with respect to increasing highlight domains and training data. To address above issues, we pro- pose a novel video highlights detection method named Global Prototype Encoding (GPE) to learn incrementally for adapting to new domains via parameterized pro- totypes. To facilitate this new research direction, we collect a finely annotated dataset termed LiveFood, including over 5,100 live gourmet videos that consist of four domains: cooking, eating, ingredients and presentation. To the best of our knowledge, this is the first work to explore video highlights detection in the incre- mental learning setting, opening up new land to apply VHD for practical scenarios where both the concerned highlight domains and training data increase over time. We demonstrate the effectiveness of GPE through extensive experiments, No- tably, GPE surpasses popular domain-incrementa. learning methods on LiveFood, achieving significant mAP improvements on all domains. The code and dataset will be made publicly available.
INTRODUCTION
The popularization of portable devices with cameras greatly promotes the creation and broadcasting of online vidcos. These sufficient video data serve as essential prerequisites for relevant researches,
e.g. video summarization (Potapov al. 2014 Song et al. 2015| Zhang et al., 2018: Fajtl et al., 2018 Zhu et al. 2021), video highlights detection (VHD) (Yng et al. 2015) Xiong et al. 2019 Lei et al. [021]] Bhattacharya et al.] 202T), and moment localization (Liu et al. 2018 Zhang al, 2020 Rodriguez et al.) 2020). to name few. Currently, most VHD methods are developed under the closed world assumption, which requires both the number of highlight domains and the size of training data to be fixed in advance. However, stated in Rebuffi et al. (2017). natural vision systems are inherently incremental by consistently receiving new data from different domains or categories. Taking the gourmet video as example, in the beginning, one may be attracted by the clips of eating foods, but lately, he/she may raise new interests in cooking and want to checkout the detailed cooking steps in the same video. This indicates that the target set the model needs to handle is flexible in the open world. Under this practical setting, all existing VHD methods suffer from the scalability issue: they are unable to predict both the old and the newly added domains, unless they retrain models on the complete dataset. Since the training cost on videos is prohibitive, it is thus imperative to develop new methods to deal with the above incremental learning issues.
Broadly speaking. there exist two major obstacles that hinder the development of incremental VHD: a high-quality VHD dataset with domain annotations and strong models tailored for this task. Recall existing datasets that are widely used in VHD research, including SumMe (Gygli et al., 2014), TVSum (Song et al., 2015). Video2GIF (Gygl et al.) 2016). PHD (Garcia del Molino & Gygli 2018), and QVHighlights (Lei et al. 2021), all of them suffer from threefold drawbacks: (1) only the feature representations of video frames are accessible instead of the raw videos, thus restricting the application of more powerful end-to-end models; (2) most datasets only have a limited number of videos with short duration and coarse annotations, which are insufficient for training deep models; (3) none of them has the video highlight domain or category labels, thus not be directly used in

---

Under review as a conference paper at ICLR 2023
Figure 1:
The LiveFood dataset. The row from top to bottom illustrates examples of vanilla clips, ingredients and presentation respectively. More samples are attached in Appendix A.2
incremental learning. In order to bridge the gap between VHD and incremental learning, we first collect a bigh-quality gourmet dataset from live videos, namely LiveFood. contains over 5,100 carefully selected videos with 197 hours in total. Four domains are finely annotated, i.e., cooking, eating, ingredients and presentation. These related but distinctive domains provide a new test bed for incremental VHD tasks.
To solve this new task, we propose a competitive model: Global Prototype Encoding (GPE) to learn new highlight concepts incrementally while still retaining knowledge learned in previous video domains/data. Specifically, GPE first extracts frame-wise features using a CNN, then employs a transformer encoder to aggregate the temporal context to each frame feature, obtaining temporal- aware representations. Furthermore, each frame is classified by two groups of learnable prototypes: highlight prototypes and vanilla prototypes. With these prototypes. GPE optimizes a distance-based classification loss under L2 metric and encourages incremental learning by confining the learned prototypes in new domains to be close that previously observed. We systematically compare the GPE with different incremental learning methods on LiveFood. Experimental results show that GPE outperforms other methods on highlight detection accuracy (mAP) with much better training efficiency, using complex exemplar selection complicated replay schemes, strongly evidencing the effectiveness of GPE.
The main contributions of this paper are summarized as follows:
We introduce new task named incremental video highlights detection, which has im- portant applications in practical scenarios. A high-quality LiveFood dataset is collected facilitate research in this direction. LiveFood comprises over 5,100 carefully selected gourmet videos in high resolution, providing new test bed for video highlights detection and domain- intemminnaa learning tasks.
We propose novel end-to-end model for solving incremental VHD, i.e., Global Prototype Encoding (GPE). GPE can incrementally identify highlight and vanilla frames in new high- light domains learning extensible and parameterized highlight/vanille prototypes. GPE achieves superior performance compared with other incremental learning methods, improv- ing the detection performance (mAP) by 1.57% on average. The above results suggest that GPE can serve as a strong baseline for future research.
We provide comprehensive analyses of LiveFood as well as the proposed GPE model for deepening the understanding of both, as well as giving helpful insight for future develop- ment. We hope our work can inspire more researchers to work in incremental VHD, finally pushing forward the application of VHD in practical scenarios.
RELATED WORK
Video Highlights Detection (VHD) is an important task in video-related problems. This line of research be roughly divided into two groups, namely the ranking-base and regression-based
2

---

Under review as a conference paper at ICLR 2023
methods. Yao et al. (2016) employs ranking model to learn the relationship between highlights and honhhighlights, assigning higher scores to the positive clips. Saquil et al. (2021) utilizes multiple pairwise rankers to capture both the local and global information. [Badamdor] al. (2021) assigns higher scores to annotated clips based on dual-modals, ie., the visual and audio streams. Based on ranking methods, lot of works aim to mitigate the expensive cost of human annotation using unsu- pervised techniques or priors, such as (Xiong et al., 2219)] [Bdannorrj et al., 2022). Different from the above methods, regression-based methods predict the locations of highlights directly, Zhu et al. (202T) presents anchor-base based and anchor-free approaches to predict the start and end timestamps, as well as the confidence score of highlights, therefore avoiding the laborious manual designne post- processing. Moment-DETR (Lei et al. (021) employs a transformer decoder to obtaín the times- tamps of specific clips based on different queries. Although the existing methods mentioned above boost the performance of VHD tasks, they substantially neglect the requirements of incremental learning in VHD, which is critical to the practical applications of VHD. In reality, numerous videos and new interests are created rapidly, thus demanding the VHD model to be capable of efficiently handling increasing highlight domains and data.
Incremental Learning (IL) is of great concern since the natural vision systems are inherently in- cremental. The main problem to solve in incremental learning is catastrophic forgetting, manifested as the forgetting of old classes domains when learning new concepts. As investigated in Langel ett al. (2022), the primary efforts deal with this issue in three aspects: using memory buffer to store the representative data (Rebuffi et al.) 2017 Isele& & Cosgun| 2018 Rolnick et al. 2019 Yan al. 2021 Lange& & Tuytelaars] 2021), adopting regularization terms to constrain the change of model's weights or outputted logits (Kirkpatrick et al. 2016, Zenke et al. 2017 Schwarz et al. 2018 and
performing parameter isolation to dedicate different model parameters for each task (Fernando et al.) N017|Malyy & Lazebnik, 2018, Rosenfeld & Tsotsos 2020). The memory buffer replays previous samples while learning new concepts to eliminate forgetting, however, it may lead to overfitting on the stored sub-set and incur heavy memory costs. The regularization-bass model if some characteristics changed during the next training stage, resulting in the domination of the so-called essential characters. Parameter isolation grows new branches for new tasks, raising prohibitive colossal architectures. To mitigate the negative effects of existing methods, GPE employs prototype learning combined with distance measurement to perform binary classification (highlight vanilla frames). Prototypes are essentially the most representative features learned across the whole training data, circumventing both the issue of overfitting on the sub-set and the unbearable costs of storing raw data. Besides, we constrain the change of prototypes between stages which is an overall refinement instead of minority domination. The prototypes from previous stages are inherited during training in the current domains, so as to maintain global consistency while leaving room for adjustment and improvements.
methods penalize the
3 PROBLEM STATEMENT
In incremental VHD, the training procedure consists of several consequent tasks built on disjoint datasets with distribution shifts. As- suming that we have T tasks in total, and this yields training data stream (Ti, T2.... } where T; n T Ø if + j. Each training where of E X' denotes the whole frame set of i-th training video in stage t. o is its correspond- ing frame-wise label (i.e., a binary vector in- dicating highlight/vanilla frames), and n, rep- resents the number of accessible training data pairs. Moreover, we use {D1, D2,..., Dr to de- scribe the corresponding domains included in training tasks (Ti, T2,..., Note D. * D, i i j. Specifically, in our proposed LiveFood, we split the training videos into four disjoint sub- set (Ti, Tz, Ts. Ta). and the corresponding domains are denoted as (D,,D2, Ds. D.}. Considering that a video may consist of more than one domain, further constrain that the domains appear- ing in D is a sub-collection of that in D,. Formally, let S, denotes all possible combinations of domains presented in D., and C, is the domain combinations appearing in videos of Tr, have
Tj
Ts
Ts
T
...... Training Stage:
A video stream
.....
Testng Stage:
Fixed lesting sel
task T, is represented as {(x (< " th
D
D2
D3
Dr
Figure 2: Illustration of conducting incremental VHD on LiveFood.
3

---

Under review as a conference paper at ICLR 2023
C1 S and C. = S, S U'=iS, More concretely, let d, denotes specific domain label, then if D1 = {d1}, D2 = [d1, d2), and D3 = {d1, d2, d3 we have C1 = {d1], C2 = {d2, (d1, d2)} and C3 {dz, (d1, dg), (d2, da), (d1, d2, d. In above example, C2 (d), (d1,d))) d2)) indicates that the videos in T2 can contain the domain of d2 or the mixture of dj and d2. Videos that merely contain the domain of d1 excluded from T2 since the intention of incremental VHD is effectively learn new concepts while remembering what are already learned in past data. The testing set con- tains mixed videos including all domains, and during task 7, only the domain appearing in D, is treated as positive when evaluating the performance.
4 THE LIVEFOOD DATASET
Video Selection. We collect online gourmet videos with high resolution. As introduced in Xiong al. (2019), shorter videos are more likely to contain attractive clips and thus have more hits, while longer videos are usually boring. Taking this prior into consideration, we filter out both the extremely short (less than 30 seconds) which may contain insufficient gourmet content learn from and long (over 15 minutes) videos which users generally pay less attention to. After that, all reserved raw videos are viewed by qualified workers to check whether the content of videos is gourmet-related
not, eliminating the effects of incorrect category annotation. Only videos that pass the aforementioned checks are selected for subsequent annotation tasks, order to guarantee the quality of LiveFood. Figure (a) shows the duration distribution of the videos across all domains.
Percentage (%)
Percentage (%)
Percentage (%))
la is n1 ase Duration of highlights (sec)
Video duration (sec)
Relative position of highlights
(b)
(c)
Figure 3: Statistical results of the proposed LiveFood. (a) shows the distribution of video duration.
(b) illustrates the distribution of highlight durations. Most highlight clips are shorter than 15 seconds. (c) shows the relative position of each attractive clip w.r.t. corresponding videos. The highlights distribute evenly across whole videos, evidencing the good diversity of LiveFood.
Highlights Annotation. We define four highlight domains that are generally presented collected videos, namely cooking, eating, ingredients and presentation. For each domain, video clip is accepted as a satisfactory highlight if it meets the following criteria in Table
Table 1: Basic description of annotated domains in LiveFood.
Domain
Description
cooking eating
The process demonstrates the chef's exquisite cooking skills.
People enjoy foods with exaggerated or satisfied expressions on their faces. The video clip shows at least three kinds of rare ingredients in high quality.
ingredients
presentation
A well-displayed meal to make it looks more appetizing.
The annotators are required to glance over the whole video first to locate the coarse position of at- tractive clips. Afterward, the video annotated at frame level from the candidate position verify the exact start and end timestamps of highlights. Meanwhile. we introduce strict double-check mechanism (cf. Appendix A..)) to further guarantce the quality of annotations. Since selecting the timestamps of highlights is partly subjective, both objective and subjective verifications neces- sary in quality control. Concerning the consistent visual feeling of videos, we restrict the highlights to be longer than three seconds, and less than two minutes avoid being tedious.
Data Statistics. Figure depicts the statistical results of our proposed LiveFood, including the distribution of video duration, the distribution of highlights duration, and the relative position of
4

---

Under review as a conference paper at ICLR 2023
center timestamp with respect to each video, etc. Besides, in Table 2 also compare the proposed LiveFood with existing VHD datasets such as SumMe Gyyli et al. 2014), YouTubeHighlightt (Sun
et al. 2014), Video2GIF (Gygh et al. 2016), PHD (Gareia del Molino & Gygli 2018) and QVHigh-
lights Lee et al. 2021) for better illustrating the differences among them.
Table 2: Comparison between the proposed LiveFood and existing datasets.
Label
Total number of
Avg. len. (sec) videos highlights
Dataset
Year
Contents
domain/ class
videos / highlights
SumMe
2014
Open
x
25 390
120.0 /-
Higghhhti
2014
Activity
x
712/-
143.0 2.0
Video2GIF
2016
Open
X
80K / 98K
332.0 5.2
PHD
2018
Open
X
119K / 228K
440.2 5.1
QVHighlights
2021
Vlog News
X 10.2K / 10.3K
150 24.6
LiveFood (ours)
2022
Gourmet
5.1K/ / 14,3K
136.5 / 6.4
As demonstrated in Table 2 the SumMe and You Highhheets only contain a small number of videos and annotations, which makes them insufficient for training deep models. The Video2GIF and PHD are edited by online users and lack strict quality control mechanisms. Thus the reliability of datasets may be undermined. The newly released QVHighlights can not be used for incremen- tal learning since it does not have domain annotations. Besides, the average length of clips in QVHighlights is pretty long: nearly one-fifth of each video is annotated as attractive clips, thus causing the selected clips less discriminative compared to the vanilla clips. Different from the above datasets, LiveFood provides gourmet videos with finely annotated domain labels, making it suitable for Gomminn-ncremmentll VHD tasks.
METHOD
GPE aims tackle forgetting while still improving by learning new concepts. As analyzed in Section 2 conventional incremental learning methods have shortcomings, such as overfitting on replayed data, limited flexibility, and unbearable growing architectures. Distinguished from them, GPE employs prototypes together with distance measurement to solve the classification problem. Prototypes are compact and concentrated features learned on the training data, mitigating the effects of overfitting the stored sub-set. In addition, by using global and dynamic prototypes, we endow the model with appealing capability for further refinement when fed with new data (M, in Figure or accommodation new domain concepts (Ma in Figure 4
Architecture. Inspired by Carion et al. 2020), GPE employs the combination of convolution- and attention- based models to extract features. Concretely, a ConvNeXt (Liu et al. 2022) pre- tre-trained on ImageNet (Russakovsky et al., (015)) is used to extract spatial features of input video frames. After that, transformer encoder with multi-heads is used to perform temporal fusion, generating global representations based on the whole video frames. With the transformation of a feedforward network (FFN) which consists of fully-connected layers, each frame is classified based on the distance to learnable prototypes. We aim to learn two groups of trainable prototypes with the same shape, namely the highlight (positive) and vanilla (negative) prototypes. By denoting the dimensionality of output feature of transformer as m and the number of prototypes within each group as k. both the highlight and vanilla prototypes can represented as a matrix with shape distance as the distance measurement between each feature and prototype, we obtain the pair- wise distance between features and each group of prototypes. Formally, we use h, H, and V to represent the transformer feature, the highlight prototypes and the vanilla prototypes. (-(-) denotes the FFN module. d(-) is the L2 distance between a naturr-prootoype pair. The distance from feature to H and V are formulated as:
m. By utilizing L22
d min (h). lik
= = min d(g) i=ikk
where the subscript i represents the i-th prototype. The distance is mapped to probability Py using the sofumax function which can be understood as the confidence of assigning feature h to highlights.
Ps = exp(-dn) + exp(-dv))
(2)
5

---

Under review as a conference paper at ICLR 2023
PFN
spatial features
temporal fusson
M
Ma
vanilla prototypes
constrain all protolypes
T
T
T2
transformer encoder
CNN
T2
highlight
constrain old prototypes
Tr
prototypes
0100 time encoding
Tr
Lele distance based clustering
Figure 4: The proposed GPE framework. {T:) indicates a training stream with T tasks. In the right-most part, Mr and Ma represents the fixed and dynamic modes of GPE. Mr defines the number of prototypes in advance and refines them across different stages. restriction on the magnitude of change amplitude is imposed during learning (cf. Eq. 5) My dynamically adds new prototypes into the learning process when dealing with new domains. The change restriction only applied on inherited prototypes. This mode is more suitable when learning a large amount of domains. Each prototype in above figure is equivalent to a row of V or H (cf. Eq. I
Then, we use cross-entropy loss to optimize the model through gradient paok-propagation:
N
Lala
Σ Ve log( (Pr) + (1) llog(1 Pи))
(3)
where N represents the size of training frames and yi equals 1 if the i-th frame is annotated as highlights otherwise
Learning with incremental domains. We detail the learning of Mr (Figure 4 in this section, and the dynamic mode Ma can be easily derived by only restricting the change of inherited proto- types and training newly added prototypes freely. We h-)-) parameterized by 0 to indicate the feature extractor jointly constructed by a ConvNeXt and transformer encoder. The FFN (-(() is parameterized by é. Recall the outputted feature is h. Both the vanilla and highlight prototypes are denoted by æ for simplicity. With the help of these notations, the classification loss built in Eq. B abbreviated as (0, Ø, π). We expand the definition of distance measurement for evaluating the distance between given two prototypes. For two sets of learned prototypes ((1) and ((++1), the distance between them is calculated as:
d(x((0) ((+1)
(1+1))2 Ji
(4)
During the training phase T, the model inherits trained prototypes .. from the former stage. For the incremental need, we tackle the catastrophic forgetting issue by restricting the change of pro- totypes, guarantecing the awareness that the model has learned towards the observed domains. With the above formulations, we consider the following constrained nonlinear optimization problems:
min 8,0,5
(0, (,)
(5)
s.t. d(r (T-1) 5
where ? is the tolerable change introduced the observed prototypes. The optimal result that meets the above restriction is (B(T), ((T), (T)). Instead of solving the complex nonlinear problem, we resort to its corresponding empirical dual formulation. With an auxiliary positive Lagrange multiplier X, the optimization objective in Eq. i transformed into the following manner:
((T) (0(T)) ((T) (T))
= max min X 0.0.
(6)
max min Less (0, @, + ([d(x
À 8,0.0

---

Under review as a conference paper at ICLR 2023
where S(T) indicates the optimal solution in training stage T. We update the trainable parameters (i.e., 8, o, and т) and the empirical Lagrangian variable À alternatively and iteratively:
0 - 0 ec (0,
00
Φ - 0 aLcia O,
(7)
π f 0L Φ, n. A)
dm
A max nld( (T- n) -
where 7j is the learning rate trainable parameters and À is the multiplier in the dual step. With the above analysis, the incremental training and inference pipeline is summarized in Algorithm
Algorithm 1: Global Prototype Encoding for Incremental Video Highlights Detection.
Input: training data X' with is stages; initial ho()), g6 (-), and x; initial 7) 7, and X.
Output: trained ha O), and π at each stage.
while Training do for stage i=l:k do
while Inference do for stage i=1:k do
Reset 17, ?y. and A;
12
repeat
repeat
Sample {x, y) from the testing set at
Sample batch data (z, y) from within stage i;
stage i;
Calculate logits: / = Do (hr(:));
Calculate outputted logits:
15
Classify inputs based on the distance between and m
(=)h)²z));
Calculate the loss function with Eq
until the testing set is 0;;
6
17
Average the mAP cross domains.
Update 0, Φ, n, and À with Eq until 6, ø. and π are converged;
Average the mAP cross all stages.
6
EXPERIMENT
We introduce the details of the evaluation protocol and experimental results in this section.
6.1 EXPERIMENTAL SETUP
Data and Evaluation Protocol. LiveFood contains 4928 videos for training and 261 videos for testing. We randomly split 15% of the 4928 videos for validation. Tı. T2, Ta and Ta consist of 3380, 854, 393, and 113 videos respectively. D1, D2, D3 and D4 (presentation), {presentation, eating), ). {presentation, eating, ingredients). and (presentation, eating, ingredients, cooking). We report the mAP testing following previous works (Yo al. 2016 Xiong al. 2019).
GPE. GPE only finetunes the last layer of ConvNeXt during training. The transformer encoder has heads and 3 layers. The feedforward module g-(() is a multi-layer perception with 3 linear layers activated by ReLU. Both the vanilla and highlight prototypes are formulated as 40 vectors with dimensions of 128. During every stage, the model is trained with 300 epochs, and the learning rate halves every 70 epochs starting with 1e-3, GPE is initialized randomly during the first stage. In later stages, trains prototypes in M, by starting from weights learned all prototypes learned in previous stages are inherited and trained as similar as in Ma. In addition to these inherited prototypes, in each stage, new prototypes are randomly initialized and added to Ma to enhance the learning of new concepts.
former stage. In Ma
Regularization-base methods. SI (Zenke et al., 2017) and oEWC (Schwarz et al., 2018) are representatives. These schemes update the model with compromised loss function, utilizing both the overall importance previous stages and the current task to eliminate forgetting.
Replay methods. ER (Rolnick al. 2019) and DER (Buzzega et al. 2020) memory buffer to store the representative data from the previous task to defend against forgetting. The buffer size is set to 200 if not specified.

---

Under review as a conference paper at ICLR 2023
Presentation
cooking
cooking
presentation
GPG
GPC
DEN 7g
Figure 5: The observed highlights across different training stages.
Lower bound (Lb). In each stage, GPE is trained without constraints (cf. Eq. 5) suffering from severe catastrophic forgetting and rendering performance drops with the increasing of tasks.
Upper bound (Ub). In each stage, GPE trained with data from all stages. Thus is free from the forgetting issue and provides the upper bound performance for all incremental learning methods.
6.2 MAIN RESULT
Comparison with existing IL methods. The experimental results are depicted in Table We highlight the upper bound results with gray background. It can be observed from Table that the vanilla GPE (M+) surpasses the lower bound with an improvement
2.91% mAP. Compared to the classic IL methods SI and oEWC, GPE outperforms them by remarkable margin, yielding at least 1.57% performance gain on mAP. Moreover, when equipped with the same replay schemes as Yan al. (202I) GPE achieves 1.06% and 2.71% mAP gain compared DER and ER. The above results clearly demonstrate the effectiveness of GPE in tackling the incremental VHD task.
Table 3: Comparison of GPR with existing incremental learning methods on LiveFood. We evaluate their Trame-wise mAP performance.
MAP
Lb
ST
oEWC
ER
DER*
Ub
GPE(M:)
GPE(M))
GPE*
T 36.13
36.16
36.13
35.79
36.17
36.15
36.14
36.21
36.17
T2
30.86
31.84
31.82
31.38
33.14
37.38
35.82
36.13
36.62
29.18
30.72
30.51
29.13
32.52
36.90
31.87
32.74
33.15
T4
25.89
28.73
28.67
29.06
30.11
36.30
29.88
30.15
30.27
Avg.
30.52
31.86
31.78
31.34
32.99
36.68
33.43
33.90
34.05
*: using memory buffer to replay samples.
Visualization of highlight scores across training stages. We investigate the effects of observed prototypes across different training stages. In Figure we present the highlight detection results of GPE the first and the last training stage. For comparison, we also provide the prediction of DER. In the curves shown in Figure 5 the blue and orange points indicate the highlight scores of each frame predicted by GPE (i.e., PH in Eq. 22 during the first task Ti and the final task T.. The green points represent the predicted scores of DER in T4. It is clear that GPE can learn e.g., cooking while keeping the memory of presentation learned in the first stage. This result is in line with our motivation that strong incremental VHD model should be able to cover both the past and new concepts. In contrast, since DER employs stored data to strengthen memory, it still has the drawback of being prone forgetting due the limited buffer size. This is demonstrated by its assigning much lower scores to the old domain of presentation when learning cooking.
concepts,
Scalability of GPE in the dynamic manner. We investigate the generalization ability of dynamic GPE (Ma) in the scenario where the model needs to handle large number of domains. We consider the R-MNIST dataset containing series of rotated digits with different degrees between [0, r), where each degree represents domain. For a fair comparison, we use the identical settings as Buzzega et al.] (2020), yielding a stream with 20 subsequent tasks. Note that no augmentation techniques used. In this experiment, GPE is simplified to be small network with 2 fully- connected layers followed by ReLU. The number of prototypes set to 5 per class. Therefore each task T, has 5t prototypes per class by inheriting from previous stages and generating 5 new prototypes for each class. The old prototypes are forbidden to change too much. À and y are and 1e-2. All other experimental settings follow Buzzega al. 2020)
8

---

Under review as a conference paper at ICLR 2023
Table 4; Average classification accuracy R-MNIST.
Method
SI
oEWC
ER
GEM
FDR
GSS
HAL
DER
GPE (M.)
GPE (M.)
Buffer Size
X
X
200
200
200
200
200
200
X
200
Avg Acc.
71.91
77.35
85.01
80.80
85.22
79.50
84.02
90.04
85.42
90.17
Results depicted in demonsteee that the dynamic GPE surpasses most conventional methods, including the regularization-basee -based and replay methods. Notably, the dynamic GPE achieves 85.42% average accuracy across 20 tasks, comparable to ER (Rolnick et al.) 2019) and FDR (Benjamin al., 2019) while outperforming other methods with considerable margin. We further adopt the al. [2021), and this helps the dynamic GPE achieve 90.17% top-1 accuracy, which established as a new comparable state-of-the-art approach.
identical replay schemes as DER (Yan et
6.3 ABLATION STUDY
Ablation on the initial number of prototypes k. The number of prototypes reflects the model's capacity. Too many prototypes increase the training cost while too few lead to underfitting. Results shown in Table 5 provide a comparison between the training cost and performance under the fixed mode of GPE (M:) LiveFood. It is observed that with the increasing initial quantity of prototypes k, the average mAP over all tasks consistently increases. However, by comparing the last two rows in Table 5 notice that the performance gain between k 40 and 50 is marginal though more parameters are introduced. Therefore, we set k to 40 throughout experiments to strike a good balance between accuracy and efficiency.
Table 5: Ablations on the initial number of prototypes k.
Table 6: Ablations on the changing con-
straints of distance
mAP (y 5.0)
mAP (k
40)
k
7
Ti
T2
Ta
T
Avg.
Ti
T2
Ts
Ti
Avg.
10
35.31
33.48
29.18
25.98
30.99
le-3
36.14
34.26
30.17
26.43
31.75
20
35.82
34.66
30.72
28.73
32.35
1.0
36.13
34.97
30.62
27.41
32.28
30
36.13
35.70
30.51
28.67
32.75
3.0
36.15
35.50
31.44
28.27
32.84
40
36.14
35.82
31.87
29.88
33.43
5.0
36.14
35.82
31.87
29.88
33.43
50
36.21
35.88
31.90
29.94
33.48
15.0
36.14
34.92
30.80
28.12
32.50
Ablation on distance constraint y,. Extremely small y hinders the model from learning new con- cepts since the prototypes are almost unchanged. In contrast, too large may lead to catastrophic forgetting since the model may heavily overfit to the newly observed data. As shown in Table we set à to 40 by default and investigate the effects of y with different values. In our experiments, find the distance between the vanilla and highlight prototypes after Ti is less than 15, so is set as that when is small, say 1e-3, GPE can hardly
the upper bound of 7. From Table [6, learn the new contents, only achieving similar mAP compared to the lower bound method as shown in Table 3 By enlarging 7, the average mAP increases consistently from 31.75 33.43. When is 15, the model suffers from the forgetting issue as explained before, resulting in a near 1.0% performance drop. Consequently, we set k and y to 40 and by default.
CONCLUSION
In this paper, we introduce new task: incremental video highlight detection, aiming to perform VHD in the practical scenario where both the highlight domains and data increase over time. To pave the road in this new direction, we collect high-quality video gourmet dataset LiveFood which contains four fine-annotated domains. Then we propose a new end-to-end model named Global Prototype Encoding (GPE) learn incrementally to adapt experimental results clearly demonstrate the effectiveness of our method. We hope this work serves to inspire other researchers work on this new and critical task.
highlight domains. Extensive

---

Under review as a conference paper at ICLR 2023
REFERENCES
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alchh--Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 11816-11825, 2019. URL http:::/proceeeaae.eieeee mewraiinnnnnnnnu.....eee.m nwwripsu.ppp.. httppps.ccppppppeee cctpappr//019// hash/
Absttr/tttbbb-ttaaat..... .ttll
Taivanbat Badamdorj, Mrigank Rochan, Yang Wang, and Li Cheng. Joint visual and audio learn- ing for video highlight detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 8127-8137, October 2021.
Taivanbat Badamdorj. Mrigank Rochan, Yang Wang. and Li Cheng. Contrastive learning for un- supervised video highlight detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14042-14052, June 2022.
Ari S. Benjamin, David Rolnick, and Konrad P. Körding. Measuring and regularizing networks in function space. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA. May 6-9, 2019. OpenReview.net, 2019. URL
net hett/forum? 2 d=5kMwpi R9Y7
Uttaran Bhattacharya, Gang Wu, Stefano Petrangeli, Viswanathan Swaminathan, and Dinesh Manocha. Highlightme: Detecting highlights from human-centric videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 8157-8167, October
2021.
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calder- ara. Dark experience for general continual learning: a strong, simple baseline. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL WoLhhtts:: http:://poooeeeeggs.ceeeeee neurips....e.....p..p........... CC http:/ppprrr//// Saaper2020//ast// b704ea2c33 3977397777 9779977777777 6
Abst htmml
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020 16th Eu- ropean Conference, Glasgow, UK, August 28-28, 2020. Proceedings, Part I, volume 12346 of Lec- ture Notes in Computer Science, pp. 213-229. Springer, 2020. doi: 10.10007998--3-030--44555 _13. URL htt.hhttt///eett org 10. 1007 0-00888------------------8-883
Hyuntak Cha, Jaeho Lee, and Jinwoo Shin. Co21: Contrastive continual learning. In 2021 IEEF/VVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-
17, 2021, pp. 9496-9505, IEEE, 2021. doi: 10. 1011000CCC4492222210000000 1ttp:///001100//C//33840
URL https:
2021 00938
Arslan Chaudhry, Mare' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with A-GEM. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https:
d Hkf2 sC5FX
Arslan Chaudhry, Albert Gordo, Puneet K. Dokania, Philip H. S. Torr, and David Lopez-Paz. Using hindsight to anchor past knowledge in continual learning. In Thirty-Fifth AAAI Conference on Ar- tificial Intelligence, AAAI 2021, Thirty-Third Conference on Inmovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelli- gence, KAA/ 2021, Virtual Event, February 2-9, 2021, pp. 6993-7001. AAAI Press, 2021. URL https: ojs aaai. org/ index. php/ http/////aaatteeee//////s///////
Jiri Fajtl, Hajar Sadeghi Sokeh, Vasileios Argyriou, Dorothy Monekosso, and Paolo Remagnino. Summarizing videos with attention. In Gustavo Careeiro and Shaodi You (eds.), Computer Vi- sion ACCV 2018 Workshops- 14th Asian Conference on Computer Vision, Perth, Australia,
10

---

Under review as a conference paper at ICLR 2023
December 2-6, 2018, Revised Selected Papers, volume 11367 of Lecture Notes in Computer Science. pp. 39-54. Springer, 2018. doi: /rgoi.org//l 1007000-0077770000-0000
URL https:
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A. Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. CoRR, abs/1 007344, 2017. URL
08734
Ana Garcia del Molino and Michael Gygli. PHD-GIF's Personalized Highlight Detection for Auto- matic GIF Creation. In Proceedings of the 2018 ACM on Multimedia Conference, MM '18, New York, NY, USA, 2018. ACM.
Michael Gygli, Helmut Grabner, Hayko Riemenschneider, and Luc Van Gool. Creating summaries from user videos. In ECCV, 2014.
Michael Gygli, Yale Song, and Liangliang Cao. Video2gif: Automatic generation of animated gifs from video. In 2016 IEEE Conference on Computer Vision and Puttern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 1001-1009, IEEE Computer Society, 2016. doi: doi org wwt..11119/..E... 201.://00..1111//.....11.....
00.1109/CVR..2016.0
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
David Isele and Akansel Cosgun. Selective experience replay for lifelong learning. In Sheila A.
Mcllraith and Kilian Q. Weinberger (eds.), Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI- 18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pp. 3302-3309, AAAI Press, 2018. URL https: www. aaai. ort/ index. AAAT AAAAAAAAAIII /ttppp///AAAAAAAAAAAAaaaee/ee/// 11C5 16054
James Kirkpatrick. Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, An- drei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinst Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic for- getting in neural networks. CoRR, abs/1612.00776, 2016. URL http::///xxv...rrrr/t///
1612. 00796)
Matthias De Lange and Tinne Tuytelaars. Continual prototype evolution: Learning online from non-stationary data streams. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC. Canada, October 10-17, 2021, pp. 8230 8239. IEEE, 2021. doi: 10. 1109/CCCV48922.2221.088
URL |tL|attps::/aoo.ooo
110.00..11101111 1wt.1110///CVVC2222222 2021.
00814
Matthias De Lange. Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory G. Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE Trans. Puttern Anal, Mach. Intell., 44())3366-3385,, 2022. doi: 10.T/09/
2021.3057446. URL /oi. htt.:/ii..ggg/oooooooooooooooeee 10 1109/ / TPAMI 2021 3057446
Jie Lei, Tamara L Berg. and Mohit Bansal. Detecting moments and highlights in videos via natural language queries. Advances in Neural Information Processing Systems, 34:1 1 1846-118881 11858, 2021.
Meng Liu, Xiang Wang, Liqiang Nie, Qi Tian, Baoquan Chen, and Tat-Seng Chua. Cross-modal moment localization in videos. In Susanne Boll, Kyoung Mu Lee, Jiebo Luo, Wenwu Zhu, Hyeran Byun, Chang Wen Chen, Rainer Lienhart, and Tao Mei (eds.), 2018 ACM Multimedia Conference on Multimedia Conference, MM 2018, Seoul, Republic of Korea, October 22-26, 2018, pp. 843- doi: 00114453224008.32405555 3240508 3240549
851. ACM, 2018.
URL hitps: /etps:///ci... /wi. http:////2.100/1111...000
Zhuang Liu, Hanzi Mao, Chao-Yure Yuan Wu, Christoph Feichtenhofer. Trevor Darrell, and Saining Xie. A convnet for the 2020s. Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition (CVPR), 2022.
11

---

Under review as a conference paper at ICLR 2023
David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vish- wanathan, and Roman Gamett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 7, December 4-9, 2017, Long Beach, CA, USA, pp. 1467-6476 2017. URL https: http::///rccceeee wtw.eeeeedidiggee.. nwwrips..p....... ec/
o html
Abstract
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to single net- work by iterative pruning. In 2018 IEEE Conference on Computer Vision and Pat- tern Recognition, CVPR 2018, Salt Lake City, UT. USA, June 18-22, 2018, pp. 7765- 7773. Computer Vision Foundation / IEEE Computer Society, 2018. doi: 10.1109/CVPR. 2018.00810.
thecvf. com/ content evpr 2018/htm1/ Mallya PackNet Addinn Multiple CVPR__ 2018 Ba1l___pppp_____ html
Danila Potapov, Matthijs Douze, Zaid Harchaoui, and Cordelia Schmid. Category-speciff video summarization. In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision - ECCV 2014 13th European Conference, Zurich, Switzerland, September 6- 12, 2014, Proceedings, Part VI, volume 8694 of Lecture Notes in Computer Science, pp. 540-555. Springer, 2014. doi: 10.1000//78-3-311 10599-4) 33... URL pontps://dee org 100///10.100 10tg//11.1007//1010 978 319 10599-4 35
Sylvestre- Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. icarl: Incremental classifier and representation learning. In 2017 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26. 2017, pp. 5533-5542. IEEE Computer Society, 2017. doi: 00.109//VVP.2211..55.. URLhttps. //w.///////// org/10.
LIC9/CVPR.2017.55 587
Cristian Rodriguez, Edison Marrese-Taylor Fatemeh Sadat Saleh, HONGDONG LI, and Stephen Gould, Proposal-free temporal moment localization of a natural-language query in video using guided attention. In Proceedings of the IEEE/CVF Winter Conference on Applications of Com- puter Vision (WACV), March 2020.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy P. Lillicrap, and Gregory Wayne. Experience replay for continual learning. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' AlchééBucc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Pro- cessing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC. Canada, pp. 348 458-358, 2019. URL https: htttps://prroceedings neur i ips. CC htt.ppaar//201//hasi html
Amir Rosenfeld and John K. Tsotsos. Incremental learning through deep adaptation. IEEE Trans. Pattern Anal. Mach. Intell., (2(3):651666) 2020. doi: 10.1109/TPAMI
URL
htttps://dii. orogo0...............oooo....... 1109/ TPAMI. 2018 2884462
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV) 115(3):211-25 2015. doi: 10.1007/s| 1263-015-0816-y.
Yassir Saquil, Da Chen, Yuan He, Chuan Li, and Yong-Liang Yang. Multiple pairwise ranking networks for personalized video summarization. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1718-1727, October 2021.
Jonathan Schwarz, Wojeiech M. Czarnecki, Jelena Luketina, Agnieszka Grabska-Barwinkka. Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable frame- work for continual learning, ArXiv, bbs//885.00670. 2018.
Yale Song, Jordi Vallmitjana, Amanda Stent, and Alejandro Jaimes. Tysum: Summarizing web videos using titles. In IEEE Conference Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pp. 5179-5187771 IEEE Computer Society, 2015, doi: 10. P00/CCVP.2015.729991 URL httpt:tttt/......... /tstts://dee //oii http:/ddooooooooooo wwo. 09/ CVPR. 2015. 2. 54
12

---

Under review as a conference paper at ICLR 2023
Min Sun, Ali Farhadi, and Steven M. Seitz. Ranking domain-specine highlights by analyzing edited videos. In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Timne Tuytelaars (eds.), Com- puter Vision ECCV 2014 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I, volume 8689 of Lecture Notes in Computer Science, pp- 787-802. Springer, 2014. doi: 978 3- 319 378-3-319-10550-11 10590-1551 51
huthht:s://dde wwg/10..... 1007/
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Bo Xiong. Yannis Kalantidis, Deepti Ghadiyaram, and Kristen Grauman. Less is more: Learning highlight detection from video duration. In Proceedings the IEEE/CVF Conference Com- puter Vision and Pattern Recognition (CVPR). June 2019.
Shipeng Yan, Jiangwei Xie, and Xuming He. Der: Dynamically expandable representation for class incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3014-3023, June 2021.
Huan Yang, Baoyuan Wang, Stephen Lin, David P. Wipf, Minyi Guo, and Baining Guo. Un- supervised extraction of video highlights via robust recurrent auto-encoders. In 2015 IEEE International Conference Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, IEEE Computer Society, 2015. doi: 10.1109//CC2201155226 URL
2015, pp. org.org//// 10 L109/ICCV.2015.....
Ting Yao, Tao Mei, and Yong Rui. Highlight detection with pairwise deep ranking for first-person video summarization. In 2016 IEEE Conference on Computer Vision and Pullern Recognition,
CVPR 2016, Las Vegas, NV, USA. June 27-30, 2016, pp. 982 990. IEEE Computer Society, 2016. URLAttps
doi: 10.11099CVRR.2016.111
doi org/ 10. 1w1.1111EEEE.. 2016............. 112
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelli- gence. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Con- ference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, vol- ume 70 of Proceedings of Machine Learning Research, pp. 3987-3995. PMLR, 2017. URL http: http://pocceeiinee .mlr www.ppressss...mmp www.eeee..eee...... http://......../ee//////////aaaa
Ke Zhang, Kristen Grauman, and Fei Sha. Retrospective encoders for video summarization. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.), Computer Vi- sion ECCV 2018 15th European Conference. Munich, Germany, September 8-14, 2018, Proceedings, Part VIII. volume 1 11212 of Lecture Notes in Computer Science. pp. 391-408 Springer, 2018 978 978-3-030-01231--
doi:
24. URL https:///cc. .org 110 1007/
Songyang Zhang. Houwen Peng, Jianlong Fu, and Jiebo Luo. Learning 2d temporal adjacent net- works for moment localization with natural language. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 12870-12877, 2020.
Weneheng Zhu, Jiwen Lu, Jiahao Li, and Jie Zhou. Dsnet: A flexible detect-to-summarize network for video summarization. IEEE Trans. Image Process., 2020.3039886. URL https: //wp:.//iiii/// org. 11.9990....111PP111 2020. 3039886
2021. doi: 10. 10.1100 109/TIP
13

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 ETHICS, DATA, AND PRIVACY
We solemnly claim that strictly hold to the highest standard to obey any regulations/rules regard- ing ethics codes, data safety, and privacy preservation. Below we introduce the main measures taken to eliminate potential risks/issues.
Data Desensitization. All videos appearing in LiveFood are crawled from public video-sharing platforms. We remove all private information related video owners to preserve privacy, including but not limited to identifiers, meta-data, addresses, and profiles.
Data Storage. We emphasize that only video owners have the right to retain or remove their up- loaded videos on the platform. To protect the intelligent property of creators, we do not store any videos locally. The LiveFood consists of video links and corresponding domain annotations. Videos appearing in LiveFood turn to invalid after being deleted by the creators.
User Privacy. We highly respect user privacy during the construction of the dataset. All videos in LiveFood are publicly available on video-sharing platforms. Consent for public usage of videos, including academic research, has been reached between the platform and users. All sensitive data about users have been removed from our dataset.
Licence. LiveFood can only be used for non-commercial purposes. Applicants must sign an agree- ment before using the dataset (cf. Appendix A.7).
Annotator Related. The annotators are skilled and experienced in data preparation. They have been sufficiently instructed to beware of how to avoid any potential dangers and risks during annotation. Their work is properly compensated per local law.
A.2 VISUALIZATION OF VIDEO ANNOTATIONS
Due to space limits, we do present samples for each video domain in the main text. To help better understand LiveFood, we show randomly selected videos video11/. //..//.' /77 from LiveFood along with domain annotations in Figures 6 to 12 To preserve privacy, all faces are blurred.
Figure 6: Samples of clips and domain annotations from video_1. The row from top to bottom illustrates vanilla clips, presentation, and eating, respectively. The faces appearing in video above are blurred to protect privacy.
A.3 ANNOTATION QUALITY CONTROL
We have applied elaborate quality control to guarantee the high quality of LiveFood. Specifically, to reduce subjectivity during annotation, an annotated video is reviewed by another experienced reviewer to double-check if it meets the requirements in Table [ An annotated video only passes double-checking if both the annotator and reviewer agree on the correctness domain labels over 90% video segments. Besides, reviewers are also required to identify the start and end timestamps of
14

---

Under review as a conference paper at ICLR 2023
Figure 7: Samples of clips and domain annotations from video.2. The row from top to bottom
illustrates examples of ingredients, cooking, and presentation, respectively,
Figure 8: Samples of clips and domain annotations from video.3. The row from top to bottom illustrates examples of vanilla clips, ingredients, and presentation, respectively.
Figure 9: Samples of clips and domain annotations from video.4 4. The row from top to bottom illustrates examples of vanilla clips, presentation, and eating, respectively. The faces appearing in video above are blurred to protect privacy.
15

---

Under review as a conference paper at ICLR 2023
Figure 10: Samples of clips and domain annotations from video_5. The row from top to bottom illustrates examples of vanilla clips, presentation, and eating, respectively. The faces appearing in video above are blurred to protect privacy.
Figure 11: Samples of clips and domain annotations from video 6. The row from top to bottom illustrates examples of vanilla clips, cooking, and eating, respectively. The faces appearing in video above are blurred to protect privacy.
Figure 12: Samples of clips and domain annotations from video_7. The row from top to bottom all illustrates the examples of presentation.
16

---

Under review as a conference paper at ICLR 2023
highlight clips, and only clips with over 90% temporal loU (Intersection-ove---niie between the annotations of the annotator and reviewer are deemed as qualified highlight clips. The review process is conducted in a batch-wist wise manner: 30 of every 100 annotated videos are selected randomly for inspection. The threshold of qualification rate is 90%. Before starting the official annotation, all annotators trained with
hundred testing videos to get fully prepared.
A.4 IN-HOUSE ANNOTATION TOOL
We use an in-house tool named LiveLabel for annotating videos. In this section, we introduce the basic functions of LiveLabel to give rough idea of how it is used in the annotation task.
Play
Capture
Scale
22441700
Load
Time mmpss
TET 00.41
Skip
Check
Figure 13: The annotation tools used in LiveFood.
As shown in the interface above, Livelabel pulls videos from a pre-defined video pipeline automat- ically and parses these videos video is played at the user-specified speed. One can play/stop the video display using the play but- ton. Pushing capture will demonstrate the detailed video contents frame-by-frame fashion, as shown in the right panel of Figure [3] These frames help the annotators select accurate timestamps for video highlight clips. The check button is used to select pre-defined domains or categories. If video is undesired, the annotator can click the skip button to remove them from LiveFood.
adjustable frame rate. On the left panel of Figure [3] the raw
A.5 MORE COMPARISON RESULTS ON R-MNIST
In Table show the comparison results between GPE and popular incremental learning ap- proaches on the R-MNIST benchmark. We note that all methods employ identical architecture for a fair comparison, i.e., multi-layer perception (MLP) with two fully connected layers. In this section, we further investigate the extensibility of GPE to stronger backbones, egg., the convolution network in (Cha et al. 2021). Since detailed structural information of the convolution network used in Cha et al. [2021) is unavailable, we use ResNet-18 (He et al. (016)) as the backbone for GPE and all other competing methods. For comprehensive comparison,
experiment with the following three settings: (1) using no memory buffer, (2) using a memory buffer of 200 samples, 3) using memory buffer of 500 samples. We directly take the results from the paper of Co (Cha al, 2021) for comparison. For references of other methods in Table please refer to: ER (Rolnick et al.
2019 GEM (Lopez-Paz & Ranzato 2017). A-GEM (Chaudhry al. 2019) FDR (Benjamin al.
2019 GSS (Aljundi et al. 2019). HAL (Chaudhry et al. 2021) DER Yan et al. 2021). DER++
Yan et al. 2021), and Co (Cha et al. 2021).
When no memory buffer is applied, GPE with ResNet-18 (He et al. 2016) backbone achieves 94.77% % top-1 classification accuracy on the R-MNIST dataset. By using a memory buffer of 200 samples and 500 samples, GPE boosts its performance by 2.29% and 3.56%. respectively. Compared to more complex incremental learning techniques such as Co (Cha et al. 2021). although GPE achieves comparable results, it is much simpler and more efficient. For example, Co counteracts
17

---

Under review as a conference paper at ICLR 2023
Table 7: Average classification accuracy on R-MNIST using convolution-based models.
Buffer
ER
GEM
A-GEM
FDR
GSS
HAL
DER
DER++
Co²L
GPE (M.)
200
93.53
89.86
89.03
93.71
87.10
89.40
96.43
95.98
97.90
97.06
500
94.89
92.55
89.04
95.48
89.38
92.35
97.57
97.54
98.65
98.33
knowledge forgetting using strong augmentations and distillation penalty, while GPE is much sim- pler and cleaner by discarding all these tricks. This appealing property makes GPE more scalable to handle practical scenarios. In comparison, Co has to design specific augmentation schemes as well as hyper-parameters appearing in the distillation procedure, all leading to the hardship of scaling large data/tasks.
A.6 VISUALIZATION OF LEARNED PROTOTYPES
To provide a better understanding of the learned vanilla/highlight prototypes in GPE, we use SNE (Van der Maaten & Hinton 2008 to illustrate their distributions. GPE in the dynamic man- ner is used in this experiment. Concretely, we conduct two experiments by setting the number of vanilla/highlight prototypes to 40 and 80 respectively. We use scatters in different colors to indicate the various training tasks. Ti, T2, Ta and T4 are used to indicate the prototypes learned at different stages. Figure 14 depicts the learning procedure.
40-
Figure 14: The learned prototypes of GPE across different training stages.
In Figure 14 the number of learnable prototypes is set to 40 in the top row and 80 the bottom row. Taking the first two images in the top row examples, GPE is trained to classify the vanilla and presentation clips as illustrated in the first image. The scatters in gray are the negative prototypes, while the pink ones are positive (highlight) prototypes of presentation. During the second task, GPE aims to learn the characters of eating while still retaining that of presentation. Therefore, GPE limits the change of prototypes observed in T (in pink) and develops new group prototypes in yellow, namely T2 positive as shown in Figure 14 We emphasize that positive prototypes for To are updated without constraints while that in pink (i.e., Tn positive) are forbidden to change too much (cf. Eq. The following tasks are optimized similarly.
Based on the depicted images, we can find that GPE adaptively adds new positive prototypes within incremental tasks and meanwhile modifies the position of negative prototypes which have fixed quantity. Since we model the task each stage as a binary classification problem (highlight vs. vanilla), the goal of GPE is thus to reduce the discrepancy across old and new domains while boost- ing the distinction between the vanilla and highlight prototypes. The visualization results shown in Figure [4 are consistent with our expectation by showing smaller inter-domain discrepancy and greater distances between positive/negative prototypes. From another view, since we allow the learned prototypes to change for adapting to the data, both the inherited and the newly de- veloped prototypes are updated simultaneously. thus helping reduce the discrepancy of prototypes learned across domains.
18

---

Under review as a conference paper at ICLR 2023
A.7 LiveFood RELEASE AGREEMENT
One must sign the following agreement in order to be permitted to use the LiveFood dataset.
Food Release Agreement
The released LiveFood can be only used for non-commercial purposes, including academic re- search and education. Applicants are forbidden use this dataset for profitability infringe- ment activities, including but not limited to advertising, selling, face-based applications, etc.
Applicants promise that they will not conduct any form of analysis with respect to bio-info in this dataset, including but not limited to faces, gender, etc.
Applicants are aware that they take full responsibility for their usage of LiveFood. They agree that we reserve the right ask them immediately stop illegal behavior and eliminate negative influences caused.
Applicants agree that we reserve the right to stop their usage of the dataset and delete its copies found any violation this agreement.
Applicants must clearly state their purposes and the potential effects of using LiveFood.
Name:
Email:
Organization:
Date:
Signature:
19


Reviews:
[{"review_id": "w8UL50pnOJr", "reviewer": "Reviewer_HURe", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "First, the paper introduces a new task of incremental video highlights detection aiming to adapt to more data and new domains. To that end, they propose a framework Global Prototype Encoding (GPE). They employ transformer-based temporal encoder to aggregate frame-level CNN features to generate temporally aware representations of a video. Next, each frame is classified by two groups of learnable prototypes -  highlight prototypes and vanilla prototypes. The utilization of prototypes help incremental learning by keeping the distance between the learned prototype in the new domain and previously observed prototypes close to each other.  A major contribution of this paper is the LiveFood dataset. Performance of their incremental GPE method are compared with other incremental methods relying on complex exemplar selection or complicated replay schemes, and the author observed that GPE produces encouraging results in the incremental setting. ", "summary_of_the_review": "The paper presents a new task - incremental video highlights detection. The framework is based on simple ideas around global prototype encodings. The dataset contribution is also significant. ", "strength_and_weaknesses": "The idea of learnable global prototypes leads to a simpler video highlight detection framework and an apt method for incremental learning. The dataset contribution is also significant, as the previous related datasets are either smaller in scale or do not have domain tags. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "j0QP1QCxFl", "reviewer": "Reviewer_LRmd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper addresses a crucial problem in computer vision: Video Highlights Detection (VHD) which aims to target the appealing domains for the given video(s). More general approaches are based on world assumptions which lead to poor scalability and to address this issue the author proposes a new method called Global Prototype Encoding (GPE) which learns incrementally for adaptation of the new domains. Additionally, the author also uses the LiveFood dataset which is a finely annotated dataset for the experiments. Together with strong and exhaustive experimental results, the authors showcase the practical usability and competitive performance against the prior work for specific LiveFood datasets. ", "summary_of_the_review": "Overall, currently at this stage, this is a good manuscript. I like the simplicity and wide applicability of the proposed GPE approach. There are some concerns which has been raised in the weakness section regarding the dataset proposal which holds me with the current rating, however, if given strong reasons and clarity, I'm happy to increase the rating for the same. A detailed literature review, a complete overview of each component, and detailed experiments and ablation studies help to give a good insight into the manuscript. ", "strength_and_weaknesses": "Strengths: \n\n++ The proposal of the GPE is incremental and end-to-end. The main advantage of GPE is that it can identify the domains via learning vanilla prototypes, whereas prior approaches fail for the same as the fixed number of highlight categories have been defined before learning. It can also be detected in the experiments which have been performed on the LiveFood dataset by improving the performance by a good margin. \n\nWeaknesses:\n\n-- After reading the manuscript and performing research on this there have some concerns my mind which is stated below: \nThe language for the use of the dataset is quite unclear in the entire manuscript, as from the first point-of-view, it looks like the author has curated/created the dataset, whereas moving ahead the dataset has been collected from some XYZ sources, and there has been no direction that from which settings/how the dataset has been curated/collected. Could the author clarify this more if I have missed something in the manuscript for the same?\n\n-- In regards to the dataset statistics, I believe the dataset does not have larger statistical effects as the average video lengths or the total number of videos are less compared to the open datasets as mentioned in Table 2. Can I ask the authors if they can give clear direction on why they need to propose a dataset? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "hcwbzo6HX4", "reviewer": "Reviewer_nvTS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Most of the paper is clearly written. A new dataset for incremental learning is provided. The fixed and dynamic modes of Global Prototype Encoding is proposed and outperform the state-of-arts. ", "summary_of_the_review": " It offers a new dataset and a new method for future research in its direction, which sounds reasonable. The details of the protype learning should be explained more clearly.", "strength_and_weaknesses": "Strength: Most parts are explained clearly for the equations.Most of the procedures seems reasonable.\n\nWeakness: \n1) How to learn the two groups of trainable prototypes by equation (3)? Do the  two groups of trainable prototypes are trained by 2 class clasiification by equation 3. More explanation should be given for k*m before equation (1) and how the k*m matrix is used in Figure 4. The distance based clustering of Figure 4 is not mentioned in the paragraphs.\n2) What does it mean by Average the metrics of current stage and Average the metrics of all stages in Algorithm 1.No explanation.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-GDeOna3ast", "reviewer": "Reviewer_6gVY", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new task of video highlight detection (VHD) in a domain-incremental setting. First, the authors define the task of incremental video highlights detection and introduce a gourmet dataset named LiveFood that they've carefully collected to facilitate research in this new task. Second, they argue that all the previous approaches fail to address this new task and have limited scalability. To this end, the authors present a new method named Global Prototype Encoding (GPE) that learns two groups of prototypes that are used to optimize a classification loss for incremental learning. The proposed method shows good performance on R-MNIST as well as LiveFood.", "summary_of_the_review": "The paper has two contributions, which are LiveFood and GPE; but both of them have non-negligible issues.", "strength_and_weaknesses": "Strength:\n- The introduced task of incremental VHD looks interesting and quite practical.\n- The proposed dataset has domain annotations that are missing in the previous VHD datasets.\n- The presented method achieves state-of-the-art performance on the dataset.\n\nWeakness:\n- W1. First of all, the paper does not provide enough information about the proposed dataset, even if it is the major contribution of the paper. The most important and distinctive aspect of the dataset is the presence of domain annotations, but there is no visualization of the four domains. For example, the \"eating\" domain is not depicted in any of the figures. Moreover, the information on the labeling interface or annotation tool is also missing. I also believe that the quality control section can be more elaborated for better understanding.\n\n- W2. According to the description of annotated domains in Table 1, the \"eating\" scene contains people and their faces. I believe that it might arouse some ethical concerns so the authors should have stated this in the paper.\n\n- W3. Apart from the dataset, the proposed method does not have a technical novelty. It seems like GPE is merely a combination of ConvNext, transformer encoder, and prototypical learning. For example, all the equations from 1 to 7 do not convey any novel concept because they explain a simple softmax function (eq. 2), BCE loss (eq. 3), L2 distance (eq. 4), etc. I wonder how exactly GPE is different from the previous incremental learning methods based on prototypes. As far as I know, prototypical networks have been used a lot in class-incremental learning, and I also would like to know what novelties GPE has when compared to [ref1] and [ref2]. Especially, [ref1] looks very similar to GPE.\n\n- W4. In addition, GPE's higher performance seems merely due to the use of the improved feature encoders, i.e ConvNext and a transformer. If that is the case, we cannot say that the proposed method using the prototypes contributes to the performance increase because other methods can also benefit from the high-performing feature encoders. At this point, I am not even sure why the authors mentioned \"Inspired by Carion et al., GPE employs a ConvNext, ...\"; I wonder how DETR inspires the usage of ConvNext.\n\n- W5. The paper lacks an in-depth qualitative analysis of the prototypes. I strongly believe that the authors should visualize the learned prototypes in the feature space to validate the effectiveness of the proposed learning strategy.\n\n- W6. There exists another important baseline called Co2L that was published in ICCV 2021 [ref3]. I don't think GPE is a new state-of-the-art method in terms of the average classification accuracy on R-MNIST, which is wrongly argued by the authors in Table 4.\n\n\n[ref1] Constrained Few-shot Class-incremental Learning, Hersche et al. (CVPR 2022)\n\n[ref2] Prototype Augmentation and Self-Supervision for Incremental Learning, Zhu et al. (CVPR 2021)\n\n[ref3] Co2L: Contrastive Continual Learning, Cha et al. (ICCV 2021)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject