Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper addresses a crucial problem in computer vision: Video Highlights Detection (VHD) which aims to target the appealing domains for the given video(s). More general approaches are based on world assumptions which lead to poor scalability and to address this issue the author proposes a new method called Global Prototype Encoding (GPE) which learns incrementally for adaptation of the new domains. Additionally, the author also uses the LiveFood dataset which is a finely annotated dataset for the experiments. Together with strong and exhaustive experimental results, the authors showcase the practical usability and competitive performance against the prior work for specific LiveFood datasets. 
strength_and_weaknesses: Strengths: 

++ The proposal of the GPE is incremental and end-to-end. The main advantage of GPE is that it can identify the domains via learning vanilla prototypes, whereas prior approaches fail for the same as the fixed number of highlight categories have been defined before learning. It can also be detected in the experiments which have been performed on the LiveFood dataset by improving the performance by a good margin. 

Weaknesses:

-- After reading the manuscript and performing research on this there have some concerns my mind which is stated below: 
The language for the use of the dataset is quite unclear in the entire manuscript, as from the first point-of-view, it looks like the author has curated/created the dataset, whereas moving ahead the dataset has been collected from some XYZ sources, and there has been no direction that from which settings/how the dataset has been curated/collected. Could the author clarify this more if I have missed something in the manuscript for the same?

-- In regards to the dataset statistics, I believe the dataset does not have larger statistical effects as the average video lengths or the total number of videos are less compared to the open datasets as mentioned in Table 2. Can I ask the authors if they can give clear direction on why they need to propose a dataset? 
clarity,_quality,_novelty_and_reproducibility: Novelty:

++ The task formulation and the proposed solution are concise, convincing, and novel. A seemingly reasonable approach has been proposed in this manuscript for the VHD. Compared to the prior work and baseline introducing GPE it achieves competitive results. 

Clarity:

++ Some of the manuscript sections have been written exceptionally well whereas there has been clear confusion in the dataset section. However, a brief insight into the problem and background information has been provided which is necessary to understand the major issue in the prior work. 
++ The manuscript also clearly describes the improvements and adequately contextualizes the contributions in such a way that it makes a good starting point for a novice reader.

Reproducibility:

++ The author has agreed to release the code and dataset made publicly available in some point of time in the future. However, looking at the algorithmic prototype and architecture, it can be reproducible at some point. 
summary_of_the_review: Overall, currently at this stage, this is a good manuscript. I like the simplicity and wide applicability of the proposed GPE approach. There are some concerns which has been raised in the weakness section regarding the dataset proposal which holds me with the current rating, however, if given strong reasons and clarity, I'm happy to increase the rating for the same. A detailed literature review, a complete overview of each component, and detailed experiments and ablation studies help to give a good insight into the manuscript.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Most of the paper is clearly written. A new dataset for incremental learning is provided. The fixed and dynamic modes of Global Prototype Encoding is proposed and outperform the state-of-arts. 
strength_and_weaknesses: Strength: Most parts are explained clearly for the equations.Most of the procedures seems reasonable.

Weakness: 
1) How to learn the two groups of trainable prototypes by equation (3)? Do the  two groups of trainable prototypes are trained by 2 class clasiification by equation 3. More explanation should be given for k*m before equation (1) and how the k*m matrix is used in Figure 4. The distance based clustering of Figure 4 is not mentioned in the paragraphs.
2) What does it mean by Average the metrics of current stage and Average the metrics of all stages in Algorithm 1.No explanation.
clarity,_quality,_novelty_and_reproducibility: The work is mostly novel.
summary_of_the_review:  It offers a new dataset and a new method for future research in its direction, which sounds reasonable. The details of the protype learning should be explained more clearly.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a new task of video highlight detection (VHD) in a domain-incremental setting. First, the authors define the task of incremental video highlights detection and introduce a gourmet dataset named LiveFood that they've carefully collected to facilitate research in this new task. Second, they argue that all the previous approaches fail to address this new task and have limited scalability. To this end, the authors present a new method named Global Prototype Encoding (GPE) that learns two groups of prototypes that are used to optimize a classification loss for incremental learning. The proposed method shows good performance on R-MNIST as well as LiveFood.
strength_and_weaknesses: Strength:
- The introduced task of incremental VHD looks interesting and quite practical.
- The proposed dataset has domain annotations that are missing in the previous VHD datasets.
- The presented method achieves state-of-the-art performance on the dataset.

Weakness:
- W1. First of all, the paper does not provide enough information about the proposed dataset, even if it is the major contribution of the paper. The most important and distinctive aspect of the dataset is the presence of domain annotations, but there is no visualization of the four domains. For example, the "eating" domain is not depicted in any of the figures. Moreover, the information on the labeling interface or annotation tool is also missing. I also believe that the quality control section can be more elaborated for better understanding.

- W2. According to the description of annotated domains in Table 1, the "eating" scene contains people and their faces. I believe that it might arouse some ethical concerns so the authors should have stated this in the paper.

- W3. Apart from the dataset, the proposed method does not have a technical novelty. It seems like GPE is merely a combination of ConvNext, transformer encoder, and prototypical learning. For example, all the equations from 1 to 7 do not convey any novel concept because they explain a simple softmax function (eq. 2), BCE loss (eq. 3), L2 distance (eq. 4), etc. I wonder how exactly GPE is different from the previous incremental learning methods based on prototypes. As far as I know, prototypical networks have been used a lot in class-incremental learning, and I also would like to know what novelties GPE has when compared to [ref1] and [ref2]. Especially, [ref1] looks very similar to GPE.

- W4. In addition, GPE's higher performance seems merely due to the use of the improved feature encoders, i.e ConvNext and a transformer. If that is the case, we cannot say that the proposed method using the prototypes contributes to the performance increase because other methods can also benefit from the high-performing feature encoders. At this point, I am not even sure why the authors mentioned "Inspired by Carion et al., GPE employs a ConvNext, ..."; I wonder how DETR inspires the usage of ConvNext.

- W5. The paper lacks an in-depth qualitative analysis of the prototypes. I strongly believe that the authors should visualize the learned prototypes in the feature space to validate the effectiveness of the proposed learning strategy.

- W6. There exists another important baseline called Co2L that was published in ICCV 2021 [ref3]. I don't think GPE is a new state-of-the-art method in terms of the average classification accuracy on R-MNIST, which is wrongly argued by the authors in Table 4.


[ref1] Constrained Few-shot Class-incremental Learning, Hersche et al. (CVPR 2022)

[ref2] Prototype Augmentation and Self-Supervision for Incremental Learning, Zhu et al. (CVPR 2021)

[ref3] Co2L: Contrastive Continual Learning, Cha et al. (ICCV 2021)
clarity,_quality,_novelty_and_reproducibility: It is unclear whether the performance increase is merely due to ConvNext/transformer. Although the dataset looks original, the paper lacks detailed information about the dataset. The proposed method does not seem to outperform the work published in ICCV 2021.
summary_of_the_review: The paper has two contributions, which are LiveFood and GPE; but both of them have non-negligible issues.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: First, the paper introduces a new task of incremental video highlights detection aiming to adapt to more data and new domains. To that end, they propose a framework Global Prototype Encoding (GPE). They employ transformer-based temporal encoder to aggregate frame-level CNN features to generate temporally aware representations of a video. Next, each frame is classified by two groups of learnable prototypes -  highlight prototypes and vanilla prototypes. The utilization of prototypes help incremental learning by keeping the distance between the learned prototype in the new domain and previously observed prototypes close to each other.  A major contribution of this paper is the LiveFood dataset. Performance of their incremental GPE method are compared with other incremental methods relying on complex exemplar selection or complicated replay schemes, and the author observed that GPE produces encouraging results in the incremental setting. 
strength_and_weaknesses: The idea of learnable global prototypes leads to a simpler video highlight detection framework and an apt method for incremental learning. The dataset contribution is also significant, as the previous related datasets are either smaller in scale or do not have domain tags. 
clarity,_quality,_novelty_and_reproducibility: The paper is well written. They provide satisfactory details to understand the concepts, and potential implementation in the manuscript. They also provided the source code, although I wasn't able to run the code due to time constraints. 
summary_of_the_review: The paper presents a new task - incremental video highlights detection. The framework is based on simple ideas around global prototype encodings. The dataset contribution is also significant.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper claims three main contributions: 1) new task (incremental video highlight detection), 2) new dataset (LiveFood dataset, which contains 5100 food related videos labeled with 4 "domain" categories), 3) a prototypical learning approach that is effective on the proposed task & dataset. The reviewers acknowledged that the proposed task is interesting and practical, and that the domain labels provided in the newly collected dataset could be useful to the community. 

However, there were concerns about the availabilities of the dataset -- the videos will be released in the form of URLs due to the copyright concerns. This itself could be OK if the dataset was large (in the millions), but at just about 5K videos a few missing videos can lead to a large swing in the performance. The authors assured that they will make every effort to make the videos available, but given that the authors do not retain the copyright the reviewers were unconvinced whether the videos will truly remain available for a potential long-lasting impact.

There was a critical concern on the limited technical novelty. The proposed approach is a combination of well-known backbones (ConvNeXt and transformers) and learning techniques (prototypical learning). Especially, prototypical learning has been studied extensively in the incremental learning literature, which is what this paper is focusing on (although in a specific application domain of video highlight detection, which hasn't been studied extensively before). 

The authors provided a rebuttal, addressing most of the minor concerns raised by the reviewers, but it didn't successfully address the two core concerns above (data availability and technical novelty).

After carefully reading the reviews, the rebuttal, and the discussion threads, this meta-reviewer did not find strong reasons to accept the paper despite the lingering concerns on novelty and the dataset availability, which are two of the three claimed contributions in this paper. Given this, we are recommending rejection at this time. 





justification_for_why_not_higher_score: The two major concerns are critical enough for rejection.
Decision: Reject