Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes FrameMaker to reduce the memory footprint of video class-incremental learning systems by condensing video examples of past tasks into single frames. Specifically, the authors used a list of learnable parameters to compute weighted average over uniformly sampled frames of each example. This average frame is then combined with a learnable prompt of same dimensionality and fed as input to the main classifier, trained under classification and distillation loss. When compared to prior work on video incremental learning, FrameMaker is shown to improve average accuracies while also reducing the memory needed per class.
strengths_and_weaknesses: **Strengths**

- The proposed solution is well motivated and reduces the memory footprint per class significantly compared to prior methods. It is somewhat surprising that weighted averages of sparsely sampled frames can be used as surrogate for full videos in the memory bank.
- Extensive ablation studies are performed which provides insight into the challenges of video incremental learning problem.

**Weaknesses**

Relation to prior work:
- The problem of frame condensation has high affinity to well-studied areas like video summarization and keyframe detection, yet there is no mention of related work or empirical comparisons. The authors should clarify key differences from these lines of work, in terms of both methodology and ultimate goal.

Concerns about frame condensing method:
- From my understanding, the proposed system requires a video model that can operate on single-frame inputs, such as TSM. For the majority of 3D CNN-based methods or video transformers it is not apparent how to perform forward pass on the condensed frame using the same video model.
- Another suspicion is that averaging the frames can be effective for appearance-biased datasets like UCF101. On Sth v2, many classes are only differentiated through the ordering of frames (move sth from left to right, right to left etc.). It is unlikely that the proposed method can generate informative examples for these temporally challenging scenarios and it might be the reason that improvements over the baseline is less significant.
- Using prompting for incremental learning is a promising direction, but having instance-level prompts of the same dimension as input image seems like an unnatural choice. Have the authors experimented with prompting at the feature level, or sharing prompts across instances of the same class? Also, do the additional parameters introduced by prompting count towards the memory budget?

Concern about experiments:
- Some natural baselines are not compared against, such as using an unweighted average of frames (table 3), or learning task-specific prompts (instead of one prompt per instance). These are necessary to show that the complexities of the proposed system are essential.
- While the visualizations of fig. 3 are nice, I'd imagine there can be more direct metrics to evaluate the quality of frame condensation, such as directly comparing the model output using original clip vs. condensed frame (similar to eq. 2, but evaluated at a later stage of continual learning).
questions: I am looking forward to the authors clarifying the concerns listed in the section above:
- Relation to video summarization/keyframe selection;
- How proposed method works for temporally challenging tasks and a 3D video backbones;
- Alternative prompting strategies;
- Missing baselines and additional evaluation metrics.

Update 8/7: Increased rating post-rebuttal. Most of my concerns and questions have been well addressed in the authors' response.
limitations: Both limitations and potential societal impacts are addressed in the paper. However, it might help to expand the paragraph to discuss some of the concerns listed above.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work introduces FrameMaker, a new method of distilling video datasets for class-incremental learning. FrameMaker consists of two components: Frame Condensation and Instance-Specific Prompt. The Frame Condensation learns an optimal weighted average of the exemplar video while the Instance-Specific Prompt learns a per-pixel bias that makes the resulting frame optimal for training a classifier on the respective class.
strengths_and_weaknesses: Strengths

Based on Tables 1 and 2, FrameMaker out-performs all previous methods on all 3 datasets.

The Figures throughout the paper do a good job explaining the method and its effectiveness.

The authors perform extensive ablation studies to highlight the importance of each component of their method.

Weaknesses

There are a significant number of grammar errors that somewhat distract from the work itself. Rather than list all of them here, I recommend running the text through a grammar-checking program.

Along with Figure 3, it would have been nice to visualize the sum of the condensed frame and the prompt to see just how far the prompt shifts the resulting image away from the weighted sum of frames.

One really tiny thing: the formatting of quotations is inconsistent throughout the paper. Opening quotes should be `` and closing quotes should be ‘’. This will give you the correct-looking curly quotation marks.

questions: Could we see some visualizations of the sums of the condensed frames and the prompts? It would be interesting to see how much this sum image differs from the condensed frame.

I think the authors would also likely be interested in the quickly growing field of “Dataset Distillation.” Several recent works have shown impressive results for this continual learning task in the image domain [A, B].

With these works in mind, I would also like to see some visualizations of an ablation where only the prompt is learned. It seems like the learning of the prompt is very similar to what these dataset distillation methods are doing while the condensed frame acts as a regularizer to ensure the resultant sum image does not overfit to the current sub-task (i.e., subset of classes). I’d like to hear your thoughts on this.

[A] Zhao and Bilen “Dataset Condensation with Distribution Matching” arXiv 2021
[B] Zhou et al. “Dataset Distillation using Neural Feature Regression” arXiv 2022

-----

Edit: raising my rating as authors answered all my initial questions.
limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes FrameMaker to produce a linearly combined condensed frame and use an instance-specific prompt to represent the lost spatio-temporal details.

strengths_and_weaknesses: Strengths: 
1. Raw videos have redundant frames for action recognition. Using a condensed frame to represent the whole video is interesting.
2. This paper is well-written and easy to follow.

Weaknesses:
1. Fig.2 $L_f^c$ is hard to optimize, as the condensed frame inherently loses spatio-temporal details. Table 4 shows that $L_f^c$ gives a marginal improvement.
2. Memory comparison is unfair, as FrameMaker stores additional instance-wise prompts. The authors should provide a detailed space analysis for prompts.
3. The compared CIL baselines are however not strong, the authors should consider stronger CIL methods (e.g., AANets[1], DER[2]).

[1] Liu, Yaoyao, Bernt Schiele, and Qianru Sun. "Adaptive aggregation networks for class-incremental learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
[2] Yan, Shipeng, Jiangwei Xie, and Xuming He. "Der: Dynamically expandable representation for class incremental learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
questions: other comments:
Line 98 "Temporal Shit Module" => "Temporal Shift Module"
limitations: The limitations and societal impact claimed in the paper are reasonable and well-acknowledged.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents an approach for video incremental learning with frame condensing and instance prompting. The overall intuition is reasonable, while condensing video into a single frame provides a memory efficiency, it looses the temporal signal, thus instance specific prompting helps to preserves some temporal signal. Experiments are done on HMDB51, UCF101, and Something-Something V2 with good results compared with current approaches. Written presentation is fairly good and enough to understand.  
strengths_and_weaknesses: ## Strength
* The proposed approach is naturally introduced and the motivations and intuitions for proposing the approach are reasonable and convincing.
* The proposed approach is simple, yet seems to be effective in the problem of incremental learning.
* Experiments show improvements over current methods are consistent across 3 different benchmarks.
* Written presentation is clear and easy to understand.
 
## Weakness
* The experiments are done using only one backbone architecture, i.e. TSM.
* The reviewer understands the incremental video learning is well-defined and recently studied, but the proposed approach will be more convincing if it works beyond this problem. One closely related problem is efficient video classification using condensed frame + instance-specific prompting?
questions: * Given the proposed architecture are simple and effective, it is natural to ask if it work for the general efficient video classification (considering more on the aspect of memory and compute efficient rather than looking at accuracy). If it works better than TSM, the proposed approach will be much more convincing.

* Are the parameters big/capital \Theta in Eq (3) and Eq (6) shared? what are the intuition? Since Eq (3) applied on only condensed frame while Eq (6) is applied on condensed frame + prompt (expected to be compensated for motions)? 

* Experiments are done only on one backbone of TSM, will the method also works on another backbone?
limitations: * the proposed FrameMaker is currently proved to work for [only] video incremental learning which may limit the impact of the work. If there is experiment to validate the useful of FrameMaker on another task (e.g., efficient video classification not necessary incremental), it will improve the scope of the impact of the work.
* current experiments are done on only one backbone of TSM.
review_title: null

Meta-review: metareview: The reviewers appreciated that the proposed idea is interesting and is well supported by sufficient empirical evidence. There were some concerns in the initial review and the rebuttal successfully addressed most of them. As a result, two reviewers upgraded their ratings. Overall, this paper tackles an important problem of incremental learning and the proposed approach is efficient (memory-wise) and effective (performance-wise). We are happy to recommend acceptance.
confidence: Certain
Decision: Accept