Learning a Condensed Frame for Memory-Efficient Video Class-Incremental Learning
Authors: Yixuan Pei
Zhiwu Qing
Jun CEN
Xiang Wang
Shiwei Zhang
Yaxiong Wang
Mingqian Tang
Nong Sang
Xueming Qian
Venue: NeurIPS 2022 Accept (2022)

Abstract: Recent incremental learning for action recognition usually  stores representative videos to mitigate catastrophic forgetting. However, only a few bulky videos can be stored due to the limited memory. To address this problem, we propose FrameMaker, a memory-efficient video class-incremental learning approach that learns to produce a condensed frame for each selected video. Specifically, FrameMaker is mainly composed of two crucial components: Frame Condensing and Instance-Specific Prompt. The former is to reduce the memory cost by preserving only one condensed frame instead of the whole video, while the latter aims to compensate the lost spatio-temporal details in the Frame Condensing stage. By this means, FrameMaker enables a remarkable reduction in memory but keep enough information that can be applied to following incremental tasks. Experimental results on multiple challenging benchmarks, i.e., HMDB51, UCF101 and Something-Something V2, demonstrate that FrameMaker can achieve better performance to recent advanced methods while consuming only 20% memory. Additionally, under the same memory consumption conditions, FrameMaker significantly outperforms existing state-of-the-arts by a convincing margin. 

Learning a Condensed Frame for Memory-Efficiett Video Class-Incremental Learning
Yixuan Pei
Zhiwu Qing Jun Cen
Xiang Wang'
Shiwei Zhang*
Yaxiong Wang'
Mingqian Tang'
Nong Sang"
Xueming Qian'
Xi'an Jiaotong University*
Huazhong University of Science and Technology
The Hong Kong University Science and Technology
Alibaba Group"
fpeiyixuansstu, wangyx qianxm@mail) .it.aaiiuiiit.uuuuuuu.... edu.
{qzw, wxiang, nsang) Chust. edu, cn,
jenaa@ccomecttuse ust usk [zhangj in. zsw, mingqian. tmq) Cal www.llbabbiinccc. com
Abstract
Recent incremental learning for action recognition usually stores representative videos to mitigate catastrophic forgetting. However, only a few bulky videos can be stored due to the limited memory. To address this problem, we pro- pose FrameMaker, a memory-efficient video class-incremental learning approach that learns to produce condensed frame for each selected video. Specifically, FrameMaker is mainly composed two crucial components: Frame Condensing and Pestaacee-Spciffe Prompt. The former is to reduce the memory cost by preserv- ing only one condensed frame instead of the whole video, while the latter aims to compensate the lost spatio-temporal details in the Frame Condensing stage. By this means, FrameMaker enables remarkable reduction memory but keep enough in- formation that can be applied following incremental tasks. Experimental results on multiple challenging benchmarks, i.e., HMDB51, UCF101 and Something- Something V2, demonstrate that FrameMaker can achieve better performance recent advanced methods while consuming only 20% memory. Additionally, under the same memory consumption conditions. FrameMaker significantly outperforms existing state-of of-the-arts by a convincing margin.
Introduction
Training video action recognition models with classes a single fine-tuning stage have been widely studied [6, 13, 30, 34, 37, 50, 51, 61] in recent years. However, in many realistic scenarios, limited by privacy matters or technologies, the different classes can only be presented in sequence, where the previously trained classes are either unavailable or partially available in limited memory. Naively training models will overfit currently available data and cause performance deterioration on previously classes, which is named catastrophic forgetting [39]. Class-incremental learning is machine learning paradigm to fight this challenge when fine-tuning single model in sequence of independent classes.
branch of caas--nccrmmnnal learning methods [5, 7, 9, 12, 21, 33, 45, 59] has achieved remarkable performances in the image domain by re-training portion of past examples. Meanwhile, some existing video incremental learning methods [41, 53] have demonstrated that storing more past
equal contribution
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Video Dataset
124b
Accunscy over vet claves (1))
25.0
Video Exemplars
Condensed Frames
ThanoM (lon
550
Franne Nko
TCD(CNS)
7CDMMB
5Fx5V=4lI Framms
1FX8V=8 Frames
TCD [37]
Memory; SM
FrameMaker
Memory:
L.M
450
0.0
1.0
Memary Consumption (Mb) (b)
2.0
3.0
4.0
5.0
6.0
Figure I: An intuitive comparison between FrameMaker and TCD [41]. (a) Compared with TCD, Frkee stores only one condensed frame for each video exemplar. (b) Illustration of memory cousumption accuracy curve. FrameMaker costs 1.2Mb memory, which slightly exceeds the performance of TCD with 6.0Mb memory.
examples can improve the ability to mitigate forgetting. Nevertheless, videos are information redundant and require high memory load to be saved, hence it is impractical to store good deal of training videos for each class. Therefore, herding strategy [45] is adopted to select a small set of representative videos for the exemplar memory [41, 533. vCLIMB [53] tries to down-sample frames for each video to reduce the memory consumption, and imposes a cemporal-consitenne regularization for better performance. Despite the remarkable performance, these methods still demand to conserve multiple (i.e., 8-16) frames for each representative video, leading to non-negligible memory overhead, which limits their further real-world applications.
To remedy the above weaknesses, we present FrameMaker, a memory-efficien video class- incremental learning approach. Concretely, the proposed FrameMaker contains two components, Frame Condensing and Instance-Specific Prompt. In Frame Condensing, FrameMaker assigns a frame
learnable weight each frame, and then these weights adjusted to make the weighted i.e., the condensed frame, share the same embedding features with the original video clip. Obviously, the collapsed temporal dimension and mixed spatial cues by Frame Condensing may degrade the accuracy of action recognition. To compensate the missing spatio-temporal details, inspired by Task- Specific Prompt [36], further propose Serance--Seeiffe Prompt, which learns set of parameters for each condensed frame to perform fine-grained pixel-wise modifications. In this way, the memory single video be remarkably compressed. An intuitive comparison is given in Figure (a)) On the premise of comparable performance with TCD [41], Frame significantly reduces memory overhead by 80%. Moreover, the comparison curve displayed Figure 1(b) further indicates that FrkeeMaker achieves a consistent performance improvement over recent TCD with the same memory consumption.
required
In summary, we make the following contributions: 1) We present a memory-efficien video class- incremental learning method Frkme which significantly reduces the memory consumption of example videos by integrating multiple frames into one condensed frame: 2) For the first time, we show novel perspective of Prompt, i.e., Instance-Specific Prompt, and demonstrate that it can embed effective spatio-temporal interactions into learned condensed frames for video class-incremental learning: 3) Experimental results reveal that FrameMaker consistently outperforms recent state-of-the- art methods with the same memory overhead on three challenging benchmarks (HMDB51, UCF101, and Something- Somettinn V2).
2 Related Works
2.1
Class-Incremental Learning
The Gaass-nncremental learning has been well-studied in image domain [1, 26, 32, 62], existing approaches could be grouped into knowledge distillation-based methods [20], memory-basee meth- ods [40, 45, 46], and architecture-based methods. In video domaín, class-incremental learning for action recognition is still an underexplored area. The existing solutions are designed for better temporal constraints, such as decomposing the spatio-temporal features [64], exploiting time-channel importance maps [41], and applying temporal consistency regularization [53]. Notably, both [41] and

---

Condensing
Instanco-Spcctfic Prompt
u
Chesifiction Sam
Coodened Proms
F( :82)
Frame
/2
samgle
V
Li
Peorrpt
My
f( :0,)
1
Pf
(PF +t)
Pisclewise Addiition Multiplication
Figure 2: An overview of the proposed FrameMaker, which mainly contains Frame Condensing and Instance- Specific Prompt. For each selected representative video V.* we first uniformly sample T frames and learn a condensing weight wt for each frame perform Frame Condensing. Then learnable prompt P is introduced for the condensed frame compensate the lost spatio-temporal information. Finally, distillation loss and cross entropy loss are both used guide optimization condensing weights and prompt.
[53] have shown that more stored examples in memory can effectively encourage the performance. However, FrameMaker attempts to minimize the stored frames for each video by pretty maneuver, which can provide decent performance with slight storage requirements.
2.2 Prompt-Bass Learning
Prompting is an emerging transfer learning technique in natural language processing [36], which adapts the language model to downstream tasks by fixed function. Since the design of prompting functions is challenging, the learnable prompts [29, 31] are proposed to learn task-specific knowledge parameters for the input. Recently, in the visual domain, some works have also emerged adapt the pre-trained models to new tasks by adding prompt tokens [24] or pixels [3] in the data space. Besides, DualPrompt [57] and 1.2P [56] get rid of the rehearsal buffer in continual learning by learning set of task-specific parameters. In this paper, we draw inspiration from prompting, and present an Instance-Specific Prompt to replenish the collapsed spatio-temporal cues for condensed frames.
2.3 Action Recognition
The approaches designed for action recognition task can be grouped into three categories: 2D CNN, 3D CNN and Transformer-basee methods. 2D CNN based methods is more efficient [19, 48J, while 3D CNN [6, 10, 15, 16, 43, 50-52, 55, 61] achieves better accuracy in action recognition the cost of more computation cost. With the rapid development of vision transformer [22, 35, 38), researchers also apply into the video recognition [2, 4, 13, 37], which is proven be remarkable effective. In this work, we follow the settings in [41, 64], and employ the Temporal Shift Module (TSM) [34] based on CNN for efficient experiments.
The video summarization and efficient action recognition tasks have provided some methods for key- frame selection and generation, which are related this work. The video summarization [54, 63, 66] task selects the most representative frames for efficient human understanding, which has a different paradigm with action recognition task, hence we will mainly discuss efficient The efficient video classification task aims to learn a strategy to improve classification accuracy with fewer discriminative frames as input. The approaches designed for this task can be grouped into categories: rule-based and model-based methods. The former [27, 65] select frames via some pre-defined rules such as motion distribution [65] or channel sampling [25], and the later design submodel to select or generate key frames [14, 17, 23, 44, 58, 60]. In this paper, we propose model-free frame condensing method for video class-incremental learning, which only learns a few condensing weights and prompting parameters for each instance.
3

---

3 Method
This section presents the general framework of FrameMaker shown in Figure 2, which mainly consists of two crucial components: Frame Condensing and Instance-Specinn Prompt. Firstly, Frame Condensing is designed integrate multiple frames into one frame for efficient storage. Then we introduce Instance-Spedd Speccffcc Prompt to compensate the collapsed temporal information and the mixed spatial knowledge. Finally, the specific ctass-incrementl training details for subsequent tasks using Condensed Frames are given.
3.1 Problem formulation
The purpose of video class-incrementee learning is to train model F(: h(f(;0); parameter- ized by θ as the tasks {T], TT, TK) arrive in a sequence, where f(: is the feature extractor
and h:: {) is the classification head. Each task T has its specific dataset D = {[d, yf (f)... y. E L*} whose labels belong to a predefined label set T and have not appeared in previous tasks. The memory-based methods [41, 45, 53] have been shown to be effective in preventing catastrophic forgetting. Specifically, after the incremental training step k; a memory bank M will be established for the dataset DE in this task to store the representative exemplars features. Exemplars stored in previous datasets 1 = M' U ...-U U..UUU'' M will be used subsequent incremental tasks to mitigate forgetting. At the incremental step k. the model F(; Qx) is trained from F(; Ox-1) with D' D* U (1)(kk-)) and evaluated on all seen classes.
3.2 FrameMaker
Frame Condensing. We follow the standard protocol of video class-incremental methods [41, 53], which are based on memory-replay strategy and knowledge distillation. Differently, devotes reducing the number of stored frames for representative videos to achieve memory-efficient video class-incremental learning. Formally, first select subset of video instances V by herding strategy [45] from D after the incremental step k. Given video V.* E V* we then sample T frames {7A, I ... Thr uniformly and integrate them to condense a more representative one. To this end, we define learnable weights W = {wi w ... w/r } for the i-th video instance V. in exemplar set yk, And the condensed frame can be calculated as follows:
e
1 Σ -E
1 E
EE ent
where C. H and W are the size of channel, height and width of input frames, respectively. Next, the condensed frame Ite is excepted to empower the same or much similar expressive ability as the original video clip. Hence, the embedding features of the condensed frame extracted from current model should be consistent with the features of the original video clip:
4i = S(I,; 0,) S(V,'; 0x) 11
where [[: (x) is the feature extractor of current model F(; θx). This consistency regularization can expressions of video clips. To further
foster the condensing weights to exploit memory-efficine improve the adaptability of the condensed frames to the current model, we employ cross entropy loss to supervise the classification confidence from the condensed frame:
L CrossEEnroppy(((13 Ox),
(3)
The full objective function for condensing weights W is given by:
L = I + Ig
(4)
With the methods described above, can learn effective representation, i.e., the condensed frame for each video based on current model.
Instance-Specific Prompt. However, condensing a video into one frame would inevitably lose the temporal dynamics and complete spatial information of the original video to certain extent. Therefore, an intriguing perspective of prompt, termed Instance-Specific Prompt, is proposed to replenish the missing spatio-temporal cues for the condensed frame.

---

Specifically, we first construct a learnable prompt P. E RC for each exemplar video Vi which shares the same spatial resolution as its original clip. The prompt P is then pixel-wise summed with the condensed frame I,N, and learns to pull the embedding feature of the summed frame (I + P.t)) and the video clip V together, which is similar to Eq. 2:
L = || f((T) P.)) f(V,"; 0%) 11
(5)
Notably, the Eq. 2 for condensing weights is difficult to learn the satisfactory spatio-temports features due to the collapse of the temporal dimension. Nevertheless, the Eq. 5 can enrich the representations of condensing frames by introducing more flexible learnable parameters in the input space. Besides, the Cross-Entroee loss is also utilized to enhance the semantic perception:
D CrossEntropy(F((1"" + Ok);
Theoretically, the condensing weights W and the prompting parameters P can be jointly updated by Lp = _ + L Interestingly, we observed in our practice that the flexible prompt Pf leads to the condensing weights W being under optimized. Hence, we give the final training objective function for frame condensing
= = a + BL + 7 + 1L
where additional Li and Lie are added to achieve stronger constraint on W for its effective training. a, B, ? and 17 are the balance weights for each term, and they are empirically otherwise specified.
1.0 unless
After the optimization of condensing weights and prompting parameters, the prompt is added directly to the condensed frame and stored together. And the learned prompts will be frozen when the corresponding task is completed. Therefore, there no extra memory cost for prompts.
3.3 Training
is worth noting that FrameMaker aims to condense the stored frames for representative videos, and other class-incremental learning steps still follow the standard pipeline. Specifically, when training the incremental step k; use the dataset r* D* U M to update the model from F(; Ox-1), where D is the dataset of the task k which consists of videos belonging to Lt, and = is memory bank which contains condensed frames generated by FrameMaker. The training samples from D and M alternately fed into the current model according to the proportion of their sample number. To further prevent the catastrophic forgetting, we also employ the knowledge distillation method proposed in PODNet [12] transfer the knowledge from the previous model F(; Ox-1) to current model F(; Os). The overall objective function for task reads:
Lai = L.e + m L
(8)
where Lå and L are the Cross Entropy losses for new task data D and condensed frame exemplars =- respectively. And L is the knowledge distillation loss [12] for condensed frame exemplars.
4 Experiments
4.1 Experimental Setup
Datasets and evaluation metries. The proposed FrameMaker is evaluated three standard action
recognition datasets, UCF101 [49], HMDB51 [28] and Something-Somethin V2 [18]. HMDB51 dataset consists of 6.8K videos belonging to 51 classes from YouTube or other websites. UCF101 dataset contains 13.3K videos from 101 classes.
V2 is a crowd-sourced dataset
that includes 220K videos from 174 classes.
For UCF101, the model is trained on 51 classes first, and the remaining 50 classes are divided into 5, 10 and 25 tasks. For HMDB51, train the base model using videos from 26 classes, and the remaining 25 classes are separated into 5 or 25 groups For Samething-Something V2, we first train 84 classes in the initial stage, and generate the groups of 10 and 5 classes.
To evaluate the performance of class-incremental learning, we infer the test videos from all seen categories after each task, and finally, report the average accuracy of all tasks. Following [41], two
5

---

Table Comparison with the state-of-the-art approaches class-incremental action recognition performance on UCFI01 and HMDB51, Our FrameMaker achieves the best performance under all experimental settings.
UCF101
HMDB51
Num. of Classes
10 X 5 stages
5 X 10 stages
2 X 25 stages
5 X 5 stages
1 X 25 stages
Classifier
CNN
NME
CNN
NME
CNN
NME
CNN
NME
CNN
NME
Finetuning
24.97
13.45
5.78
16.82
4.83
LWFMC [33]
42.14
25.59
11.68
26.82
16.49
LwM [9]
43.39
26.07
12.08
26.97
16.50
iCaRL [45]
65.34
64.51
58.73
40.09
33.77
UCIR [21]
74.31
74.09
70:42
70.50
63.22
64,00
44.90
46.53
37.04
37.15
PODNet [12]
73.26
74.37
71.58
73.75
70.28
71.87
44.32
48.78
38.76
46.62
TCD [41]
74.89
77.16
73.43
75.35
72.19
74.01
45.34
50.36
40.47
46.66
FrameMaker
78.13
78.64
76.38
78.14
75.77
77.49
47.54
51.12
42.65
47.37
Table 2: Comparison with the top approaches over caassinncrementl action recognition performance on Something Something V2.
Table 3: Ablations for Frame Condensing (FC) and
Instance- Speccffc Prompting (ISP) UCF101 with 10 steps and HMDB51 with steps.
Num. of Classes
10 9 stages
5 18 stages
UCFI0L
HMDB51
Classifier
CNN
NME
CNN
NME
Frames
FC
ISP
CNN
NME
CNN
NME
All
72.09
75.70
43.38
47.00
UCIR [21]
26.84
17.98
20.69
12.57
Random
X
X
68.64
73.96
39.59
43.48
PODNet [12]
34.94
27.33
26.95
17.49
Random
V
70.71
75.04
39.81
43.74
Average
70.82
75.45
41.84
45.45
TCD [41]
35.78
28.88
29.60
21.63
Average
X
V
71.51
76.23
42.59
46.46
Condensed
X
72.29
76.42
42.18
46.27
FrameMaker
37.25
29.92
30.98
22.84
Condensed
V
72.93
76.64
43.39
46.88
different metrics are reported, i.e., CNN and NME. CNN refers to training fully-connected layer for extracted features, which is standard classification protocol. NME is proposed by iCaRL [45], which assigns the labels for test data by comparing the feature embeddings with the -feemlls-
Implementation details. TSM [34] is employed as backbone, and we follow the data prepro- cessing procedure of TSM. For UCF101, we train ResNet-34 TSM for 50 epochs with batch size 256 from an initial learning rate 0.04. For HMDB51 and Something-Somethiin a ResNet-50 TSM for 50 epochs with batch size of 128 from an initial learning rate of 1e-3 and 0.04, respectively. All used networks are first pre-trdined on ImageNet [8] for initialization. These settings are consistent with TCD [41]. We train all models on eight NVIDIA V100 GPUs and use PyTorch [42] for all our experiments.
V2, train
4.2 Comparison with State-of-the-art Results
This section presents the qualitative comparison our proposed FrameMaker with the existing class incremental learning approaches under multiple challenging settings three datasets: UCF101 [49], HMDB51 [28] and Someehing-SSmetthin V2 [18]. For a fair comparison, we use the same exemplar memory size per class, model structure and pre-training initialization the existing methods [41].
Table summarizes the results on UCF101 and HMDB51, which shows that FrameMaker outperforms other methods consistently under different configurations in terms of both CNN and NME scores. The average accuracy of our FrameMaker surpasses TCD by around 3.0% on UCF101 and 2.0% on HMDB51 in terms of CNN, respectively. These results demonstrate that one condensed frame can also be equipped with an effective spatio-temporal representation.
FrameMakkrr is compared with recent advanced methods on Someehing-SSmethinn V2 in Table 2. Frkee sets new state-of-the-a performance under multiple challenging settings. Although FrameMaker only uses one condensed frame for future replay, it still achieves decent performance on the large-scale motion-sensitive dataset. We speculate that these gains mainly come from two aspects: (L) The proposed Spsccfc Prompt effectively rescues the lost spatio-temporal details, which

---

Table 4: Ablations for objective functions Frame Condensing on UCF101 with 10 steps.
Table 5: Ablations for objective func- tions Instance-Specific Prompt on UCF101 with 10 steps.
Table 6: The number of
frames for Frame Condensing on UCF101 with 10 steps.
L
T
CNN
NME
L
T
CNN
NME
T
CNN
NME
X
X
70.82
75.45
X
V
/
71.83
75.94
2
71.62
75.89
71.35
75.76
/
/
72.35
76.48
71.70
76.12
71.96
76.07
x
72.58
76.57
72.93
76.64
/
72.29
76.42
/
/
72.93
76.64
16
72.42
76.11
Table 7: Ablations for different backbones for FrameMaker HMDB51 with steps.
Table 8: Ablations for alternative prompting strategics for FrameMaker HMDB51 with steps.
Backbone
Frames
CNN
NME
Mem.
Position
Type
CNN
NME
Mem.
TSM
43.38
47.00
6.00Mb
Feature
T.-Spe.
41.67
45.71
77.07Mb
TSM
FC+ISP
43.39
46.88
0.75Mb
Feature
C.eep
41.72
45.93
385.35Mb
R3D50
All
39.85
45.64
6.00Mb
Feature
I.-pee.
42.19
46.53
1926.75Mb
R3D50
FC+ISP
39.88
45.08
0.75Mb
Frame
T-- T--Spc.
41.16
45.34
0.75Mb
VIT
All FC+ISP
35.34
39.46
6.00Mb
Frame
C.ecp
42.01
45.47
0.75Mb
VIT
35.25
39.58
0.75Mb
Frame
I.-peec
43.39
46.88
0.75Mb
also discussed Section 4.3. (2) Memory-effiiien Frame allows us to store more exemplar videos with less memory consumption, which greatly alleviates catastrophic forgetting.
4.3 Ablation Study
In this section, we present ablation studies to analyze the properties and the effectiveness of Frkme If not specified, the ablation studies performed on UCF101 with 10 steps and HMDB51 with 5 steps. For comparison with the case of storing all frames, we select 5 exemplar videos for each class.
Frame Condensing and Instance-Speeifie Prompt. To show the effectiveness of Frame Condensing and Instance-Ste -eeancceSppccficc Prompt, we compare them with the following alternative methods to produce stored frames for future replay: (1) All Frames, which simply saves the whole video. (2) One Random Frame, which randomly selects one frame for each exemplar video. (3) One Average Frame, which averages the input frames for each exemplar video. (4) One Condensed Frame, which is generated by our Frame Condensing procedure. The results are reported in Table 3. From the table, in terms of CNN, we observe that SSsecic--Specfiic Prompt yields an improvement of 2.07% and 0.22% with randomly sampled frames, 0.69% and 0.75% with averaged frames on UCF101 and HMDB51, 1, respectively, which implies that prompts can always replenish the ignored spatio-temporal cues in video frames. Further, Frame Condensing with Sestace--Specff, Prompt leads to at least on par or better performance than preserving all frames while using just one condensed frame under all settings on both datasets.
Loss terms. To preserve the best spatio-temporal information in the condensed frames, the distillation loss L; and the cross entropy loss Lee are both applied for Frame Condensing and Instance- SSpeciffe Prompt. Table 4 presents the results with Frame Condensing from several different combinations of loss terms. L.= provides more explicit semantic supervision, making the accuracy step further. Similar experiments for Settance-Speciff Prompt are shown in Table 5. One noticeable thing is that once introducing prompt, we only need to calculate the final L, and LL to realize the joint update of condensing weights W.A. and prompt Pk However, we observe that the learned weights, this case, tend to be average, which indicates that W. under-optimized. We speculate that this is due the strong plasticity of prompt, which leads to the insufficient optimization of W.k. To this end, further employ L and Lg for W the guidance. The improvement in Table 5 and the visualizations in Figure 3 show that W receives effective learning.
7

---

Table 9; Analysis for memory budget on UCF101 with 10 steps. Here 'xF yV=zMb' indicates that sampled frames from y different videos are stored for each class, and æ frames are sampled from each video. assume that the spatial resolution of frames is x24, and the total memory consumption is =Mbytes.
Memory Per Class Classifier
8F x CNN
1V=1.2Mb
SF x CNN
2V=2.4Mb
8F x 5V=6
NME
NME
CNN
NME
iCaRL [45]
58.05
60.50
64.51
UCIR [21]
61.92
65.52
66.43
67.58
70.42
70,50
PODNet [12]
63.18
70.96
65.93
72.78
71.58
73.75
TCD [41]
64.52
71.96
68.40
73.30
73.43
75.35
Memory Per Class
1F X 1 M0
1F x 2V=0.3Mb
1F x 5V=0.75Mb
FrameMaker
49.37
70.78
62.06
74.18
72.93
76.64
Memory Per Class
1F x 8V=1.2Mb
1F x 16V=2,4Mb
1F x 40V=6.0Mb
FrameMaker
73.64
76.98
75.19
77.43
76.38
78.14
The number of frames used for Frame Condensing. As discussed in Section 3.2, the T frames are uniformly sampled from representative video for Frame Condensing. In Table 6, we show the effect of T in video class-incremental learning. We observe that the performance increases as increases for both CNN and NME. However, the performance is saturated when using 16 frames, which is in line with the conclusion of existing methods [41, 53], i.e., storing more frames in a video does not necessarily deliver performance improvement. This phenomenon reveals that FrameMaker can empower one Condensed Frame to absorb plenty of spatio-tempose features from original clips.
Different backbones. To evaluate the applicability of proposed FrameMaker, we replace the back- bone with 3D CNN-based method R3D50 [16] and video transformer VIT I], as shown in Table 7. In these experiments, we simply replicate the condensed frames multiple times along the the tem- poral dimension to meet the 3D models and Transformers. From the results, we can find that our FrameMaker can consistently achieve comparable performance on the three different backbones with only 12% memory cost. These results demonstrate the better generalization of FrameMaker.
Alternative prompting strategies. We conduct experiments explore the types and positions of the prompts in Table 8. (i) We compare our Instance Speecifc (I.-Spec.) Prompt (ISP) with the Task- (T.-Spec.) and Class-Specific (C.-Spec.) prompts and observe that our ISP achieve the best performance when operating on both features and frames. Since the actions in videos have great intra-class variance, simply sharing task and class-specific prompts is insufficient reserve the important spatia-temporal cues for each instance; (ii) Positions of prompt: For the feature settings, add the prompt on the features of the 4 stages of the ResNet. The feature- prompt is worse than our frame-based prompt, which may be caused by the mismatch between the running model and fixed prompts during incremental procedure. Moreover, additional features require additional storage space because the corresponding prompt cannot be directly added on the changed feature.
Memory budget comparison. We now compare the memory budget of FramMaker with existing approaches. To make a direct comparison, we use the definition of the working memory size in terms of stored frames, following [53]. The results our FrameMaker only costs 1. 1.2Mb memory, which surpasses TCD [41] using 6.0Mb by 0.21 1% and 1.63% in terms of CNN and NME. FrameMaker saves 80% of memory space with the comparable performance. Meanwhile, we further increase the stored videos but keep the same memory with existing methods. FrameMaker can still effectively promote performance and shows superior abilities to fight catastrophic forgetting. Our results are about 3% ahead of TCD in both indicators. Besides, we also attempt to select the same number of videos as the existing methods. However, the accuracy of the trained linear classifier, i.e., CNN, is weaker with small number of video instances. Interestingly. NME evaluated by the exemplar class centre yielded fairly or even better results, implying that the model suffers only slight forgetting. We hypothesize that this is caused by the poor feature diversity that a few condensed frames are provided. Therefore, the classifier tends to overfit fixed sample features, resulting in poor generalization.
summarized in Table 9. We can that
8

---

E2215
Cradened Fraee
Pronet
Crelknsd Frene
Pranpt
(2)
(b)
Condensd Freez
Figure 3: Visualization of Condensing Weights and Instance-Specific Prompts HMDB51.
Puzzo (CNNN) ..... Alfraancs (ME
All Frones (ChN)
All Franes (MMP)
83
Accoracy ever apen clasess
(CNN)
INME)
65
31
25
63
05
80
95
101
25
31
76
41
51
Number of classes
Namber clases
(n) Per-task Accuracy on UCFF10
(b) Per-tnsk Accumcc on HMDB51
Aucteasy over ciffeem ciffeemteeees clases
Aarend
(c) Perccotgyry Accuracy on UCF101
(8) Per-eutegoyy Accuracy HMDB5)
Figure 4: Per-category and per-task accuracy on UCF101 and HMDB51. 'FC' and 'ISP' refer to Frame Condensing and Instance-Specific Prompt, respectively. The plots in (a) and (b) indicate that the introduction of prompt improves accuracy in almost all tasks. The results in (c) and (d) show that our Instance Specific Prompt significantly improves performance from Frame Condensing all categories both datasets.
Visualization of condensing weights and instance-specifie prompts. To intuitively understand FrameMaker, we provide visualization condensing weights, -nstance-Specific Prompts, condensed frames and learned prompts in Figure 3. To save space, uniformly select the most typical 4 frames from T frames for display. Based on this, we have the following fascinating observations: (1) For frames with strong scene bias, the learned weights are not. For example, the action classification for (a) and (d) in Figure 3 does not depend on a specific frame. Hence, the condensing weight for each frame roughly the same. contrast, stair climbing and hugging in (c) and (b) only occur in some specific frames, and those keyframes are highlighted by condensing weights. This demonstrates that it is reasonable condense multiple frames into one frame since our proposed Frame Condensing can assist the learning of which frame is meaningful for the action recognition task. (2) is difficult for humans to absorb useful knowledge from the learned prompts shown in Figure 3, and even they seem to be the same. It should be emphasized that the prompts are different. We try to share the same prompt for all videos to verify this. The experiment conducted on UCF101 with 10 steps, and the performances decrease by 0.83% and 0.78% in terms of CNN and NME, respectively. This fully validates that our Instance-Specific Prompt is to replenish the specific information for different instances.
more uniform, while those with motion bias
Per-category and per-task analysis. We depict the task-wise and category- wiise accuracy with different forms of stored frames Figure 4, i.e., "All Frames". "Frame Condensing without Instance- Specific Prompt" and "Frame Condensing with Instance-Specific Prompt". As expected, integrating multiple frames into one condensed frame does undermine performance in almost all cases unless Instance-Specifie Prompt is introduced. As shown in Figure 4(a) and (b), the gains incremental

---

tasks yielded by Instance- Spercffcc Prompt mainly fall in the later tasks, which implies that the spatio-temporal knowledge absorbed by prompt can effectively alleviate forgetting. Meanwhile, from Figure 4(C) and (d), we also observe that Frame Condensing can lead to the performance degradation of actions with large temporal and spatial variations or short duration, such as "CliffDiving" "Ride horse" and "Hug", etc. Therefore, Frame Condensing with only few parameters is difficult to capture these complex spatio-temporal interactions. However, -etancceSSpecffic Prompt helps alleviate this issue since it recovers the lost spatio-temporal details in Frame Condensing for each instance.
5 Discussions
Limitations. Compared with the previous video incremental learning approaches, FrameMaker provides simple framework that significantly reduces the amount memory required for each exemplar video. Nevertheless, FrameMaker may fail when the memory budget is highly constrained, which needs to be further explored in future work.
Societal Impacts. Although FrameMaker only stores condensed frames in memory buffer for experi- ence replay, which still retains some information about the original video. Therefore, FrameMaker may be used in applications with privacy concerns [47].
Conclusion. This paper proposes FrameMaker, memory-efficient approach for video class- incremental learning. It explores how to learn an effective condensed frame with less memory for each video that can be applied to the future incremental task. FrameMaker mainly consists of two key components, i.e., Frame Condensing and Pustance-Specity Prompt. The former learns group of condensed weights for each video integrate multiple frames into one condensed frame, while the latter learns to retrieve the collapsed spatio-temporal structure for the condensed frame. In this way, FrameMaker offers better results with only 20% memory consumption compared recent advanced methods and sets a new state-of-the- class- inssemnncrmmttll learning performance challenging datasets. Memory-baat based class-incremend learning is one of the effective methods to fight forgetting. The efficient utilization of limited space is worth exploring in future work.
multiple
Acknowledgments. This work is supported by the National Key R&D Program of China under Grant No. 2018AAA0101501, the National Natural Science Foundation of China undergrant 61871435 and 62272380, Fundamental Research Funds for the Central Universities no. 2019kfyXKJC004 the Science and Technology Program of Xi'an China under Grant 21RGZN0017, and by Alibaba Group through Alibaba Innovative Research Program.
References
[1] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuyte- laars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Conference on Computer Vision, pages 139-154, 2018
[2] Anurag Amab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučié, and Cordelia Schmid. Vivit: A video vision transformer. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 6836-6846, 2021.
[3] Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Visual prompting: Modifying pixel space to adapt pre-trained models. arXiv preprint arX14:2203.17274, 2022.
[4] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding. arXiv preprint arXiv:2102.005095, 2(3):4, 2021.
[5] Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: strong, simple baseline. Advances in neural information processing systems, 19900000,, 2020.
[6] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2299-6308, 2017.
10

---

[7] Francisco M Castro, Manuel J Alahari. End-to-end incremental learning. In Proceedings of the European conference on computer vision, pages 233-248, 2018.
Nicolás Guil, Cordelia Schmid, and Karteek
[8] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255. leee, 2009.
[9] Prithviraj Dhar, Rajat Vikram Singh, Kuan-Chuan Peng, Ziyan Wu, and Rama Chellappa. Learning without memorizing, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5138 5 5146, 2019.
[10] Ali Diba, Mohsen Fayyaz, Vivek Sharma, M Mahdi Arzani, Rahman Yousefzadch, Juergen Gall, and Luc Van Gool. Spatio-temporal channel correlation networks for action classification. In Proceedings of the European Conference on Computer Vision, pages 284- 299, 2018
[11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 19929, 2020.
[12] Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. Podnet: Pooled outputs distillation for small-tasks incremental learning. European Conference on Computer Vision, pages 86-102. Springer, 2020.
[13] Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6824-6835, 2021.
[14] Hehe Fan, Zhongwen Xu, Linchao Zhu, Chenggang Yan, Jianjun Ge, and Yi Yang. Watching a small portion could be as good as watching all: Towards efficient video classification. In IJCAI International Joint Conference on Artificial Intelligence, 2018
[15] Christoph Feichtenhofer. X3d: Expanding architectures for efficient video recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 203-213, 2020.
[16] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6202-6211, 2019.
[17] Shreyank N Gowda, Marcus Rohrbach, and Laura Sevilla-Lara. Smart frame selection for action recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 1451-1459, 2021.
[18] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag. al. The" something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE international conference computer vision, pages 5842- 5850, 2017.
[19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[20] Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in a neural network. arXiv preprint arXiv: arXiv:1503.02:: 5200002001,22777 2015.
[21] Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning unified classifier incrementally via rebalancing. Proceedings of the IEFF/CVE Conference on Computer Vision and Pattern Recognition, pages 831-839, 2019.

---

[22] Han Hu, Zheng Zhang, Zhenda Xie, and Stephen Lin. Local relation networks for image recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019.
[23] De-An Huang, Vignesh Ramanathan, Dhruv Mahajan, Lorenzo Torresani, Manohar Paluri, Li Fei-Fei, and Juan Carlos Niebles. What makes a video a video: Analyzing temporal information in video understanding models and datasets. In Proceedings the IEEE Conference on Computer Vision and Pattern Recognition, pages 7366-7375, 2018.
[24] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. arXiv preprint arXiv:2203.12119 2022.
[25] Kiyoon Kim, Shreyank N Gowda, Oisin Mac Aodha, and Laura Sevilla-Lara. Capturing temporal information in single frame: Channel sampling strategies for action recognition. arXiv preprint arXiv:2201113304 2022.
[26] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaune Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Barnnka,, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 1 (14(13)3551-33222 2017.
[27] Bruno Korbar, Du Tran, and Lorenzo Torresani. Scsampler: Sampling salient clips from video for efficient action recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2019.
[28] Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: a large video database for human motion recognition. In 2011 International conference on computer vision, pages
IEEE, 2011.
[29] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2 2021.
[30] Kunchang Li, Yali Wang, Peng Gao, Guanglu Song, Yu Liu, Hongsheng Li, and Yu Qiao. Uniformer: Unified transformer for efficient spatiotemporal representation learning. arXiv preprint arXiv: 04676,, 2022.
[31] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190 2021.
[32] Zhizhong Li and Derek Hoiem. Learning without forgetting, IEEE transactions on pattern analysis and machine intelligence, 40( 2947, 2017.
[33] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-22 2947, 2017.
[34] Ji Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient video understand- ing. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 7083-7093, 2019.
[35] Chengxu Liu, Huan Yang, Jianlong Fu, and Xueming Qian. Learning trajectory-aware trans- former for video super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[36] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: systematic survey of prompting methods in natural language processing. arXiv preprint arXiv: 13586 2021.
[37] Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu, Video swin transformer. arXiv preprint 100i 13230, 2021.
[38] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, Furu Wei, and Baining Guo. Swin transformer v2: Scaling up capacity and resolution, In International Conference Computer Vision and Pattern Recognition, 2022.
12

---

[39] Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109-165. Elsevier, 1989.
[40] Oleksiy Ostapenko, Mihai Puscas, Tassilo Klein, Patrick Jahmichen, and Moin Nabi. Learning to remember: A synaptic plasticity driven framework for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11321-11399 2019.
[41] Jaeyoo Park, Minsoo Kang, and Bohyung Han. Class-incremental learning for action recognition in videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13698 13707, 2021.
[42] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, al. Pytorch: An imperative style, sigh-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[43] Zhaofan Qiu, Ting Yao, and Tao Mei. Learning spatio-temporat representation with pseudo-3d residual networks, In proceedings of the IEEE International Conference on Computer Vision, pages 5533-5541, 2017.
[44] Zhaofan Qiu, Ting Yao, Yan Shu, Chong Wah Ngo, and Tao Mei. Condensing sequence to one informative frame for video recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1631 1-16320, 2021.
[45] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2001-2010, 2017.
[46] Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. Advances in neural information processing systems, 30, 2017.
[47] Reza Shokri and Vitaly Shmatikov. Privacy-pree precevyipeerringgggg deep learning. Proceedings of the 22nd ACM SIGSAC conference computer and communications security, pages 1310-1321, 2015.
[48] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409 1556, 2014.
[49] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos the wild. arXiv preprint
2012.
[50] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference computer vision, pages 4489-4497 2015.
[51] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. closer look at spatiotemporal convolutions for action recognition. In Proceedings of the IEEE conference on Computer Vision and Puttern Recognition, pages 6450-6459, 2018.
[52] Du Tran, Heng Wang, Lorenzo Torresani, and Matt Feiszli. Video classification with channel- separated convolutional networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5552-5561, 2019.
[53] Andrés Villa, Kumail Alhamoud, Juan León Alcázar, Fabian Caba Heilbron, Victor Escorcia, and Bernard Ghanem. velimb: A novel video class incremental learning benchmark. arXiv preprint arXiv:2201. 009381 2022.
[54] Junbo Wang, Wei Wang, Zhiyong Wang, Liang Wang, Dagan Feng, and Tieniu Tan. Stacked memory network for video summarization. In Proceedings of the 27th ACM International Conference Multimedia, pages 836 844,844,20 2019.
13

---

[55] Limin Wang, Wei Li. Wen Li, and Luc Van Gool. Appearance appearance-and-relatoo networks for video classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1430-1439, 2018.
[56] Zifeng Wang, Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. arXiv preprint arXiv: / 12.06554 2021.
[57] Zifeng Wang. Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen- Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal free continual learning. arXiv preprint wrXiv:220.004799, 2022.
[58] Wenhao Wu, Dongliang He, Xiao Tan, Shifeng Chen, and Shilei Wen. Multi-agent reinforcement learning based frame sampling for effective untrimmed video recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6222-6233 2019.
[59] Yue Wu, Yinpeng Chen, Lijuan Wang. Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 374-382, 2019.
[60] Zuxuan Wu, Caiming Xiong, Chih- Yao Ma, Richard Socher, and Larry S Davis. Adaframe: Adaptive frame selection for fast video recognition. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1278-1287, 2019.
[61] Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy, Rethinking spa- tiotemporal feature learning for video understanding. arXiv preprint 1rXiv:1712.04851, 1(2):5, 2017.
[62] Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. International Conference on Machine Learning, pages 3987-3995, PMLR, 2017.
[63] Bin Zhao, Haopeng Li, Xiaoqiang Lu,, and Xuelong Li. Reconstructive sequence-graph network for video summarization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44 (5):2793-2801, 2021.
[64] Hanbin Zhao, Xin Qin, Shihao Su, Yongjian Fu, Zibo Lin, and Xi Li. When video classification meets incremental classes. In Proceedings of the 29th ACM International Conference Multimedia, pages 880-889 2021.
[65] Yuan Zhi, Zhan Tong, Limin Wang, and Gangshan Wu. Mgsampler: An explainable sampling strategy for video action recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15 15---25 2021.
[66] Wencheng Zhu, Jiwen Lu, Jiahao Li, and Jie Zhou. Dsnet: A flexible detect-to-summaric network for video summarization. IEEE Transactions on Image Processing, 30:948-962, 2020.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A] You strongly encouraged to include justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/amswers below.
1. For all authors...
14

---

(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [No]
(b) Did you specify all the training details (c.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [No]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "mERAlhJpGzx", "reviewer": "Reviewer_pcyo", "summary": "The paper proposes FrameMaker to reduce the memory footprint of video class-incremental learning systems by condensing video examples of past tasks into single frames. Specifically, the authors used a list of learnable parameters to compute weighted average over uniformly sampled frames of each example. This average frame is then combined with a learnable prompt of same dimensionality and fed as input to the main classifier, trained under classification and distillation loss. When compared to prior work on video incremental learning, FrameMaker is shown to improve average accuracies while also reducing the memory needed per class.", "questions": "I am looking forward to the authors clarifying the concerns listed in the section above:\n- Relation to video summarization/keyframe selection;\n- How proposed method works for temporally challenging tasks and a 3D video backbones;\n- Alternative prompting strategies;\n- Missing baselines and additional evaluation metrics.\n\nUpdate 8/7: Increased rating post-rebuttal. Most of my concerns and questions have been well addressed in the authors' response.", "limitations": "Both limitations and potential societal impacts are addressed in the paper. However, it might help to expand the paragraph to discuss some of the concerns listed above.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The proposed solution is well motivated and reduces the memory footprint per class significantly compared to prior methods. It is somewhat surprising that weighted averages of sparsely sampled frames can be used as surrogate for full videos in the memory bank.\n- Extensive ablation studies are performed which provides insight into the challenges of video incremental learning problem.\n\n**Weaknesses**\n\nRelation to prior work:\n- The problem of frame condensation has high affinity to well-studied areas like video summarization and keyframe detection, yet there is no mention of related work or empirical comparisons. The authors should clarify key differences from these lines of work, in terms of both methodology and ultimate goal.\n\nConcerns about frame condensing method:\n- From my understanding, the proposed system requires a video model that can operate on single-frame inputs, such as TSM. For the majority of 3D CNN-based methods or video transformers it is not apparent how to perform forward pass on the condensed frame using the same video model.\n- Another suspicion is that averaging the frames can be effective for appearance-biased datasets like UCF101. On Sth v2, many classes are only differentiated through the ordering of frames (move sth from left to right, right to left etc.). It is unlikely that the proposed method can generate informative examples for these temporally challenging scenarios and it might be the reason that improvements over the baseline is less significant.\n- Using prompting for incremental learning is a promising direction, but having instance-level prompts of the same dimension as input image seems like an unnatural choice. Have the authors experimented with prompting at the feature level, or sharing prompts across instances of the same class? Also, do the additional parameters introduced by prompting count towards the memory budget?\n\nConcern about experiments:\n- Some natural baselines are not compared against, such as using an unweighted average of frames (table 3), or learning task-specific prompts (instead of one prompt per instance). These are necessary to show that the complexities of the proposed system are essential.\n- While the visualizations of fig. 3 are nice, I'd imagine there can be more direct metrics to evaluate the quality of frame condensation, such as directly comparing the model output using original clip vs. condensed frame (similar to eq. 2, but evaluated at a later stage of continual learning).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "gBXwUknaaCp", "reviewer": "Reviewer_VmcB", "summary": "This work introduces FrameMaker, a new method of distilling video datasets for class-incremental learning. FrameMaker consists of two components: Frame Condensation and Instance-Specific Prompt. The Frame Condensation learns an optimal weighted average of the exemplar video while the Instance-Specific Prompt learns a per-pixel bias that makes the resulting frame optimal for training a classifier on the respective class.", "questions": "Could we see some visualizations of the sums of the condensed frames and the prompts? It would be interesting to see how much this sum image differs from the condensed frame.\n\nI think the authors would also likely be interested in the quickly growing field of “Dataset Distillation.” Several recent works have shown impressive results for this continual learning task in the image domain [A, B].\n\nWith these works in mind, I would also like to see some visualizations of an ablation where only the prompt is learned. It seems like the learning of the prompt is very similar to what these dataset distillation methods are doing while the condensed frame acts as a regularizer to ensure the resultant sum image does not overfit to the current sub-task (i.e., subset of classes). I’d like to hear your thoughts on this.\n\n[A] Zhao and Bilen “Dataset Condensation with Distribution Matching” arXiv 2021\n[B] Zhou et al. “Dataset Distillation using Neural Feature Regression” arXiv 2022\n\n-----\n\nEdit: raising my rating as authors answered all my initial questions.", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nBased on Tables 1 and 2, FrameMaker out-performs all previous methods on all 3 datasets.\n\nThe Figures throughout the paper do a good job explaining the method and its effectiveness.\n\nThe authors perform extensive ablation studies to highlight the importance of each component of their method.\n\nWeaknesses\n\nThere are a significant number of grammar errors that somewhat distract from the work itself. Rather than list all of them here, I recommend running the text through a grammar-checking program.\n\nAlong with Figure 3, it would have been nice to visualize the sum of the condensed frame and the prompt to see just how far the prompt shifts the resulting image away from the weighted sum of frames.\n\nOne really tiny thing: the formatting of quotations is inconsistent throughout the paper. Opening quotes should be `` and closing quotes should be ‘’. This will give you the correct-looking curly quotation marks.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "PQg1IAcVlN", "reviewer": "Reviewer_cK7P", "summary": "This paper proposes FrameMaker to produce a linearly combined condensed frame and use an instance-specific prompt to represent the lost spatio-temporal details.\n", "questions": "other comments:\nLine 98 \"Temporal Shit Module\" => \"Temporal Shift Module\"", "limitations": "The limitations and societal impact claimed in the paper are reasonable and well-acknowledged.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1. Raw videos have redundant frames for action recognition. Using a condensed frame to represent the whole video is interesting.\n2. This paper is well-written and easy to follow.\n\nWeaknesses:\n1. Fig.2 $L_f^c$ is hard to optimize, as the condensed frame inherently loses spatio-temporal details. Table 4 shows that $L_f^c$ gives a marginal improvement.\n2. Memory comparison is unfair, as FrameMaker stores additional instance-wise prompts. The authors should provide a detailed space analysis for prompts.\n3. The compared CIL baselines are however not strong, the authors should consider stronger CIL methods (e.g., AANets[1], DER[2]).\n\n[1] Liu, Yaoyao, Bernt Schiele, and Qianru Sun. \"Adaptive aggregation networks for class-incremental learning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n[2] Yan, Shipeng, Jiangwei Xie, and Xuming He. \"Der: Dynamically expandable representation for class incremental learning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "H8qr0CnYfKa", "reviewer": "Reviewer_SSbH", "summary": "This paper presents an approach for video incremental learning with frame condensing and instance prompting. The overall intuition is reasonable, while condensing video into a single frame provides a memory efficiency, it looses the temporal signal, thus instance specific prompting helps to preserves some temporal signal. Experiments are done on HMDB51, UCF101, and Something-Something V2 with good results compared with current approaches. Written presentation is fairly good and enough to understand.  ", "questions": "* Given the proposed architecture are simple and effective, it is natural to ask if it work for the general efficient video classification (considering more on the aspect of memory and compute efficient rather than looking at accuracy). If it works better than TSM, the proposed approach will be much more convincing.\n\n* Are the parameters big/capital \\Theta in Eq (3) and Eq (6) shared? what are the intuition? Since Eq (3) applied on only condensed frame while Eq (6) is applied on condensed frame + prompt (expected to be compensated for motions)? \n\n* Experiments are done only on one backbone of TSM, will the method also works on another backbone?", "limitations": "* the proposed FrameMaker is currently proved to work for [only] video incremental learning which may limit the impact of the work. If there is experiment to validate the useful of FrameMaker on another task (e.g., efficient video classification not necessary incremental), it will improve the scope of the impact of the work.\n* current experiments are done on only one backbone of TSM. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength\n* The proposed approach is naturally introduced and the motivations and intuitions for proposing the approach are reasonable and convincing.\n* The proposed approach is simple, yet seems to be effective in the problem of incremental learning.\n* Experiments show improvements over current methods are consistent across 3 different benchmarks.\n* Written presentation is clear and easy to understand.\n \n## Weakness\n* The experiments are done using only one backbone architecture, i.e. TSM.\n* The reviewer understands the incremental video learning is well-defined and recently studied, but the proposed approach will be more convincing if it works beyond this problem. One closely related problem is efficient video classification using condensed frame + instance-specific prompting?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept