StPR: Spatiotemporal Preservation and Routing for Exemplar-Free Video Class-Incremental Learning
Authors: Huaijie Wang
De Cheng
Guozhang Li
Zhipeng Xu
Lingfeng He
Jie Li
Nannan Wang
Xinbo Gao
Venue: ICLR 2026 Poster (2026)
Keywords: Video Class-Incremental Learning，Frame-Shared Semantics Distillation， Temporal Decomposition-based Mixture-of-Experts
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Video Class-Incremental Learning (VCIL) seeks to develop models that continuously learn new action categories over time without forgetting previously acquired knowledge. Unlike traditional Class-Incremental Learning (CIL), VCIL introduces the added complexity of spatiotemporal structures, making it particularly challenging to mitigate catastrophic forgetting while effectively capturing both frame-shared semantics and temporal dynamics. Existing approaches either rely on exemplar rehearsal, raising concerns over memory and privacy, or adapt static image-based methods that neglect temporal modeling. To address these limitations, we propose Spatiotemporal Preservation and Routing (StPR), a unified and exemplar-free VCIL framework that explicitly disentangles and preserves spatiotemporal information. We begin by introducing Frame-Shared Semantics Distillation (FSSD), which identifies semantically stable and meaningful channels by jointly considering channel-wise sensitivity and classification contribution. By selectively regularizing these important semantic channels, FSSD preserves prior knowledge while allowing for adaptation. Building on this preserved semantic space, we further design a Temporal Decomposition-based Mixture-of-Experts (TD-MoE), which dynamically routes task-specific experts according to temporal dynamics, thereby enabling inference without task IDs or stored exemplars. Through the synergy of FSSD and TD-MoE, StPR progressively leverages spatial semantics and temporal dynamics, culminating in a unified, exemplar-free VCIL framework. Extensive experiments on UCF101, HMDB51, SSv2 and Kinetics400 show that our method outperforms existing baselines while offering improved interpretability and efficiency in VCIL.

Published as a conference paper at ICLR 2026
STPR: SPATIOTEMPORAL PRESERVATION AND ROUTING FOR EXEMPLAR-FREE VIDEO CLASS-
INCREMENTAL LEARNING
Huaijie Wang', De Cheng', Guozhang Li Zhipeng Xu Lingfeng He Jie Li Nannan Wang', Xínbo Gao Xidian University, XI an, China BBejjing Normal University, Beijing, China
xidian. edu . cn, dcheng@xidian
ABSTRACT
Video class-nncremental Learning (VCIL) seeks develop models that continu- ously learn new action categories over time without forgetting previously acquired knowledge. Unlike traditional Class-Incremental Learning (CIL), VCIL intro- duces the added complexity of spatiotemporal structures, making it particularly challenging to mitigate catastrophic forgetting while effectively capturing both trame-shared semantics and temporal dynamics. Existing approaches either rely on exemplar rehearsal, raising concerns over memory and privacy, or adapt static image-based methods that neglect temporal modeling. To address these limitations, we propose Spatiotemporal Preservation and Routing (StPR) mechanism, unified and exemplar-free VCIL framework that explicitly disentangles and preserves spatiotemporal information. We begin by introducing Frame-Shared Semantics Distillation (FSSD), which identifies semantically stable and meaningful channels by jointly considering channel-wiss sensitivity and classification contribution. By selectively regularizing these important semantic channels. FSSD preserves prior knowledge while allowing for adaptation. Building on this preserved semantic space, we further design Temporal Decomposition- (TD-MoE), which dynamically routes task-specific experts according to tempo- ral dynamics, thereby enabling inference without task IDs or stored exemplars. Through the synergy of FSSD and TD-MoE, StPR progressively leverages spatial semantics and temporal dynamics, culminating in a unified, exemplar- ffree VCIL framework. Extensive experiments on UCF101, HMDB51, SSv2 and Kinetics400 show that our method outperforms existing baselines while offering improved interpretability and efficiency in VCII..
fiixture-of-Experre
INTRODUCTION
Ilase-nncremnnal Learning (CIL) Li & Hoiem (001)) 7): Belouadah et al. (2021) : De Lange et al. (202T) Masana et al. (2022); Zhang al. (2024) Wang al. (2025) critical direction in deep learning Spolaore & Wacziarg (2009); Jhuang et al. (2013); Cheng et al. (0244): Xu et al. (2025); Cheng et al. (2026) that develops models capable learning from a sequence of tasks without forgetting previous knowledge, recognizing ever-grove growing set of classes without past task data or identifiers. key challenge catastrophic forgetting McCloskey
Cohen (1989): Rateliff (1990), where new knowledge overwrites old. While well studied for images, extending CIL to videos: Video Class-Incremental Learning (VCIL) Park et al. (2021); Villa et al. (2022), remains underexplored. VCIL differs from CIL by requiring continual learning of new categories while modeling frame-shared semantics and temporal dependencies, unlike CIL's focus on static images. This spatiotemporal complexity is critical for understanding actions, motion, and scene dynamics in real-world applications like surveillance, driver monitoring, and robotics. Further, memory and privacy constraints often prohibit storing past data, demanding continual learning without rehearsal.
"Corresoonding author.

---

Published as a conference paper at ICLR 2026
(a)
(b)
Learning
Class Ilasemnncremettl Learning
Current
Data
Extend
Spatial
Spatiotemporal Encoder
Encoder
Current Data
Old Data
Spatial
Spatiotemporal
Encoder
Encoder
Frame shared
ww.............................. Temporal Dynamics
Semantics
Figure I: (a) Prior methods rely on exemplar rehearsal or naively stack video and CIL modules. (b) Our StPR framework explicitly decouples and reuses spatiotemporal semantics to mitigate forgetting.
The central challenge of VCIL lies in mitigating catastrophic forgetting while effectively leveraging frame-shared semantics and temporal dynamics to incrementally learn new categories. Existing
methods can be broadly categorized into two types, as illustrated in Figure Ta):
1) Exemplar-
based methods Rebuffi et al. (2117): Hou et al. (2019); Douillard et al. (2020): Park et al. (2021)
Pei et al. (2022) y Villa et al. (2022) : Alssum et al. (2023); Liang et al. 2024); Chen al. (2025)
store a portion of previous data (video clips, frames, or features) and apply rehearsal to reduce
forgetting. However, storing exemplars incurs memory and privacy costs and typically emphasizes
frame-level learning without explicitly modeling temporal dynamics. 2) CIL-based methods [i & Hoiem (2017): Dhar et al. (2019); Cheng et al. (2024b) adapt algorithms developed for static images, using techniques like regu leguzaazttioon subspace projection. While avoiding exemplar storage, they often overlook spatiotemporal propertics by flattening or underutilizing temporal features. In contrast, method StPR (b)) explicitly decouples video features into Frame shared semantics and temporal dynamics, and reuses these decomposed components to enhance the model's 's ability to adapt continually, thereby reducing forgetting without storing extensive exemplars.
Specifically, we propose unified, exemplar-fred VCIL framework that fully exploits the spatiotempo- ral nature of videos. Our method integrates both spatial semantic consistency and temporal variation to mitigate forgetting and improve adaptation across tasks. Separately, we introduce: 1) Frame- Shared Semanties Distillation (FSSD). To preserve frame-shared semantics and alleviate forgetting. we quantify the semantic importance of each channel using a combination of semantic sensitivity and classification contribution. This ensures that semantically meaningful and stable channels are preserved, achieving a better trade-off between stability and plasticity. 2) Temporal-Decomposiiie based Mixture-of- Efpexxssss (TD-MoE). To exploit temporal dynamics for continual adaptation, we decouple task-specific temporal cues for each expert. At inference, expert routing depends solely on the temporal dynamics of the input, without requiring task identities or stored exemplars. This enables dynamic assignment of weights to experts according to the temporal dynamics of the input, facilitating incremental learning of new categories.
Our framework uniquely bridges the gap between video-specific spatiotemporal representation and class-increment incremental adaptation. By disentangling and leveraging both spatial semantic channel consistency and temporal dynamics, it offers an effective and explainable solution for continual video understanding. Our main contributions are: 1) We propose Trame-Shared Semantics Distillation method (FSSD) that preserves frame-shared, semantically aligned spatial channels through semantic importance-aveer aware regularization, optimizing the stability-plastiiie trade-off in continual learningl; 2) We design a Temporal Decomposition based coxxtureoof-Expert strategy (TD-MoE) that decomposes spatiotemporal features and uses temporal dynamics for expert combination, enabling task-id- fref and dynamic adaptation; 3) We present a unified, exemplar-fres VCIL framework that achieves state-of-ther the-trterr results on UCF101, HMDB51, SSv2 and Kinetics400, demonstrating the effectiveness of integrating spatial semantics and temporal dynamics VCIL.
2 RELATED WORK
2.1
CLASS-INCREMENTAL LEARNING
flass-Incrementas Learning (CIL) aims to enable models to continually learn new classes without forgetting previously learned ones. Existing approaches typically fall into three categories: (1)
2

---

Published as a conference paper at ICLR 2026
regularization-basse methods Kirkpatrick et al. (2017); Zenke al. (2017); ;; Xiang et al. 2022);
Zhou et al. (2023); ; Li et al. (2024a); Wang et al. (2025). which constrain parameter updates to preserve prior knowledge, often via knowledge distillation Li & Hoiem (007)); Hou et al. (2019); (2)
exemplar-based methods |Bang et al. 20 Chaudhry et al. 22: Rebutti et al. (2017). which store or generate past data reduce forgetting: and (3) structure-based methods (Serra al. (2018) Mallya & &Lazebnik]/ (2018); Mallya et al. 2018); LLing & (2024); | Yu et al. (2024); He et al. (2025b). which expand model capacity or isolate task task-specific components. Recently, CIL combined with pre-trained vision transformers ( (ViTs) Ermis et al. [2022); Smith et al. (2023); Wang et al. (202bb c) He et al. (2025a) has achieved impressive results by leveraging transterable representations and modularity, Some methods fully fine-tune pre-trained backbones [Boschini et al. (2022): Zhang et al. (2023), but this is computationally expensive. To address efficiency, parameter-effiiien fine-tuning (PEFT)
methods have been introduced. Prompt pool-based approaches Wang et al. (2022c); |Smith et al. (2023); Wang et al. (2024); Zhang et al. (2023) maintain task-specific prompts. while udapter-based methods Zhou al. (2024a); Tan et al. (2024); ) Gao et al. (2024); Liang & Li (2024); Zhou et al.
(2024b) adapt ViT's to new classes with minimal updates. While effective, most CIL strategies were
originally developed for static image domains and do not generalize well to video-based scenarios, where temporal dynamics play critical role.
2.2 VIDEO CLASS-INCREMENTA LEARNING
Action recognition has been widely explored with 2D CNNs using temporal aggregation Lin et al. (2019); Wang et al. (2016) and 3D CNNs or transformer for joint spatiotemporal modeling Carreira Zisserman (2017) Tran et al. (2015); Li et al. (2024b). More recent work focuses on improving temporal sensitivity and efficiency Feichtenhofer (2020); Fan et al. (2020); Li et al. (2021) or text information Li et al. (202)). ). However, these models are trained in static setups and do not address continual adaptation forgetting. To address these challenges, Video Class- Incremenall Learning (VCIL)) extends conventional Incremencrllttlll Learning (CIL. to spatiotemporal data, introducing additional challenges such as managing temporal variations across tasks. Several recent methods, including TCD Park et al. (2021), FrameMaker Pei et al. (2022), and HCE Liang et al. )2024), address this setting by storing videos or compressed exemplars. However, these strategies raise concerns related to memory efficiency and data privacy. While SMILE Alssum et al. (2023) effectively extracts image features from individual frames, it does not explicitly capture temporal information, which may limit its ability to leverage the distinctive decision cues present in video data. Exemplar-fir free methods such as STSP/Cheng et al. (2024b) mitigate forgetting via orthogonal subspace projections, but they mainly adapt image-domain strategies to video tasks. In contrast, our approach decouples and models the spatiotemporal structure of videos, proposing a unified VCIL framework that preserves spatial consistency via Frame- Shared Semantics Distillation (FSSD) for knowledge retention without exemplars, while leveraging temporal dynamics for expert routing through Temporal Decomposition--asse based Mixture-of-Expeee Experts (TD-MoE).
METHOD
Problem Definition: In the Video Class-Incrementee Learning (VCIL) setting, model is trained
across B stages with sequentially arriving datasets (D),..., DD.. Each dataset D {(V), v/) [P'] corresponds the b-th task, where V is the j-th video and 3 is its class label. Here, videos primarily represent human action recognition scenarios, where the spatiotemporal dynamics capture motion patterns, subject interactions, and scene context. [D') represents the number of samples in the b-th task. Let )d be the label space of the b-th dataset. For all b * b', the label spaces are disjoint: ndb n no = Ø. The objective of VCIL is to incrementally train a model over B tasks while maintaining high performance across all accumulated classes (J/1, ,. ,,,,...,,)))
Overall framework. We propose a unified, exemplar-free framework for Video Clase-lnc Learning (VCIL) built upon the CLIP model Radford et al. (202I) Our goal is to mitigate catas- trophic forgetting while effectively leveraging frame-shared semantics and temporal dynamics to incrementally learn new categories. The frozen visual encoder F(-) extracts spatial features, while adapters A are updated for each task b. A spatiotemporal encoder G(-) models temporal dynamics. Our framework introduces two key components: 1) Frame-Shared Semantics Distillation (FSSD) identifies semantically stable channels across frames by combining Semantic Sensitivity and Classifi-
3

---

Published as a conference paper at ICLR 2026
Spatiotemporal Encoder G2
Concatenate
CLIP Encoder
b
dy
Multi-head Self-A
A
(a)
FSSD
Temporal
Dynamics
Coot
VH
Lyso
Anchor Pool
"Ride Horse"
Spatial Encoder
b-1
V
Text Encoder
Inference
(b)
A ssign Weight
Expert
(II
Video Feature
V
Spatial Encoder Fg
V2
Router
TD-MoE
Trainable
+
Frozen
Frame-Shared Semantics Distillation
O0O: Channel of Spatial Feature
Temporal Decomposition
Figure 2: Overview of our proposed framework. (I) During training, the (a) Frame-Shared Seman- tics Distillation (FSSD) module retains past knowledge via frame- semantics importance-aware regularization. (II) At inference, the (b) Temporal Decomposition- MoE) dynamically routes input videos to expert branches by leveraging the disentangled temporal component of the spatiotemporal representation, enabling adaptive predictions.
Mixture-of-Expeer Expertss (TD-
cation Score, applying selective regularization to preserve critical spatial semantics while maintaining plasticity. Temporal Decomposition based Mixture-of-Exxee (TD-MoH). To exploit temporal dynamics for continual adaptation, we decouple shared static components and temporal dynamics. During inference, temporal dynamies are used assign dynamic weights expert temporal encoders, enabling task-id-free adaptation without requiring task identifiers or stored exemplars.
3.1 SPATIAL AND SPATIOTEMPORAL ENCODER
Spatial Encoder. The shared adapter module Chen et al. 2022) A'' = {A))) utilized with frozen CLIP-VIT model with N layers of transformer module, serving as the spatial extractor. An adapter is an encoder-decode architecture embedded into the residual of each transformer layer, facilitating transfer learning and enhancing downstream task performance. Typically, it consists of a down-sampling MI.P Woown E dRdx,, a ReLU activation Φ(.), and an up-sampling MLP W Rd where d is the input and output dimension, and d, is the hidden dimension. For adapter, if input is v Rd, which is the i-th frame-level feature after the Multi- Head Self-Attention and residual connection in CLIP ViT, the output of adapter is:
vf ((v W down W up-
The spatial feature of i-th frame V F(V;; A*), V E Rdor, where is the i-th frame of the video, dut is the dimension of the aligned video-text features.
Spatiotemporal Encoder. To obtain the spatiotemporal representation, we feed both the frame-level spatial features and learnable [CLS] token into multi-head self-attention based spatiotemporal encoder (()). Specifically, the input
consists of the frame features Vi...., Vū, and [CLS]
token V E Rd.,, where N₂ represents the number of sampled frames. The temporal encoder outputs the spatiotemporal feature Vst, corresponding to the transformed [CLS] token:
G(V* aea Vi:
V D E Ru
(2)
where [0] selects the output associated with the [CLS] token after attention brast aggregation.
4

---

Published as a conference paper at ICLR 2026
3.2 FRAME-SHARED SEMANTICS DISTILLATION
In VCIL, shared adapter modules inevitably drift in feature space when adapting to new tasks, leading forgetting. Directly applying classic mniforn-weighted distillation from CIL to video tasks ignores the differences in semantic importance and temporal stability across video features. This leads to a suboptimal balance between stability and plasticity. To address this, we propose Frame-Shared Semanties Distillation (FSSD), which identifies stable cross- framme channels capturing semantics and regularizes them to preserve prior knowledge while allowing adaptation
Frame-Shared Semanties Distillation (FSSD). To mitigate semantic drift across tasks, we introduce a distillation loss weighted by the frame-shared semantics importance:
1C
No
dut
LESSD
Volc.es
DDb
dut
C
where [Col is the number of classes in the b-th task, and Ne is the number of samples per class. 16-1. denotes the frame-shared importance the j-th channel for class c from task (b-1). Vo-1c and V "...i.j are the j-th channel outputs of the i-th sample in class extracted from the spatial encoders of task (b-1) and b, respectively, calculated current data, with the previous model frozen.
Frame-Shared Semantics. To quantify the importance of frame-shared semantics, we assess each channel based two criteria: 1). Semantic Sensitivity. It measures the responsiveness to activation changes, thereby reflecting its reliability in representing consistent semantic information. and 2) Classification Score. It reflects the channel's contribution the final classification.
For semantic sensitivity, we employ Fisher Information to estimate how sensitively a channel's , Ny
V aggregate frame-wise variations, the Central Limit Theorem suggests each channel's distribution approximates a Gaussian (Belong to the same category). Thus, we assume the j-th channel activation for class e follows (For simplicity, we omit the subscripts for task and sample.):
activation influences the output. As spatial features V N 2i=1
Vi ~ (Peaj 2
where He., and oa denote the mean and variance across frames. The Fisher Information I(He,)) (Detailed derivations are provided in the appendix B.I with respect to Pe.j is computed as:
27
I(po.) =
VEJ How 2
1 T
L(VGs - Hes)?l
=
1 Fod
(5)
For classification score, we compute the cosine similarity between the spatial video feature V E dgdut and its corresponding text feature T. Rddr, Specifically, for the j-th channel, the classification score defined as To.j =
VT, [V where To,; denotes the j-th dimension feature of Te We then take the expectation of Yo.j across frames to obtain stable channel-level contribution estimate:
Vž Tej |
V Taj
Te Me.j
x
E
(6)
|TaIl
|
where || V X treated as a constant after normalization.
Combining semantic sensitivity and classification score, the semantic importance for the j-th channel of the c-th class is defined as:
Tej Pe. 02
Iaj
FSSD accumulates Trame-shared semantic importance as distillation weights, retaining key chan- nels for old tasks while allowing less important ones to adapt, thus balancing stability and plasticity.
5

---

Published as a conference paper at ICLR 2026
KPSS Mean p-value Distribution (UCF101)
KPSS Mean p-value Distribution (HMDB51)
6000
0.05 (Strict)
1400
0.05 (Stnct)
(Shortterm)
(Short-tarm)
5000
1200
Frequency
4000
Frequency
1000
800
600
2000
400
1000
200
KPSS p-value (mean)
KPSS p-value (mean)
Figure 3: For each video, we uniformly sample 8 frames to compute the p-value defining p > 0.05 as strictly stationary and p > 0.02 as weakly stationary in the short term.
3.3 TEMPORAL DECOMPOSITION BASED MIXTURE-OF-EXPERTI
Given the high forgetting tendency of deep transformers in VCIL, we allocate a dedicated spatiotem- poral encoder for each task. task IDs are unavailable during inference, we allocate a spatiotemporal encoder per task and design routing mechanism that dynamically weights experts based on temporal patterns, ensuring relevant experts contribute more to the final representation.
Task-Specifie Expert. For each task, we train dedicated expert based on the spatiotemporal encoder. The spatiotemporal features V Rdd captured by each expert are computed as Eq.
Temporal Decomposition-basee Router.
design this routing mechanism based on temporal dynamics, we consider two aspects: 1) Temporal residuals. These reflect the subtle temporal differences within redundant frames. 2) Inter-frame information. This captures abstract temporal concepts between frames, based on the knowledge learned by each expert.
For temporal residuals, we observe that redundant frames, where backgrounds and subjects remain consistent, cause minimal variation between adjacent frames Kim & Choil (2024); Liu al. (2021). This leads to short-term temporal stationarity, which we further validate on the UCF101 and HMDB51 datasets Fig. 3 Thus, each frame feature decomposed V = 6,, with shared static components and as temporal residuals. The spatial representation then the mean across frames:
1 N N =11
Σ 4.
- ==
(8)
For the inter-frame information, since the spatiotemporal feature Vat is computed by the atten- tion module, it can be approximated as V ΣMa, a; Vs, where a, attention score. After normalization, we can obtain Σ Ug 1. Substituting Eq. we obtain:
N a. E,
a V
Since v is difficult to estimate, and to decouple the temporal residual e; and inter-frame information ag naturally address this by using the difference between Vat and V*, effectively isolating the temporal dynamics, which can be represented as:
Vea
Σ
a4 2 EE.
(10)
This formulation reveals that fiem quantifies the deviation between the model's intention-weightee temporal dynamics and the uniform temporal mean, effectively disentangling temporal variations from static semantics. This enables routing to exploit temporal cues while avoiding background interference, thereby mitigating forgetting and enhancing continual learning.

---

Published as a conference paper at ICLR 2026
Inference. During inference, first compute the decoupled temporal representation Viem R' for each imput video. For all categories in the current task, we calculate the mean temporal representation and store it in the anchor pool Vtemm R where represents the c-th class. For each expert k, we compute a similarity- score as the router:
Tk
max cos (Viem, Vtem), cEC₁
(11)
where Cs represents the set of classes assigned to expert k. Then, we combine the adapter-tuned spatial features V with the expert outputs weighted by rı the final video representation:
V= V Σ mh Vg.
(12)
&
The final video representation is matched with text embedding via cosine similarity for classification.
3.4 LOss FUNCTION AND OPTIMIZATION
Our loss function includes: 1) contrastive loss between video features and text descriptions for classification; 2) contrast loss between video features under adapter fine-tuning and text features for spatial optimization: and 3) FSSD loss to mitigate forgetting in shared adapter modules.
Contrastive Loss Formulation. We use symmetric contrastive loss for video-to-text and text-to-video alignment. Given batch of N samples, let V; and T, denote the video and text features, respectively, The similarity between video i and text j is computed as the cosine similarity Sr cos (V. T2), forming similarity matrix S E RNXN, Let M {0, N be the label mask, where M, = 1 if y: yj and Mr 0 otherwise. Then, the Video-to-text contrastive loss can be caculated by:
i= Maj exp(S..) E exp(St.) +
Lu N
log
(13)
where is a small constant added avoid division by zero. The Text-to- loss Lay is similarly defined by swapping video and text in the equation. Symmetric total contrastive loss is:
LCont 2 (fur + fez)).
(14)
For LS Com the embeddings are the spatiotemporal video feature v and corresponding text features. For LS we use the CLIP adapter feature V and corresponding text features.
Total Loss. The overall training loss is:
L St
S
+ w LISSD:
(15)
where w is hyperparameter. This design aligns both spatial and spatiotemporal semantics with text supervision, while the FSSD term preserves critical frame- semantics to mitigate forgetting.
4
EXPERIMENTS
4.1
EXPERIMENTAL DETAILS
Dataset. We evaluate our method on four benchmark datasets: UCF101 Soomro (2012).
V2 (SSv2) et al. (017) and Kinetics400 Carreira Zisserman (207)) All experiments are conducted in an exemplar-free setting. For the TCD benchmark Park et al. (2021) on UCF101, HMDB51, and SSv2, pretraining the model on 51, 26, and 84 base classes, respectively, with the remaining classes split into tasks. For Kinetics-400, we follow the vCLIMB benchmark Villa et al. (022) with 10- or 20-task splits, each containing the same number of classes.
HMDB5 Kuehne et al. (201T). Something-Sometine
fair comparison,
Evaluation Metries. We adopt three widely-used metrics to evaluate performance in VCIL: 1).
Final Accuracy (Ace) Villa et al. (2022). which measures the overall classification accuracy on all learned classes after the final task is completed; 2). Average Accuracy (Acc) Park et al. 2021 which
7

---

Published as a conference paper at ICLR 2026
Table 1: Average Accuracy (Acc) of the UCFI01 and HMDB51 under the TCD benchmark.
UCF101
HMDB51
Method
Exemplar
Venue
10 x 58
5 10s
2 25s
5 5s
1 x 25s
iCaRL Rebufhi et 2017
CVPR'T7 17
65.34
64.51
58.73
40.09
33.77
LwFMC
La
Hoıem
2017
X
18
42.14
25.59
11.68
26.82
16.49
LwM
Dhar et al. 2019
X
CVPR'1 19
43.39
26.07
12.08
26.97
16.50
UCTR Hou ct al. 2019
CVPR' 19
74.09
70.50
64.00
46.53
37.15
PODNet
Douillad et al. 2020
FCCV
74.37
73.75
71.87
48.78
46.62
PCD el 2021
ICCV'21
77.16
75.35
74.01
50.36
46.66
FrameMaker el al. (2022
22urlPS'22
78.64
78.14
77.49
51.12
47.37
L2P Wang ct al. 20022c
X CVPR'22
81.24
80.09
78.58
49.98
45.87
S-iPrompps Wang el al 2022a
X
22
80.60
80.27
80.43
53.11
53.89
ST-Prompt Per el al. 202.3
X
CVPR 23
84.75
85.54
85.67
60.14
60.54
STSP et
20244
X
2C
81.15
82.84
79.25
56.99
49.19
HCB Liang et al.
20224
AAAI'24
80.01
78.81
77.62
52.01
48.94
CoSTEO Zou al. 2225
CVPR 25
86.05
86.71
86.95
61.70
61.84
StPR (Ours)
X
94.67
92.13
88.52
68.12
67.01
Table 2: Average Accuracy (Acc) of the SSv2 under the TCD benchmark, with best results in bold.
Method
Exemplar
Venue
10 x 9s
x 18s
iCaRL Rebulli et al. (2017
CVPR'T7
20.41
16.62
UCIR Hou et al. 2019
CVPR' 19
24.32
19.31
PODNet
Douillad et al. 2020
ECCV'20
27.63
20.14
TCD Park PCD
al. 202
ICCV'21
29.32
21.69
FrameMaker Pei
2022
NeurIPS'22
31.41
26.57
L2P Wang et al. 2022c
Wang
CVPR'22
26.02
21.33
S-iPrompts Wang of al. 2202a
NeurlPS'22
33.69
30.84
ST- ST-Prompi Pci ct al. 202.3
X
CVPR'23
39.98
35.44
HCE Liang el al. 2024
AAAI'24 36.88
32.82
CoSTEO Zou SIPR (Ours)
2025
CVPR'25
41.44
36.60
40.79
37.30
measures the mean classification accuracy over all incremental stages after the final task is completed; 3). Backward Forgetting (BWF) Villa et al. (2022), which quantifies the average drop in performance on previously Iearned tasks, reflecting how well the model retains past knowledge.
Implementation Details. All experiments conducted We adopt the CLIP ViT-B/16 model Radford et al. (2021) ) as the backbone, with all its parameters frozen during training. The spatial and spatiotemporal encoders
single NVIDIA RTX 3090 GPU.
the only trainable components in our framework. For optimization, we employ Stochastic Gradient Descent (SGD) with an initial learning rate of 0.01 and a batch size of 40. Each task is trained for 60 epochs in the first incremental session and 30 epochs in each subsequent session. The weighting hyperparameter w in Eq. 15 is to 1 10 The multi-head self-attention module within the spatiotemporal encoder consists of three transformer layers, each employing two attention heads. Video clips are sampled using the TSN strategy Wang et al. (2018), selecting 8 frames per video uniformly across the temporal dimension.
4.2
MAIN RESULTS
Table. 2 and 3 report results on UCF101, HMDB51, SSv2 and Kinetics400, covering different action complexities and temporal dynamics. Based on their strategies to mitigate forgetting, existing methods are categorized into two groups: 1) Exemplar-based methods (iCaRL, UCIR, PODNet, TCD, FrameMaker, HCE, vCLIMB, SMILE, CSTA). They store video clips, frames, or compressed features and apply rehearsal to reduce forgetting. However, these methods face scalability and privacy challenges due to their reliance stored exemplars. 2) ClL-based methods (LwFMC, LwM, L2P, S-iPrompts, ST-Prompt', STSP). This group adapts techniques from image- class-incred inss-innrrmeetal learning, such unified distillation and subspace projection, without storing exemplars. While avoiding exemplar storage, their performance tends to be lower, especially as task difficulty increases and lacking explainable spatiotemporal disentanglement. In contrast, Our method (StPR) without
8

---

Published as a conference paper at ICLR 2026
Table 3: Results of the Kinetics-400 under the vCLIMB benchmark at 10 and 20 task settings.
Kinetics400-
Kinetics400-222
Method
Exemplars
Venue
Acc 1
BWF
Ace t
BWF
VCLIMB+BiC
Villa et al.
2022
CVPR'22
27.
51.96
23,06
58.97
VCLIMB+ +IBB+iCCRRIII Villa Villa
2022
/
CVPR'22
32.04
38.74
26.73
42.25
SMILE+BiC Alssum et 2023
/
CVPR'2)
52.24
6.25
48.22
0.31
SMILE+iCaRL Alssum
2023
CVPR'23
46.58
7.34
45.77
4.57
CSTA (Vivit)
Chen el al.
2025
/
TCSVT'25
54.98
5.06
51.01
6.91
CSTA (Times)
Chen ct al.
2025
V
TCSVT"25
56.09
4.97
52.20
6.89
Ours
X
57.83
14.01
53.95
15.09
Table 4: Ablation Study on UCFI01 and HMDB51, with best results in bold.
A FSSD TD-MoE
UCFI0I(5
10s)
HMDB51
58)
UCF101(10 x 58)
HMDB51 1s)
Acc f Acr
BWF.)
4
Acc
Āer:
BWF 4 Acc
Arc
BWF.
Acc
Are
BWF J
70.65
74.67
7.01
43,30
43.62
6.18
70.14
72.72
5.33
43.71
47.48
8.74
/
74.23
79.21
8.88
48.99
55,95
10.47
74.71
78.68
8.54
52.92
57.10
9.99
/
/
77.55
81.84
5.76
53.23
55.67
7.73
77.63
82.06
4.86
54.63
60.83
9.01
89.36
12.38
61.14
11.75
85.94
93.47
7.40
62.54
68.88
10,20
/
83.07
91.28
10.52
57.47
63.37
21.30
88.03
94.14
8.39
64.71
73.02
21.72
V
V
85.79
92.13
5.63
63.04
68.12
11.04
88.85
94.67
6.31
69.61
75.07
7.02
storing exemplars, surpasses all baselines across datasets and settings. On the TCD benchmark, our method outperforms the state-of-the, aar approach (ST-Prompt") as well as all exemplar- based methods UCF101, HMDB51, and SSv2. On the vCLIMB benchmark, exemplar-based methods can alleviate forgetting by replaying stored samples,
which makes forgetting lower. Nevertheless, method achieves higher final accuracy, surpassing the current state-of-the-art (CSTA) and all exemplar-based counterparts.
Baseline
UCF101(5 x 10s)
UCF101(10 x 5s)
100
302
302
Accuracy (96)
Task
Figure 4: Task wise ablation analysis across incremental tasks on UCF101 and HMDB51.
4.3
ABLATION STUDY
We perform ablation studies evaluate the contribution of each component: the adapter tuning (A*),, Frame-Shared Semantics Distillation (FSSD), and Temporal Decomposition-basee Mixture-of- Experts (TD-MoE). Results are summarized Table The baseline (pretrained CLIP) model exhibits limited performance on downstream tasks, as it lacks adaptation to new task-specific categories. Introducing FSSD alone moderately improves performance by preserving spatial semanties and reducing semantic drift, while TD-MoE independently enhances adaptation by leveraging temporal dynamics. However, using either module alone yields suboptimal performance. Combining adapter tuning with TD-MoE provides further improvements but still lacks sufficient stability in preserving spatial semantics. The full model (StPR), integrating both FSSD and TD-MoF, achieves the most stable performance across tasks, demonstrating the complementary strengths of spatial semantic preservation and temporal dynamic modeling.
As shown in Figure 4 we progressively add our modules (FSSD and TD-MOE) on the UCF101 and HMDB51 datasets. Significant improvements are observed at each incremental stage, with more pronounced gains the number of tasks increases, especially in the long-term scenario (10 tasks). This further validates the effectiveness of our proposed method and the superior performance of our model.

---

Published as a conference paper at ICLR 2026
4.4 FURTHER ANALYSIS
Analysis of remporal-decomppoition routing strategies. Table compares our TD-MoE with several alternative Mixture-of-Exee Experts (MoF) al. (991)) routing strategies. Simple averaging (Avg-MoF) and static weight assignments-inclinge includinn CLIP-MoR, which uses frozen CLIPP visual moderate improvements but fail to dynamically leverage task-specific temporal cues, often resulting in higher forgetting. In contrast, TD-MoE enables adaptive expert weighting based on temporal dynamics, consistently improving both accuracy and stability across tasks. This highlights the importance of modeling temporal variability explicitly, rather than relying on static or feature -aguostc routing.
features for routing, and Adapter-MoE, which uses adapter- tuned CLIP features-
Table 5: MoE Method Results on UCFI01 and HMDB51, with best results in bold.
UCFI01(5
10s)
HMDB51(5
58)
UCF101(10 58)
HMDB51 1s)
Method
Acc f
BWF
Acc
BWF 4
Acc f
BWF
Acc
BWF +
Avg-MoE
81.14
9.95
59.46
8,49
84.04
9.60
62.69
9.14
CLIP-MoE
83.59
7.85
58.82
10.01
85,80
7.27
65.43
8.08
Adapter-MoB
83.26
6.07
61.99
7.75
84.31
7.82
65.57
7.68
TD-MoB(OUrs)
85.79
5.63
63.04
11.04
88.52
6.39
69.61
7.02
Table 6: Distillation Method Results on UCFI01 and HMDB51, with best results in bold.
UCF101(5
10s)
HM
5s)
UCFI0I(10 5s)
HMDB51 1s)
Method
Acc f
BWF
Acc 1
BWF 1
Acc 1
BWF
Acc
BWF |
w/o Distillation
83.07
10.52
57.47
21.30
88.03
8.39
64.71
21.72
Distillation
84.27
7.45
61.74
13.33
88. 12
7.95
67.54
13.38
FSSD(Ours) 85.79
5.63
63. 63.04
11.04
88.52
6.39
69. 61
7.02
Effectiveness of FSSD over Uniform Distillation. Table @ compares our FSSD method with the no-distillation baseline (w/o Distillation) and standard uniform distillation (Distillation) across four VCIL settings. While uniform distillation improves accuracy and reduces backward forgetting (BWF) over the naive baseline, FSSD consistently outperforms both, achieving the highest accuracy and lowest BWF in all settings. These results highlight the benefit of selectively preserving ftame-shared semantics, validating the importance-awwut design of FSSD for continual video learning. For more experiments, see the appendix C
5 CONCLUSION
In this work, we propose StPR, a unified and exemplar-free framework for Video Class- Inaseeencremttall Learning (VCIL) to tackle the spatiotemporal challenges in continual video learning. By disentangling spatial semantics and temporal dynamics, StPR effectively balances stability and plasticity without relying on stored exemplars. Our method combines Trame-Shared Semanties Distillation (FSSD), which selectively preserves meaningful and stable semantic channels, protecting model's plasticity. based contuure-ofEEpperte (TD-MoE), adaptively routes inputs based on temporal cues, reducing forgetting in deep networks. Extensive experiments UCF101, HMDB51, and SSV2 validate the effectiveness and efficiency of our approach, establishing new ofate-of-the-ae results for continual video recognition. In future work, we plan explore more realistic application scenarios, such as open-world settings, and investigate the deployment of our method on resource- constrained edge devices. See Appendix for our statement on LLM usage.
6 ACKNOWLEDGMENTS
This work was supported by the New Generation Artificial Intelligence-Natione Science and Technology Major Project (2025ZD0123601), in part by the National Natural Science Foundation of China under Grants 62576262, U22A2096, in part by the Key Research and Development Program of part by the Fundamental
Shaanxi Province under grant 2024GX- YBXM135, 2024SF-YBXM-66 Research Funds for the Central Universities under Grant QTZX25083, QT7X23042.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Lama Alssum, Juan Leon Alcazar, Merey Ramazanova, Chen Zhao, and Bernard Ghanem. Just a glimpse: Rethinking temporal information for video continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2474-2483, 2023,
Jihwan Bang, Heesu Kim. YoungJoon Yoo, Jung- Woo Ha, and Jonghyun Choi. Rainbow memory:
Continual learning with a memory of diverse samples. In Proceedings of the IEEF/CVE conference on computer vision and pattern recognition, pp. 8218 88888,,27 2021.
Eden Belouadah, Adrian Popescu, and loannis Kanellos. A comprehensive study of class incremental learning algorithms for visual tasks. Neural Nenworks, 135:38-50 55, 2021.
Matteo Boschini, Lorenzo Bonicelli, Angelo Porrello, Giovanni Bellitto, Matteo Pennisi, Simone Palazzo, Concetto Spampinato, and Simone Calderara. Transfer without forgetting. In European Conference on Computer Vision, pp. 692-709. Springer, 2022.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6299-6308, 2017.
Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the European conference on computer vision (ECCV), pp. 532-547, 2018.
Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. Advances in Neural Information Processing Systems, 35: 16664-16676 2022.
Tieyuan Chen, Huabin Liu, Chern Hong Lim, John See, Xing Gao, Junhui Hou, and Weiyao Lin. Csta: Spatoal-temporal causal adaptive learning for exemplar-free video class-incremental learning. IEEE Transactions Circuits and Systems for Video Technology, 2025.
De Cheng, Zhipeng Xu, Xinyang Jiang, Nannan Wang, Dongsheng Li, and Xinbo Gao. Disentangled prompt representation for domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23595-23604, 2024a.
De Cheng, Zhipeng Xu, Xinyang Jiang, Dongsheng Li, Nannan Wang. and Xinbo Gao. Prompt disentanglement via language guidance and representation alignment for domain generalization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026.
Hao Cheng, Siyuan Yang, Chong Wang, Joey Tianyi Zhou, Alex C Kot, and Bihan Wen. Stsp: Spatial- temporal subspace projection for video class-incremental learning. In European Conference Computer Vision, pp. 374-391. Springer, 2024b.
Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence, 44(7):3366 3385, 2021.
Prithviraj Dhar, Rajat Vikram Singh, Kuan-Chuan Peng, Ziyan Wu, and Rama Chellappa. Learning without memorizing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5138-5146, 2019.
Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. Podnet: Pooled outputs distillation for small-tasks incremental learning. In Computer vision-ECCC) 2020: 16th European conference, Glasgow, UK, August 23-28, 2020, proceedings, part XX 16, pp. 802102. Springer, 2020.
Beyza Ermis, Giovanni Zappella, Martin Wistuba, Aditya Rawal, and Cedric Archambeau. Memory efficient continual learning with transformers. Advances in Neural Information Processing Systems, 35:10699-10642. 2022.
Linxi Fan, Shyamal Buch, Guanzhi Wang, Ryan Cao, Yuke Zhu, Juan Carlos Niebles, and Li Fei-Fei. NetiksNet: learnable 3d-shift for efficient video action recognition. In ECCV, 2020.
11

---

Published as a conference paper at ICLR 2026
Christoph Feichtenhofer. X3D: Expanding architectures for efficient video recognition. In CVPR, 2020.
Xinyuan Gao, Songlin Dong, Yuhang He, Qiang Wang, and Yihong Gong. Beyond prompt learning: Continual adapter for efficient rehearsal-free continual learning. European Conference on Computer Vision. pp. 89-106. Springer, 2024.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuma Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mlueller-Freitage et al. The" something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE international conference on computer vision, pp. 5842-5850, 2017.
Lingfeng He, De Cheng, Zhiheng Ma, Huaijie Wang, Dingwen Zhang, Nannan Wang, and Xinbo Gao. Ckaa: Crossesubspace knowledge alignment and aggregation for robust continual learning. arXiv preprini urXiv:2507.09477 2025a.
Lingfeng He, De Cheng, Di Xu, Huaijie Wang, and Nannan Wang. Harnessing textual semantic priors for knowledge transfer and refinement in clip-driven continual learning. arXiv preprint arXiv:2508. 01579, 2025b.
Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning a unified classifier incrementally via rebalancing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 831-839, 2019.
Robert A Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.
Hueihan Jhuang, Juergen Gall, Silvia Zuffi, Cordelia Schmid, and Michael Black. Towards understanding action recognition. In Proceedings of the IEEE international conference on computer vision, pp. 3192-3199, 2013.
Hyun-Wee Woo Kim and Yong-Suk Choi. Fusion attention for action recognition: Integrating sparse-dense and global attention for video action recognition. Sensors (Basel, Switzerland), 24621):6842, 2024.
James Kirkpatriek, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barr Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521-3526, 2017.
Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: large video database for human motion recognition. In 2011 International conference on computer vision, pp. 2556-2563, IEEE, 2011.
Guozhang Li, Jie Li, Nannan Wang, Xinpeng Ding, Zhifeng Li, and Xinbo Gao. Multi-hierarchical category supervision for weakly- temporal action localization. IEEE Transactions Image Processing, 30:9332-9233 -3444, 2021. doi: 10. (09/TIP.2021.312466)
Guozhang Li, De Cheng, Xinpeng Ding, Nannan Wang, Xiaoyu Wang, and Xinbo Gao. Boosting weakly-supervised temporal action localization with text information. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10648-10657, June 2023.
Guozhang Li, De Cheng. Xinpeng Ding. Nannan Wang, Jie Li, and Xinbo Gao. Weakly supervised temporal action localization with bidirectional semantic consistency constraint. IEEE Transactions on Neural Networks and Learning Systems, 1309)) 2024a. doi: 10.1 109/TNNSS. 2023.3266062.
Guozhang Li. De Cheng, Nannan Wang, Jie Li, and Xinbo Gao. Neighbor-guided pseudo-label gener- ation and refinement for single-frame supervised temporal action localization. IEEE Transactions on Image Processing, 33:2419-244 2430, 2024b. doi: 10.100991P.20443338777.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-2947, 2017.
12

---

Published as a conference paper at ICLR 2026
Sen Liang, Kai Zhu, Wei Zhai, Zhiheng Liu, and Yang Cao. Hypercorelattio evolution for video insseminceeettal learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 3315 3315-3323, 2024.
Yan-Shuo Liang and Wu-Jun Li. Inflora: Interference-free low-rank adaptation for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23638 23647, 2024.
Lin, Chuang Gan, and Song Han. TSM: Temporal shift module for efficient video understanding. In ICCV, 2019.
Xin Liu, Silvia L Pintea, Fatemeh Karimi Nejadasl, Olaf Booij, and Jan C Van Gemert. No frame left behind: Full video action recognition. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 14892-14901, 2021.
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 7765-7773, 2018.
Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting a single network to multiple tasks by learning to mask weights. In Proceedings of the European conference computer vision (ECCV), pp. 67-82, 2018.
Mare Masana, Xialei Liu, Bartlomiej Twardowski, Mikel Menta, Andrew D Bagdanov, and Joost Van
De Weijer. Class-incrementte learning: survey and performance evaluation on image classification.
IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5):5513-555 5533, 2022.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of Learning and Motivation, volume 24, pp. 109-165. Elsevier, 1989.
Jaeyoo Park, Minsoo Kang, and Bohyung Han. Class-incremental learning for action recognition in videos. In Proceedings of the IEEF/CVE international conference on computer vision, pp. 13698-1333 13707, 2021.
Yixuan Pei, Zhiwu Qing, Jun Cen, Xiang Wang. Shiwei Zhang, Yaxiong Wang. Mingqian Tang, Nong Sang, and Xueming Qian. Learning condensed frame for memory-efficient video class- incremental learning. Advances in Neural Information Processing Systems, 35:31002-31016, 2022.
Yixuan Pei, Zhiwu Qing, Shiwei Zhang, Xiang Wang. Yingya Zhang, Deli Zhao, and Xueming Qian. Space-time prompting for video class-incremental learning. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 1932-11994 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8766 PmLR, 2021.
Roger Ratcliff. Connectionist models of recognition memory: Constraints imposed by learning and forgetting functions. Psychological review, 97(2)2285, 1990.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In International conference on machine learning, pp. 4548 4557. PMLR, 2018
James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Basillte-Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. CODA-Prompt: COntinual Decomposed Attention Attention-Based Prompting for Rehearsal-Free Continual Learning, In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 199909-19999 June 2023. doi: 10.1 PA.109//VVPR577722233010
13

---

Published as a conference paper at ICLR 2026
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: 2012.
Enrico Spolaore and Romain Wacziarg. The diffusion of development. The Quarterly journal of economics, 124(2):469-55) 529, 2009.
Yuwen Tan, Qinhao Zhou, Xiang Xiang, Ke Wang, Yuchuan Wu, and Yongbin Li. Semantically- shifted incremental udapter-tuning is continual vitransformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 23252-23262, 2024.
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks. In ICCV, 2015.
Andrés Villa, Kumail Alhamoud, Victor Escorcia, Fabian Caba, Juan León Alcázar, and Bernard Ghanem. velimb: novel video class incremental learning benchmark. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 19035-19044, 2022.
Huaijie Wang. De Cheng, Lingfeng He, Yan Li, Jie Li, Nannan Wang, and Xinbo Gao. Ekpe: Elastic knowledge preservation and compensation for class-incremental learning. arXiv preprint arXiv:2506.12355 2025.
Limin Wang, Yuanjun Xiong, The Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In ECCV, 2016.
Limin Wang, Yuanjun Xiong, The Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks for action recognition in videos. IEEE transactions on pattern analysis and machine intelligence, 41(11):2740-2755 2018.
Liyuan Wang. Jingyi Xie, Xingxing Zhang, Mingyi Huang, Hang Su, and Jun Zhu. Hicrarchical decomposition of prompt-based continual learning: Rethinking obscured sub-optimallty. Advances in Neural Information Processing Systems, 36, 2024.
Yabin Wang, Zhiwu Huang, and Xiaopeng Hong. S-prompts learning with pre-trained transformers: occam's 's razor for domain incremental learning. Advances in Neural Information Processing Systems, 35:5682-5695, 2022a.
Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen- Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-fre free continual learning. In European Conference on Computer Vision, pp. 631-648. Springer, 2022b.
Zifeng Wang, Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 139-149, 2022c.
Xiang Xiang, Yuwen Tan, Qian Wan, Jing Ma, Alan Yuille, and Gregory D Hager. Coarse-to- fine incremental few-shot learning. In European Conference on Computer Vision, pp. 205-222. Springer, 2022.
Zhipeng Xu, De Cheng, Xinyang Jiang, Nannan Wang, Dongsheng Li, and Xinbo Gao. Adversarial domain prompt tuning and generation for single domain generalization. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18584-18595, 2025.
Jiazuo Yu, Yunzhi Zhuge, Lu Zhang, Ping Hu, Dong Wang, Huchuan Lu, and You He. Boosting continual learning of vision-language models via mfxture-of-exppeeee adapters. In Proceedings of the IEEF/CVF Conference on Computer Vision and Patern Recognition, pp. 23219-23230, 2024.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International conference on machine learning, pp. 3987-3995. PMLR, 2017.
Dingwen Zhang, Yan Li, De Cheng, Nannan Wang. and Junwei Han. Center-sensitive kernel optimization for efficient on-device incremental learning. arXiv preprint arXiv:2406.088300 2024.
14

---

Published as a conference paper at ICLR 2026
Gengwei Zhang, Liyuan Wang, Guoliang Kang, Ling Chen, and Yunchao Wei. Slea: Slow learner with classifier alignment for continual learning on a pre- rreitraied model. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp.
2023.
Da-Wei Zhou, Zi-Wen Cai, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu, Revisiting class-incremental learning with pre-trained models: Generalizability and adaptivity are all you need. International Journal of Computer Vision, pp. 1-21, 2024a.
Da- Wei Zhou, Hai-Long Sun, Han-Jia Ye, and De- De-Chuan Zhan. Expandable subspace ensemble for pre-trained model-based class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23554-23564, 2024b.
Qinhao Zhou, Xiang Xiang, and Jing Ma. Hierarchical fask-incremental learning with feature-space initialization inspired by neural collapse. Neural Processing Letters, pp. 1-17, 2023.
Xiaohan Zou, Wenchao Ma, and Shu Zhao. Learning conditional space-time prompt distributions for video class-incremental learning. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 4862 4873, 2025.
15

---

Published as a conference paper at ICLR 2026
A ALGORITHM
Algorithm 1: Frame-Shared Semantics Distillation (FSSD)
Input: Current task data Dy; frozen model from task 1; text features {Tc}
Output: FSSD loss LESSD for each class с E 3 do Compute mean spatial features:
Vi.- ,, N; EM Viaas 11. = N, EN Vi
// Current model
// PPrevious model
for each channel j = to d do Estimate Me,ji o2. from V - Compute semantic sensitivity:
// /cross frames
I(He.)) o2.
// Fisher Information
Compute classification contribution: E[re,]] Te.:
// Cosine-aligned score
Compute importance score:
19-1.. Te.jHeeei
// Weighted relevance
0,5
Compute weighted distillation loss:
CrSSD 120
[(ss Σi Egee I
Ib-1, 1 VV
return LPSSD
A.1 ALGORITHM OF TD-MoE
Algorithm 2: Temporal Decomposition based Mixture-of- Inference
Input: Video frames (xx] "
1=
Task-specific experts (9x K
1=
Temporal anchors {Vrem c=1 for current task
Output: Final representation V
Compute spatial mean: V N7 ΣM x
// Mean of frame features
for each expert A: = 1 to K do Concatenate CLS token and frame features V = :xx;
// Spatiotemporal feature
Compute temporal representation:
Viem V V
// Temporal decomposition
Compute routing score:
Tk = max.ey, cos (Vi.m, Viem): // Similarity to temporal anchors
Compute final representation:
V V + Σ Fk Val
// Residual fusion
return
Frame-Shared Semantics Distillation (FSSD). Algorithm [ mitigates forgetting by selectively preserving spatial feature channels that are semantically important and temporally stable across frames. Importance is computed per channel using two criteria: (1) Semuntic sensitivity, quantified by Fisher Information, and (2) Classification contribution, measured by cosine similarity with text features. These weights are used in weighted distillation loss between the frozen previous model and the current task model, enabling exemplar- free knowledge retention while allowing plasticity.
(TD-MoE). Algorithm routes video inputs to task-specific experts based on temporal relevance. Each expert encodes spatiotemporal features from video frames, from which temporal dynamics are isolated via residual decomposition.
Temporal Decomposition-based
16

---

Published as a conference paper at ICLR 2026
Temporal features are then compared to precomputed class anchors to compute routing scores. Final representations are generated by combining the expert outputs with the spatial feature via residual fusion. This enables dynamic, task ID-annostic inference driven by temporal structure.
B THEORETICAL SUPPLEMENT
B. FRAME- SHARED SEMANTICS
FRAME-SHARED
Semantic Sensitivity. The Fisher Information with respect to the mean parameter μ4 is defined as:
[,(Pes) = Byz, [
a
27
log p V
(16)
where p(V25 Me.)) ) is the probability density function of the Gaussian:
(Vé, Me.s)
p(V (o..)
exp
(17)
/2T022
20
Taking the derivative of the log- likelihood with respect to pi:
(Vo Mag )2
log pVVVVii - log log(2To²
(18)
20
Vo Mc.j
log log (V. (eji
(19)
Öpe.
Then, the Fisher Information becomes:
I(Pe.) = E
V cJj
- Hod
27
B[Ves =
E
0
(20)
0
Thus, we obtain:
1
I(He.) =
(21)
Classification Contribution. As cosine distance as classification basis, the c-th classification decision score ye.j of the j-th channel is modeled as:
V
Tes Tal
Taj
%o.
(22)
I
|V&l|
Therefore, the score function can be approximately written as:
Tej
Tei
Ne.
Σ
VG
ajj
æj
(23)
[
X
For the expected score Bl it can be calculated as:
E|yas x E
Ee Vä Toa
xx |l
Σ aj
us
Tej Maj
X
(24)
Then, the joint measure of informativeness is:
Te Pej X 02
Ta X Ojj Paj I(pas)
(25)
This expression provides a theoretically principled and interpretable metric for frame-shared seman- ties. It reflects the intuition that informative channel should (i) be strongly activated on average
17

---

Published as a conference paper at ICLR 2026
(Pe.j large), and (ii) exhibit consistent activation patterns samples (o small). Therefore, define the importance channel j as:
Te,j Pa.j. dai
Lo.j
(26)
This formulation also aligns with the the signal-to-noise ratio theory (SNR), providing unified theoretical justification.
C ADDITIONAL EXPERIMENTS
C.I ANALYSIS HYPER- EYPER-PARAMETER.
Table 7: Analysis of hyperparameter w on HMDB51 and UCFI01 datasets. Best results are in bold.
Dataset
Metric
1 10
1x x 10a
2.5 x 104
5 x 10*
1 10a
Acc 1
59.51
63.04
62.10
62.29
63.05
HMDB(5 x 58)
BWF
16.90
11.04
11.14
12.32
12.77
Acc 1
85.10
85.79
84.54
84.88
84.16
UCF101(5 x 10s)
BWF 4
7.93
5.63
6.95
5.67
6.83
Table. 7 analyzes the sensitivity the hyper- parameter w, which controls the strength of FSSD regularization. Across wide range of values, our framework maintains stable performance, indicating robustness to hyper-parameter variations. Moderate w values achieve the best trade-off between knowledge retention and adaptability, avoiding regul-reguliiization or excessive constraint on model plasticity.
C.2 COMPARISON OF MoE ROUTING STRATEGIES (TASK BY TASK)
CUPMEE
Adaptal
10
UC1101 15 106)
UC110L (10 361
HMDD33 (3 x
755
3.0
-30
Tadk1D
Task D
Teal
Figure 5: MoE Methods Performance per Task on UCF101 and HMDB51
Figure presents the relative accuracy change (%) different Mixture-of-Eaaee Experts (MoE) routing strategies three benchmarks: UCF101 with two task configurations (5 x 10s and 10x5s), and HMDB51 (5x The horizontal axis represents the incremental task ID, while the vertical axis shows the accuracy change relative to the Avg-MoB baseline.
We observe that TD-MoE consistently outperforms all baselines across datasets and task gran- ularities. Its performance advantage becomes more pronounced as the number of tasks increases, reaching up to 6-8% improvement on later tasks. contrast, Adapter-MoE and CLIP-MoE exhibit only marginal gains, which tend to saturate early, suggesting limited ability model task-specific dy- namics. These findings confirm that temporal dccomposition is effective for guiding expert selection in task ID-agnostic manner, and helps mitigate forgetting by capturing relevant spatiotemporal
As shown in 6 temporal decomposition based router is effective at task boundary decisions.
As shown in Figure 4 maintains highest accuracy time and effectively mitigates forgetting, demonstrating the complementary strengths of spatial semantic preservation and temporal dynamic modeling.
18

---

Published as a conference paper at ICLR 2026
Fandomly select
Randamy aslect
label per
Sek-2
Tandomly select
3:55
0.54
832
0.33
0.36
0.39
320
0.3
647
0.52
2.37
660
1:44
0.37
0.53
0.40
0.30
Aspacimann
Sciggg
935
0.22
0.67
0.25
6.2
6:5
0.44
336
Metue
0.33
2440
659
2.0
0.53
3.3
6:00
6.22
3.40
644
6.9
0.79
3.2
643
0.1
3.54
0.32
122
335
342
3.68
128
Figure 6: Heatmap of task selection by temporal decomposition- based router the UCF101.
To investigate the performance TD-MoE on temporally challenging datasets, we added an ablation study on SSv2 with and without the TD-MoE module(Table [8. Since SSv2 is highly sensitive to temporal cues, TD-MoE brings a clear performance improvement, which aligns with our design motivation. Without TD-MoE, the model relies Because the dataset itself' is difficult to train and each new task initializes the MSA module with the parameters from the previous task, the model's ability to learn new tasks is inherently weakened. This leads to poor performance on new tasks; thus, as more tasks are added, forgetting does not further deteriorate simply because the initial performance already low. Nevertheless, the shared deep MSA module still experiences substantial forgetting. In contrast, with TD-MoE, the model learns temporal patterns more effectively, achieving higher accuracy on new tasks while not exhibiting noticeable increase in forgetting as the number of tasks grows.
shared MSA module in the deeper layers.
Table 8: SSv2 Performance Across Task Numbers (w/ vs. w/o ID-MoE), Best results are in bold.
Task Numbers
TD-MoE
Acc +
BWF J
10s
X
17.64
56.32
10s
/
40.79
18.86
20s
X
18.39
49.49
20s
/
37.30
19.44
C.3
EFFECTIVENESS FRAME-SHARED SEMANTICS DISTILLATION
w/o Distilation
Distilation
iSSO
UCF101 10s)
HMDB51 5s)
Relltive Inrrovameenttoo
2-
7.5
2.5
2.5
10
Thik D
Thick
Figure 7: Distillation Methods Performance per Task on UCF101 and HMDB51
Figure 7 compares the impact of different distillation strategies, including w/o Distillation, uni- fied Distillation, and our proposed Frame-Share Semantics Distillation (FSSD), on UCF101 and HMDB51. The vertical axis reports the relative improvement over the non-distillation baseline.
The results demonstrate that FSSD delivers the most consistent and significant improvements, particularly on HMDB51. While unified distillation offers slight improvements, it lacks consistency across tasks. This suggests that uniform constraints fail to address the heterogeneous semantic importance of feature channels. Moreover, the increasing gap between FSSD and other methods over time confirms that adaptive regularization based on frame-shared semantic importance is critical for preserving relevant knowledge across tasks in VCII.
19

---

Published as a conference paper at ICLR 2026
To compare with gradient-based regularization methods and clarify the difference between approach and EWC Kirkpatrick et al. (2017), we provide the following explanation and comparison experiments. Gradient-basse based methods such EWC rely parameter-level regularization. We reproduced EWC and compared it with FSSD (Table 99. On both UCF101 and HMDB51, our imprattii-mmotttnceeeeeeeee based approach consistently outperforms EWC. This is because EWC approxi- mates the Fisher information using only diagonal elements, ignoring cross-parameter correlations and assuming model stability during estimation, which introduces approximation error. Moreover, EWC does not accumulate importance across tasks; it assumes that the model after tine-tuning already significant, preserves previous knowledge. As the number of tasks grows, feature drift becomes more causing the Fisher-based estimate to become increasingly inaccurate. In contrast, FSSD computes importance via an analytic, semanrc- formulation and accumulates knowledge over sessions, allowing it to better preserve early-task information and achieve superior performance. In contrast, FSSD computes importance analytically via feature statistics (mean and variance) aligned with the textual representations, requiring no backpropagation and thus incurring negligible computational cost. Furthermore, the importance vector in FSSD has the same dimensionality as the output features, resulting in minimal memory footprint. EWC, however, must compute gradients for all samples and store importance values for all trainable parameters, leading to significantly higher computational and memory overhead.
Table 9: Performance of EWC and our method on UCF101 and HMDB51.
Methods
Acc f
Acc f
UCF101 (10x5s)
EWC
Kirkpatrick et al.
2017
93.49
86.37
UCF101 (10x5s)
FSSD
94.67
88.85
UCF101 (5x10s)
EWC
Kirkpatrick et al.
2017
90.00
80.29
UCF101 (5x10s)
FSSD
92.13
85.79
HMDB51 (Ix25s)
EWC Kirkpatrick al.
2017
66.49
55.60
HMDB51 (Ix25s)
FSSD
67.01
57.89
C.4 PARAMETER COUNT ANALYSIS
We compute the parameter count each module: the CLIP ViT-B/16 backbone, the inserted adapters, and the spatiotemporal encoder module.
Table 10: Parameter counts for each module per 8- 8-frame video.
Module
Parameters
CLIP (VIT-B/16) (frozen)
86M
Adapter
1.17M (1.36%)
Spatiotemporal Encoder
9.45M (10.99%)
Table 10 provides detailed breakdown of parameter count for each component in framework, evaluated on 8-frame video inputs. The backbone CLIP (ViT-B/16) dominates the overall cost with 86M parameters. The inserted adapter modules, despite being integrated into every transformer layer, introduce only 1.17M additional parameters (1.36%), demonstrating their lightweight nature. Fur- thermore, our spatiotemporal encoder used capture dynamic information, adds 9.45M parameters (10.99%).
Table 11: Performance of TD-MoE with Different Frame Counts UCF101 (10x5s).
Frames
Acc 1
Acc 1
BWF 4
2
91.29
83.75
8.91
92.72
85.29
8.60
8
94.67
88,85
6.31
16
94.50
89.54
5.58
32
93.39
89.48
4.72
20

---

Published as a conference paper at ICLR 2026
To analyze TD-MoE's ability to handle different degrees of temporal information, conducted additional experiments on UCF101 with input clips containing sparse (2, frames), adequate (8, 16 frames). and redundant (32 frames) temporal cues. The results shown in Table [T We observe that performance increases as more temporal information becomes available, peaking at 8-16 frames. Notably, when the frame count is doubled 32, the accuracy remains stable, indicating that TD-MoE can effectively exploit longer temporal sequences without being negatively affected by redundancy. These results confirm that TD-MoE maintains strong long-term temporal modeling capability.
Table 12: Performance of different layers on UCF101 (10x5s).
Acc f
Acc
Parameters 4
CLIP (backbone)
74.67
70.65
149M
layer
91.29
83.75
3M (2.01%)
2 layers
92.72
85.29
6M (4.02%)
3 layers
94.67
88.85
9M (6.03%)
Table 13: Performance of selecting latest 5 experts on UCF101.
Select latest
Acc Acc f
Linear Growth
UCF101 (5x10s)
88.77
74.97
UCF101 (5x10s)
X
92.13
85.79
UCF101 (2x25s)
82.15
71.10
UCF101 (2x25s)
X
88.52
82.67
We conducted preliminary studies on expert design. As shown in Table 12 reducing expert depth lowers parameters and slightly affects temporal modeling, but even single-layer expert with TD-MoE still achieves oftte-of-hhe-at performance (Ace 84.75). We further evaluate efficient design: we tested simple selective-expert strategy by keeping only latest five experts Table [3 Because the inputs of all experts and temporal anchors rely on shared spatial features, newly trained experts naturally retain part of the old knowledge, Under this mechanism, the 2x25 configuration falls slightly short of the current state-of-the-art (Acc = 85.67), while the 5x10 setting still exceeds it (Acc 84.75). This demonstrates that selective expert retention is feasible. In future work, we plan to further explore more principled expert-selection mechanisms and investigate how to transfer knowledge across experts reduce the performance gap when using fixed number of experts.
D LLM USAGE STATEMENT
In accordance with the ICLR 2026 policy large language models (LLMs), clarify that LLMs were employed solely to assist in polishing the language and improving readability of the manuscript. The conception of the research problem, development of the methodology, algorithmic design, code implementation, experimental setup, and result analysis were entirely carried out by the authors without reliance LLMs.
21


Reviews:
[{"review_id": "AOtHD8AStB", "reviewer": "Reviewer_vccK", "summary": "The paper addresses the challenges of catastrophic forgetting and spatiotemporal modeling in Video Class-Incremental Learning (VCIL) by proposing an exemplar-free framework named StPR. Its contributions are twofold: 1）It proposes a Frame-Shared Semantics Distillation method (FSSD) that preserves frame-shared, semantically aligned spatial channels through semantic importance-aware regularization, optimizing the stability-plasticity trade-off in continual learning; 2）It designs a Temporal Decomposition based Mixture-of-Experts strategy (TD-MoE) that decomposes spatiotemporal features and uses temporal dynamics for expert combination, enabling task-id-free and dynamic adaptation.", "questions": "Seen above.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1) The paper proposes an exemplar-free setting, which avoids the memory and privacy issues associated with storing historical data, making it more practical for real-world applications.\n2) The theoretical analysis is substantial, providing derivations for the channel importance metrics (Fisher Information, classification contribution) used in FSSD.\n3) The experimental analysis is thorough, validating the method's effectiveness across multiple datasets and various task partitions, and includes comprehensive ablation studies and analysis.", "weaknesses": "1) Line 70: \"reuses these decomposed components to enhance the model’s ability to adapt continually, thereby reducing forgetting without storing extensive exemplars.\" Is this statement problematic, as the subsequent text does not mention reusing these decomposed components? Furthermore, the claim that decomposed components help enhance adaptability is also debatable.\n\n2) Why are channels with high semantic importance preserved? The semantic importance from a previous session does not necessarily guarantee importance in subsequent sessions.\n\n3) If the mixture-of-experts system can effectively address catastrophic forgetting and adaptive learning, why is the distillation module necessary? Moreover, mixture-of-experts has already been used in class-incremental learning[1], and no significant distinction from your method is evident. \n[1] Yu J, Zhuge Y, Zhang L, et al. Boosting continual learning of vision-language models via mixture-of-experts adapters[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 23219-23230.\n\n4) On the Kinetics-400 dataset, there is a lack of comparison with results from the latest top-tier conference papers.\n\n5) It is recommended to supplement experiments with longer video sequences (e.g., 16 or 32 frames) or higher frame rate inputs to analyze TD-MoE's capability in long-term temporal modeling and its impact on computational cost.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vw0XNoSVAi", "reviewer": "Reviewer_VZo2", "summary": "This paper proposes StPR (Spatiotemporal Preservation and Routing), a unified, exemplar-free framework for Video Class-Incremental Learning (VCIL). The key challenge in VCIL is mitigating catastrophic forgetting while preserving both frame-shared spatial semantics and temporal dynamics across incrementally arriving action classes—without storing past video exemplars due to memory/privacy constraints.\n\nTo address this, the authors introduce two core components:\n\nFrame-Shared Semantics Distillation (FSSD): A channel-wise importance-aware distillation mechanism that preserves semantically stable and classification-relevant spatial features by combining semantic sensitivity (via Fisher Information) and classification contribution (via cosine alignment with CLIP text embeddings).\n\nTemporal Decomposition-based Mixture-of-Experts (TD-MoE): A task-agnostic routing mechanism that decomposes spatiotemporal features into static (mean) and dynamic (residual) components. At inference, it routes inputs to task-specific temporal experts based solely on temporal dynamics, eliminating the need for task IDs or stored exemplars.\n\nThe framework is built on a frozen CLIP ViT backbone with lightweight adapters and a small spatiotemporal encoder per task. Experiments on UCF101, HMDB51, SSv2, and Kinetics400 under standard VCIL benchmarks (TCD and vCLIMB) show state-of-the-art performance, outperforming both exemplar-based and exemplar-free baselines. Ablation studies and analyses support the design choices.", "questions": "Scalability: How does StPR scale to 50+ incremental tasks? Is there a plan to compress or merge experts (e.g., via expert pruning or clustering) to avoid linear parameter growth?\n\nAnchor Storage: Are the temporal anchors  stored permanently after training each task? If yes, how much memory do they consume? Does this violate the “exemplar-free” claim, even if minimally?\n\nBackbone Generality: Have you tested StPR with non-CLIP backbones (e.g., VideoSwin, TimeSformer)? Is the performance gain primarily due to CLIP’s strong priors or the StPR mechanism itself?\n\nTask Boundaries: TD-MoE assumes known task boundaries during training (to assign experts). How would StPR handle task-agnostic or online VCIL where task boundaries are unknown?\n\nComparison to Prompt-Based Methods: ST-Prompt† (Pei et al., 2023) also uses spatiotemporal prompting. How does TD-MoE fundamentally differ? Is the gain due to routing or the disentanglement design?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Strengths\n\nOriginality: The disentanglement of frame-shared semantics and temporal dynamics for VCIL is novel. While MoE and distillation exist in CIL, their adaptation to video via temporal decomposition and semantic importance weighting is creative and domain-aware.\n\nQuality: Rigorous experiments with strong baselines, multiple datasets, and thorough ablations. The use of CLIP aligns with modern vision-language trends.\n\nClarity: The framework is easy to follow (Figure 2), and algorithms are provided (Appendix A). The writing is concise and technical depth is balanced well.\n\nSignificance: Addresses a critical gap in VCIL—exemplar-free learning with temporal modeling. The method is efficient (low FLOPs), scalable, and applicable to real-world scenarios like edge devices (mentioned in conclusion).", "weaknesses": "Task-Specific Expert Scaling: TD-MoE allocates one spatiotemporal encoder per task. While efficient per expert (~9M params), this leads to linear growth in parameters with the number of tasks (e.g., 10 tasks → ~90M extra params). This may limit scalability in long-task sequences. The paper does not discuss parameter efficiency in the long run or compare total model size vs. baselines like L2P or ST-Prompt.\n\nDependence on CLIP: The method relies heavily on frozen CLIP features. While common, this raises questions about generalizability to non-pretrained or non-ViT backbones. A brief experiment with a non-CLIP backbone (e.g., VideoMAE, TimeSformer) would strengthen claims about framework generality.\n\nTemporal Anchor Construction: During inference, TD-MoE uses class-wise mean temporal features (“anchors”) from the current task. This assumes access to current-task data during inference to build anchors. Clarification is needed: are anchors computed once after training and stored? If so, does this constitute a form of “exemplar” (albeit aggregated)? The paper claims “exemplar-free,” but storing per-class statistics could be seen as a lightweight memory buffer.\n\nKinetics-400 Results: On Kinetics-400 (Table 3), StPR achieves higher final accuracy but also higher BWF than some exemplar-based methods (e.g., CSTA). This trade-off should be discussed—does higher accuracy come at the cost of more forgetting on earlier tasks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FB5msD7iuU", "reviewer": "Reviewer_SftT", "summary": "This paper introduces Spatiotemporal Preservation and Routing (StPR), a novel mechanism designed to tackle the challenging problem of Video Class-Incremental Learning (VCIL) in an exemplar-free (memory-free) setting. The proposed method explicitly disentangles and preserves critical spatial and temporal representations, effectively mitigating catastrophic forgetting while maintaining adaptability. StPR consists of two key components: Frame-Shared Semantics Distillation (FSSD) and the Temporal Decomposition-based Mixture of Experts (TD-MoE). FSSD identifies and preserves semantically stable and meaningful spatial channels by jointly considering channel-wise sensitivity and classification contribution. TD-MoE dynamically routes task-specific spatiotemporal experts based on decomposed temporal dynamics, enabling adaptive learning without task identifiers or stored exemplars. Experimental results demonstrate that this memory-free approach achieves superior performance compared to rehearsal-based baselines such as BiC and iCaRL.", "questions": "1. How does memory and compute cost scale with the number of tasks? Do you freeze old experts or allow fine-tuning?\n2. How does FSSD compare to gradient-based importance as a measure of stability? \n3. Can you discuss the different between FSSD and the pricipales of EWC, which also use the Fisher information?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper introduces an effective rehearsal free method that can obtain state-of-the-art performance in different benchmarks like vCLIMB and TCD, outperforming both exemplar-based (e.g., FrameMaker, HCE) and exemplar-free (e.g., STSP, ST-Prompt) methods.\n2. Temporal Decomposition-based Mixture-of-Experts (TD-MoE) provides a routing based on temporal residuals and attention-weighted deviations which allows the inference without task IDs.\n3. They provide a well-designed distillation loss FSSD that leverages the Fisher information and classification contribution to compute per-channel relevance which help preserving model stability-plasticity. \n4. The paper provides extensive evaluations across multiple benchmarks and metrics, along with a thorough ablation study isolating the contributions of each component.", "weaknesses": "1. The proposed TD-MoE instantiates a separate expert for each new task. Consequently, both memory and computation costs are expected to scale linearly with the number of tasks. The paper does not provide an explicit analysis or discussion on this scalability issue, nor on potential mitigation strategies.\n2. It would be valuable to understand the limitations of the TD-MoE in temporal challenging datasets like SSv2. For instances, analizing the forgetting of the model with and without the TD-MoE module when the number of tasks increase.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AjbpgG3ua7", "reviewer": "Reviewer_s53Z", "summary": "The topic of this paper is about video class-incremental learning(VCIL). The authors propose an exemplar-free VCIL framework with the spatio-temporal Preservation Routing(STPR) mechanism. It consists of a frame-shared semantics distillation(FSSD) and a Temporal Decomposition-based Mixture-of-Experts(TD-MoE) strategy. The STPR framework has been evaluated on several datasets.", "questions": "My major concerns are inclued in the above weaknesses.\n\nQ1. Different parameter-efficient finetuning(PEFT) strategies(e.g., prompt, adapter, MoE) have been utilized in VCIL scenarios. How to select appropriate PEFT strategy in different practical VCIL scenarios?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "+ The idea of exploring temporal-based MoE in VCIL is interesting.\n+ The performance of the proposed method StPR outperfoms that of other methods by a margin on several datasets.", "weaknesses": "- Concerns about the FSSD module. The proposed FSSD strategy is a plug-in method can be integrated to other VCIL methods. To show the effectivess of FSSD, can it be evaluated on different VCIL baselines? The authors have conducted ablation studies in Table 4 and the performance with only employing the FSSD method is needed.\n\n- The presentation can be improved. For example, there are some minor spelling mistakes. “suppl.”->supplemental. The formulation “$A^b$” is not presented in Figure 2 of the overview framework.\n\n- Lack of some important comparisons. The idea of decoupling spatiotemporal knowledge in VCIL has been discussed in previous methods (e.g., [a][b] ). Compare to these works and make a discussion in the related work. Besides, some works published in 2025(e.g., [b]-[c]) can be compared and discussed. For instance, the proposed method can utilize the complex large-scale video continual learning benchamrks [c] for evaluation.\n\n[a] When Video Classification Meets Incremental Classes, MM2021\n\n[b] Learning Conditional Space-Time Prompt Distributions for Video Class-Incremental Learning, CVPR 2025\n\n[c] CRAM:Large-scale Video Continual Learning with Bootstrapped Compression, ICCV 2025", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)