TS-ILM:Class Incremental Learning for Online Action Detection

Authors: jian  cheng
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 08:25:08

Abstract: Online action detection aims to identify ongoing actions within untrimmed video streams, with extensive applications in real-life scenarios. However, in practical applications, video frames are received sequentially over time and new action categories continually emerge, giving rise to the challenge of catastrophic forgetting - a problem that remains inadequately explored. Generally, in the field of video understanding, researchers address catastrophic forgetting through class-incremental learning. Nevertheless, online action detection is based solely on historical observations, thus demanding higher temporal modeling capabilities for class-incremental learning methods. In this paper, we conceptualize this task as Class-Incremental Online Action Detection (CIOAD) and propose a novel framework, TS-ILM, to address it. Specifically, TS-ILM consists of two components: task-level temporal pattern extractor and temporal-sensitive exemplar selector. The former extracts the temporal patterns of actions in different tasks and saves them, allowing the data to be comprehensively observed on a temporal level before it is input into the backbone. The latter selects a set of frames with the highest causal relevance and minimum information redundancy for subsequent replay, enabling the model to learn the temporal information of previous tasks more effectively. We benchmark our approach against SoTA class-incremental learning methods applied in the image and video domains on THUMOS'14 and TVSeries datasets. Our method outperforms the previous approaches.

Main paper: 
 introduction: 
 Online action detection (OAD) aims to identify ongoing actions in video streams without foreknowledge of the future. This task holds significant sway across various real-life applications, including but not limited to autonomous driving [22, 24] , video surveillance [1, 20] , and anomaly detection [34, 36] . Researchers in related fields have also devised numerous efficacious solutions to tackle this challenge [8, 9, 13, 14, 28] .However, video frames are received sequentially over time in most real-world scenarios, and new action categories continually emerge. Constrained by memory and privacy considerations, the model can only access the current data, with previously observed classes being unavailable or partially accessible. Under these circumstances, a straightforward resolution is to fine-tune the mode sequentially, enabling the model to assimilate knowledge from the continuously arriving new data and categories. Nevertheless, this approach may result in the model overfitting to the current categories, leading to a marked decline in the recognition capabilities for previously learned categories (a phenomenon known as catastrophic forgetting [32] ). This adverse characteristic has catalyzed extensive research within the domain of continual learning [2, 26, 44, 56] . Our focus lies on a specific variant of continual learning, Class-Incremental Learning (CIL), wherein labels across tasks are mutually exclusive, and the model lacks access to task IDs during inference. Prior works have delved into the CIL paradigm within the context of video domains [4, 37, 39, 40, 51, 52, 61] , with these studies predominantly focused on scenarios where the model has the capacity to observe the entire video at any given moment. In contrast, the challenge of OAD rests on the reliance solely on historical observations without the possibility of accessing future video frames. This restriction places a more significant demand on the model's capacity for long-duration modeling and causal reasoning and leads to an exacerbated occurrence of catastrophic forgetting. This paper first presents this problem and designates it as Class-Incremental Online Action Detection (CIOAD) (as shown in Figure 1 ).Within the setup of CIL, previous methodologies have primarily tackled the issue of forgetting through two strategies [7, 46, 57] : (1) preserving old knowledge whilst acquiring new information, which typically involves limited expansion of the network architecture, and (2) employing a restrained memory budget to select and store representative samples from old classes for rehearsal. However, prior approaches tend to disrupt temporal relations to a certain extent by storing additional frames, which mitigates their effectiveness in CIOAD (as illustrated in Figure 2 ). To address these issues, we introduce a Time-sensitive Incremental Learning Method (TS-ILM). This method is comprised of two components: Task-Level Temporal Pattern Extractor (TPE) and Temporal-Sensitive Exemplar Selector (TES).First, given that each action has a different temporal pattern, the backbone network might only concentrate on the temporal patterns of the action class of the current task during training, inadvertently overlooking those from past tasks. This could result in the network developing a temporal bias, leading to a decline in its performance on previous tasks. In response to this challenge, we propose the TPE, engineered to extract and preserve the temporal patterns of actions across different tasks, thereby facilitating an exhaustive temporal assessment of data prior to its entry into the backbone network.Second, video streams are composed of a sequence of background and action frames, among which there exists a simultaneous existence of causality and information redundancy. The selection of exemplary representative samples is paramount [6, 23] . To address this, we propose TES to select and preserve a set of frames that maximize causal relationships while minimizing information redundancy on the temporal level, allowing the network to more effectively learn temporal information from previous category samples during exemplar replays. To obtain a more compelling evaluation, we apply state-of-theart CIL methodologies from the image and video domains to this task, establishing a baseline for the CIOAD challenge and contrasting our approach with them. We adapted the THUMOS'14 and TVSeries datasets-both frequently employed for OAD task evaluations-to align with the CIL settings. Subsequent thorough evaluations across various configurations on these adapted datasets demonstrate the effectiveness of the proposed framework.The contributions of this paper can be summarized as follows:• We observe that models designed for OAD experience catastrophic forgetting seriously when actually deployed. Furthermore, due to the task's need for strong temporal modeling, CIL methods applied in video domains fail to address the problem adequately. We are the first to propose this issue and name it Class-Incremental Online Action Detection. • We propose a novel method named TS-ILM to tackle the new task. Specifically, it comprises two critical components: TPE and TES. The former extracts and preserves temporal patterns of actions corresponding to different tasks, while the latter selects a set of frames with maximal causal relevance and minimal information redundancy for subsequent replay. • We benchmark our approach against SOTA CIL approaches applied in the image and video domains on two widely used OAD datasets that comply with the CIL settings. Our method significantly outperforms the previous approaches. 
related work: 
 In this section, we present methods related to CIOAD, which cover image class-incremental learning, video class-incremental learning, and online action detection.The problem of class-incremental learning has received widespread research attention in recent years and has been extensively applied within the image domain [10, 31, 35] . For image classification, current methods can be broadly divided into three categories: parameter isolation methods, regularization methods, and rehearsal methods. Parameter isolation methods alleviate catastrophic forgetting by segregating the parameter space of the neural network.Some of these approaches dynamically change the network architecture to adapt to new information [3, 12, 45] , while others implement parameter isolation by masking individual parameters or layers [29, 30] . Rehearsal methods typically involve the preservation of a representative subset of original training data [41, 44, 54, 56] or the use of generative models to emulate the training data of previous tasks [33, 47, 60] . These examples are subsequently used as supplementary data to guide the model while training new tasks, thus mitigating catastrophic forgetting to some extent. The methods based on regularization attempt to protect old knowledge from being overwritten by new knowledge by penalizing drastic changes in the weights related to previous tasks. Among these methods, some measure the importance of each parameter in the network and regularize the weights [2, 26, 27] , while others focus on using regularization terms to avoid forgetting the feature representations of previous tasks [43, 49] .Recent works have started to explore the problem of class-incremental learning within the video domain. [61] mitigates catastrophic forgetting by decomposing and transferring spatiotemporal knowledge. TCD [37] identifies the most contributory channel subsets within feature maps through an importance mask. vCLIMB [52] introduces a temporal consistency regularization to diminish the influence of subsampled instances on the model. FrameMaker [39] compresses videos into a single frame to conserve memory. SMILE [4] posits that in class-incremental learning tasks for action recognition, storing a single frame per video to ensure sample diversity is more efficacious than retaining the temporal information of whole videos. However, within CIOAD tasks, we are required to perform frame-by-frame recognition of actions in untrimmed videos without prior knowledge of future information, which demands greater temporal precision compared to the recognition of actions in pretrimmed videos. TS-ILM enhances performance by extracting and saving temporal modalities of actions from various tasks, as well as by choosing and saving downsampled videos that demonstrate the strongest causal relationships and the least information redundancy on a temporal level.Distinct from action recognition tasks with the luxury of observing entire videos, the objective of OAD tasks is to identify the class of the current frame without foresight. Hence, researchers concentrate on employing robust temporal modeling to address this conundrum. TRN [58] employs the latent states of LSTM [15] to model the historical and contextual relationships of videos in order to predict future actions and detect current actions via the temporal correlation between future and current actions. OadTR [55] leverages the long sequence modeling capability of Transformers [50] to capture historical data while concurrently modeling extensive global temporal information, thereby forecasting future contexts to identify the present action. LSTR [59] has instituted a mechanism for long-term and short-term memory for protracted sequence data, enabling the analysis of more historical video content. HCM [28] identifies the suitable division between action and background clips through deep metric learning [21] , thus distinguishing action frames from the background ones better. Although researchers have proposed many solutions for the OAD task, how to address the catastrophic forgetting issue that arises in practical applications due to the continuous influx of data has not yet been actively explored. 
method: 
 In this section, we first illustrate the formulation of CIOAD (Sec. 3.1), and then outline the framework of TS-ILM in Sec. 3.2. Following this, we propose the Task-Level Temporal Pattern Extractor (Sec. 3.3) to ensure a comprehensive temporal-level observation of data before it is input into the backbone network. Concurrently, we introduce the Temporal-Sensitive Exemplar Selector (Sec. 3.4) to select a set of frames that have the most significant temporal causality and minimal information redundancy for example replay. Finally, in (Sec. 3.5), we thoroughly detail the specifics of using TS-ILM for class incremental training.Similar to vCLIMB benchmark [52] , CIOAD requires the training of a neural network 𝑓 Θ : 𝑋 → 𝑌 , which is moderated by the parameter Θ. The network's objective is to discern the class 𝑦 0 of the current frame 𝑥 0 within video stream 𝑥 = [𝑥 𝑡 ] 0 𝑡 =-𝑇 +1 , where x 𝑖 ∈ 𝑋 and y 𝑖 ∈ 𝑌 ∀𝑖 ,in the case that future frames 𝑥 1 , 𝑥 2 , ..., are not accessible. The parameter Θ of the model is learned from a sequence of m tasks, collectively referred to as T 1 , T 2 , . . . , T 𝑘 , . . . , T 𝑚 , where each T 𝑘 is associated with a unique dataset corresponds to the specific action categories germane to the extant task and 𝑞 represents the number of categories in the task. It is worth emphasizing that in this arrangement, the action categories for each task do not overlap.𝐷 𝑘 = 𝑉 𝑘 1 , 𝑉 𝑘 2 , . . . ,Figure 3 illustrates the overall framework of our method. In incremental step 𝑘, given an input video 𝑉 𝑘 𝑖 ∈ 𝐷 𝑘 , following the convention [28, 55, 59] , we first use the temporal feature extractor to extract the motion information of video frames, which are then concatenated with the appearance information extracted by the spatial feature extractor to form the feature sequence 𝐹 𝑘 𝑖 ∈ R 𝑇 ×𝐷 . Then, we input the feature sequence into Task-Level Temporal Pattern Extractor(Sec. 3.3) to ensure it is thoroughly observed on a temporal level before entering the backbone network, avoiding temporal attention bias. Afterward, we input the output 𝑋 𝑘 𝑖 ∈ R 𝑇 ×𝐷 from the Task-Level Temporal Pattern Extractor into the backbone to identify the action of the current frame.After incremental step k, given a video 𝑉 𝑘 𝑗 ∈ 𝐷 𝑘 , we utilize the Temporal-Sensitive Exemplar Selector (Sec. 3.4) at a specific ratio to filter out a set of frames 𝑉 𝑘 ′′ 𝑗 to be stored in memory bank 𝑀 𝑘 . This set of frames maximizes causal relationships on the temporal level and minimizes information redundancy. Replaying exemplars from 𝑀 𝑘 in subsequent incremental steps allows the model to better learn the temporal information of previous tasks. To prevent catastrophic forgetting, we extract the unique temporal information of each task through the Task-Level Temporal Pattern Extractor (TPE) and preserve it for use during the training of subsequent tasks. This enables videos to be thoroughly observed on a temporal level before they are put into the backbone network. Formally, for each task T 𝑘 , we design a self-attention layer 𝐴𝑡𝑡𝑒𝑛𝑡𝑖𝑜𝑛 𝑘 (𝐹 𝑘 𝑖 ; 𝜑 𝑘 ) , which is a Standard Self-Attention layer [50] , regulated by a weight matrix 𝜑 𝑘 . This layer is deployed to identify which temporal segments should be paid more attention to within this task. Given the feature sequence 𝐹 𝑘 𝑖 extracted from 𝑉 𝑘 𝑖 ∈ 𝐷 𝑘 , the output after passing through the self-attention layer associated with the task 𝑘 can be elucidated as follows:𝐼 𝑖 𝑘 = 𝐴𝑡𝑡𝑒𝑛𝑡𝑖𝑜𝑛 𝑘 (𝐹 𝑖 𝑘 ; 𝜑 𝑘 ) ∈ R 𝑇 ×𝐷 (1)where 𝐼 𝑘 𝑖 represents the output of 𝑘-th self-attention layer, 𝐹 𝑘 𝑖 ∈ R 𝑇 ×𝐷 denotes the input feature sequence, 𝑇 is the length of the feature sequence, and 𝐷 corresponds to the dimension of each feature. After the training of T 𝑘 is completed, the parameters of 𝐴𝑡𝑡𝑒𝑛𝑡𝑖𝑜𝑛 𝑘 are frozen. Subsequently, a new self-attention layer with adjustable parameters is set to focus on the temporal information of the forthcoming task. In addition, we define a learnable weight for each self-attention layer to integrate the features output from each self-attention layer. This integration allows the fused feature to encompass the temporal modalities of various tasks comprehensively. The sequence of weights up to T 𝑘 is denoted by 𝑊 = {𝜔 1 , 𝜔 2 , . . . 𝜔 𝑘 }, and the integrated feature sequence can be calculated as follows:EQUATIONTo circumvent interference with previously preserved information by the temporal modalities of subsequent tasks during training, we introduce a regularization term designed to penalize changes in the weight parameters of self-attention layers from prior tasks:EQUATIONwhere 𝜔 𝑘 𝑗 represents the weight parameters of the self-attention layer for task 𝑗 at the incremental training step 𝑘, and || • || denotes ℓ 2 -normalization. Following the inspiration from [16] , we employ a method of residual summation to amalgamate the integrated features with the original features. This process enhances the representation capability of the features and improves training stability. Subsequently, we normalize to generate the final feature 𝑋 𝑘 𝑖 , which is then input into the backbone network. Generally, this process can be formulated as follows:EQUATIONWe can learn an effective feature representation that integrates the temporal modalities of past tasks and the current task. This allows a comprehensive observation of the video before it is input into the backbone network.In addition, untrimmed videos contain both background frames and action frames. There might be a dense overlap of information between successive frames of the same type, which leads to a significant amount of redundant information. If we store these frames simultaneously, it cannot make effective use of the replay memory. Meanwhile, there exists a strong causal relationship in transitions between adjacent frames of different types, such as from background to action or from action to the background, which also contains a substantial amount of information. Therefore, we utilize the Temporal-Sensitive Exemplar Selector (TES) to maximize the temporal causality between frames of stored videos while simultaneously minimizing their informational redundancy. This enables the network to better learn the temporal information of past category samples through a replay mechanism. Formally, Given a video 𝑉 𝑘 𝑗 ∈ 𝐷 𝑘 ∀𝑗 after incremental step 𝑘, we classify each frame into two collections based on their labels: action frames set 𝐴 𝑘 𝑗 and background frames set 𝐵 𝑘 𝑗 , where all frames in 𝐴 𝑘 𝑗 have non-zero labels, and all frames in 𝐵 𝑘 𝑗 have labels of zero. At a considerable ratio 𝛼, we select respective representative subsets of frames for 𝐴 𝑘 𝑗 and 𝐵 𝑘 𝑗 using the herding strategy [44] , and arrange them in their original temporal sequence𝑉 𝑘 ′ 𝑗 = 𝑣 𝑘 ′ 𝑗 1 , 𝑣 𝑘 ′ 𝑗 2 , . . . , 𝑣 𝑘 ′ 𝑗 (𝑎+𝑏) ×𝛼to form a temporally downsampled version of 𝑉 𝑘 𝑗 , where 𝑎 and 𝑏 represent the number of frames of 𝐴 𝑘 𝑗 and 𝐵 𝑘 𝑗 respectively. Afterwards, we will further refine the selection on 𝑉 𝑘 ′ 𝑗 to ensure that the ultimately selected frames have a strong temporal association. Specifically, we have defined the inter-class distance as:EQUATIONwhere 𝑣 𝑘 ′ 𝑗 𝑛 and 𝑣 𝑘 ′ 𝑗 𝑚 are of the same class, that is, both are either background frames or action frames, and ||•|| represents ℓ 2 -normalization. In addition, we define the intra-class distance as:EQUATION)where 𝑣 𝑘 ′ 𝑗 𝑛 and 𝑣 𝑘 ′ 𝑗 𝑚 are of different classes, meaning one frame is a background frame and the other one is an action frame, || • || represents ℓ 2 -normalization and 𝐹 𝑚𝑎𝑥 represents the maximum ℓ 2normalization between frames. Then, at a ratio 𝛽, we use the Prim algorithm [42] to select a set of frames 𝑉 𝑘 ′′ 𝑗 with the minimum distance from 𝑉 𝑘 ′ 𝑗 for storage to replay in subsequent training. Frames of the same class in this set are spatially the farthest apart and contain the maximal amount of diverse information, thus maximizing the total information content of the frame subset. Meanwhile, frames of different classes are spatially the closest, and the correlation between the background frame and the action frame achieves its best, thereby strengthening the causal relationship between frames.During the training of incremental step 𝑘 , we use dataset 𝐷 ′𝑘 = 𝐷 𝑘 𝑀 1:(𝑘 -1) to update model 𝑓 𝜃 𝑘 -1 , where 𝐷 𝑘 is the dataset for T 𝑘 , consisting of untrimmed videos that belong to 𝐿 𝑘 . 𝑀 1:(𝑘 -1) is the memory bank, which is composed of the temporally downsampled videos selected by TES from 𝐿 1 to 𝐿 𝑘 -1 , where 𝐿 𝑘 represents the collection of untrimmed videos that belong to the k task, assisting the model in learning the temporal information from previous tasks better. Moreover, during the training of incremental step 𝑘, only the parameters of 𝐴𝑡𝑡𝑒𝑛𝑡𝑖𝑜𝑛 𝑘 in the TPE are adjustable, while all other parameters of the self-attention layers remain frozen, with gradients set to zero. In addition, the final objective function for T 𝑘 is formally defined as:EQUATIONwhere 𝐿 𝐷 𝑘 𝑐𝑒 and 𝐿 𝑀 1:𝑘 𝑐𝑒 respectively represent the cross-entropy loss from the training of samples for the new task data 𝐷 𝑘 and the samples from the memory bank 𝑀 1:𝑘 , 𝐿 𝑘 𝑟𝑒𝑔 is the regularization loss on weights in the TPE module, and 𝛾 and 𝜂 are weights used to balance the different loss terms.During inference, for each new video, we first use the temporal feature extractor to extract motion information from the video frames. Then, we concatenate it with the appearance information extracted by the spatial feature extractor to form a feature sequence. Subsequently, we input the feature sequence into the TPE module to ensure a thorough observation at the temporal level before entering the backbone, to avoid temporal attention bias. After that, we input the output of the TPE into the backbone to recognize the action in the current frame. 
experiments: 
 In this section, we apply state-of-the-art CIL methods from the image and video domains to this task, constructing a baseline for the CIOAD task and comparing our method against them. We also present ablation experiments demonstrating the effectiveness of different components and analyze the results under different practical designs. Finally, we analyze the generalization ability of the model and conducted a qualitative analysis of its performance.Datasets We evaluate our model on two publicly available datasets, which are the standard OAD datasets: THUMOS'14 [18] and TVSeries [9] . The THUMOS'14 dataset consists of a training set, a test set, and a validation set, containing background frames and 20 types of action frames. We use its validation set for training and its testing set for evaluation. The TVSeries dataset is composed of 27 untrimmed long videos with actions labeled under 30 categories, with the remaining parts corresponding to the background class. Following the conventions [5, 55] , we select 20 videos for training and use the remaining 7 for evaluation. The performance of various methods on THU-MOS'14 with 10 steps at each incremental step. In most incremental steps, TS-ILM achieved higher accuracy, indicating its strong capability to preserve past knowledge.Benchmark We follow the basic guidelines of CIL [11, 17] , and in line with the setup for background in object detection incremental learning [48] , we divide the OAD dataset into settings conforming to CIL and establish a benchmark. Specifically, for the training set, we consider action frames not belonging to the current task's categories as background frames and discard untrimmed videos that only consist of background frames, which are not used in the training of the current task. For the test set, we keep frames of all action categories that have appeared up to the current task and consider other frames as background frames. For the THUMOS'14 dataset, we split the 20 action categories into 10 and 20 tasks. For the TVSeries dataset, we split the 30 action categories into 10 and 30 tasks. Training and evaluation were conducted across all four scenarios. Evaluation Protocal Following the OAD task conventions [28, 55] , we assess the single incremental step of the THUMOS'14 and TVSeries datasets using per-frame mean average precision (mAP) and per-frame mean calibrated average precision (cAP) [9] , respectively. The calibrated average precision can be formulated as: where 𝐼 (𝑡) is equal to 1 if frame 𝑡 is 𝑇 𝑃. The coefficient 𝑤 is the ratio between negative and positive frames. After each incremental step, we evaluate the model on all seen classes and assess the THU-MOS'14 and TVSeries datasets separately through the average mAP and average cAP across all tasks (denoted by "mAP" and "cAP"). In addition, we follow [61] and report the rate of forgetting during the incremental process and the memory overhead of the saved sample frames (denoted by "Forget" and "Mem. Frame Instances(G)"). Implementation Details TS-ILM was implemented on PyTorch [38] using Nvidia RTX 3090 during the training and testing phases. LSTR [59] is used as our backbone, and we follow LSTR's data preprocessing process. For feature extractors, We employ a two-stream network [53] pretrained on TSN-kinetics [5] , where ResNet-50 [16] and BN-Inception [19] were used for spatial and temporal subnetworks respectively. To learn the model weights, we use an Adam optimizer with a weight decay of 5 × 10 5 [25] . Each incremental step in the training phase lasts 25 epochs with a batch size of 16.EQUATIONEQUATIONFor a fair comparison, the training settings for all methods and backbone are the same, with the loss weight 𝛾 and 𝜂 set to 0.95 and 0.45, respectively.This section delineates the comparative assessment of our proposed TS-ILM against existing class incremental learning methods applied in the image and video domains under multiple challenging settings on two datasets. Specifically, We test the regularization methods MAS [2] and EWC [26] , exemplar replay techniques iCaRL [44] and BiC [56] in the image domain, and the class-incremental learning approaches vCLIMB [52] and TCD [37] in the video domain, adapting all of these methods to the CIOAD task setting. For fair comparisons, all methods use the same feature extractor and backbone, with the exemplar replay methods using the same exemplar memory. Table 1 reports the performance of various methods under different settings on the THUMOS'14 and TVSeries datasets, from which we can draw the following conclusions. First, TS-ILM significantly outperforms the other methods, particularly in the 20-task setting on the THUMOS'14 dataset, with the mAP increase of over SOTA by 7.08 % while reducing the forgetting rate by 22.95%, substantially mitigating the catastrophic forgetting issue in OAD tasks. Second, we find that the catastrophic forgetting on the THUMOS'14 dataset is generally more severe than on TVSeries, which we speculate is due to its more complex action patterns and greater likelihood of confusion between actions. Third, we observe that on the TVSeries dataset, storing only 5% of the data, our method reached 90.91% of the performance of joint training, i.e., the upper bound, indicating that our method utilizes replay memory effectively and significantly reduces the demand for memory capacity. Figure 4 shows the performance of various models at each incremental step. TS-ILM achieves higher accuracy in most incremental steps, indicating its robust capability to preserve past knowledge.The proposed method specifically focuses on the issue of catastrophic forgetting that occurs during the continual learning process and is agnostic to the type of backbone, thus allowing TS-ILM to be easily integrated into any OAD method. We evaluate the generalization ability of TS-ILM using three different backbones, as shown in Table 2 . The results indicate that across the three different backbones, the performance of our method consistently surpasses that of other class-incremental learning methods,demonstrating that the superior performance of TS-ILM on CIOAD tasks is independent of the backbone category, evidencing robust generalization capabilities.Effect of each component To demonstrate the effectiveness of the Task-Level Temporal Pattern Extractor (TPE) and Temporal-Sensitive Exemplar Selector (TES), we conducted experiments to test various combinations of these two modules on the THUMOS'14 and TVSeries datasets. The experimental results for each combination are presented in Table 3 , which indicates that each introduced component contributes positively to the performance, and their amalgamation yields the best performance. Effect of memory size To substantiate the robustness of TS-ILM in memory consumption, we assessed the performance of various methods under different memory capacities on two datasets, with the results delineated in Table 4 . It is observable that on the THUMOS'14 dataset, our TS-ILM incurs merely 0.12GB of memory expenditure, which is comparable to the performance of BiC [56] that utilizes 0.41GB, thereby economizing 70.7% of memory under equivalent performance metrics. This underlines the efficient utilization of memory by our method. Additionally, the table reveals that regardless of the memory budget, TS-ILM consistently surpasses other methodologies, indicating its outstanding capability to counteract catastrophic forgetting. Effect of regularization loss As discussed in Section 3.3, within the TPE module, we define a learnable weight for the self-attention layer associated with each task, denoted by 𝜔, and introduce a regularization loss 𝐿 𝑟𝑒𝑔 to penalize variations in the weight parameters of the self-attention layers that pertain to prior tasks. To evaluate the efficacy of this loss, we assessed performance with and without the loss on two datasets, as shown in Table 5 . The results indicate that this additional loss has led to performance gains of 2.25% on THUMOS'14 and 3.06% on TVSeries, while the forgetting rate decreased by 2.85% and 8.15% respectively. These findings suggest that this loss function can effectively preserve information from past actions and mitigate catastrophic forgetting. Effect of each component in TES As described in Section. 3.4, our TES module can be roughly decomposed into two components: initially, frames are coarsely filtered using the herding strategy [44] , and then further refined and stored through Minimum Distance Selector. Table 6 enumerates the results of testing different combinations of the two parts of TES on two datasets. It is noteworthy that the tests were conducted with the same memory budget and that a random sampling strategy was employed to select the frames to be saved when both components were disabled. The findings corroborate that each component individually contributes to performance improvement, and their conjunctive application leads to superior performance outcomes.Visualization of action scores Figure 5 visualizes video clips and their corresponding action scores. These action scores were inferred from videos in the dataset of the 10-th task after training on 10 tasks on THUMOS'14. The category "Javelin Throw" displayed in the figure is part of the training set of the 7-th task and did not appear in subsequent training. The results indicate that both components of our proposed TS-ILM, as well as their combination, significantly enhance the model's ability to remember previous action categories, effectively overcoming the issue of catastrophic forgetting. Visualization of self-attention layers in TPE As detailed in Section 3.3, within the TPE module, we integrate a self-attention layer for each task to memorize its temporal action patterns. Figure 6 displays the visualized results of the self-attention layers for different tasks, where higher values indicate greater attention by the self-attention layer at that specific time. The results suggest that the focus of self-attention layers varies across different tasks. Integrating these observations can effectively prevent biases in the backbone of the temporal dimension caused by discrepancies in the categories of actions in different tasks. 
conclusion: 
 In this paper, we introduce the novel task of Class-Incremental Online Action Detection and propose an innovative framework to address this challenge. Specifically, our framework comprises two key components. The first one extracts and saves the temporal patterns of divergent actions in different tasks, allowing for a comprehensive temporal analysis before the data enters the backbone network. The second one selects a set of frames that maximize temporal causality and minimize information redundancy for subsequent replay, enabling the model to learn the temporal information of previous tasks better. Finally, we establish a benchmark for this task and conduct a thorough evaluation of our approach in comparison to state-of-the-art class-incremental learning methods previously applied in the image and video domains, demonstrating the effectiveness of our method. 
 
 Appendix:  


Decision: In this work the authors focus on class-incremental learning for online action detection. It is a novel task and the authors propose a new method to solve this. This work received 3x borderline accept ratings before the rebuttal and the authors have provided a response to reviewers comments. The main concerns were lack of clarity/motivation on proposed components, motivation behind this new task, and limited datasets for evaluation. The rebuttal addressed most of the concerns and the reviewers kept their original borderline scores. The motivation behind why class-incremental learning is relevant for online action detection is still not well explained and non-multimodal aspect of this work weakens its relevance to multimedia community. Despite these limitation, introduction of this new task and extensive evaluation is a strong point of this work. Considering these points, the AC recommends acceptance of this work.

Reviews: review: paper_summary: paper_summary: This paper proposes the task of Class-Incremental Online Action Detection and introduces the TS-ILM framework to deal with this task.
strength_weakness: strength_weakness: (1) The analysis of Class Incremental Learning for Online Action Detection is interesting and reasonable.
(2) Empirical results are extensive and demonstrate the effectiveness of the method.
limitations: limitations: (1) For the task-level temporal pattern extractor, the design of the parameter adjustable module is not clearly illustrated. What is the motivation to add another self-attention module on top of the previous self-attention modules?

(2) For the temporal-sensitive exemplar extractor, how do the background and action be distinguished?
paper_summary: paper_summary: This paper proposes an approach to solve the task of Online Action Detection, specifically the newly introduced task of Class Incremental Online Action Detection (CIOAD). To solve this task, a new framework is proposed, namely TS-ILM. TS-ILM contains two parts: a task-level temporal pattern extractor and a temporal-sensitive exemplar selector. The temporal pattern extractor extracts the temporal
patterns of actions in different tasks and saves them. The temporal-sensitive exemplar selects frames with high relevance to learn temporal
information of previous tasks more effectively. The proposed model is evaluated on THUMOS'14 and TVSeries dataset and report improvements over the current SOTA.
strength_weakness: strength_weakness: --> The introduction of class incremental learning to the problem of online action detection is new. 
--> The components of the proposed architecture, the task-level temporal pattern extractor, and the temporal-sensitive exemplar selector are explained well in Fig 3. 
--> Ablation experiments are thorough, covering all the component and the hyper-paraments in the method. 
--> The paper is well-organized and well-written. Overall it is not hard to read and understand.
--> Results on both datasets beat the baselines across different settings.
limitations: limitations: --> The motivation to combine Class Incremental Learning with Online Action Detection is not clear. In Online Action Detection, during inference, the constraint is to detect an ongoing action without accessing the future frames. I do not understand how catastrophic forgetting is relevant here. 
--> Addition symbol in Fig 3, should be element-wise addition and pixel-wise addition. 
--> What are the x and y-axis in the attention maps of Fig. 6?
--> It is mentioned in lines 571-574, that during evaluation, the number of tasks is set to 20 and 30 for THUMOS' and TVSeries datasets respectively. Why are these different from the number of tasks during training? 
--> The THUMOS'14 and TVSeries are the only two datasets used for evaluation. Any reason for not evaluating on HDD dataset or EpicKitchen like the recent works on OAD?
paper_summary: paper_summary: In this paper, the authors propose ClassIncremental Online Action Detection (CIOAD), which is class-incremental learning in online action detection. Due to the need for more precise temporal modeling in online action detection, the authors found that applying existing class-incremental learning algorithms to CIOAD does not yield good performance. They have introduced a framework more suitable for CIOAD, called TS-ILM. In this framework, the task-level temporal pattern extractor extracts the temporal patterns from different tasks before the data is input into the backbone, allowing for a more comprehensive observation of the data on a temporal level. The temporal-sensitive exemplar selector selects a subset of frames from the video for subsequent replay. TS-ILM has demonstrated excellent performance on several challenging benchmarks.
strength_weakness: strength_weakness: 1. This paper introduces the task of ClassIncremental Online Action Detection (CIOAD) for the first time, which focuses on class-incremental learning within Online Action Detection (OAD) and has practical value for the application of OAD
2. The proposed TS-ILM is easy to follow and has demonstrated excellent performance on the task of ClassIncremental Online Action Detection (CIOAD)
3. Numerous ablation studies and qualitative comparisons help in understanding the setting
4. A large number of existing methods have been replicated on the task of CIOAD, and the code is easy to read
limitations: limitations: 1. In Equation 3, the authors propose a regularization term. However, as shown in Figure 3, the self-attention parameters from 1 to k-1 are frozen during task k. Does this imply that the regularization term will only optimize the self-attention for k? If so, as shown in Equation 3, it calculates the regularization for 1 to k-1, but would it not be effective for optimization? Despite the results in Table 5 showing a clear effect of the regularization loss, I remain confused about this part.
2. The symbols and definitions in some parts of the article are difficult to understand. For example, what does q represent in \( C^k_{q-1} \) at line 303? And what is \( L^k \) at line 521?
3. In Online Action Detection (OAD), data is generally fed into the network frame by frame. However, as the authors have described in the text, the TS-ILM uses a two-stream network to extract features, and it is usually necessary to extract optical flow in the Temporal branch. Would this affect the response speed of OAD in practical applications?
4. The paper lacks an inference step.