Prior-free Balanced Replay: Uncertainty-guided Reservoir Sampling for Long-Tailed Continual Learning

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-06 10:34:21

Abstract: Even in the era of large models, one of the well-known issues in continual learning (CL) is catastrophic forgetting, which is significantly challenging when the continual data stream exhibits a long-tailed distribution, termed as Long-Tailed Continual Learning (LTCL). Existing LTCL solutions generally require the label distribution of the data stream to achieve re-balance training. However, obtaining such prior information is often infeasible in real scenarios since the model should learn without pre-identifying the majority and minority classes. To this end, we propose a novel Prior-free Balanced Replay (PBR) framework to learn from long-tailed data stream with less forgetting. Concretely, motivated by our experimental finding that the minority classes are more likely to be forgotten due to the higher uncertainty, we newly design an uncertainty-guided reservoir sampling strategy to prioritize rehearsing minority data without using any prior information, which is based on the mutual dependence between the model and samples. Additionally, we incorporate two prior-free components to further reduce the forgetting issue: (1) Boundary constraint is to preserve uncertain boundary supporting samples for continually re-estimating task boundaries. (2) Prototype constraint is to maintain the consistency of learned class prototypes along with training. Our approach is evaluated on three standard long-tailed benchmarks, demonstrating superior performance to existing CL methods and previous SOTA LTCL approach in both task- and class-incremental learning settings, as well as ordered- and shuffled-LTCL settings.

Main paper: 
 introduction: 
 Over the last decade, deep neural networks (DNNs) have demonstrated remarkable performance in various multi-media tasks, such as image segmentation [50] , video caption [25] , and audio-visual learning [37] . However, these tasks are usually performed in a static environment where all data is available in a single training session. In a dynamic environment where data arrives phase by phase, the Figure 1 : Illustration of Ordered-LTCL (bottom) and Shuffled-LTCL (top). 𝜃 𝑖 denotes the model parameters at the incremental step 𝑖. Ordered-LTCL assumes that all tasks are ordered by the sample number per task, i.e., old tasks contain majority classes while new tasks have minority classes. Shuffled-LTCL assumes that all classes are randomly distributed. model trained on a new task tends to forget a significant amount of information of old tasks, which is commonly known as catastrophic forgetting issue [19, 27] . Besides, it was reported that recent advanced large vision and language models still face challenges in forgetting knowledge when dealing with complex sequential tasks [4] , resulting in the potential loss of previously learned information during the fine-tuning process for novel tasks [14] . To overcome the catastrophic forgetting, significant efforts have been made to learn from the sequential data stream without forgetting previously acquired knowledge, which is called continual learning (CL) [58] .Despite recent significant progress in conventional CL, it is typically based on the assumption that the training data is drawn from a balanced distribution. However, real-world data often exhibits a long-tailed distribution [13, 53, 55] , where only a few classes dominate the most samples. For instance, in autonomous driving [49, 57] , anomaly accidents often occur with lower probabilities than the frequent safe events. When constructing a medical dataset [39] , it is an usual phenomena that common symptoms are easily collected while it is difficult to collect enough rare symptoms. In dynamic environments characterized by realistic scenarios, minority classes often incrementally emerge as new tasks, posing a great challenge for the adaptation ability of DNNs [19] . Besides, existing CL methods exhibit severe performance degradation over the long-tailed data stream. Therefore, it is essential to investigate continual learning over the long-tailed data stream.Inspired by [30] , we consider two different long-tailed continual learning (LTCL) settings, i.e., ordered-and shuffled-LTCL settings as shown in Figure 1 . To address the LTCL problem, several straightforward solutions have been proposed [15, 21, 30] to combine existing CL methods with re-balancing techniques, such as data re-sampling [10] , data re-weighting [38] , and two-stage training [20] . For instance, [11] explored a balance sampling strategy to keep a balanced memory buffer for the imbalance continual learning. [36] proposed to utilize data augmentation for the memory buffer to alleviate the class imbalance issue in the class incremental learning. However, these approaches may not be practical for tasks evolving over time, as most re-balance techniques require label distribution information of the entire stream, while obtaining such information is often infeasible due to unknown prior of new tasks emerging in the future. In fact, it is usually unknown whether the incoming data or the emerging class is from majority or minority classes in the real world. This makes most existing methods, which are to solve the class imbalance problem unsuited for real-world applications.To further cast light on challenges for the LTCL problem, we conduct a motivating experiment (details can be seen in Sec 3.1) and observe that: (1) Minority samples are more likely to be forgotten than majority samples; (2) The classifier weights are easily biased to the old majority classes; and (3) Minority data is usually distributed around the task boundaries with higher uncertainty. Besides, with the limited storage and computing resource, one should utilize a constrained buffer size throughout the entire training phase to address the LTCL problem [5] .To address the LTCL issue, motivated by the above experimental findings, we propose a novel Prior-free Balanced Replay (PBR) framework to incrementally learn an evolved representation space for the LTCL problem. More precisely, we design an uncertaintyguided reservoir sampling strategy to prioritize storing minority samples in the replay memory, which is based on the mutual information between changes in model parameters and prediction results. Besides, two prior-free components are newly designed to effectively alleviate the catastrophic forgetting issue under LTCL, especially for minority classes. In detail, prototype constraint ensures all classes have balanced magnitudes by maintaining the consistency of class prototypes learned at different times, while boundary constraint prevents forgetting task boundary information by preserving boundary supporting samples of old tasks.In summary, key contributions of this work are threefold:(1) We propose a novel PBR framework to address the LTCL problem without relying on prior information (i.e., label distribution), which utilizes an uncertainty-guided reservoir sampling strategy to achieve a balanced replay with less forgetting.(2) We design two new prior-free components (i.e., boundary and prototype constraints) to further reduce the forgetting of minority data, which can be integrated into the PBR framework based on the uncertain samples.(3) Extensive experiments are conducted to evaluate the proposed method on three popular datasets under the LTCL setting, i.e., CIFAR-10 [24] , CIFAR-100 [24] , and Tiny ImageNet [2] . Experimental results indicate that our method can achieve state-of-the-art (SOTA) performance for both task-and class-incremental learning settings, as well as ordered-and shuffled-LTCL settings, surpassing previous works by a significant margin. 
related work: 
 Continual Learning. To enhance the adaption ability of deep neural networks, existing solutions could be roughly divided into four groups: rehearsal-based [40, 44, 45] , distillation-based [26, 41] , architecture-based [34, 48] , and regularization-based methods [23] .Rehearsal-based methods [1, 5, 32] store a data subset of the old tasks and replay these samples to alleviate catastrophic forgetting.The key is to achieve effective sample selection for rehearsal, such as experience replay [40, 44] and gradient-based sample selection [1, 32] . Another solution is to imitate the previous tasks' behaviors when learning new ones. The main idea is knowledge distillation [17] taking past parameters of the model as the teacher. Besides, it is a common choice to combine rehearsal and distillation by self-distillation learning [41] . Besides, regularization-based methods mainly focus on preventing significant updates of the network parameters when learning new tasks, such as elastic weight consolidation [23] , synaptic intelligence [47] and Riemannian walk [7] . Furthermore, architecture-based methods [34, 48] distinct different tasks by devoting distinguished parameter sets. For instance, Progressive Neural Networks [46] incrementally introduce a new set of parameters for incoming tasks to tackle the forgetting problem.Long-Tailed Learning. To address the long-tailed problem, rebalancing strategies are the most common solutions, including re-sampling [10] and re-weighting [38] . However, these methods easily lead to performance degradation for head classes and overfitting issues for tail classes. Two-stage based methods are proposed to further improve the re-balancing strategies, such as decoupled training [20] and deferred re-balancing schedule [6] . Besides, to learn a high-quality representation space based on imbalanced data, regularization-based approaches are proposed to increase interclass differences, such as margin [6] , bias [35, 42] , temperature [54] or weight scale [20] . Recent works explore flexible ways for reweighting by hard sample mining [28, 29] , meta-learning [43] , and influence function [38] , which target to measure the importance of each training sample. Other studies propose to transfer useful knowledge from head to tail classes via designing memory module [31] or translation [22] .Long-Tailed Continual Learning. There are some recent works exploring the imbalance issue in continual learning, e.g.. Partitioning Reservoir Sampling (PRS) [21] and LT-CIL [30] . PRS [21] proposed a balance sampling strategy for head and tail classes along with the sequential tasks to preserve balanced knowledge. LT-CIL [30] utilized a learnable weight scaling layer to decouple representation learning from classifier learning. However, these methods ignore the relationship between the tasks of imbalanced and continual learning and rely on the label distribution for re-balance strategies, while our work is orthogonal with them to learn an evolved feature space for the long-tailed continual learning without label distribution. Besides, [36] proposed to utilize data augmentation for the memory buffer to alleviate the class imbalance issue, which only focused on the class imbalance in the current incremental step. [11] explored the class imbalance for online continual learning, but it ignores the unequal roles for different samples in the memory buffer. 
methodology: 
 Problem Formulation. A standard learning agent sequentially observes a data stream {(𝐷 0 , 𝑡 0 ), . . . , (𝐷 𝑖 , 𝑡 𝑖 ), . . . , (𝐷 𝑛-1 , 𝑡 𝑛-1 )}, where 𝐷 𝑖 = {(x 𝑖 𝑘 , y 𝑖 𝑘 )} 𝑠 𝑖 𝑘=1 is a labeled dataset of task 𝑡 𝑖 . 𝑛 is the time index indicating the task identity. 𝐷 𝑖 consists of 𝑠 𝑖 pairs of samples The experiment is conducted on the long-tailed CIFAR-10 using stochastic gradient descent (SGD) and cross-entropy (CE) loss. It is observed that the performance drop of minority classes is larger than majority classes. Therefore, the long-tailed data may further aggravate the catastrophic forgetting, as the minority classes are more likely to be forgotten than the majority classes.with corresponding targets from the data space X × Y. To reflect the general long-tail phenomena, we assume that the sequence {𝑠 0 , 𝑠 1 , . . . , 𝑠 𝑛-1 } exhibits a power-law distribution, i.e., 𝑠 𝑖 = 𝐶𝛼 𝑡 𝑖 , where 𝐶 is the exponent of the power and 𝛼 is the imbalance ratio for general long-tailed settings [31] . This assumption of powerlaw behavior is commonly observed in empirical distributions, and reflects how frequently samples from each task are observed.For the classification task, the learning agent predicts the label for a given input x as 𝑓 𝜃 (x), where 𝑓 𝜃 (•) is a mapping function from the input X to the output Y parameterized by 𝜃 . Let ℓ : Y × Y → R be the loss function between a prediction 𝑓 𝜃 (x) and the target y. Our goal is to learn the optimal parameter 𝜃 with strong continual adaptation ability to correctly classify samples from any observed tasks. The training and inference processes do not rely on task identities 𝑡 𝑖 . The optimization objective for the parameter 𝜃 over the data stream is given by the follows:EQUATIONAs a motivation, we conduct an empirical experiment as the motivation to observe how the representations changes under the ordered-LTCL setting. We focus on two main factors in a representation space, i.e., class prototypes and task boundaries. Overall, we empirically investigate the reason for the severe performance degradation of existing CL methods under the LTCL setting, i.e., the deterioration of the representation space caused by biased prototypes and easily forgotten task boundaries. Here, the baseline is ResNet18 [16] trained by the SGD optimizer without any further operations (e.g., memory buffer). Main results are visualized in Figure 3 and Figure 4 (b), respectively. Forgotten Minority Data. As shown in Figure 2 , different color denotes different tasks, i.e., task-1 has the most data and task-5 4 : Forgotten Task Boundaries. We visualize the feature distribution of task 𝑖 at the stream end. Colorful and gray points denote the forgotten and non-forgotten samples, respectively. We found that forgotten samples contain more minority data and are generally located near the task boundary in the feature space, leading to confusing task boundaries.has the least data, where the tendencies can reflect the forgetting degree of each task. We observe that the tendency for minority classes (yellow curve) is steeper than majority classes (blue curve), indicating that the minority classes are more likely to be forgotten than the majority classes.Biased Prototypes. As shown in Figure 3 , the weight magnitudes of the classifier are irregularly distributed. The weight magnitudes of old majority classes are significantly higher than those of new minority classes, while the bias magnitudes of new classes are higher than those of the old classes. Based on such phenomena, the features of each class prototypes are generally clustered with the lack of discrimination in the representation space.Forgotten Task Boundaries. As shown in Figure 4 , the boundary supporting samples of each task (i.e., colorful points) are easily forgotten along with the continually arrived data, which would lead to confusing task boundaries in the representation space. In particular, the boundary supporting samples of the tasks with minority classes are more likely to be forgotten than majority classes due to insufficient training data.Motivated by the above observations, we propose a novel PBR framework based on the uncertainty-guided reservoir sampling and two constraints. The proposed approach is shown in Figure 5 .Experience Replay. Experience replay aims to preserve useful knowledge of the previous tasks. Here, we explicitly store the most uncertain samples of the old tasks along the incremental trajectory to improve the continual adaption ability of neural networks. To this end, we seek to minimize the following objective at the time 𝑡 𝑐 :EQUATIONwhere 𝜃 * 𝑖 is the optimal parameters after time 𝑡 𝑖 , and 𝛼 is a hyperparameter. L 𝑘𝑙 is the knowledge distillation loss. To overcome the unavailability of 𝐷 𝑖 from old tasks, we introduce a small memory buffer M to retain previous experiences. The objective seeks to replay the learned experiences by resembling the teacher-student trick. To save resources, we merely store the latest model state at 𝑡 𝑐 -1 rather than a checkpoint sequence from 𝑡 0 to 𝑡 𝑐 -1 . In this work, we aim to maintain prototype knowledge and task boundary information by incorporating cosine normalization for different tasks. As follows, we will present the details in turn.Uncertainty-guided Reservoir Sampling. To further keep a balance between old and new tasks, we present an uncertaintyguided reservoir sampling to guarantee that uncertain samples are stored in the buffer M in priority, since uncertain samples are more likely to belong to the minority classes. Different from vanilla reservoir sampling [52] , the uncertainty-guided sampling process is conducted at the end of each task to maintain a well-rounded knowledge. At the end of task 𝑡 𝑐 , the training samples 𝐷 𝑐 are sorted as D𝑐 according to their mutual dependence with the network. Then the sampling process iterates 𝑠 𝑐 times between sample-in and sample-out for each (𝑥, 𝑦) ∈ 𝐷 𝑐 . The sample-in decides whether to sample a data point into the memory, while the sample-out removes a sample from the memory.(1) Candidate: To store uncertain samples, for each iteration, we first generate a candidate sample by:EQUATIONwhereI [•|•]indicates the mutual information (MI) between the prediction and the posterior over parameters. If K is already stored in the memory, we will generate a new candidate from 𝐷 𝑐 \{𝐾 }, otherwise keeping it fixed.(2) Sample-In: We design a probability function P (K) to decide whether moving K into the memory M:EQUATIONwhere 𝑁 𝑐 is the total sampling iteration number from the end of task 𝑡 𝑐 up to now. 𝑠 𝑖 is the running frequency of class 𝑖. This condition implicitly achieves a trade-off between old and new tasks.(3) Sample-Out: If the memory is out of buffer size, a sample will be removed when a candidate is entered into the memory M. The probability that an sample is removed follows the uniform distribution over the memory size, i.e., 1 | M | , since sample-in works towards achieving a balanced partition for old tasks.Note that due to the lack of sufficient knowledge about the minority classes, the uncertainty degree of minority data is generally higher. Therefore, our method implicitly encodes the rule to preferentially store minority samples, rather than directly adjusting the label distribution to alleviate data imbalance.Uncertainty-guided Mutual Information. Given a network with the limited capacity, uncertainty can be utilized to estimate the importance of each training sample and identify easily forgotten minority samples with boundary information. In this work, we utilize Monte-Carlo dropout [12] for uncertainty quantification, which interprets dropout regularization as a variational bayesian approximation. Conditioning on a neural network on finite random variables, when training with data 𝐷, we could calculate the posterior probability distribution 𝑝 𝑦 * 𝑡 | 𝒙 * 𝑡 , 𝐷 by marginalizing out the posterior distribution, which can be approximated using Monte Carlo integration with dropout regularization:EQUATIONwhich is achieved by 𝑇 sampled dropout masks and the masked parameters. This formulation implicitly endues the network with the ability to quantify the confidence of predicted results. Here, we employ the predictive entropy as an indication of the amount of information in the prediction distribution of the network:EQUATIONTo capture the trustworthiness of the network in training samples, we compute the mutual information (MI) between the prediction and the posterior over parameters as the mutual dependence:EQUATION) Based on mutual dependence, minority data with boundary information of previous tasks are stored in the buffer via an efficient way, maintaining the previously learned knowledge and maximize dissimilarities among all seen tasks.Based on the selected samples in the memory, we propose two new prior-free components to further alleviate the forgetting issue.Prototype Constraint. Prototype constraint is to maintain the consistency of learned class prototypes along with training. A typical classifier produces the predicted probability of a sample x by:EQUATIONwhere w 𝑖 is the 𝑖-th weight vector and 𝑏 𝑖 is the 𝑖-th bias term in the classifier. As shown in Figure 3 , the weight magnitudes are irregularly distributed, resulting in biased prototypes in the feature space. To address this issue, we propose utilize two types of statistical information i.e., class prototype and cosine similarity to preserve useful class-wise information.Concretely, given an input data point x, the mapping function 𝑓 𝜃 (x) is to map x as a hidden representation before the final linear projection for classification. Inspired by cosine normalization [18, 33, 51] , we utilize a scaled cosine classifier to extract normalized embeddings of samples by 𝑓 𝑛 𝜃 (x) = 𝑓 𝜃 ∥ 𝑓 𝜃 ∥ , which produces the predicted probability as follows:EQUATIONwhere ŵ denotes the normalized weights in the classifier and s is the scaling factor. Instead of computing the average feature over all samples, this formulation allows us to interpret the weight vectors of the classifier as class prototypes during training, which could save the costs for computing average features. It is also noteworthy to preserve cosine similarity scores among previously learned class prototypes. Thus, we further enforce the newly updated classifier to mimic the behavior of previously learned classifier, which could produce approximately consistent similarity scores for each task along with newly coming data. Formally, we propose to exploit a distillation loss to preserve prototype information as follows:EQUATIONwhere ŵ𝑖, is the weight for previous task 𝑖 in the prototype-based classifier. Different from previous cosine normalization encouraging the similar angles between the features and the class prototypes [18] , such a distillation loss enhances the learned prototypes to be approximately preserved in the current model.Boundary Constraint. Boundary constraint is to preserve uncertain samples with boundary supporting information for continually re-estimating task boundaries. Denote the incoming data by X 𝑖𝑛 and data stored in the memory buffer by X 𝑏 𝑓 , we use a modified cross-entropy (MCE) loss to link prototypes and logits:EQUATIONwhere ŵ𝑖 is 𝑖-th weight vector (prototype) of the classifier and 𝜏 1 is a scaling factor. The prototype is normalized so that ŵ𝑇 𝑖 𝑓 𝑛 𝜃 is a cosine similarity metric. Note that class prototypes are explicitly updated where samples of the same class lie close by each other. Beyond to the prototypes, we further consider uncertain samples to preserve effective boundary information via the knowledge distillation loss:EQUATIONwhere 𝜃 * is the parameters after the task 𝑡 𝑐 -1 . 𝜏 2 is the temperature scale. We can rewrite Equation 2 as follows:E x∼{𝐷 𝑐 ∪M } L 𝑡 𝑐 (x) + 𝛼E x∼M [L 𝑘𝑙 (𝑓 𝜃 * (x)∥𝑓 𝜃 (x))] + 𝛽L 𝑑 𝑐 . (13)We approximate the expectation on batches sampled from the current task and the buffer, respectively. 
experiments 4.1 experimental setup: 
 4.1.1 Benchmarks. We consider two common CL scenarios [5] :(1) Task Incremental Learning (Task-IL), where task identities are provided to select the relevant classifier for each sample during evaluation; (2) Class Incremental Learning (Class-IL), where task identities are not provided during evaluation. This difference makes Task-IL and Class-IL the easiest and hardest scenarios. We present three benchmarks for LTCL: Seq-CIFAR-10-LT, Seq-CIFAR-100-LT, and Seq-TinyImageNet-LT, where the training data yields Table 1 : Accuracy for previous SOTA method on Seq-CIFAR-100-LT under the ordered-LTCL setting. IR is 0.01.Class-IL Task-IL PODNET+ [30] 23.90 46.00 Ours 25.05 52.23standard long-tailed distribution as defined by [31] . Besides, motivated by [30] , we also consider order-and shuffled-LTCL settings.4.1.2 Implementation Details. For a fair comparison, we try our best to maintain the experimental setting as consistent as possible. We implement our framework with Pytorch and Torchvision libraries and use NVIDIA TITAN 2080 Ti GPU to train the deep neural network. Following [5] , we employ ResNet18 as the basic backbone for all methods, and all networks are randomly initialized rather than pre-trained. We used stochastic gradient descent with a batch size of 32 and a learning rate of 0.03. 𝑠 is assigned as 10. 𝜏 1 is set as 0.1 and 𝜏 2 is 2. It is important that no pre-trained model is used in all our experiments. As for the testing phase, we utilize 128 as the batch size for validation.Baselines. SGD-LT is using SGD under long-tailed distribution. SGD-BL is under balanced distribution. Both SGD-LT and SGD-BL are under the CL setting. JOINT-LT is to jointly train all tasks under long-tailed distribution, and JOINT-BL is to jointly train all tasks under balanced distribution. CL Methods. To provide fair comparisons, we compare 14 models involving regularization, distillation, architecture and rehearsal based methods. Regularization-based methods include Elastic Weight Consolidation (oEWC) [23] and Synaptic Intelligence (SI) [56] . Two methods leveraging knowledge distillation are Incremental Classifier and Representation Learning (iCaRL) [41] and Learning Without Forgetting (LwF) [26] . One architectural method is called Progressive Neural Networks (PNN) [46] and eight rehearsal-based methods include Experience Replay (ER) [40, 44] , Gradient Episodic Memory (GEM) [32] , Averaged-GEM (A-GEM) [9] , Gradient based Sample Selection (GSS) [1] , Function Distance Regularization (FDR) [3] , Hindsight Anchor Learning (HAL) [8] , Dark Experience Replay [5] , and DER++ [5] ). ( 3) Conventional long-tailed algorithms are unavailable due to compatibility with the scenarios.LTCL Methods. We also include the previous SOTA method PODNET+ [30] as the comparison, which adopts a two-stage strategy to learn a balance classifier for different tasks.We adopt two evaluation metrics. The first is the average accuracy (ACC) at the end of all tasks. We run 5 times experiments and report the mean accuracy and the standard deviation. The second is backward transfer (BWT), indicating how much a new task influences the performance of previous tasks. Namely, higher negative values for BWT suggest catastrophic forgetting. ACC and BWT are calculated as follows:EQUATIONwhere 𝑅 𝑗,𝑖 is the accuracy for task 𝑖 at the end of task 𝑗 in the sequence, and T is the total number of tasks.Comparisons with SOTA LTCL Method under Ordered-LTCL.As seen in Table 1 , compared with previous SOTA method, our method can outperform PODNET+ [30] by a large margin for both Class-IL and Task-IL settings. The main reason is that our method can focus on uncertain samples to achieve imbalanced learning and reduce forgetting.Comparisons with CL methods under Ordered-LTCL. As shown in Table 2 , our method could achieve SOTA results compared with CL algorithms. PNN achieves the best results among non-rehearsal methods but attains lower accuracy than rehearsalbased ones. Most rehearsal-based approaches achieve higher results than regularization-based ones because of replaying old samples and learning new samples together. Distillation-based rehearsal methods aim to output similar logits for old samples when learning for new tasks, but the logit information is biased due to long-tailed distribution. Additionally, the methods resorting to gradients (GEM, A-GEM, GSS) seem less effective under this setting, since class imbalance negatively impacts the gradient. Although existing CL methods could reasonably deal with the forgetting issue, they still perform poorly under long-tailed distribution.Comparisons under Shuffled-LTCL. We report the results on shuffle-LTCL scenario (Seq-CIFAR-100-LT with 5-task setting). The buffer size is 200. Due to the superior performance of the strong baseline DER++ on the continual learning task, we utilize DER++ as the baseline. The proposed method outperforms DER++ by a large margin for both Class-IL and Task-IL settings. With varying IRs, our method can still improve the final performance.Comparisons with Different Reservoir Sampling Strategies. To verify the effectiveness of uncertainty-guided reservoir sampling, we report the comparison results of our method with random reservoir sampling under ordered-LTCL. As indicated in Table 2 , our method with random reservoir sampling can even outperform previous CL methods by a large margin. Furthermore, the proposed method with random reservoir sampling performs slightly lower accuracy than that with uncertainty-guided reservoir sampling, demonstrating the effectiveness of uncertainty estimation.Effect of LTCL.. Table 2 reports the average accuracy results at the end of all tasks under the ordered-LTCL setting. We observe that the task-IL accuracy of SGD-LT is better than SGD-BL. The main reason is that new tasks with minority samples could reduce forgetting of old tasks with majority samples. Furthermore, since alleviating forgetting may induce imbalanced impacts on the new tasks, some approaches exhibit lower accuracy than SGD-LT. For instance, regularization-based methods (e.g., oEWC, Lwf, PNN, and GEM) suffer from the imbalance issue on the new tasks, which arises from the learned regularization information from the old task. Therefore, by considering the motivating experiment in Section 3.1, our method can well address the LTCL issue by adopting prototype and boundary constraints.Effect of Imbalance Ratio. We evaluate the comparison methods with different imbalanced ratios following [31] . As shown in Table 3 and Appendix, our approach remarkably outperforms previous SOTA method DER++, where more detailed results are shown in Appendix. It is observed that the performance of existing methods is improved along with increasing imbalance ratio, which mainly benefits from the increasing sample number for each task. We notice that the imbalance ratio slightly impacts the final performance of these methods. Thus, when adopting boundary supporting samples in the buffer, the gain from increasing the imbalance ratio for our method is much larger than other methods. Effect of Buffer Size. As shown in Table 2 , our method outperforms other rehearsal-based methods remarkably using different buffer sizes. The general rule is that a larger buffer size benefits the final performance. Besides, GEM, A-GEM-R, and iCaRL exhibit higher sensitivity to the buffer size, indicating that these methods are easily influenced by the random samples stored in the buffer.Evolution of Task. To further justify the effectiveness of our method, we show the class-IL accuracy curves along with sequential tasks in different datasets under the ordered-LTCL setting. As shown in Figure 6 , the overall accuracy tendency is downwards along with incoming new tasks due to forgetting. It can be seen that our method performs the best accuracy at the end of each task, while other approaches forget a lot with incoming new tasks.Ordered-LTCL Accuracy for Tasks with Minority Classes. To indicate the performance improvement of our method on the minority data, we report the class-IL accuracy for the tasks with minority 4 , our method can obtain a significant performance improvement on the tasks with minority classes, while the baseline (i.e., SGD) performs worse due to catastrophic forgetting. BWT Comparisons under Ordered-LTCL. BWT computes the difference between the current accuracy and its best value for each task. Lower negative values of BWT indicate that new tasks lead to more catastrophic forgetting of the previous tasks. As shown in Appendix, previous methods still forget a lot with lower negative values of BWT, while our method maximizes BWT with minimal forgetting. Our method performs significantly better than other CL methods like iCaRL, DER, and ER. The main reason is that our method can learn an well evolved feature space based on the prototypes and boundary supporting samples.Importance of Prototype-based Classifier. We analyze different components of our method to verify their effects. The linear classifier is a fully-connected layer with the bias, and the cosine classifier is a normalized fully-connected layer without the bias. As shown in Table 5 , our method could obtain the best results using uncertainty quantification and cosine classifier. The linear classifier without rehearsal yields the worst performance because of both catastrophic forgetting and imbalance. As the cosine classifier retains the prototype and similarity information among classes, the class-IL accuracy can be improved compared to the linear classifier. The linear classifier with uncertainty performs lower accuracy results without prototype information, although uncertainty is used to select boundary supporting samples.Importance of Boundary-supporting Sample. In this part, we analyze the effect of boundary-supporting samples (uncertainty) on 5 reports the results of random reservoir sampling and uncertainty-guided reservoir sampling. It is observed that the performance is significantly reduced when using random reservoir sampling. The main reason is a lack of important boundary information due to catastrophic forgetting, although prototypes can be maintained via knowledge distillation over random samples. Based on the uncertainty estimation, the decision boundaries between old and new classes can be well modeled via replaying boundary supporting samples.More experimental analysis can refer to Appendix. 
conclusion: 
 In this work, we propose a novel Prior-free Balanced Replay (PBR) framework based on the newly designed uncertainty-guided reservoir sampling strategy, which prioritizes rehearsing minority data without using prior information. Additionally, we incorporate two other prior-free components to further reduce the forgetting issue including prototype and boundary constraints, which can maintain effective feature information for continually re-estimating task boundaries and prototypes. Compared with existing CL methods and SOTA LTCL approach, the experimental results on three standard long-tailed benchmarks demonstrate the superior performance of the proposed method in both task and class incremental learning settings, as well as ordered-and shuffled-LTCL settings. We believe this would be an important step towards real-world scenarios by combining continual learning with long-tailed distribution.  
 
 Appendix:  


Decision: This paper proposes a framework for Long-Tailed continual learning which prioritizes rehearsing minority data without prior information.

In reviews of reviewers, the common concerns raised by most reviewers were insufficient evaluation. In the response, the authors added the latest methods for comparison, adding experiments on a balanced dataset, adding experiments on a shuffled-LTCL task. The method outperformed most of the latest methods except DualHSC, which additionally introduces the task relationships as prior. Currently, Reviewer xh48 has concerns about assumptions. The authors have acknowledged that these assumptions are commonly used in existing works. 

The AC believes that major concerns have been reasonably solved, there are only minor concerns about clarification for the assumptions and consistency about problem settings, which can be easily solved in the camera-ready version. Therefore, the AC suggests accepting this paper.

In the camera-ready version, the authors should follow the suggestions from Reviewer xh48 to add a section to clearly discuss the assumptions and limitations caused by the assumptions. Additionally, the paragraph under the problem settings should be revised to follow the settings in the introduction and experiments to avoid confusion.

Reviews: review: paper_summary: paper_summary: This work proposes a novel framework called Prior-free Balanced Replay (PBR) for long-tailed continual learning. It designs an uncertainty-guided reservoir sampling strategy to prioritize rehearsing minority data without using prior information. Meanwhile, PBR includes two other prior-free components to further reduce the forgetting issue including prototype and boundary constraints, which can maintain effective feature information for continually re-estimating task boundaries and prototypes. Extensive experiments on three standard long-tailed datasets demonstrate the superior performance of the proposed PBR over various state-of-the-art methods in both task and class incremental learning settings, as well as ordered- and shuffled-LTCL settings.
strength_weakness: strength_weakness: 1.	The elaboration of the paper is clear and reasonable. The authors make it easy to understand the problem of predicting opioid overdose.
2.	The related work does help readers to understand the essence of the idea and to follow the work easily.
3.	The experimental results are well presented and prove that the model is indeed effective.
limitations: limitations: 1.	The meaning of Figure 4 is not very clear, why the position near the edge is the task boundary?
2.	The experiment lacks a comparison with the latest methods from 2023 and 2024. I believe that comparing with more recent methods can better highlight the progressiveness of the method proposed in this paper.
3.	In Table 2, ER is better than the proposed method when the buffer size is 500 on the Seq-CIFAR-10-LT (Class-IL). It is necessary to analyze the reason.
4.	If the data is balanced, will the performance of the proposed method also be better than other baselines? Strictly speaking, whether a data is balanced also belongs to prior knowledge. If your method is superior to other baselines even in data balancing, then I believe that your method is indeed not based on prior knowledge. Otherwise, this method is based on prior knowledge.
paper_summary: paper_summary: The author, through experimental analysis, identified three major drawbacks associated with prior methods for addressing long-tail continuous learning:
1.  The tendency for minority data to be rapidly forgotten.
2.  The propensity for classifier weights to be biased towards the old majority class.
3. The increased likelihood of minority class samples near the decision boundary being forgotten relative to majority class samples in the process of learning following the introduction of new data.

To address these issues, the author introduced the Prior-free Balanced Replay (PBR) framework. This framework enhances the model's memory capacity and mitigates the problem of forgetting. Furthermore, to specifically counter the aforementioned problems, the author proposed the Prototype Constraint module and the Boundary Constraint module. These modules are designed to alleviate the issues of forgetting and bias. 

Ultimately, extensive experimental analysis confirmed that the proposed method yields satisfactory results in both ordered- and shuffled-long-tail continuous learning tasks.
strength_weakness: strength_weakness: 1. The discourse logic of the article is notably lucid. Firstly, the issue is identified through experimentation, followed by the proposition of a solution for the issue, and finally, the efficacy of the method is substantiated. This organizational structure is exceedingly clear.
2. The Prior-free Balanced Replay method suggested by the author displays a certain degree of innovativeness.
3. When compared to other methodologies, the performance of the method in this article on the ordered-LTCL task is notably enhanced.
limitations: limitations: 1. The study asserts in the abstract and introduction that the proposed method exhibits superior performance on ordered-LTCT and shuffled-LTCL tasks. However, the experimental section only provides results related to the shuffled-LTCT task in Table 3, with all other experiments focusing on ordered-LTCT. This does not adequately demonstrate the method's effectiveness on the shuffled-LTCL task.
2. The Prior-free Balanced Replay (PBR) framework, proposed by the author, aims to alleviate the problem of forgetting and enhance the model's performance on older data. In the context of the ordered-LTCT task, the head class is the first to be encountered, meaning that the model's initial interaction is with the highest-performing class. Consequently, one would expect improved performance on the ordered-LTCT task after employing the PBR method. Nevertheless, to substantiate the method's efficacy in mitigating tail class forgetting, the author should conduct ablation and comparative experiments on the shuffled-LTCL task, which would further substantiate the method's effectiveness.
3. The majority of the comparative methods employed in the experiments and the studies followed by this article are from papers published prior to 2022, thus lacking novelty.
paper_summary: paper_summary: In continual learning (CL), catastrophic forgetting is a major issue, especially with long-tailed data distributions (LTCL). Traditional methods need label distribution for re-balancing, which is impractical in real scenarios. The authors propose a Prior-free Balanced Replay (PBR) framework that uses an uncertainty-guided reservoir sampling strategy to prioritize minority data without prior information. The framework also includes boundary and prototype constraints to further reduce forgetting.
strength_weakness: strength_weakness: 1. Catastrophic forgetting is a practical and noteworthy problem.
2. The proposed method seems to show a significant improvement compared to the baseline algorithms discussed in the paper.
limitations: limitations: 1. In Figure 1, the long-tail distribution appears to refer to the distribution of classes, while in the Problem Formulation, the long-tail distribution seems to pertain to the number of samples for each task. This discrepancy may appear inconsistent.

2. What is the rationale behind requiring the sample counts for each task to be arranged in descending order? This assumption appears excessively restrictive.

3. What is the rationale behind assuming that the long-tail distribution adheres to an exponential distribution? This assumption also seems overly restrictive.

4. Equation 1 should be formulated in terms of empirical loss for given finite sample datasets, rather than using the expectation symbol. This principle should be followed for subsequent equations, such as Equation 2.

5. For Figures 2, 3, and 4, despite their role in elucidating the motivation, the authors do not present evidence of the proposed method's effectiveness in addressing these issues.

6. A number of the methods compared appear to be outdated, predominantly predating 2020, while the most recent ones are from 2021. The authors should consider evaluating against more recent and competitive methods.