Continual Learning on a Diet:  Learning from Sparsely Labeled Streams Under Constrained Computation
Authors: Wenxuan Zhang
Youssef Mohamed
Bernard Ghanem
Philip Torr
Adel Bibi
Mohamed Elhoseiny
Venue: ICLR 2024 poster (2024)
Keywords: Continual Learning
Semi-supervised Continual Learning
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We benchmark the current continual learning under sparse label and constraint computation in large scale dataset, and propose a efficient solution to such settings.

Abstract: We propose and study a realistic Continual Learning (CL) setting where learning algorithms are granted a restricted computational budget per time step while training. We apply this setting to large-scale semi-supervised Continual Learning scenarios with sparse label rate.  Previous proficient CL methods perform very poorly in this challenging setting. Overfitting to the sparse labeled data and insufficient computational budget are the two main culprits for such a poor performance. Our new setting encourages learning methods to effectively and efficiently utilize the unlabeled data during training. To that end, we propose a simple but highly effective baseline, DietCL, which utilizes both unlabeled and labeled data jointly. DietCL meticulously allocates computational budget for both types of data. We validate our baseline, at scale, on several datasets, e.g., CLOC, ImageNet10K, and CGLM, under constraint budget setup. DietCL outperforms, by a large margin, all existing supervised CL algorithms as well as more recent continual semi-supervised methods. Our extensive analysis and ablations demonstrate that DietCL is stable under a full spectrum of label sparsity, computational budget and various other ablations.

Published as a conference paper at ICLR 2024
CONTINUAL LEARNING ON A DIET: LEARNING FROM SPARSELY LABELED STREAMS UN- DER CONSTRAINED COMPUTATION
Wenxuan Zhang', Youssef Mohamed', Bernard Ghanem', Philip H.S. Torr
Adel Bibi Mohamed Elhoseiny
KKig Abdullah University of Science and Technology
2 University of Oxford
(wenxuan. zhang, youssef mohamed, bernaadd.ghnnen)@kaal
[philip.torr, adee. bibi Beng ox. ac. uk, mohamed. www.haamme..lhhssiiiealaaaa...
ABSTRACT
We propose and study realistic Continual Learning (CL) setting where learn- ing algorithms are granted a restricted computational budget per time step while training. We apply this setting to large-scale semi- cani-superviee Continual Learn- ing scenarios with sparse label rate. Previous proficient CL methods perform very poorly in this challenging setting. Overfitting to the sparse labeled data and insufficient computational budget are the two main culprits for such a poor performance. Our new setting encourages learning methods to effectively and efficiently utilize the unlabeled data during training. To that end, propose simple but highly effective baseline, DietCI., which utilizes both unlabeled and labeled data jointly. DietCL. meticulously allocates computational budget for both types of data. We validate our baseline, at scale, on several datasets, e.g.. CLOC, ImageNet10K, and CGLM, under constraint budget setup. DietCL outperforms, by a large margin, all existing supervised CL algorithms as well as more re- cent continual semi-supervised methods. Our extensive analysis and ablations demonstrate that DietCL is stable under a full spectrum of label sparsity, compu- tational budget and various other ablations. Our code is available here: ht https: / //githubs com wx- zhang/ c maaaalllearnnne
a-diet
INTRODUCTION
In the era of abundant information, data is not revealed in its entirety but rather sequentially from a non-stationary environment. For example, social media platforms, such as YouTube, Tok,ok, Snapchat, and Facebook, receive huge amounts of data every day. The content of the data and its distribution depend greatly on social trends and focuses on existing platforms (e.g., Facebook, Snap, Twitter), thus shift over time. For instance, Snapchat, in 2017, reported the influx of over 3.5 billion short videos daily from users across the globe (Snap). These videos had to be instantly processed for various tasks, from image rating and recommendation to hate speech and misinformation detection.
Continual learning attempts to address such challenges, focusing on designing training algorithms that accommodate new data streams while preserving previously acquired knowledge. Diverse so- lutions have emerged, spanning from regularization-ber based (Kirkpatrick et al., 2017), architecture- based (Ebrahimi et al., 2020), to memory-base methods (Chaudhry et al., 2019b).
Nevertheless, the huge scale of data in most practical applications needs to be processed in real time. Such constraint imposes budget limitations the continual learning algorithms. To better demonstrate these constraints, imagine a learning algorithm takes 10 days to learn from a dataset of 3.5 billion samples (accumulated in single day for Snapchat). The ongoing data stream would have generated 35 billion new samples during the training period. This reality renders the model severely outdated by the time it's put to practical use. Such time constraints on processing pose limitations not only on the number of labels but also on the computation time for the algorithm.
"Equal Advising

---

Published as a conference paper at ICLR 2024
Unlabeled
Labeled Data
f
Data
|
Point
Annotation Budget
Budget
Allocation
Computational Budget
Time
Time to1
Figure I: DietCL considers the computation budget due to effective computational time restrictions and very sparse label rate due to annotation cost. At each time step, we propose allocate sufficient computation for labeled data and utilize the diverse unlabeled data with remaining computation to migrate the overfitting.
Existing literature has recognized the problem and put effort into finding solutions. While some online continual learning approaches attempt to formulate the online data stream (Chaudhry et al., 2019a; Cai et al., 2021), they prioritize the formulation in batch-wise training and evaluation, and often overlook the regularization of computation and time in these algorithms. Additionally, the majority of online continual learning works assume that a full set labels accessible. On the other hand, certain advancements offline continual learning have been achieved by incorporating unlabeled data (Pham et al., 2021; Fini et al., 2022). These approaches often require a full pass of the unlabeled data, thereby neglecting the expensive computational demands associated with large amounts of unlabeled data. Only recently, some work started to consider budgeted continual learning (e.g., (Prabhu et al., 2023)), which aims to regularize the computational requirements for each task to enable the applicability of continual learning algorithms under the aforementioned wolld scenarios. Nevertheless, these endeavors still concentrate on fully labeled data stream.
In this work, we extend the budgeted continual learning to semi-supervised manner. This scenario marked by constrained computational resources and sparse labeling; hence we term "CL on Diet". The crux of the challenge is illustrated in Figure I, where substantial data volumes are un- veiled during each time step; however, only a fraction of this data accompanied by labels. Subse- quently, the learning algorithms train on these data under the constraint of time of computational resources, contingent on the time interval of data reception.
Under CL on Diet, first study the capacity of existing methods cope with such challenging setting. Indeed, our findings reveal that current solely supervised methods tend to overfit to the limited labels available. On the other hand, approaches utilizing unlabeled data require extensive computational resources to perform multiple full passes over the vast unlabeled dataset, leading to significant degradation in performance when computational restrictions are imposed.
To address these challenges, we present our effective baseline, named DietCL.. It incorporates a computation budget allocation mechanism harmonize the learning of current and prior distribu- tions. Moreover, we introduce a unified training strategy that assimilates insights from labeled and unlabeled data concurrently. The efficacy of our baseline substantiated through evaluations on prominent large-scale continual learning datasets, namely ImageNet10k, CLOC, and CGLM. Our results showcase state-of-the-art performance this realistic scenario. Additionally, we demon- strate the robustness of our baseline across varying stream lengths, computational constraints, and label rates. Our contributions can be summarized in three folds:
1. We proposed challenging large-scale remi-supervised continual lcarning setting, dubbed "CL on Diet", under sparse label scenario and constrained computation budget. We explore problems of the existing methods in this setting.
2. We propose DietCL, a continual learning baseline that utilizes budget efficiently with joint optimization of labeled and unlabeled data.
3. We conduct extensive experiments in large-scale datasets, ImageNet10k, CLOC, and CGLM data and class incremental settings. Our experiments demonstrate that our sim- ple solution, DietCL, outperforms both supervised and semi-supervised continual learning
2

---

Published as a conference paper at ICLR 2024
algorithms in sparsely labeled streams by 2% to 4%. We also show superior this baseline in varying levels of stream length, label ratio, computational budget.
2 RELATED WORK
Semi-supervised Continual Learning. Following the recent success of self-supervised learning for pretraining, an increasing body of work is now studying their application for continual learning. Caccia & Pineau showed that self-supervised loss outperformed supervised loss in the meta-learning stage for continual learning. Fini et al. (2022) performed self-supervised pretraining for each task in offline continual learning. Gomez- Villa et al. (2022); Pham et al. (2021); Boschini et al. (2022) used unlabeled data for distillation or regularization loss to migrate forgetting. Most approaches validate their methods small datasets with moderate labeled-unlabelled split. They do not consider the computational expense incurred, and often treat labeled and unlabeled data points equally in computation allocation. When we scale up these approaches cope with the vast amount of real- world unlabeled data and sparse labels, as we shall show in section 3.2, we observe difficulties in learning meaningful label-related information due their limited computation allocated to labeled data. Additionally, these approaches struggle to learn from unlabeled features due to the constraints of total available computation.
Scenarios and Budget Constraint in Continual Learning. Conventional continual learning ba- sically focuses on task incremental learning, class incremental learning, and domain incremental learning. Due to the diversity of the real-world data flow, there has been lot of work exploring how to eliminate specific constraints in continual learning (Wang et al., 2024), thus making them applicable. Some work considers releasing the task boundary constraint and explores algorithms with unknown boundaries (Aljundi et al., 2019) or blurry task boundaries (Koh et al., 2021). Others consider cases where data is the fly, and tasks arrive 2019a). Most of these considerations start from the format of data arrival. In this paper, however, we consider constraints originating from the relation between the processing agent and the data flow.
one-pass data stream (Chaudhry et al.,
Only from recent, a question was posed regarding the time limit for training in continual learning. It is demonstrated that if each task can be trained for an unlimited amount of time, a non-continual algorithm can achieve comparable results with continual learning algorithms (Prabhu et al., 2020; Ghunaim et al., 2023). While some online continual learning methods (Koh al., 2021) report performance based on a fixed number of updates, this is mainly for fair comparisons and not fully explore the impact of the training budget on the algorithm. Recent work (Prabhu al., 2023) has demonstrated the effectiveness of offline continual learning under limited budgets, showing that learning from balanced distribution helpful when budgets are insufficient. Motivated by this, we propose to impose constraints on the training time for semi-supervised continual learning. However, in our work, the budgeted setup is in conjunction with the sparse labeled setting of the stream, which poses novel more challenging but realistic problem.
3 CONTINUAL LEARNING ON A DIET
3.1 PROBLEM FORMULATION
In the semi-supervised continual learning under budget, we seek to learn a function fo : X' - y parameterized by 0 that maps images X' to class labels y E y. At each time step f, the stream samples n² images { i=1 X and then only reveals the ni labels of them fe. In contrast to prior work, continual learning algorithms seek to update the parameters 0 with per time step predefined computational budget such that fo performs well on all seen distributions. Throughout this paper, we define the computational budget in terms of the total FLOPPs normalized in terms of the number of forward-backwart passes, i.e., the training iterations for a given batch size. The computation budget amounts to all FLOP iterations the training requires, from forward- -bakwadd pass updating model parameters to any other operations, e.g., importance weights as in Aljundi et al. (2018). We follow Prabhu et al. (2023) for the storage assumption in budgeted continual learning, where the buffer is large enough to store all labeled data while the amount of data used every time is constrained according to the computational budget.
3

---

Published as a conference paper at ICLR 2024
Awerape Accuracy
17
(a)
15
15
10
DietCL
ER
CoSSLe
5-
400 600
800
500 1000
1500
2000
250d
Computational Budget
Computational Budget
Figure 2: Average accuracy of ER, Ca and DietCL on 1% labeled ImageNet with varying computational steps. Left: supervised method, ER, starts to overfit after 400 steps. Right: semi- supervised method, CaSSLe, converges slowly. DietCL converges fast and alleviates overfitting.
3.2 OPPORTUNITIES FOR IMPROVEMENT
Most prior supervised and semi-supervised continual learning works presume sufficient computa- tional resources and labels they need. Nevertheless, such assumptions may not always be satisfied in some real-world scenarios, including the diet scenario that we propose in this paper. In this sec- tion, explore the potential bottleneck of existing supervised method, ER, and semi-supervised method, CaSSLe, in diet scenario with experiments of varying computational budget per time step on 20-split ImageNet1OK,, as shown in Figure 2. We compare their performances against our pro- posed algorithm, DietCL, which shall be introduced in Section 4. Each point of the figure shows the average accuracy the end of continual learning stream following Chaudhry et al. (2019a), given the corresponding time step budget.
How does Supervised CL behave under Low Budget? As shown in region (a) in Figure left where the per step computational budget is less than 400, ER has large performance degradation as the budget goes down. This is partially due to the stability gap (Lange et al., 2023) in ER; during the learning of the new task, the model overfits to the new data first and then recovers knowledge Appendix C.I. In the low-budget scenario, the training for the next time step could start before the finish of previous knowledge recovery. Furthermore, the limited number of available labels can also lead to the model capturing narrow distribution of the current task. Our remi-supervine continual learning method can effectively eliminate overfitting and stability gap issues with the unlabeled data as a regularizer, resulting in improved continual learning performance.
from replayed data of previous tasks. See more details this phenomenon
Can Supervised CL Fully Utilize the Available Computational Budget? Supervised CL not only struggles to learn efficiently under limited budget, but suffers overfitting, particularly with large computational budget, as illustrated in region (b) in Figure 2 when the budget exceeds 400. However, the behavior is different in the semi-supervised method, where overfitting does not happen much. This encourages us to effectively spend the redundant computational resources on unlabeled data.
Is Unlabeled Data Necessary? We demonstrate that even when the computational budget does not reach the maximum capacity for a supervised learning algorithm, e.g., in Figure 2 left, region (a) when the budget is less than 400, we can achieve better results by leveraging unlabeled data to im- prove generalization when labeled data is limited. Additionally, when the budget is far from enough, e.g., as Figure left region (b) when the budget is bigger than 400, a purely supervised continual learning algorithm may not fully utilize the computational budget. In this scenario, allocating extra budget to unsupervised data can help capture the current distribution.
Challenges Facing CaSSLe. Learning from unlabeled data can be computationally expensive, CaSSLe, as an instance, originally proposed to train for 500 epochs per task. Our experiments reveal how CaSSLe greatly suffers with low budgets in large-scale datasets. As shown in Figure right, the accuracy of CaSSLe remains very low even when we increase the budget to 2500 steps (around 3 epochs). In contrast, the required budget for DietCL to converge is only about 500 steps.
4 PROPOSED SOLUTION: DIETCL
We now present approach that leams from labeled data and unlabeled data jointly to efficiently use budget and capture the shifting distribution. Throughout, we assume that we learn a model fe which can be decomposed feature encoder fo. X - 2, where 2 is the feature space, and
4

---

Published as a conference paper at ICLR 2024
classification head fa. 2 + V, Le., fo for o fe.- The learning of each task is constrained with total budget = Bu + Bt, where B, is the budget for unlabeled data and Bt is the budget for labeled data.
Learn In Distribution Relationship. In a large-scale data stream, cach time step corresponds to a diverse distribution, which requires significant effort to learn the in-distribution relationships. To achieve this, we take use of unlabeled data from the current distribution by self-supervised learning (SSL). We discuss how integrate SSL and which type of SSL integrate in CL now
SSL pre-training can be expensive in CL. as observed in Figure (right). Upon further examination in Appendix C.2, we found that the features learned purely from unsupervised loss evolve extremely slowly and are not oriented towards the labels. To address this, we jointly learn from the current unlabeled and labeled data, where the dense unlabeled data acts as a regularizer to prevent overfitting to the sparse labeled data. In terms of the specific SSL algorithms, contrastive learning and masked modeling are two mainstreams. Contrastive learning algorithms typically necessitate augmenting input images into dual views and updating gradients for two distinct backbones. Consequently, with a budget of B,, for unlabeled data, only B./2 gradient steps are feasible, leading to a significant budget underutilization. As such, we employ an efficient masked modeling method, MAE (He et al., 2022), to capture the current distribution by reconstructing the masked patch of input image. We add reconstruction head to the encoder, denoted as fe. Z - X, which maps the encoded features to the image space. At each time step L, we use the unlabeled stream samples {or the reconstruction loss
) iimnt1 to compute
Σ Elle o fo. (mw)) - (p((r))]"
C
(1)
1=n'43 PETP
where the operator dhp extracts the pdh patch from the set of masked patches I from every Tu-
Furthermore, proposed by Caccia al. (2022) that continual learning benefits from learning the current distribution in isolation, we mask out the logits of classes that are not shown in the current time step and compute the following loss function on the labeled data from the current distribution:
Cm
Σ ((
CE(fo, I' (21)). vi).
where 7' indicate mask function that zeros out all indices of classes that are not introduced in the current time step and here CE indicates a cross entropy loss.
To overcome the forgetting problem, we further maintain task balanced buffer M that contains only labeled data from the current and previous time steps. The loss can expressed as follows:
La bm
Σ CB(fo(z.),..)
(3)
(11-81)~-
To that end, our final joint loss function to train under budgeted sparsely labeled stream is:
C a,C,+ L. + Lo.
Here a, is the scaling factor adjusting the mean square loss L to match the scale of the entropy loss Lm. In the experiment, we fix it to 50.0.
Budget allocation. In our final loss function Equation 4, the model is trained on the unlabeled data from the current time step (C,), labeled data from the current time step (Lm),, and labeled data from the balanced buffer (Ls), all jointly. As shown in Figure 2, classic methods have two stages as the budget goes up. iee. learning stage in the region (a) and overfitting stage in the region (b). The main reason for the overfitting with the second stage is the overuse of the very sparse labeled data of the current task. Therefore, we allocated different training budgets to each data source according to the total budget. We divide equally among the labeled (C..,, unlabeled (L,), and buffer data (Ls) if the budget is less than a threshold B. Our algorithm converges quickly in this budget, and extending the training duration does not yield significant improvement for current classes. Consequently, we opt to invest the extra budget solely on the buffer data with Equation 3 as our loss function. This stage mainly balances the learning of current and previous classes; refer to Appendix C.3 for detailed analyses. In practice, we choose the threshold B by cross validation; more details are in Appendix B.3. Please refer to Appendix for the overall training procedure and semantic code.
5

---

Published as a conference paper at ICLR 2024
EraggNettO0
20-
toperese
MAS
10-
Replay
CDumb
L2P
DRACCE
10-
DualNet
20-
CCIC
DieCCL
Training
Tieas
Tiaes
Figure 3: Accuracy at each time step of the baselines on ImageNet1Ok, CLOC, and CGLM dataset. Our algorithm surpasses the supervised methods by using unlabeled data and outperforms semi- supervised methods due to effective allocation of budgets.
5 EXPERIMENTS
In this section, we conduct experiments on various large-scale datasets with our proposed setting, CL on Diet, that the stream is partially labeled and algorithms are granted limited computational budget per time step. We start with explaining the experimental setup, then we present the com- parison of our proposed DietCL. against several other methods. At last, we demonstrate that our proposed approach is robust under varying label rate, computational budget and stream length.
5.1 EXPERIMENT SETUP
Benchmark and Metrics. We use three large-scale continual learning datasets, ImageNet 10,, CLOC, and CGLM following Prabhu et al. (2023) to evaluate the performance of DietCL along with other methods. We split ImageNet1Ok into 20 class-incremental tasks and split CLOC and CGLM into 20 tasks according the image uploading time shown in the meta information of each image. All the detailed statistics of the datasets can be found in Appendix B.I.
We report the accuracy in the last time step and the average accuracy across the stream. That is to say, let a, = Ei A E, [/((z()=v( J. where n is the number of samples per time step, be the accuracy of the model trained after time on the valid data until time Then, we report the accuracy of the union test sets in the last time step I as A(T) ar following Chaudhry et al. (2019a); Fini et al. (2022); Wang et al. (2022). We further report the averaged performance AA ΣT= a to show the trend along the whole stream, following (Douillard et al., 2022).
the VIT model (Dosovitskiy al., 2020) for classi- fication and the MAE decoder (He et al., 2022) for reconstruction. Both models are pre-trained on the ImageNetlk dataset, released by (He et al., 2022). We set the base learning rate 10 with base batch size of 256, and scale the learning rate linearly according to the effective batch size. We use multiple Nvidia 1000 GPUs for training, with batch size 256 each device. We accumulate the loss and perform the backward step when the accumulated batch size reaches 1024. All other learning parameters are adopted from (He al., 2022).
Training. Throughout all experiments,
Baselines. We compare against supervised and semi- supervipee continual learning methods in our setting. Supervised methods include ER (Chaudhry et al., 2019b; Prabhu al., 2023), EWC (Kirk- patrick et al., 2017), MAS (Aljundi et al., 3018), GDumb (Prabhu et al., 2020). L2P (Wang et al., 2022). Semi-supervised methods include CaSSLe (Fini et al., 2022), DualNet (Pham et al., 2021), and CCIC (Boschini et al., 2022). Implementation details of all these methods are left for Ap- pendix B.2. We use balanced sampling for ER; discussions of sampling is in Appendix B.2. We also compare against the joint training, where pre-training and fine-tuning are performed on the whole dataset, following (Hu et al., 2021) to show the upper bound. The per- per-time-step computational budget for all baselines is the same.

---

Published as a conference paper at ICLR 2024
Table 1:
Accuracy at the last step of the continual learning sequence. DietCL shows superior perfor- mance over the supervised methods when evaluating both the last model and the whole sequence.
ImageNet10k
CLOC
CGLM
A(T)
A(T)
A(T))
A
Upper Bound
18.23
8.56
38.15
EWC (Kirkpatrick et al., 2017)
2.19
7.69
1.83
2.28
6.16
9.28
MAS (Aljundi et al,, 2018)
6.73
11.81
1.65
2.03
1.73
3.47
Replay (Chaudhry et al. 2019b)
14.89
22.84
4.64
4.59
22.81
19.40
GDumb (Prabhu et al,, 2020)
4.64
16.25
3.15
3.3
8.81
10.11
ERAACE (Caccia 2022)
10.72
21.28
4.57
4.53
21.55
19.28
1.2P (Wang et al., 2022)
6.30
10.23
2.68
2.43
16.69
13.57
DualNet (Pham et al., 2021)
0.50
0.92
0.80
0.83
5.20
3.97
CaSSL e (Fini et al., 2022)
5.78
13.25
1.52
1.60
7.10
6.67
CCIC (Boschini al,, 2023)
0.20
0.62
1.32
0.72
4.27
4.02
DietCT
16.82
24.90
5.63
4.98
24.34
20.26
5.2
MAIN RESULTS
We conduct experiments on 20-split ImageNet with 1% label rate and 500 computational steps, 20-split CLOC with 0.5% label rate and 1000 computational steps, and 20-split CGLM with 5% label rate and 600 computational steps for main comparison. The results of DietCL and the base- lines described in Section 5.1 are shown in Figure 3 and Table 1. The figure shows the evaluation accuracy A(t) at each step, and the table shows the accuracy of the last step A(T) and the averaged performance A.
DietCL against Supervised CL. The first row in Figure 3 shows the comparison between DietCL and supervised CL methods. DietCI. consistently outperforms all the supervised methods we com- pared across all datasets, namely ImageNet10K, CLOC, and CGLM. The most competitive method among the supervised ones Replay, which involves jointly utilizing current labeled data and pre- viously labeled data uniformly sampled from memory for supervised training. However, the incor- poration of unlabeled data in DietCL significantly enhances performance, resulting in accuracy of 16.82% 5.98%, and 24.34% on the respective datasets, as shown in Table 1.
This demonstrates that DietCL can effectively leverage the unlabeled data even under the restricted per step computation towards improving the learning in the stream.
DietCL against Semi-Supervised CL. The second row in Figure shows the comparison of Di- etCL against recent semi-supervised continual learning methods and SSL upper bounds. These semi-supervised methods were originally proposed without computational constraint, and encounter large performance drop with fixed computational budget. Among them, as shown in Table 1, the best performing, CaSSLe, achieves only 5.78% Average Accuracy at the last task on ImageNet dataset compared to ours of 16.82%. CaSSLe originally proposed to perform 500 epochs of unla- beled training on their evaluated benchmarks, which roughly equals 50,000 steps in our setting. Our limited 500-step budget causes their performance collapse. Furthermore, when evaluating semi- supervised baselines in our large-scale dataset, we find some methods, that rely on the inter-class relationship, such DualNet and CCIC, struggle to cope with such large number of classes. At the same time, as shown in Figure 3, our fine performance on the three datasets is much closer to the joint training performance than other semi-supervised methods.
Learning trend. In the class-incremental benchmark of ImageNet1Ok, which has around 10K classes, the model required learn about 500 new classes at each time step. This poses a signifi- cant challenge for most methods to remember the previous massive classes while learning this large amount of new classes. While the time-incremental benchmark of CGLM and CLOC, the data volume and content of each class vary over time. This requires the model to learn the essence of the category rather than overfitting to the most recent training distribution to overcome the inference bias in the recognition of previous tasks. The overall learning trend of various methods can be seen from the slope of the curves in Figure and the A in Table 1. Notably, certain methods depicted in the Figure, such EWC, MAS, and CaSSLe, experience significant drop in accuracy in the first several tasks, while DietCL avoids large performance declines and gradually enhances accuracy in the CLOC and CGLM datasets. In the table, our method still maintains the highest A on all three
7

---

Published as a conference paper at ICLR 2024
160 Graddenn Stege
500 Senclent Steps
1300 Gradiitt Steps
2500 Gradient Stops
40
itnal Aoc. 4.76
50
Final Ace:
34
Pinal Ao0: 5.30
-00
10
20-
20-
10
16
15
20
10
10
20
10
155
Time Stop
TTuze Step
Siuue Stap
Figure 4: Varying the Computation per Time Step. Accuracy DietCL, CaSSLe, and GDumb with different computational steps at each time step in 1 % ImageNet10k. The top right boxes show the average accuracies DietCL in corresponding computational step settings.
0.5% Label-Rate Segronoe
1h: abbil-Stte Serperccnn
5%s Label-Rate Serpanoee
10% Label-Botte Sequoore
30-
Flnal Arc: 13.09
Piaal Aotal 16.82
Final Are: 21.56
60
Pinal Mot. 21
30
20
20
10
10
20
10
10
10
Time Stop
Time Stare
Time Step
Time Stere
Figure 5: Varying the Label Rate per Time Step. Accuracy of DietCL, CaSSLe, and GDumb in data streams with different label rates in 20-split ImageNet10k, with 500 computational steps for each experiment. The top right boxes show the average accuracy of DietCL in corresponding label settings.
10 Timee Stages
20 Tinse Staps
all Taan Stags
1:00 Tame Slapel
Final Acei: 17:31
hoc.:
Final Ace: 14.06
Vindl Ami: $2.05
40
20-
20-
10
10
10
15
10
130
111
400
700
20
400
800
20
100
Time Stop
Timeo
Timnc
Ticre Stop
Figure 6: Varying the Number of Time Steps. Accuracy of DietCL, CaSSLe, and GDumb with different number of time steps in 1 1% labeled ImageNet10k. We keep the total computational bud- get the same for each experiment. The top right boxes show the average accuracy of DietCL in corresponding computational time step settings.
datasets, which 24.9%, 4.98%, and 20.26%. We believe this is due the proper allocation of budgets on current and previous data, which allows our method to efficiently leverage the available resources and adapt to the new classes.
Taking the Snap example we mentioned the introduction, we further connect the results to the real-world problem Appendix D.
5.3 ABLATING EQUATION 4
We conduct ablation study of Equation 4 under 500 computational steps in 1 % labeled 20-split ImageNet10K benchmark in Table 2. We start with a Replay baseline. and show the effect by adding the masked classification loss, the bal- anced buffer, and the reconstruction loss. By adding masked classification loss, the final ac- curacy increased by 0.9%. By maintaining the balanced buffer, the final accuracy increased by 0.13% With the help of unlabeled data, the fi- nal accuracy increased by 0.86%. This demon- strates that the learning of current distribution
Replay
Lm
Balanced Buffer
C,
Avg. Acc.
15.83
15,96
16.82
Table Ablation study of our loss function Equa- tion 4. We use 1% labeled stream with 500 com- putational steps on ImageNet1Ok.
8

---

Published as a conference paper at ICLR 2024
in isolation, both with masked loss and with unlabeled data, contributes most to the scenario with limited training budget. We perform ablation with other orders in Appendix B.4
5.4 COMPUTATIONAL BUDGET AND LABEL RATE
We study the stability DietCL along with the most robust supervised method GDumb and the best performed semi-supervised method CaSSLe on ImageNerl0k against varying level of label rate, computational budgets, and stream length.
Varying the Computational Budget. We conduct experiments with varying the computational budget (100, 500, 1500, 2500) iterations per time step. Across all following experiments, we keep all other parameters unchanged as before, i..., a label rate of 1% for 20 time steps. The results are summarized in Figure 4.
When the budget is reduced to 100 iterations per step, in performance across all methods. This effect is most pronounced in the case of the supervised method, GDumb, highlighting the minimum resource requirements for achieving satisfactory perfor- mance in supervised learning is relatively large. However, it's worth noting that DierCI. outperforms the other two methods even under this tight budget constraint, and maintains a stable performance across various budgets, including the scenario of a per-time-step budget of 2500 iterations. Al- though 2500 iterations might be considered large for DietCL to converge, it remains a constrained setup. particularly in comparison to semi-supervisee ously mentioned, CaSSLe initially proposed a training regimen involving 50,000 steps, which is 20 times larger than the maximum number of steps we examined. Consequently, expecting CaSSLe to complete its training within just 5% of its original budget is clearly an impractical expectation.
clearly observe significant decline
learning approaches like CaSSLe As previ-
Varying Label Rate. Figure 5 shows experiments with various levels of sparsity of the of labeled samples in the data stream, i.e., 0.5%, 1%. 5%. 10%. We keep the computational budgets to 500 steps per time step over 20 time steps. We observe that when raising the label rate from 0.5% to I 1%, and from 1% to 5%, we can see obvious increment of the average accuracy in our method, although the computational budget remains the same. However, the performance of GDumb and CaSSLe do not improve a lot. Additionally, the performance gaps between our methods and others are also becoming large, which indicates that our use of labeled data is more efficiently thanks to unlabeled data. When we involve more labeled samples from 5% to 10% in our method, where the budgets are insufficient for the later stream, we do not observe the performance improvement. We conclude that when the budget is sufficient, the labeled data can serve as a better guide for the unsupervised training. and thus allows for improved performance.
Varying the Number of Time Steps. We examine the role of the length of the stream on Ima- geNet1Ok with experiments on the spectrum of 10. 20, 50, and 100 time steps. This mimics the speed at which data is presented. We use label rate 1% for these experiments. Among the experi- ments with different stream length, we keep the total number of computational iterations identical to previous experiments by proportionally reducing increasing the per time step budgets accordingly, so as the total iterations are equal to 20 x 500. The results are shown in Figure 6. With the same number of passes of the labeled data, the average accuracies of DietCL of over the sequence similar, i.e., 24.4%, 24.9%, 23.7%, 22.2%. However, when the task length is longer, both GDumb and CaSSLe have worse performance. This implies that method is capable of dealing with small-batch high-frequency streams compared to previous methods.
6 CONCLUSIONS
We rethink the computational budgets and the sparsity of labels in the real world, which did not get much attention in previous continual learning works. To solve this challenge, we designed an efficient and effective continual learning algorithm, DietCI. that trains the model with labeled and labeled data jointly. We benchmark the setting with large-scale benchmarks ImageNet ImageNet2k, and CGLM. Our method surpasses classic methods and achieves double of the average accuracy of those methods. We also show the superior performance of our method on two other hard benchmarks and analyze the influence of computational budget, the length of the data stream, and the sparsity of the labeled data on the method. We believe DietCL can serve as a starting point towards exploring new CL. algorithms under limited computational budget and sparse labels.

---

Published as a conference paper at ICLR 2024
REFERENCES
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Mareus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. European Conference on Computer Vision (ECCV), 2018.
3, 6, 7,
Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars. Trsk-free continual learning. In IEEF/CVF Confer- ence on Computer Vision and Puttern Recognition (CVPR), 2019.
Matteo Boschini, Pietro Buzzega, Lorenzo Bonicelli, Angelo Porrello, and Simone Calderara. Continual semi- supervised learning through contrastive interpolation consistency. Pattern Recognition Letters, 2022, 3, 6,
Lucas Caccia and Joelle Pincau. Special: Self- Self-supervised pretraining for continual learning. Semi-Supervised Learning, pp. 91.
Lucas Caccia, Rahaf Aljundi, Nader Asadi, Tinne Tuytelaars, Joelle Pincau, and Eugene Belilovsky. New insights on reducing abrupt representation change in online continual learning. In International Conference on Learning Representations (ICLR), 2022. 5,7, 7,
Zhipeng Cai, Ozan Sener. and Vladlen Koltun. Online continual learning with natural distribution shifts: An empirical study with visual data. In International Conference on Computer Vision (ICCV), 2021. 2, 12
Arslan Chaudhry, Mare' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learn- ing with a-gem. In International Conference on Learning Representations (ICLR), 2019a. 2, 3, 4. 6. 12, 14
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K. Dokania, Philip H. S. Torr, and Mare" Aurelio Ranzato. On tiny episodic memories in continual learning. In MTLRL Workshop, International Conference on Machine Learning (ICMLW). 2019b. 1, 1, 13
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition (CVPR), 2009. 12
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. In International Conference on Learning Repre- sentations (ICLR), 2020. 6
Arthur Douillard, Alexandre Ramé, Guillaume Couairon. and Matthieu Cord. Dytox: Transformers for con- tinual learning with dynamic token expansion. In IEEE/CVF Conference on Computer Vision and Putlern Recognition (CVPR), 2022.
Sayna Ebrahimi, Franziska Meier, Roberto Calandra, Trevor Darrell, and Marcus Rohrhach. Adversarial con- tinual learning. In European Conference on Computer Vision (ECCV), 2020. 1
Enrico Fini, Victor Turrisi da Costa, Xavier Alameda-Pineda, Elisa Ricci, Karteek Alahari, and Julien Mairal. Self-supervised models are continual learners. In JEEF/CVF Conference on Computer Vision and Pottern
Recognition (CVPR), 2022. 2, 3, 6, 7, 12, 14
Yasir Ghunaim, Adel Bibi, Kumail Alhamoud, Motasem Alfarra, Hasan Abed Al Kader Hammoud, Ameya Prabhu, Philip HS Torr, and Bernard Ghanem. Real-time evaluation in online continual learning: A new hope. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 11888-11897, 2023.
Alex Gomez-Vill Villa, Bartlomicj Twardowski, Lu Yu, Andrew D. Bagdanov, and Joost van de Weijer. Contin- ually learning self-supervised representations with projected functional regularization. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2022. 3
Kaiming Hc, Xinlci Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 5, 6, 14, 16
Dapeng Hu, Shipeng Yan, Qizhengqiu Lu, HONG Lanqing, Hailin Hu, Yifan Zhang, Zhenguo Li, Xinchao Wang, and Jiashi Feng. How well does self-supervised pre-training perform with streaming data? In International Conference on Learning Representations (ICLR), 2021. 6
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu,
Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Gtrassa-Barwwing et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 2017. 1, 6, 7, 13
10

---

Published as a conference paper at ICLR 2024
Hyunseo Koh, Dahyun Kim, Jung-Woo Ha, and Jonghyun Choi. Online continual learning class incremental blurry task configuration with anytime inference. In International Conference on Learning Representations (ICLR), 2021.
Matthias Lange, Gido M van de Ven, Tinne Tuytelaars. Continual evaluation lifelong learning: Identifying the stability gap. In International Conference on Learning Representations (ICLR), 2023. 4,
Vincenzo Lomonaco, Lorenzo Pellegrini, Andrea Cossu, Antonio Carta, Gabriele Graffieti, Tyler L. Hayes, Matthias De Lange, Mare Masana, Jary Pomponi, Gido van de Ven, Martin Mundt, Qi She, Keiland Cooper, Jeremy Forest, Eden Belouadah, Simone Calderara, German I. Parisi, Fabio Cuzzolin, Andreas Tolias, Si- mone Scardapane, Luca Antiga, Subutai Amhad, Adrian Popescu, Christopher Kanan, Joost van de Weijer, Tinne Tuytelaars, Davide Bacciu, and Davide Maltoni. Avalanche: an end-ta-end library for continual learning. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2nd Continual Learning in Computer Vision Workshop. 2021. 13
Quang Pham, Chenghao Liu, and Steven Hoi. Dualnet: Continual learning, fast and slow, Advances Neural Information Processing Systems (NeurIPs), 2021. 2, 3, 6, 7, 14
Ameya Prabhu, Philip Torr, and Puncet Dokania. Gdumb: simple approach that questions our progress in continual learning. In European Conference on Computer Vision (ECCV), August 2020. 3, 6, 7, 13
Ameya Prabhu, Hasan Abed Al Kader Hammoud, Puncet Dokamia, Philip Torr, Ser-Nam Lim, Bernard Ghanem, and Adel Bibi. Computationally budgeted continual learning: What does matter? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3698-370800 2023. 2, 3. 12,13
Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. Imagenet-21k pretraining for the masses. In Thirty-fifth Conference Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021. 12
Snap. Quarterly results of Snap Inc. Q4, 2017. http presentatiens httprr///nn..reeeeeeteoooooea/ae wtw.rr/aooooe------aaa----------
/doo_.
pdf. Accessed: 2022-11-06 1
Liyuan Wang, Xingxing Zhang. Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI). 2024.
Zifeng Wang, Zizhao Zhang, Chen-Yr Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 139-149--000 149, 2022. 6, 7,
11

---

Published as a conference paper at ICLR 2024
A SEMANTIC CODE OF THE ALGORITHM
See Algorithm 1 for the overall training procedure. Here, during ModiityClassficaiionneaa we expand the last layer of the classification head according to the total number of current seen classes, and initialize the previous dimensions with previously learned weights. During SplitBudget, we split the total budget joint training stage and fine-tuning stage.
Algorithm Diet Continual Learning
procedure DIETCL(S, P1rr) Input computational steps per time step S, total time step T,,
distributions Dı.T
M < ()
> Init Buffer
for all / E [I,...,T]. do
D1, Dr Dr
Get offline data
(aaallCCCalssiittooHaa((()) M + M MID;
Update Balanced Buffer
S1 SplitBuugeet(S)
for all 8 E (I,...,,S) S do
B₁ D:
D Unlabeled batch B,, memory batch B,,
B, ~ D..
Unlabeled batch B,
Bm M
>> Memory batch B..
8 (B(, Bu, Bm)
> Update model parameters by loss function (4)
end for
for all 8 E {1,.... S2) } do
b Optionally fine-tune stage
Bm M
0- ()(
b Update model parameters by loss function (3)
end for end for end procedure
B MORE DETAILS OF EXPERIMENT
B.1 BENCHMARK STATISTICS
ImageNetIOO: class-incremental. We created large-scale, sparsely labeled ImageNet1Ok bench- mark from the ImageNet21k V2 dataset (Ridnik et al., 2021). To climinate the possibility of bias to ImageNetlk, particularly when using pretrained models, we first remove the classes present in ImageNet 1k (Deng al., 2009) from ImageNet 21k V2. To that end, the resulting ImageNetl Ok benchmark contains 9459 classes with a total of 9,822,675 labeled images. We then split the bench- mark into T splits representing time steps of a stream. We randomly select a fixed ratio of each class in each split as separate labeled data. This procedure of CL data set construction is standard in prior work Chaudhry et al. (2019a); Fini al. (2022). In each of the time steps, since we assume the stream is only partially labeled, we only use 1% of the labels in each time step.
CLOC: domain-increment incremental We evaluate the domain adaptation ability of our method on CLOC (Cai et al., 2021). This benchmark contains 10788 classes and a total of 38m images for the geo- localization tasks. The images are ordered according mimicking a natural distribution shift. the stream from this dataset is presented over 20 time steps, with per step 0.5% the labels are available. That is to say, the stream reveals around 1.9m images, only 1000 of which are labeled.
the timestamps
when they were taken,
We evaluate the domain adaptation ability of our method on CGLM (Prabhu et al., 2023). This benchmark contains 10788 classes and total of 581.100 of land- marks from Google Map. The images are ordered according to the timestamps at when they were taken, mimicking natural distribution shift. Similarly, the stream from this dataset is presented over 20 time steps with per step 5% of the labels are available.' That to say, the stream reveals around
CGLM: domain-incrrmennae
We increase the label rate in CGLM 5% since CGLM is long tail distribution, and 30% of the classes will only have less than 3 images in 1% labeled stream.
12

---

Published as a conference paper at ICLR 2024
30k images, only 600 of which are labeled. In Table 3, show the statistics of ImageNet10k and ImageNet2k, and compare them with other popular semi supervisee continual learning benchmarks. Both benchmarks have much larger scales and sparser labels than the previous benchmark. In Ta- ble 4, we show the statistics of CGLM and compare with other recent semi-supervised domain incremental benchmarks, where the domain shifts according the time. CGLM has much larger number of classes and sparser labels, and thus is much harder than CLEARIO and CLEARIO0.
Classes per split
Labeled data
Unlabeled data per split
Inference data per split
Dataset
Split
per split
CIFAR100-senm
10
10
1.5k
14k
1k
ImageNetlOk
473
500k
12k
CLOC
20
712
1k
1.9m
19k
Table 3: Statistics of ImageNet10k with 1% label rate, and comparison with previous class- incremental semi-supervised benchmarks.
Labeled data per split
Unlabeled data per split
Inference data per split
Dataset
Split
Classes
CLEARIO
11
10
3k
700k
0.5k
CLEAR 100
11
100
10k
3.6m
5k
CGLM CLOC
20
10788
0.6k
30k
3k
20
713
lk
1.9m
19k
Table 4: Statistics of 20-split CGLM with 5% label rate, 20-split CLOC with 0.5% label rate, and comparison with previous innremenaanre semi-supervised benchmarks.
B.2 IMPLEMENTATION DETAILS OF BASELINES
ER (Chaudhry et al., 2019b). The original proposed methods are one-epoch algorithm. We follow Algorithm 2 Reservoir sampling update in Chaudhry et al. (2019b) to sample from and update the replay buffer for the first epoch every time step in our training. We assume the buffer size is sufficient large that store all the training samples. For the remaining epochs, we sampled from the buffer without update. Prabhu et al. (2023) highlighted the uniformly sampling in ER. We compared uniformly sampling and balanced sampling in Table and finally chose to perform balanced sampling that samples 50% data points from the buffer in every mini-batch.
ER-ACE (Caccia et al., 2022). We the same buffer update strategy as in ER. We adopt the masking strategy and metric learning method for the classification head from the origi- nal paper.
EWC (Kirkpatrick et al., 2017) and MAS (Aljundi et al., 2018). We follow Avalanche library (Lomonaco et al., 2021) implement offline EWC and MAS. We searched the hyperparameters and presented the best results.
GDumb (Prabhu et al., 2020). We follow the standard implementation of the original paper. The mask is set to the seen classes up the current time step.
1.2 (Wang et al., 2022). The original paper used model weights pre-trained on ImageNet 21k and fine-tuned on ImageNet 1k. We load the ImageNetlk pre-trained weights and
Method
Uniformly Sampling
Balanced Sampling
A(T)
12.73
14.97
Table 5: Different sampling strategies in ER. Experiments are on 20-split 1% labeled ImageNet10k benchmark with 500 steps computational budget.
13

---

Published as a conference paper at ICLR 2024
Supervised Budget
300
350
400
450
Task
32.44
32.7
32.91
32.75
Task
29.30
29.75
29.80
30.02
Task
26.81
27.10
27.44
27.09
Table 6: Average Accuracy up to the corresponding task under varying budget. No matter which task is used for selecting budget threshold, the budget threshold is always set around 400-450.
find suddenly performance drops. The authors proposed to train 5 epochs for every time step without replay buffer, and provided limited buffer size cases well. However, we find that when we modify the model in our setting to have unlimited replay buffer but limited gradient steps, the performance even better. The results we present the modified version.
CaSSLe (Fini et al., 2022). We chose the Barlow Twins SSL bascline to report the results. The original paper proposed to pre-train for 400 epochs and fine-tune for 100 epochs on ImageNet100. We adopt this ratio 4 of pretraining and finetuning steps in set- tings where the total budget is fixed. In the finetuning stage, we adopt CaSSLe incremental way by training one linear classifier for all the classes seen far for a fair comparison. The performance drop comes from the class incremental classifier and limited computation.
class
DualNet (Pham al., 2021). Given that the slow net comprises only few convolutional layers, we solely load the ImageNet 1k pre-trained weight for the fast net. In the original pa- per's semi-supervised implementation, the author employed a variable sampling approach to determine whether to the supervised unsupervised loss for the current batch, based on a comparison with the label rate. This method substantially augmented the diversity of labeled data. In implementation, unlabeled subsets and then adopt the variable sampling approach of the original paper to decide whether to sample from the labeled unlabeled subset. We determine the thresh- old to be 0.5, which yields the best results, but lower than the original ones. Additionally, we include all the gradient steps, including the two view transformation of the contrastive loss, in the computation and fix the total number of gradient steps, which leads to fur- ther performance degradation. Although the original paper proposed task-agnostic and task-free training strategies, we experimented with both but evaluated the model in class- incremental manner. Neither strategy yielded favorable results in the context large-scale class-incremental streams with limited computational resources.
first segregate the current data into labeled and
Upper bound (He al., 2022). In particular, SSL Joint Training trains through self super- vision on all the unlabeled data once and then fine-tunes on all the available labeled data. The total computational budget given is the effective budget of continual learners, which is 20 time steps x budgel per time step. 80% of this budget is spent on self supervision, while the other is for fine-tuning.
B.3 BUDGET THRESHOLD
We chose conduct cross validation choose the threshold to separate the balanced training stage and learning on buffer stage. The cross validation is only on the initial three tasks out of 20 tasks; this is standard and widely accepted in the field of continual learning (Chaudhry et al., 201
To show the robustness of such selection, we performed experiments in ImageNet1Ok benchmark with different supervised budgets and compared their performance in tasks 2, 3, and 4 in Table 6. We present the average accuracy up the corresponding task. When choosing the supervised budget from either task 2, or 4, the chosen budget threshold is consistently similarly from 400 to 450.
B.4 ABLATION WITH OTHER ORDER
Here in ablation study, we show different orders as well as the observations and report the average accuracy of the last model.
14

---

Published as a conference paper at ICLR 2024
Table 7: Ablation study with different orders 20-split ImageNet1Ok benchmark
Replay
4m
Balanced Buffer
C
Avg. Acc.
Replay
Cm
Balanced Buffer
4,
Avg. Acc.
14.97
14.97
15.01
15.01
15.96
15.24
/
16.82
16.82
The first table shows that the task-balanced buffer marginally improve the Replay method. Then, both Lm and &, can improve the algorithm by around %. This consistent with our original observation.
The second table shows that without masked loss, the unlabeled data can only marginally improve the performance. This further validates the point that the loss of the unlabeled data should be guided by proper supervised loss, as stated in section 4.
C ANALYSIS
C.I ER AND DIETCL IN LOW COMPUTATIONAL BUDGETS
In Figure
present the validation loss and accuracy of classes originally introduced from time steps O, 1, and 2, during the training of time step 2 for DietCI. and ER when the total computational step is 300. Our results indicate that the classical continual learning algorithm, ER, exhibits the stability gap phenomenon, as proposed by Lange et al. (2023). Specifically, during the learning of new classes, the accuracy of previously learned classes initially drops, before eventually resuming an upward trend once the learning of new classes stabilizes. Notably, in ER, although the accuracy of current classes initially rises to approximately 40%, it subsequently declines significantly later time step, before gradually improving again. Moreover, when the budget is fully expended, the learning of previous classes remains unfinished.
However, in our algorithm, the accuracies of previously learned classes, as depicted by the blue curves in the figure, do not exhibit a significant drop, and the learning of new classes is smoother. In other words, the accuracy of current classes does not undergo the "up-and-down detour" observed in ER, which results in significant budget savings. We postulate that in ER, since the labeled data is sparse, the model is more susceptible overfitting to the space represented by the sparse data from current classes. Conversely, the incorporation of unlabeled data in our algorithm guides the learning towards a more generalized space, thereby eliminating the overfitting issue and the stability gap. As a consequence, our algorithm is efficient in learning under low-budget scenarios, especially when the labeled data sparse.
Figure 7: Validation Loss and Accuracy of classes introduced from time step 0, during the train- ing of time step 2. The training is conducted on ImageNet1Ok, with label rate 0.01 and budget of 300 steps.
15

---

Published as a conference paper at ICLR 2024
C.2 DIRECTLY PERFORM PRE- TRAINING AND FINETUNING IN CL
We conducted empirical study the effectiveness of using SSL methods in continual learning- One naive approach is to perform two-stage training iteratively, consisting of pre-training and fine- tuning stages, which leverages unlabeled data in the pre-training stage. To establish baselines, we implemented the Otget and TwoStage methods, both of which replay all the labeled data from the current time step. In the OneStage method, we randomly sample batches from all seen classes and use cross-entropy loss for classification. In the TwoStage method, we first perform MAE pre- pre-training with the unlabeled data of the current time step using fixed ratio of compu- tational budgets, followed by fine-tuning the model with labeled data randomly sampled from the buffer using the remaining budgets. We perform this experiment in an extremely sparse labeled stream with sufficient large computational budget. Note that this setting is already highly permis- sive towards semi-supervised training compared to supervised training scenarios, as the budget is relatively large and the amount of labeled data is extremely sparse.
Our results, as shown in Figure 8, reveal that an extra unlabeled stage training in the TwoStage method improves the classification accuracy of the samples from the classes, which the exact classes from which the unlabeled images used in the pre-training stage are obtained. This observation is consistent with the success of self-supervised learning works (He et al., 2022). How- ever, we did not observe any advantages of the pre-training stage when measuring the performance among all seen classes. We hypothesize that the pre-training phase harms the label-related represen- tations learned in previous time steps, resulting in a drop in the Average Accuracy of the TwoStage method.
Evaluation on All Seen Classes
Evaluation on New Classas
30
25
25
Aocuracy
15
10
15
5
10
15
20
10
15
20
Time Step
Time Step
OneSlage
TwoSlage
Figure 8: Accuracy of the One Stage MAE and Two MAE baseline on 0.5% labeled streams. Each baseline has 2500 steps computational budget each time steps.
C.3 EFFECT OF THE FINE-TUNING STAGE
In Figure 2, we utilize the initial 400 computational steps to perform joint training of the model using labeled and unlabeled data along with data from the balanced buffer. For budgets that exceed 400, the remaining steps are allocated for fine-tuning using solely data from the buffer. The selection of 400 steps was based on the observation that classical methods tend to sacrifice performance beyond this threshold, signifying that it is adequate for learning the current distribution. As an illustration, we consider the experiment involving the 20-split ImageNet100 10k with label rate of 0.01 and 600 computational steps, where we compare the training outcomes with and without fine-tunine in Figure The figure demonstrates that the balanced fine-tune applied during the final stages of training leads to more balanced, thus improved, results.
16

---

Published as a conference paper at ICLR 2024
Accuracy classes from different distributions at time 19
with balanced fine-tune
20
----
w/o balanced fine-tune
Accuracy
18
16
14
1
3
4
6
8
10
11
12
13
15
16
17 18
19
20
Time Step
Figure 9: Accuracy of classes from each of the previous distribution after the training of the last time step. After fine-tuning the model on the balanced buffer, the accuracies across the stream become more balanced.
D CONNECTION TO REAL-WORLD PROBLEMS
We aim to study a scenario where we cannot conduct a full epoch on all incoming data, but can allocate enough computation for the sparsely labeled data. Our algorithm discusses how to take of extra budget from labeled data and put it in unlabeled data.
In experiment, need (at least) 0.5 GPU hours 1% labeled 20-split ImageNet1 1k stream. Each task contains a total of 1m images and 10k labeled images, and we perform 500 gradient steps with batch size of 1024.
80G Nvidia A A100 train each task the
Now, we want now to translate the computational requirement that we used in this paper (as detailed above) the scale of Snap example. As mentioned the introduction, Snap receives around 3.5B videos every day. Take the ranking tasks by classifying N snapshots of each video as an example, there will be 3.5N billion image data every day. We assume there are L annotators for this task and each can produce 4000 labels per day. To match our 500 computational steps, we expect the budget 4000L/10000 500 2007. As we need GPU hours for 500 steps in our experiment, this 2007. amounts to around 1007. GPU hours. That is, when B/I, the computation budget per labeling labor, around 100 GPU hours (almost a whole day running on 4 GPUs), our algorithm can outperform other baselines in the pre-defined ranking tasks. And the upper bound for the performance is related to the label rate 4000L/ (3.5 10
at least
We note that the time required is closely related to the GPU type, well as data loading time, GPU communication time, and other factors. For instance, in a system with slow data loading, we may need up 18 GPU hours. Through collaboration with various experts in acceleration, the B/L (budget per labeling labor for the algorithm to converge) can be further reduced.
17


Reviews:
[{"review_id": "SwIzpszPqv", "reviewer": "Reviewer_GSPx", "summary": "This paper presents DietCL to conduct semi-supervised continual learning under a label and computation budget. The main idea is to formulate a loss function that considers a reconstruction loss, a masking loss, and a budget loss at the same time.", "questions": "1. What is the motivation for the loss function in (4), where each term is weighted equally? How would this generalize to other settings where equal-weighting may not lead to desired behavior (e.g., it might overemphasize the reconstruction loss over others, or the budget loss over others)?\n2. What is the key novelty that enables the method to outperform prior approaches? Is it the explicit consideration of the budget in the loss function?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* Continual learning under a constrained labeling and computational budget is an interesting problem.\n* The approach seems to perform well in the evaluated scenarios.", "weaknesses": "* The paper lacks clarity and is not well-organized. For instance, there is a related work section (Sec. 2), but the discussion of prior methods continues all the way until page 5 and there is a separate section (Sec. 3.2) for additional coverage of prior work. The challenges of prior work is reiterated numerous times in the first 5 pages, and despite their mention in the abstract and introduction, they are mentioned again on page 4-5 “Challenges facing existing semi-supervised continual learning algorithms.” It is not until page 5 that DietCL is introduced and its coverage is limited to less than a page, which leads to an incomplete presentation of the method.\n* The method’s novelty and clear exposition is lacking. What is the key insight here? It seems that the method boils down to using an existing self-supervised learning method (MAE) coupled with masking out logits of classes not shown in the current time step. \n* The loss function (Eq. 4) simply adds the three different loss terms without any hyper-parameters in front of them to scale them appropriately. For instance $\\mathcal L_r$ is the reconstruction loss (Euclidean distance squared of vectors), and it is being added to the loss function with two other losses that are cross entropy. It seems like there will inevitably be a scaling issue. Even if additional hyper-parameters, e.g., $\\alpha_r, \\alpha_m, \\alpha_b$ were added in front of the loss terms, this would entail hyperparameter optimization to find the right values.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZNzIrZ62TC", "reviewer": "Reviewer_TT9Y", "summary": "This paper explores a problem setting of data efficient continual learning, where only a small portion of available data is labeled and the use case demands fast training compute. The paper introduces a continual learning algorithm that combines self-supervised on the unlabeled data with the conventional supervised continual learning strategy on the labeled subset. The algorithm divides the compute budget equally between these components to demonstrate that the proposed method outperforms existing approaches and helps to reduce overfitting.", "questions": "What is the non-monotonic behavior of DietCL in Fig 4?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The problem setting is specific and there is detailed discussion on the challenges of existing methods. The proposed method is simple and seems to perform well under many ablations.", "weaknesses": "1. The overall method is quite simple, which in itself is fine, but would warrant extensive ablation on the design choices. For instance, the algorithm depends on MAE as a way to generate good pre-trained features. How would the algorithm perform with alternative representation learning strategies? Furthermore, the buffer threshold is determined via cross-validation. How difficult is this cross-validation to perform and how many time steps / tasks are needed to learn it?\n2. The ablations provided seem to compare against the semi-supervised algorithms. However, it is clear from Table 1 that the comparable semi-supervised algorithms are outperformed by DietCL, whereas supervised techniques such as ER and ER-ACE are competitive and almost as good as DietCL. It would have been more interesting to see ablations against these supervised learning baselines, for example to demonstrate the diet limits under which the proposed method is challenged.\n3. Notation in page 6 defining A(t) and A seem to be incorrect due to overloading $t$", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2WrPkeI1l2", "reviewer": "Reviewer_dSE4", "summary": "This paper proposes “CL on Diet”, which is a large-scale semi-supervised continual learning setting, which is an extension of budgeted continual learning to a semi-supervised setting.\nDietCL utilizes budget efficiently with joint optimization of labeled and unlabeled data, and comes with a computation budget allocation mechanism to harmonize the learning of current and prior distributions.\nThe experiments demonstrate that DietCL outperforms both supervised and semi-supervised continual learning\nalgorithms in sparsely labeled streams by 2% to 4%.", "questions": "see weaknesses", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. By asking several questions, Section 3.2 successfully gives the motivation of DietCL by showing the shortcomings of existing methods under low budget scenario.\n2. DietCL is well-designed with sufficient details given. For example, using unlabeled data as a regularizer, DietCL can effectively eliminate overfitting and the stability gap issues.\n3. Sufficient details are given for the experiment part.", "weaknesses": "1. Comparison with (Prabhu et al., 2023) is needed in the experiment section, since it belongs to the supervised continual learning methods.\n2. need more connections to real-world problem: It is unclear how far is a real-scenario from the low budget setting given in this paper.  It would be better to further discuss a real example (such as snapchat example), by quantifying how many budget per time step is actually possible for a real scenario (with typical hardware).\n3. The Ablation study uses 'Replay -> Lm -> Balanced Buffer -> Lr', how about trying 'Replay -> Lm -> Lr -> Balanced Buffer'  or another order. some different observations may be obtained.\n4. small presentation issues\n  (1) Confused notation. In Appendix A, subscriptions {l,u,m} are used to demonstrate labeled, unlabeled and buffer data. While in section 4, {r,m,b} are used.\n   (2) typo in Figure 6 'to make them easier to compare?'", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zx6D9zwrno", "reviewer": "Reviewer_uZcN", "summary": "The authors study continual learning with a restricted computational budget (defined by FLOPs) per time step. Demonstrating that pre-existing supervised and semi-supervised methods struggle in this setting with overfitting or lack of adequate computation resources, the new semi-supervised method DietCL is proposed.", "questions": "- In the task-balanced buffer $\\mathcal M$ introduced in Section 4, why do you include just data from the current and previous time step? If budget allows it, why not include data from more time steps or ensure you have an adequate diversity of examples from different classes included in the buffer?\n- Questions about the reported empirical numbers: How many runs are these reported across (for instance to produce the figures 2-6) and how large is variance across the runs? Does the ordering examples are presented during training impact learning a lot, and are some methods more robust to this than others?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The setting of limited compute and labeled data is important and realistic, and the proposed method is well-adapted to this setting\n- Defining compute budget by FLOPs allows us to consider being unable to even complete a full epoch, which is challenging for learning but useful and realistic for large datasets and small compute capabilities\n- Evaluation of other prior methods seems to be fair and complete", "weaknesses": "- Fig 2 shows that DietCL does still suffer from overfitting, just not as bad as other methods (particularly ER) and takes much larger computational budget for the effect to be bad. In the caption and paper text authors claim DietCL doesn't suffer from overfitting, which seems to be a bit of an overstatement", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)