Online Boundary-Free Continual Learning by Scheduled Data Prior
Authors: Hyunseo Koh
Minhyuk Seo
Jihwan Bang
Hwanjun Song
Deokki Hong
Seulki Park
Jung-Woo Ha
Jonghyun Choi
Venue: ICLR 2023 poster (2023)
Keywords: Continual learning
data prior
boundary-free

Abstract: Typical continual learning setup assumes that the dataset is split into multiple discrete tasks. We argue that it is less realistic as the streamed data would have no notion of task boundary in real-world data. Here, we take a step forward to investigate more realistic online continual learning – learning continuously changing data distribution without explicit task boundary, which we call boundary-free setup. As there is no clear boundary of tasks, it is not obvious when and what information in the past to be preserved as a better remedy for the stability-plasticity dilemma. To this end, we propose a scheduled transfer of previously learned knowledge. We further propose a data-driven balancing between the knowledge in the past and the present in learning objective. Moreover, since it is not straight-forward to use the previously proposed forgetting measure without task boundaries, we further propose a novel forgetting measure based on information theory that can capture forgetting. We empirically evaluate our method on a Gaussian data stream, its periodic extension, which assumes periodic data distribution frequently observed in real-life data, as well as the conventional disjoint task-split. Our method outperforms prior arts by large margins in various setups, using four popular benchmark datasets – CIFAR-10, CIFAR-100, TinyImageNet and ImageNet.

Published as a conference paper at ICLR 2023
ONLINE BOUNDARY-FREE CONTINUAL LEARNING BY SCHEDULED DATA PRIOR
Hyunseo Koh
Minhyuk Seo!
Jihwan Bang"
Hwanjun Song Jonghyun Choi
Deokki Hong!
Seulki Park
Jung-Woc Woo Ha
Yonsei University
2 GIST NAVER AI Lab
4NAVER Cloud
Seoul National University
ABSTRACT
Typical continual learning setup assumes that the dataset is split into multiple dis- crete tasks. We argue that it is less realistic as the streamed data would have no notion of task boundary in real-world data. Here, we take step forward to in- vestigate more realistic online continual learning - learning continuously chang- ing data distribution without explicit task boundary, which we call boundary-fiee setup. Due to the lack of boundary, it is not obvious when and what information in the past to be preserved for better remedy for the stability-plaate plasticity dilemma. To this end, we propose a scheduled transfer of previously learned knowledge, In addition, further propose data-driven balancing between the knowledge in the past and the present in learning objective. Moreover, since it is not straightfor- ward use the previously proposed forgetting measure without task boundaries, we further propose novel forgetting and knowledge gain measure based on in- formation theory. We empirically evaluate our method on a Gaussian data stream and its periodic extension, which is frequently observed in real-life data, as well as the conventional disjoint task-split. Our method outperforms prior arts by large margins in various setups, using four benchmark datasets in continual learning lit- erature - CIFAR-IO,,, 10, CIFAR- 100, IinylmageNet and ImageNet. Code available at https:: / com www.s/inosseeeeeeeeeeeeoo /dl.
INTRODUCTION
In real-world continual learning (CL) scenarios (He et al., 2020), data arrive in streamed man- ner (Aljundi et al., 2019a; Cai et al., 2021) whereas typical continual learning setups split the data into multiple discrete tasks whose data distributions differ from each other. Moreover, most CL al- gorithms are studied in an offline CL setup (Kirkpatrick et al., 2017; Rebuffi et al., 2017; Saha et al., 2021), where the model can access data multiple times. While being prevalent in the literature, this setup has number of issues far from the realistic scenario. Although the task setup have been partly addressed by (Prabhu et al., 2020; Koh et al., 2021; Kim et al., 2021b; Bang al., 2022), the re- vised setups still have the notion of task boundary whereas real-world data may not have the explicit task boundaries as the data distribution changes continuously. Despite that many methods update manner, called task-free CL (Aljundi et al., 2019b; Koh et al.,
the model in a boundary-agnone 2021), they still leverage the notion of task boundary for knowledge transfer and evaluation, e,g. leveraging the fact that distribution shift in data stream occurs only at task boundaries. In addition, the definition of forgetting depends on the notion of 'old' and 'new" tasks, which are defined by the task boundary.
We argue to address an online CL setup where data are learned online (allowing only a single access to data) with continuous distribution shift without explicit task boundaries. We refer to the setup as online boundary-free continual learning. In this setup, a small set of data is streamed to the model one by one, and the model only has access to the current data batch only (Aljundi et al., 20 2019c;a) without the notion of task boundary.
For the distribution of continuous data stream, following (Shanahan et al., 2021; Wang et al., 2022), we consider Gaussian distribution as an instance of data streaming distributions. The Gaussian
indicates the corresponding author.

---

Published as a conference paper at ICLR 2023
online data stream models the frequency of each class as Gaussian distribution over time. Note that classes do not recur after their initial Gaussian mode in this setup. However, in real-world data, the frequency of class may have multiple recurring modalities over time, rather than single mode, as depicted in Fig. 1. To further address such scenario, we investigate a Seriodic-Gaussiin online stream, where each class would recur and the recurrence is periodic. To the best of our knowledge, this is the first work to study CL in continuous data distributions either in periods or not.
The boundary-free setup poses several challenges as follows. In the CL setup with explicit task boundary definition, methods using both episodic memory and distillation (i.e, using data prior (Buzzega et al., 2020; Wu et al., 2019; Hou al., 2019)) show compelling perfor- mance (Masana et al., 2020). They store the model weights at each task boundary and use the stored model as a distillation teacher for mitigating catastrophic forgetting. However, in the continuous data stream, it is challenging to determine which past models should be stored to be used as data prior. To determine which data prior to transfer the knowledge from, we propose to combine dif- ferent exponential moving average (EMA) distributions to have particular schedule of transferring past knowledge.
In addition, the past knowledge is now from diverse contexts, it is not trivial to balance the supervisory signal from the past and the present. Instead of using a fixed balancing hyperparameter, propose to learn to balance them for better generalization in multiple scenarios, i... datasets. In our empirical studies, we observe that our method outperforms comparable prior arts in Gaussian, and disjoint task-split data stream on 4 popular benchmarks in CL. literature.
Moreover, conventional performance metrics for CL methods including forgetting is not trivially applicable to our setup as they are defined on the task boundary. Here, we propose a new metric for measuring forgetting using information theory. In contrast to the conventional forgetting metric, it captures loss and gain of intra-class knowledge, appropriate for periodic data distribution where the model has to accumulate different knowledge about the same class over multiple periods.
We summarize our contributions follows:
Extensively studying online CL with continuous data stream setup without explicit task bound- ary, including newly proposed periodic CL setup.
Proposing an online boundary-free CL method that uses scheduled transfer of past knowledge.
Proposing to learn to balance amount of using past and present knowledge.
Proposing new metrics that can measure loss of past knowledge (i.e., forgetting) and gain of new knowledge (i.e., the opposite of intransigience) based on information theory.
2 RELATED WORK
Setups for Continual Learning. With the increasing popularity of CL, there have been several proposals learning configurations to be realistic. As the first task setup to mimic real-world data stream that continuously changes over time, prior arts have employed the notion of task-split, where the entire data is split into multiple subsets for different continuous tasks (Rebuffi al., 2017; Castro et al., 2018; Wu et al., 2019). When each subset arrives, it is stored and provided as a set of examples to learn model. They use multiple epochs learn model (Kirkpatrick et al., 2017; Rebuffi et al., 2017; Saha et al., 2021), which is referred as offline setup, while a small batch sample of the subset is streamed and used only once in online setup (Rolnick al., 2019; Chaudhry al., 2019; Aljundi et al., 2019a). In the task-split setup, task boundaries are available and well exploited in various offline/online CL methods (Lopez-Paz & Ranzato, 2017; Rebuffi et al., 2017).
In recent literature, however, there has been efforts to question whether the task-split setup is realis- tic. To enforce the class distribution of each split differently, disjoint task-split confines each class to be assigned to only a single task (Castro et al., 2018). As the disjoint setup is rather artificial as the data stream arrives in class agnostic manner. Blurry task-split allows every task shares all classes but with different dominance (Aljundi et al., 2019c; Bang et al., 2021). The i-Blurry task-split further guarantee some classes to be added incremental to the blurry task-split (Koh et al., 2021). However, these task configurations have explicit task boundary, which is still artificial. For more realistic scenario, task-fres CL (Aljundi et al., 2019b) has been studied, where models are not allowed to use task boundary information during training. However, they still train and evaluate methods task-
2

---

Published as a conference paper at ICLR 2023
Swimsuit
Coat
Christmas Gift
100
Search Interest
80
60
40
09 jan
10-jan
11-Jan
12-jan
13-jan
14-jan
15-Jan
16-Jan
17-jan
18-jan
19-jan
20-jan
23-jan
22 J2nan
Figure I: Search interest trends of three items (iee, swimsuit, coat, Christmas gift) from 2009 to the present in Google Trends. Each item follows a periodic distribution with its own mode and duration.
split setup such as disjoint, blurry, and i-blurry. Thus, task-free is more of a restriction on methods rather than a setup.
In contrast the prior work, we propose boundary-free setup. removes the notion of artificial task boundary, rather, the data arrival follows a certain continuous distribution over time. (Shanahan al., 2021; Wang et al., 2022) investigate data stream following Gaussian distribution, they still split the data into micro-tasks and neglect a periodic property of real-world data, which we study further here.
Online Continual Learning Methods. One of the goals of successful CL learners is not to forget knowledge obtained from the preceding tasks; here, catastrophic forgetting has become the main challenge with deep neural networks for CL (French, 1999). To mitigate the issue, there are four main directions in the recent literature, namely distillation, memory replay, parameter isolation, and regularization. For a more comprehensive review, we refer the reader to surveys (De Lange et al., 2021; Mai et al., 2022).
The key difference between online and offline CL is the accessibility of streaming inputs; the model can only see the entire stream once except for the samples in the episodic memory. To tackle this online constraint, several methods have been developed based on the similar ways in offline CL. GEM (Lopez-Paz & Ranzato, 2017) leverages the gradient samples in the available memory so that they alleviate forgetting the knowledge of the previous tasks. A-GEM (Chaudhry et al., 2018b) proposes to utilize the average of gradients for each task instead of using the projection of all the gradients. It further saves memory usage and reduces the computational cost. GDumb (Prabhu et al., 2020) proposes greedy balance selection which randomly selects the samples while balancing the number of selected samples per class. GSS (Aljundi et al., 2019c) and RM (Bang et al., 2021) utilize the gradient and uncertainty of each sample, respectively, to increase diversity of selected samples. CLIB (Koh et al., 2021) makes up informative samples in the memory by discarding the least informative samples in the memory for further training.
Unlike offline CL, however, online CL understudied the methods which utilize knowledge distilla- tion. DER (Buzzega et al., 2020) utilized the knowledge distillation between logits of an original image and augmented image from origin, which well-known and widely used. In our proposed setup, which is boundary-fres it is not trivial to determine when and what knowledge should be transferred mitigate the forgetting. Here, we propose a new knowledge distillation method which is suitable for online boundary-free CL by using scheduled data prior.
3 ONLINE TASK BOUNDARY-FREE CONTINUAL LEARNING SETUP
Motivated by a real data stream that changes continuously (e.g., Google search trend of 'swimsuit' 'coat', and 'Christmas gift' during 13 years, depicted in Fig. 1'), we propose new CL setup called online boundary-free. We argue that our setup is more realistic than the task-split setup for the following considerations: (1) an ever-changing distribution with (periodic) Gaussian online stream, (2) no notion of explicit task boundaries, and (3) any-time inference for online continual learning.
he nendee - gong com'tt render leepp ore tewn Eigeoe[/Seeeechh strme K20gi fit, coat, axineut
3

---

deviation a is the same for all classes for simplicity (we use = 0.1 for our empirical validations Published as a conference paper at ICLR 2023
3.1 GAUSSIAN ONLINE STREAM
For easeness of modeling, here we assume that a real data stream for each class follows Gaussian distribution. First, consider single mode stream, then extend it multi-modal periodic stream. Gaussian-distributed task modeling has been addressed by Shanahan et al. 12 Wang al. (2022). However, they differ from ours since their distribution change is not fully continuous as they still split data into multiple micro-tasks and overlook the periodicity of the online stream.
Specifically, we consider the 'arrival time' of samples to be modeled by the Gaussian distribution. In order to create an online stream without task boundary; samples are streamed in an increasing order of their arrival time. Formally, let us assume that the distribution of arrival time for class follows the Gaussian distribution NV ((ss, (²)) For each class i, the mean μ4 (=mode) of the distribution is exclusively chosen from {0, N. ..... NN'
N-11 where N is the number of classes, and the standard
and provide analysis on different values in the appendix).
Periodic Gaussian Online Stream. By a simple modification, we can extend the Gaussian stream to a periodic one. Now, the distribution of arrival time of a class i not only follows the Gaussian distribution but also is repeated multiple times:
R-1 Nom +r,
where R is the number of repetition (i.e., periods, we use R = 5 in our experiments). The mean p4 and standard deviation a are the same as the ones in the non-periodic Gaussian online stream.
EVALUATION METRICS FOR CONTINUAL LEARNERS IN THE BOUNDARY-FREE DATA STREAM
For evaluating the overall performance in a boundary-free CL, we use the area under the curve of accuracy (AAuc) proposed in (Koh et al., 2021) for measuring area under the curve of accuracy- to-(# samples) curve, and last accuracy (Atast) that measures the final accuracy after learning all samples. We cannot measure Aavy metric since there is no task boundary.
Previous CL research use Forgetting and Intransigence metric (Chaudhry et al., 2018a) to investigate stability and plasticity of CL algorithms. However, these metrics require measure the accuracy of the previous task, so they are not readily measurable in the boundary-free setup due to lack of task boundary. Here, we propose new metrics of measuring forgetting and ability learn new knowledge by computing loss and gain of knowledge based on information theory.
Knowledge Loss Ratio. Specifically, we want to measure loss and gain of knowledge between arbitrary two points in the training, L and f2. Let Yer be ground truth label for randomly selected sample from a data distribution and Y, be the model's prediction for that sample at time We define the Total Knowledge (TK(t) at time t as:
TK(t) Vorr
P(x, ((a))(y)
where I(X;Y) EEEEEE P(a, u) v) log
is mutual information that measures the
BEY ZEX
quantity of information one variable has about the other. In other words, measures how much information about ground truth we can obtain by observing the model's prediction at time t.
To measure the loss of knowledge between i and fz, we quantify the knowledge in Yr, but not in Yt,. The knowledge loss can be measured by the knowledge difference between having both output (Yn, Yo,) and having only Yz,- Thus, we define Knowledge Loss (K Ltt, t2)) between i and tz as: KL f2) = I(Y,, Yts: YGT) I(Y,; YoT) I(Y,; Yor)Y,,), where (X;Y Z) = [X, Y) - [(2; Y) is the conditional mutual information. By dividing the knowledge loss by the total knowledge at é1, obtain Knowledge Loss Ratio (K (KLR tz)) as:
(3)
KL 12) = kL(t1,t2)
TK(ti) = I(Y,: Yorl
(4)
1(Y,,; YoT)
4

---

Published as a conference paper at ICLR 2023
KLR measures the ratio of the past knowledge that lost between 8 and 82, which will use an equivalent measure for forgetting in continuous data stream.
Knowledge Gain Ratio. Similarly, can define the Knowledge Gain (KGG) between & and t2 as:
KG I(Y, Y2; I(Y,; :or))YYr)))) I(Y;; Yor(Vi,): Similar to KLR, we define Knowledge Gain Ratio (KGR) measure the ratio of the potentially obtainable knowledge obtained between ( and f2. The amount of information in GT label is H (Yar) where HY) - P(y) log P(y) is the entropy, so the potentially obtainable knowledge at time t1 is HYGrr) - (t((t Thus, (GR(t1, (2) is defined as:
(5)
KG(t1.tz)
I(Y Y,) H(Yer) - I(2; YGr)
(2) = H(YGr) - TK(h)
(6)
Implications. We illustrate the implication of the de- fined KL and KG in Venn-Diagram of Fig. 2. Upper circle, lower left circle, lower right circle represent YGr. Yt1. and Yta. respectively. KL is knowledge about GT that is in model at f but not t2. KG is the knowledge about GT that is in model at t2 but not in t,. The triple in- tersection TY,,; Yı, YGT) represents retained knowledge about GT that is in model at both l and l2. Finally, the lower intersection, [(, Yr,, YGT), represents useless in- formation shared in both models that are not relevant to GT labels, such as bias in the model.
GT Label
R., X.)
(orr:Yo (KL)
a [..)
(KG)
HI (r,,, Yor) Yar)
JIY,, Yo, Yar)
r (4,, (Yar)
One advantage of KL is that it can be interpreted as intra- class forgetting. Since conventional 'foggetting' is mea- sured using task-wise or class-wise accuracy, if forgetting and knowledge gain simultaneously happens within task class, its net effect is zero, thus its effect will not be measured. For example. if we assume there is class A
Model Oulput al Model Oulput at
Figure 2: Relation of knowledge
learned by model to the Knowledge Loss (K LKL) and Knowledge Gain (KG). with features a and 6 and the model forgot about feature a and learned about feature b so that over- all accuracy for class A is the same, it counts zero forgetting in the conventional measure. In contrast, when using the proposed Knowledge Loss (KL), the current model only has information about feature b, but past and current models combined have information about both a and b so it captures loss of information about feature a.
APPROACH
The existing task-split setup partitions the data stream into multiple discrete tasks. Prior arts in CL literature that have been developed in this setup focus on extracting information from prior models learned in the previous tasks. However, in the online boundary-free setup whose data stream is not partitioned into explicit tasks, is necessary to consider information at which previous moment we have transfer to the current time step for better stability and plasiticity trade-off. Although there is no task boundary, a model learned in the past can be stored when each sample arrives and used as data prior, i.e., teacher in distillation framework. However, it is not clear which previous models to be used to transfer knowledge to the current time step. We consider various weighting functions to transfer information from the previous data stream in online and continuous fashion, and summarize the empirical results in Table 1.
5.1 SCHEDULED DATA PRIOR
To determine the amount of past knowledge to transferred in a continuous fashion, we consider schedule of transfer of past knowledge by a composite function of exponential moving average (EMA). EMA calculates a weighted average in an online manner with exponentially decaying weight to place higher weight recent datapoints. EMA model (t(t) with EMA ratio timestep t is defined recursively as
(1( (1 a) a1 1) 0())
5

---

Published as a conference paper at ICLR 2023
14c.5
14e-5
12e-5
EMA
12e-5
SDP
10e-5
weight
8e-5
weight
8e-5
6e-5
4e-5
4e-5
2e-5
2e-5
5k
10k
15k
20k
25k
30k
30k
Present
timesteps
Past
Present
timesteps
Past
(a) EMA weights with a = 0.0001
(b) SDP weights with = 10, 000 and c² = 0.75
Figure 3: Comparison between weight distribution of EMA and SDP.
where (t) is the online model's parameter at time In particular, the EMA update emphasizes the recent knowledge more than the past knowledge by exponentially decreasing weight distribution, as shown Fig. 3-(a). However, we argue that recently learned information is well maintained in the model being currently trained (i.e., not yet forgotten) and need shift the focus to a slightly farther past (as depicted in Fig. 3-(b)).
To implement such weighting scheme. propose a hypo-cxponential distribution, which has skewed-bell curve shape to transfer the knowledge from the past. We configure the hypo-exponential distribution by weighted average of two EMA eurves with different hyperparameters. The resulting distribution has a mode in the distant past while the vanilla EMA has the mode at the nearest past then the weight monotonically decreases.
Specifically, to construct the scheduled data prior (SDP), we take a weighted sum of two EMA models 0g (4) and (t(() with EMA ratios of and B where a B. Using the coefficient for the hypo-exponential distribution, the proposed SDP model (s (t) is defined as:
3
@SP .1 (t)
a 0 a - (t).
Its weight are non-negative and summed up to 1, and form skewed-bell as depicted in Fig. 3- (b). Instead using a and B, we define SDP with mean μ and coefficient of variation c² f of weight distributions, where o² is the variance of the distribution, that hyperparameters are interpretable. The values a and 3 can be calculated from μ and o as:
1 +
20 + 2 P 111
1 1
2c
2 e2)-1
8
111
where and 3 are positive real values when found in appendix.
< 2 <
1
The derivation for these can be
Teacher Model
AAuC
Atast f
We empirically validate our hypothesis by a compara- tive study of using differ- ent scheduling distribution of past knowledge transfer in Tab. 1. As shown in the table, the proposed weight distribu- tion (SDP) outperform other scheduling functions by no- ticeable margin.
None (Bascline)
62.37+0.52
64.22+0.97
Periodically Saving Model (Offline)
63.44+0.10
65.67+1.38
Saved Logits (DER)
59.82++.30 30
58.92#2.36
Saved Features (DER)
61.55+0.22
58.78=2.38
EMA Model
63.75+:0.22
66.55#3.32
SDP Model (Ours)
64.04+0.25
69.47-1.79
Table 1:
Comparison of past knowledge transfer functions for con- tinual learners on Non-Periodic Gaussian data stream in CIFAR-
10. The proposed method outperforms other functions including EMA Model.
5.2 LEARNING TO BALANCE PAST AND PRESENT KNOWLEDGE LEARNING
While transfer the knowledge from the past experience by the SDP distribution, balancing the amount of information from the past and the current time step is remaining challenge for continual learners (Mai al., 2022). The balancing hyperparameter is usually tuned by hand-crafted manner

---

Published as a conference paper at ICLR 2023
and is dataset specifie. Moreover, fixed parameter is not always optimal, as importance of classifica- tion and distillation may vary over the course of training. Thus, we propose to learn to balance them for better generalization to different data context and adapting in different phases of training.
With the learnable balancing parameter X we can write the final objective as:
(10) where Leg (z. y) is the cross-entropy loss for current time step's knowledge and LKD(2) ll { is the L.2 distillation loss for the past knowledge transfer and fogor is the neural network with parameters OSDP obtained by equation 5. Before balancing the two terms for each task, since model parameter update proportional gradient, we normalize the scale of the two terms by ml
[(x, y) = A (cn(2, y) + (1 Ax) . 7) LKD(Z),
So (0)
VyLcEl
7
V CkDl
(11)
where f is the feature layer of the model and & a batch index. Note that we use the gradient norm at the feature layer since that of the earlier layers will be proportional to it thanks to the chain rule.
ICueel A is typically defined as [Cau++CCal (Wu al., 2019), where Cnew the new classes, i.e., classes in the current task and Cola is old classes, i.e., classes from the previous tasks. In the online boundary- task is not defined, the notion of new old is not available for each class. Instead of hard assignment of new and old classes, we measure S 7 S that represents how new class is. The % is measured by the inverse of past model's average confidence over samples in class i. since the model will have low confidence for new classes. If a past model predicts class i with p(t) 1, consider that the model has completely learned class so 7/ 0. If it predicts class with p(i) < N where N is the total number of classes, Le., no better than a random model, we consider that the model has learned nothing about class c, so % To compute %. we use the samples' confidence on currently learned model at the current time step before using them for training. as a proxy of validation accuracy.
free setup, unfortunately,
Finally, we define the online balancing parameter Ae by averaging confidence for all the samples for each class in the current time step as:
Li=o 7 N
if p(t) < 1/N, otherwise.
A
where 71 = N
(12)
N-1 (1))())),
6 EXPERIMENTS
Experimental Setup. For empirical validations, following (Koh et al., 2021; Guo et al., 2022), we use four benchmark datasets; CIFAR-10, CIFAR-100, TinylmageNet, and ImageNet. Most of our experiments are on Gaussian and Periodic-Gaussial online stream defined in Sec. 3.1 for boundary- free setup except in Sec. 6.3 where we evaluate methods disjoint stream using E-splitCIIAAR We use = 0.1 for Gaussian setup, and a = 0.1, R = 5 for Periodic- -Gausiiann setup.
For evaluation metrics, we use the area under the curve of accuracy (AAuc) and last accuracy (Atest) for overall performance, and knowledge loss ratio (KLR) and knowledge gain ratio (KGR) for sta- bility and plasticity, as defined in section 4. KLR and KGR are measured every 10, 000 samples for CIFAR-10 10 and CIFAR- 100, every 20, 000 samples for TinyImageNet, every 100, 000 samples for ImageNet. All results are averaged over different random seeds, except ImageNet (Bang et al., 2021; Koh et al., 2021) for computational cost. We will publicly release the implementation of our method and the continually learned models.
Method Detail. SDP an episodic memory, updated by the Greedy Balancing Sampler (Prabhu et al., 2020). For training, we use only the samples randomly selected from the memory, following (Koh et al., 2021). We provide a pseudocode of SDP in Appendix Sec. A.4. For the hyperparameters, we use / 10, 000 and c² = 0.75 found by hyperparameter search on CIFAR-10 non-periodic Gaussian setup, for all experiments. Details can be found in Appendix Sec. A.9
Baselines. We compare our method with other baselines that can be used in task-free setup: ER (Rolnick et al., 2019). DER++ (Buzzega et al., 2020), ER-MIR (Aljundi et al., 2019a), Gdumb (Prabhu et al., 2020), and CLIB (Koh et al., 2021). Note that since our setup is boundary- free, only the task-free methods can be applied.
7

---

Published as a conference paper at ICLR 2023
CIFAR-I0
CIFAR-100
Methods
AAuC 1
Ana
KLR
KG
AxUC
Aba f
K 4
KGR
ER
55.97400.5
58.5942.56
23.36+1.39
21.52++0.92
36,93=0.77
40.9340.35
51.77+0.87
17.42-4.50
DER++
56. 16.1244.11
60,3412.37
23.6042.12
22.57=0.98
28.77=0.67
34.48:0.62
48.83++1.49
17.35=0.16
56.48-40.18
59.3842.81
25.04+1.67
21.72+0011
34.76-1.20
37.20+22.66
57.2341.14
16.46-00.88
GDumb
46.45=00.68
42,62=0.64
35.47=0.51
11.69-0.32
28.81=0.74
25.38:40.26
86,75+2.07
8,54=0.34
CLIB
63.01:10.31
65,2412.10
20.7921.64
22.52+11.2
42.79=0.65
43.8140.58
48.59
15.9640.26
SDP (Ours)
66.51:10.42
76.2910.50
10.75=11488
11.19:1.16
46.34=0.64
54.93:0.566
36.5610.92
18.77:0.27
TiyIInageNet
ImageNet
Methods
AAuC f
Aua f
KLR,,
KGR
AauC
Aba f
KLRma 4
KGRon
ER
21.78:+0.6
22.64:0..4
66.39:1.36
9.04:0.88
24.16
26.64
6.19
DER++
17.13+00.8
16.85+0.63
69.65-4..1
7.51440.64
19.34
21.17
40.64
5.71
ER-MIR
21.64:40.77
22.55:00.66
66.51=1.18
9,43+0.46
24.70
8.23
GDumb
18.81=0.64
15.08=0.25
91,58+2.15
3.60:0.10
13.74
10.24
64.23
2.95
CLIB
27.53:40.94
26.10:0..3
63.97:22.18
6.16++0.9
34.11
31.60
36.45
6.20
(Ours)
30.49-14.69
32.59+0.20
54.85+0.300
8.30+0.10
8.25
Table Accuracy of continually learned model in Non-Periodic Gaussian data stream in CIFAR-
10, CIFAR-100, TinylmageNet and ImageNet. The results except ImageNet are the average values of the three random seeds.
Implementation Detail. We use ResNet-18 (He et al., 2016) the network architecture for all experiments. We set training hyperparameters following (Koh et al., 2021; Bang et al., 2021; Prabhu al., 2020). For CIFAR-10, CIFAR-100, TinylmageNet and ImageNet, 16, 32, 256, number of updates per sample of 1, 3, 3, 0.25, memory size of 500, 2000, 4000, 20000, respectively. We use Adam optimizer with LR of 0.0003 for all datasets and setup. Constant LR schedule is applied unless the method specifies the LR schedule (Koh et al., 2021; Prabhu et al., 2020). For data augmentation, some prior works (Koh et al., 2021; Bang al., 2021) use AutoAug- ment (Cubuk et al., 2019a). However, AutoAugment requires searching the augmentation policy using the dataset, so it not applicable for our setup. Instead, we use CutMix (Yun et al., 2019) and RandAugment (Cubuk et al., 2019b) with fixed hyperparameters for requiring no policy search.
batch size of 16,
6.1 RESULTS ON NON-PERIODIL GAUSSIAN DATA STREAM
As shown in Table 2, our method outperforms the other baselines on all benchmark datasets. Espe- cially, observe that the last accuracy (Atss) is better than the other baselines by large margins, which verifies that our method can quickly adapt to the current task online boundary-frea setup.
In addition, the lower LR score shows that our method is more robust forgetting.
6.2 RESULTS PERIODIC GAUSSIAN DATA STREAM
The results of our method and other baselines on periodic-gaussian online stream is displayed in Table observed that our method is superior other baselines on various datasets with higher accuracies (AAuC, Atas). It is noteworthy that our forgetting scores (K LR) is considerably lower compared to other baselines. We believe that our method can transfer information from the appro- priate past, and thus be adventageous when the data distribution repeats periodically.
6.3 RESULTS ON DISJOINT TASK SPLIT
As our method is not specifically designed for the boundary-free task setup, we also compare our method prior arts in the disjoint setup. We summarize the results 5-split-CIFAR-10 (i.e., partitions 10 classes into 5 tasks) and 5-split- Asplit-CIFAR-100 in Table 4 on the conventional task-split setup. As shown in the table, the proposed SDP outperforms other methods even in the setups with clear task boundaries.
6.4 ABLATION STUDY
We now ablate the model for the two components, and summarize the results Table 5. We ob- serve that both the scheduling function and adaptive loss balancing contribute in improving the performance. Baseline is a simple episodic- based method using Greedy Balancing Sam- pler (Prabhu et al., 2020) and memory-only training (Koh et al., 2021). By using SDP distillation,
8

---

Published as a conference paper at ICLR 2023
CIFAR-I0
CIFAR-100
Methods
AAuC 1
Ana
KLR
KG
AAuC
A f
KL 4
KGR
ER
61.1740.97
67.2942.58
17.87+00.48
22.44+1.005
37.90-0.31
46,45+042
48.83+1,499
17.35-+0.16
DER+
60.70:11.07
70.41
16.47:40.53
23.45=00:9
23.07=1.22
17
48.72=
15.56=+0.50
60.16++1.70
67.86+11.06
18.49+1.01
22.37+00.11
35.87-0.28
45.08-+0.35
51.26+0.60
16.7340.12
GDumb
43.01=00445
42.97=0.91
39,88+1.90
12.87=0883
25.01=0.13
25.37+:1911
88.84+1.91
7.82=0.11
CLIB
66.18:10.64
73.8621.38
16.52400.43
24.9911.00
42.48=0.54
10.9111.01
46.9410.8
17.08:0.27
SDP (Ours)
69.68:10.3
78.6610.91
5.5410.22
18.54
4..90=0.32
55.62:1.166
33.30=0.41
21.41:0008
TinIInageNet
ImageNet
Methods
AAuC f
Aua f
KLR,,
KGR 1
AAuC
Aba f
KLRmn 4
KGRun
ER
20.67:0025
24.8410.49
66.65=1.117
7.14+0.12
25.25
34.93
43.04
9,39
DER++
16.6740..4
21.67+0.86
66.39+1.36
7.63+0.13
20.57
27.85
44,96
9.87
ER-MIR
20.74440.36
24.57+1.000
68.40-11772
7.10+0.06
19.52
31.85
44.73
8.65
GDumb
15.75=0.112
15,40=0.76
95.73=3.75
3.15:0.17
11.22
11.38
67.32
2.59
CLIB
25.45:+0.39
30.77+11.40
61,24:1.37
7.4740.16
32,90
38.96
39.37
8.25
SDP (Ours)
30.14+00.32
37.55+0.43
49.95-+0.30
10.68-00.09
33.38
Table 3: Accuracy of continually learned model in Periodic Gaussian data stream. The results
except ImageNet are the average values of the three random seeds.
CIFAR-I0
CIFAR-100
Methods
AAue f
Aua
KL
KGR
AAuC
Asa 1
K KRu L
KG
ER
73.52-0.87
58.95+2.11
2..95+1.41
25.37++121
49.21-1.05
39.86+0..52
58.02+ 0.99
13.83+00.45
DER++
71.01:1.26
55.3712.47
30.0243.22
23.6241..8
4..03=1.29
37.60:10.75
19.9511.18
13.79:10.63
ER MIR
71.69+0.91
56.01 +2206
28.68
23.69+0.52
48.48-1.19
37.60+0.34
62.95+1.31
12.88-00.22
GDumb
62.23-40.70
44.69:1..11
25.6441.45
11,44+0.33
37.80=0.31
24.77:40.68
79.3343.13
9.04-0.144
CLIB
73.62:10.87
56.21.:3.30
28.68 14.43
23.69::0.2
50.26 1.06
40.05:10.71
59.48:12.00
12.23:10.46
SDP (Ours)
76.89:11.22
66.1911..48
15.86:1..44
25.68::2.61
55.94
50.26:11.41
46.35:12.44
16.04:10.80
Table 4: Results of disjoint task split on CIFAR-10/ / 100. Each of datasets are split into tasks
we have 1.7~~5.6% improved AAUC and 5.2-8..4% improved Atest compared to baseline. Since we didn't t apply adaptive loss balaneing, we use fixed balaneing parameter optimized by grid search on CIFAR-10 10 non-periodic Gaussian stream. By applying adaptive loss balancing, we have further improved performance, except for Atast in CIFAR- -00 periodic-Gaussian setup. still outperforms baseline and other methods by large margins.
CIFAR-10
CIFAR-100
Methods
Goussian
Periodic Goussian
Goussian
Periodie-Gaussian
Auu: 1
Arext
Aauc
Atooe
Awux
Auost
Aaue
Auore ?
Buseline
62.37:10.52
64.22.0.97
64.95=
72.55:0..71
42.02-0.558
43.1910.12
40.54
48.15110.39
Schecluling Function
64.04:00.21
69.47+1.7
67 07.79-0.21
71.87+0.24
45.73-4040
50.53+0.18
46.14-0.00
56.56+00.66
+ Datn-driven Balancing
66.51:10.42
76.29:0.50
69.68 0.33
78.66::0.91
46.34
54.93:0.56
46.90
55.62:0..16
Table 5: Ablation Study
7 CONCLUSION
For working better realistic continual learning scenario, we propose a continual learning setup of continuous data stream without explicity task boundary defined, called boundary-free continual learning, with the constraint of learning the data in online manner. We also investigate the periodicity of the data distribution. As the existing evaluation metries for continual learning setup assumes the explicit task boundaries, they are not readily applicable in the boundary- setup. Thus, we propose two new information theoretic evaluation metrics quantify the amount of information loss and gain over the data stream, named knowledge loss and knowledge gain. To address the continuous data stream to incrementally update the model, we propose method leverage the previously leameed knowledge by skewed bell shaped weighting function in a distillation framework, named scheduled data prior. For further generalization to different benchmarks, we propose to balance the pace of learning of knowledge in the past and the present in data-driven manner. In our empirical evaluations, the proposed method outperforms comparable prior arts that update the weights per each data batch in stream in the proposed boundary- -free setup well as the conventional disjoint task-split setup.
Limitations. Since method does not leverage any notion of periodicity, it is expected to im- prove the method by considering the fact that the data stream has a periodic, though we may not know the duration of the period.

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENT
Continual learning (CL) including our approach aims to address real-world scenarios where data distributions continuously change in a non-stationary manner. Therefore, it can alleviate some eth- ical limitation of models resulting from lack of latest knowledge by continuous training and eval- uation on newly observed data. This effort can be more effective handle recent large language models (Brown et al., 2020; Kim et al., 2021a; Chowdhery et al., 2022) or foundational vision models (Bommasani
al., 2021). On the other hand, CL methods might be vulnerable to model bias (Rae et al., 2021) caused by various bias inherent in the changing data due to its continuous tracking. This ethical issue a future research topic. This may expose the public discrimination by the deployed deep models due to unsolved issues in deep learning such.
REPRODUCIBILITY STATEMENT
We take the reproducibility in deep learning very seriously and highlight some of the contents in the manuscript that might help reproducing our work. We will definitely release our implementations, learned models and newly derived datasets used in our experiments as mentioned in Sec. 6. Also, we have included relevant implementation details in Sec. 6. Finally, we present our final optimization objective (Eq. 10) with necessary details to reproduce the methodology.
ACKNOWLEDGEMMEN
This work is partly supported by the NRF grant (No. 200.2222RAA2440020000 HITP grants (No.2020-0- 01361-003, Al Graduate School Program (Yonsei University) 5%, No.2021-0-02068, AI Innovation Hub 5%, 2022-0-00077, 20%. 2022-0-00113, 20%, 2022-0-00959, 15%, 2022-0-00871, 15%, 2022- 0-00951, 15%) funded by the Korea government (MSIT).
REFERENCES
Rahaf Aljundi, Rugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia. Online continual learning with maximal interfered retrieval. NeurIPS, 32:1 1849-11866. 2019a. 1.
Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars. Task-free continual learning. In CVPR, pp. 11254-1 1263, 2019b. 1,2
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. In NeurlPS, pp. 11816-11825, 2019c. 1,
Jihwan Bang, Heesu Kim, YoungJoon Yoo, Jung- Woo Ha, and Jonghyun Choi. Rainbow memory: Continual learning with a memory of diverse samples. In CVPR, pp. 8218-8227, 2021. 2, 3, 7,
Jung-Woo
Jihwan Bang, Hyunsco Koh, Seulki Park, Hwanjun Song, Jung- Woo Ha, and Jonghyun Choi. Online continual learning on contaminated data stream with blurry task boundaries. In CVPR, June 2022.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportu- nities and risks of foundation models. arXiv preprint arXiv:2108.0723 2021. 10
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NeurlPS, 33:1877-1901, 2020. 10
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: strong, simple baseline. NeurlPS, 13:15590-15930, 2020. 2, 7
Zhipeng Cai, Ozan Sener, and Vladlen Koltun. Online continual learning with natural distribution shifts: An empirical study with visual data. In ICCV, pp. 8281-8290, 2021. 1
10

---

Published as a conference paper at ICLR 2023
Francisco M. Castro, Manuel J. Marin-Jimenez, Nicolas Guil, Cordelia Schmid, and Karteek Ala- hari. End-to-end incremental learning. In 2CCVV2018 2
Arslan Chaudhry, Puneet K. Dokania, Thalaiyasingam Ajanthan, and Philip H. S. Torr. Riemannian
walk for incremental learning: Understanding forgetting and intransigence. ECCV, 2018a. 4, 17
Arslan Chaudhry, Mare' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. In ICLR, 2018b. 3
Arslan Chaudhry, Mare' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with A-GEM. In ICLR, 2019. 2
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 2022. 10
Ekin D. Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V. Le. AutoAugment: Learning augmentation strategies from data. In CVPR, June 2019a. 8
Ekin Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical data aug- mentation with no separate search. arXiv preprint arXiv: 1909. 13719, 2(4):7, 2019b. 8
Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey; Defying forgetting in classification tasks. TPAMI, 2021. 3
French. Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences, 1999.
Yiduo Guo, Bing Liu, and Dongyan Zhao. Online continual learning through mutual information maximization. In International Conference on Machine Learning, pp. 8109-8126. PMLR, 2022.
He, R. Mao, Z. Shao, and F. Zhu. Incremental Learning In Online Scenario. CVPR, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pp. 770-778, 2016. 8
Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning a unified classifier incrementally via rebalancing. In CVPR, pp. 831-839, 2019. 2
Boseop Kim, HyoungSeok Kim, Sang- Woo Lee, Gichang Lee, Donghyun Kwak, Dong Hyeon Jeon, Sunghyun Park. Sungju Kim, Sconhoon Kim, Dongpil Seo, et al. What changes can large-scale language models bring? intensive study on hyperclova: Billions-scale korean generative pre- trained transformers. arXiv preprint 1rXiv:2109.04400 109.04650, 2021a. 10
Chris Dongjoo Kim, Jinseo Jeong, Sangwoo Moon, and Gunhee Kim. Continual learning on noisy data streams via self- oolf-purified replay. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 537-547, 2021b.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Barnskkaaaaiiiaaaaa.aa al. Overcom- ing catastrophic forgetting in neural networks. PNAS, 114(13)3521-3326, 2017. 2, 17
Hyunseo Koh, Dahyun Kim, Jung-Wer Woo Ha, and Jonghyun Choi. Online continual learning on class incremental blurry task configuration with anytime inference. In ICLR, 2021. 1, 3, 4, 7,
Zhiqiu Lin, Jia Shi, Deepak Pathak, and Deva Ramanan. The clear benchmark: Continual learning on real-world imagery. In Thirty-fitth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. 21
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. In NeurIPS, 2017. 2,3
11

---

Published as a conference paper at ICLR 2023
Zheda Mai, Ruiwen Li, Jihwan Jeong, David Quispe, Hyunwoo Kim, and Scott Sanner. Online continual learning in image classification: An empirical survey. Neurocomputiing. 2022. 3, 6, 21
Marc Masana, Xialei Liu, Bartlomiej Twardowski, Mikel Menta, Andrew D Bagdanov, and Joost van de Weijer. Class-incremental learning: survey and performance evaluation on image classifi- cation. arXiv preprint wrXiv22000.15777 2020. 2
Ameya Prabhu, P. Torr, and Puneet K. Dokania. GDumb: A simple approach that questions our progress continual learning. In ECCV, 2020. 1,3, 7, 15
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:21 1446, 2021. 10
Sylvestre-AIvee Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. iCaRL:
Incremental classifier and representation learning. In CVPR, 2017. 1.2
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Greg Wayne, Experience replay for continual learning. In NeurlPS, 2019. 7,7
Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning. In ICLR, 2021. 1,2
Murray Shanahan, Christos Kaplanis, and Jovana Mitrovic. Encoders and ensembles for task-free continual learning. arXiv preprint MXXiv:2105.13221 2021. 1,
Zifeng Wang, Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vin- cent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In CVPR, 2022. 1, 3,4 4
Yue Wu, Yan-Jia Chen, Lijuan Wang. Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In CVPR, 2019. 2, 7, 17
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, pp. 6023-6032, 2019. 8
12

---

Published as a conference paper at ICLR 2023
A
APPENDIX
A.1
DETAILS ABOUT GAUSSIAN AND PERIODIC GAUSSIAN DATA STREAM SETUP
1.2
class
class 2
class 3
class
class
class
class 3
class
1.0
frequency
frequency
0.8
2
0.0
time
(a) Gaussian data stream
(b) Periodic-Gauue Gaussian data stream
Figure 4: Distribution for (a) Gaussian and (b) Periodic Gaussian data streams
The resulting distribution with N = 4 is depicted in Fig. 4-(a). The periodic Gaussian distributions with = and are illustrated in Fig. 4-(b).
A.2 DERIVATION OF SDP RATIOS FROM MEAN AND VARIANCE
The weights of EMA is same as the pmf of geometric distribution, with k-th weight (a(k) obtained as:
wa(k) = ax1 ax)* k-1
(13)
when is the EMA ratio. Using the known mean and variance of geometric distribution, Thus, mean Ma of EMA is obtained as:
Na kw (k)
(14)
The variance of is:
1-a
0 wa (k)
11aa
(15)
a²
Thus,
Σ
00
2 Wa (k(k)
1
1
2
(16)
a²
a²
Using the definition of SDP, 5, we obtain 1 ((a,g) the k-th weights of SDP with two ratios a and as:
a a (b(k) a - wa(k)
B
W (()
(17)
The mean ((a,8) of SDP is calculated as:
P(a,8)
kw (a (k)
α
8
kwo(k)
kw (k)
3
k
k
3
Bla - 3)
ala - 3)
a+
1
+
(18)
a3
a
3
13

---

Published as a conference paper at ICLR 2023
Also, the variance (a of SDP is calculated as:
Σ
k2 ()) p(a,8)
k
a
Σ
k wwp(k)
B
Σ
k (k(k) ((a
a 3
a-
&
&
a(2 ()
B(2 0)
(a + B)
8(a 5)
a²
B]
a²
a² + 8 a + B)
(19)
a²
Thus, the squared coefficient of variation
a² 22
n²² is:
2² (a + 3)2 ar3(a + 3 +
(20)
(a + B)²
Recall
P(a,0) a+ B
(21)
aB
Solving these equations for and aB by substitution, we get
aB
(22)
6 122 + 1
2p
Q + B
(23)
2² 2 +
Solving this equations for a and 3 using quadratic formula, we get:
1 + 2 +
1- 26
a= 111
8
(24)
111
c²)
A.3 ANALYSIS ON STANDARD DEVIATION OF GAUSSIAN DISTRIBUTION
Sigma=0)
Sigma=10
AAue 1
Apa
KLR
KGRmg
Axic 1
Age 1
K KRl. 1
KG
ER
73.52-10.87
58.95
24.95:11.41
25.37::1.21
55.97=0.85
58.5911256
23.3611.39
21.52:10.22
DER++
71.01:1126
55.37=2.47
30.0243.22
23.62+11.8
12.124
60.54:2.37
23.60#2.12
22.57:10.98
ER-MIR
71.69:10.91
56.01
28.68:14.43
23.69:10.52
56.48=0.18
59.38:12.83
25.04:11.67
21.72:10.16
GDumb
62.23:10.70
44.6911.11
25.64:11.45
11.44:0.33
46.45=0.68
42.62.0.64
35.4710.51
11.69:1.322
CLIB
73.62:10.87
56.21
28.68:1443
23.69:1.52
63.01=0.31
65.24:12.10
20.79:1.64
22.52:11.92
SDP (Ours)
76.89-11.22
66.1913.48
15.86:11.74
25.68:12.61
66.51=0.42
76.29-1.50
10.75:1.48
23.19:11166
Sigma=25
Sigma=50 50
Methods
Axu f
Apa
K.ll
KG
Asuc
Aua f
K. 1
KG
ER
60.42:10.80
68.7412..5
1..32
26.89:1181
58.25 -8.25
58.59112566
23.36::.39
21.52:10.2
DER++
58.25:10.73
68.58:10.79
16.96:1.677
27.49-2.10
64.39=0.91
75.82++1.88
12.5610.88
26.39-10.49
ER-MIR
60.94:10.92
69.21110.99
17.9410.99
25.67:10.3
66.82-0.44
76.3811.31
13.8410.11
25.20:10.69
GDumb
41.19-40.18
41.79+0.82
43.04-2.111
12.45+0.24
41.65-0.16
42.6240.59
39.204-.200
12.98-00.75
CLIB
66.39:10.09
71.41:1.32
19.74:10.77
26.94.1069
69.62:00422
72.31:11.21
7.42-1.122
12.874151
(Ours)
69.97.10.37
81.95:0.89
4.2311064
20.67::2.30
73.44=0.26
82.33:10.26
3.57110.2
16.24:1.59
Table 6: Comparison Accuracy for different values standard deviation for Gaussian data Stream
in CIFAR-10
We test CL methods on 4 different values of standard deviation for Gaussian data stream in CIFAR- 10 and CIFAR-100. SDP outperform other methods in all tested values of sigma, showing that our model is robust to how spread out the distribution is.
14

---

Published as a conference paper at ICLR 2023
Sigma=0
Sigma=10 10
Methods
AAuC 1
Aba
K.
KGR
AxuC
Aba f
Kla 4
KGRua
ER
49.21+-1.05
39.86+00.52
58.02+0.977
13.83+0045
36,93=0.77
40.9340.35
51.77+0..7
17.42-4.50
DER++
46.03:11.9
37.60:0.75
59.95=1.18
13.79:10.33
28.77=0.67
34,48:0..2
48.83++149
17.35=0.16
18.48++1.19
37.60+0.34
62.95+1.31
34.76-1.20
37.20+22.06
57.2341.14
16.46-00.88
GDumb
37.80=0.31
24.77=0.68
79.33=3.13
9.04+0.14
28.81=0.74
25.38:00.26
86,75=2.07
8,54=0.34
CLIB
10.26:1:006
40.05:0.71
59.4822.00
12.2340046
42.79=0.65
43.8140.58
48.5921.78
15.9640.26
SDP
55.9411.55
50.2611144
46.35-2.44
16.04:1.80
46.34=0.64
54.93:10.56
36.361:0922
18.77:0.27
Sigma=25
Sigma=50
Methods
AAuc 1
Aua
KLR
KGR,.
AAUC
Abai 1
K.R 4
KGRing
ER
19.21=1105
39.86+0.522
58.02-40.97
13.83=0045
32.86=0.399
45.11=11111
50,75=1.055
17.38 17.38=0.43
DER++
16.03:1129
37.6010.75
19.95=1.11
13.79:0.63
31.2551.559
1..00:1:..5
47.7620.64
17.5740.06
ER-MIR
1.19
37.60+0.34
62.95+1.331
12.88:0032
31.85=0.17
44.60+0.17
53.32+00.9
16.40++0.30
GDumb
37.80:10.31
24.77:0.68
79.33=3.13
9.0440.14
23.98=0.13
24.30:0055
90.32+1..50
7.66=0.17
CLIB
40.06:10:20
46,90:10877
49.71=0.60
16.51110.09
50,26:1.06
40.05:10.71
59.4812.00
12.23:0.46
SDP (Ours)
43.88:10.57
55.32:10.50
32.92:10.68
20.00:10.24
46.50=00.42
55.97:10.33
32.89
21.57:0.100
Table 7: Comparison of Accuracy for different values of standard deviation for Gaussian data Stream in CIFAR-100
Algorithm Pseudocode for SDP
Input model fa. EMA models fo.. feg. EMA ratios a, Memory M, Training data stream
Learning rate μ
2: 8a 60,, 0g +-
EMA models initialize
3: for (I.y) E D do
sample arrives from training data stream D
4:
Update M f- GreeddyBalnncingSanpler [M, (2r, v))
> Update Memory
5: @sDe(t) = a- 3 (t -- (t).
9
Calculate SDP model parameters
6:
Update 7v + UpdateConfidenceMen (fo (x), W- %
Update class-wise confidence
7:
Update A
Update balancing parameter A Get batch (X, Y) from Memory Calculate cross-entropy loss Calculate distillation loss
8:
Sample (X, Y) f RandomSample(MM)
9:
CCe(X XX Y) CrossEntropy (fo,
10:
CKD = [f(X) - fer (
V f
11:
Th
Obtain batch balancing factor тı
12:
C(X, Y) = d Lcu (X X) + +0)++11-01
(1- de) (1-xe) 7k (X)(X)
> Calculate total loss Update model
13:
9 0 μ VoL(X,
14:
Update An + (1 a) -0 +
b Update EMA models
Update 0g3 (1 3) 3-0
16:
end for
Output fe
A.4 PSEUDOCODE FOR THE SDP FRAMEWORK
Algorithm provides detailed pseudocode for SDP. When the sample enters time order, the mem- ory updated by Greedy Balancing Sampler (Prabhu et al., 2020). In addition, update the balancing parameter based on the confidence obtained by running inference on new samples using SDP model. SDP model parameters
of two EMA model parameters. The updated SDP serves as a data prior for distillation during training. After training the model using the sampled batch (X,Y7 Y). update the EMA model fe.. fea.
obtained by weighted
A.5 PSEUDOCODE FOR MEASURING KLR AND KGR
Algorithm 2 show how KLR and KGR are measured in practice. We obtain predictions for the test set at two different timestamps, t and tz. Then we calculate the joint probability for GT, prediction at ti and prediction tz. KL and KG are calculated as the conditional mutual in- formation. Line 6 and line are derived from the equation for the conditional mutual informa- tion (X;Y T(X, Y) T(Z; Y), and equation for computing the mutual information
15

---

Published as a conference paper at ICLR 2023
Algorithm 2 Computing KLR and KGR
1: Input Test set (X, Yor), Previous model output Yi, Current model fo(t2)+ Set of Possible Labels
V1, ,1,3/2
2: Y2 [arg max Se(tz) (3:) for E X
Inference set with model
3: for (ygs M1, 12) (Ver, 3/1 3/2 do
P(Uge V1, (2) R(ve. (a)r Yorl
4:
D. Calculate joint prohability
5: end for
P() ( P y2)
6: (2)
Σ
Σ
Σ
P(ug: 122 log
P(yg: y1 P(y,,32
By Ear yn E3i y2E
Calculate KL by Conditional Mutual Information
((y2) PWyas v1)
7: KG(1)
= Σ
Σ
Σ
Plugt Un, 82) log
P(ug, V2) (111. U2)
Ear WEE) 92E32
CCacculate KG by Conditional Mutual Information
P(yg, v1)
TK(t)) E
E
Ply 31 log
P(yn)
Wer yn EiM
D Calculate TK at tl by Mutual Information Calculate the amount of information (entropy) in GT label
9: H(Gor) =- Σ
P(yge log P(ya))
10: t2) = (2)
D. Obtain KLR
(t(
(G(t1.t2)
11: KGR(tI,,t) = H(YGT) - (K)))
P Obtain KGR
Output Y2. KLR((,, 12), KGR(h1,4 42)
ER
+ER+
ER-MIR
CLIB
EMA only
SDP (Ours)
68
78
66
64
AAUC
62
69
66
60
58
60
57
54
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
Training Time (GPU Hrs)
Training Time (GPU Hrs)
Figure 5: AAUC and Ausst of various methods compared to training time, on Non-Periodic Gaussian data stream in CIFAR-10.
I(X;Y) ΣE P(e log P(x.y) v) ) P(r)
The marginal probabilities obtained by sum-
DEJ mation over other dimensions.
A.6 COMPUTATIONAL COMPLEXITY
We compare the computational complexity by reporting the training time on a single NVIDIA GeForce RTX 2080Ti. 'EMA only' is method using EMA model for distillation, without
16

---

Published as a conference paper at ICLR 2023
1.0
0.8
0.6
0.4
Gaussian
0.2
Perisdic-Gaussian
0.0
10k
20k
30k
40k
50k
# samples
Figure 6: Visualization of balancing parameter X over time, on CIFAR-10 Gaussian and Periodic- Gaussian data stream.
ing data-driven loss balancing. We did not report GDumb since GDumb does not train at training time. Instead, GDumb trains from scratch at inference time, their computational cost depends on the frequency of inference queries. SDP shows good tradc-off between computational complexity and performance, compared other methods.
A.7 DISCUSSION ON THE ADDITIONAL MEMORY COST
Method
Additional Memory
Theoretical
ImageNet (MB)
ER
N, N
3,010.6
DER+
N. N N
3,090.6
ER-MIR
N₂ Ng +
3,057.3
GDumb
N,- -
3,010.6
CL.BB
N, -
3,010.6
SDP (Ours)
N. S 2
3,100.0
Table 8: Additional memory cost for storing models, replay buffers, and logits. We report theoretical values and actual memory cost on ImageNet.
We compare the additional memory required to save models or samples in Table. 8. SDP has an additional memory cost of N, N: S + 2 No. where N, the number of exemplars stored class, N₂ is the number of classes, S is the size of an image, and No is the size of model parameters. Overall, SDP uses more memory than other compared methods since requires storing model weights for SDP while other methods only store samples in episodic memory.
Since the resolution of the image in the real world is larger than similar to the resolution of the ImageNet images, we compare the memory costs in the ImageNet as the closest baseline for the real world applications. As shown Table, 8, the extra memory cost of storing the model checkpoints is negligible compared to the memory cost of episodic memory.
Storing model parameters from previous tasks for regularization or distillation is a common prac- tice in CL with task boundaries (Kirkpatrick et al., 2017; Chaudhry et al., 2018a; 18;; Wu et al., 2019). However, such methods were not actively studied in task-free CL, since one cannot decide which checkpoint to store. For comparison with other regularization patrick et al., 2017) requires 2 T. No where T is the number of tasks, EWC++ (Chaudhry et al., 20188 8a) requires N, Nr S Na (31 (3150.6MB quires N (3244. 1BB on ImageNet), and BiC (Wu et al., 2019) requires N (3057.3MB ImageNet).
distillation methods, EWC (Kirk-
ImageNet), RWalk (Chaudhry et al., 201 8a)
A.8 VISUALIZATION LOSS BALANCING PARAMETER OVER TIME
In this section, we discuss the behavior of the data-driven balancing parameter X, over time. A leamss to adjust the weight of distillation loss to follow the performance the SDP model. Thus, in the early phase where SDP model's performance is low, the distillation loss has small weights (high A₂), and as the model learns and SDP model's 's performance increases, the weight for the distillation loss becomes larger (lower X,). We provide visualization of X over time in CIFAR-10 10 Gaussian
17

---

Published as a conference paper at ICLR 2023
70
68
AAAC
64
62
60
2k
10K 12k
14k
18k
0.60
0.70
0.75
0.80
0.85
0.90
SDP
mean
Coefficient of Variation
Figure 7: Comparison of different values of mean and coefficient of variation, evaluated on CIFAR-
10 non-periodic Gaussian data stream.
65.0
64.5
64.0
63.5
62.5
62.0
10
10
10
EMA ratio
Figure 8: Comparison of different values of EMA ratio for EMA model in Table 1, for CIFAR-10 Gaussian non peniperioiic Gaussian data stream.
and Periodic Gaussian data stream in Fig. 6, where this behavior is observed. Since the model's predictions are almost random in the initial phase, the coefficient tends to start with and decrease with the training progress.
A.9 HYPERPARAMETER TUNING ON SDP
We study the effect of two hyperparameters of SDP, mean and coefficient of variation c², by grid search on CIFAR-10 non-pent non-periodic Gaussian setup, and summarize the results in Fig. 7. We observe some dependence on SDP mean, but not much on coefficient of variation. Note that the valid range coefficient of variation 0.5 < c²
Since it is not much relevant to performance,
we choose the value in the middle of valid range, c² 0.75 for all experiments. SDP mean has some impact on the performance, so we choose the optimal value found on CIFAR-10 non-periodic Gaussian setup, 10, 000. Since dataset or setup-specifie hyperparameter search is not desirable in CL scenario, we use the same value, μ = 10, 000, for all datasets and setups.
A.10 HYPERPARAMETER TUNING ON EMA
We also report hyperparameter tuning result for EMA model in Table. 1, searched on CIFAR-10 non-periodic Gaussian setup, which is same as the setup on Table. 1. The performance did not vary much depending on the EMA ratio a, and we used 0.001 as showed highest performance.
A.11 ONLINE CL WITHOUT REPLAY
We consider extreme scenario where episodic memory is not available. Since other CL methods covered this work depend on memory replay, we compare only with basic fine-tuning as baseline. We summarize the result Table 9. We observed that applying SDP the baseline (fine-tuning) improves AAUC by 3.07% for Gaussian data stream and 5.03% for Periodic Gaussian data stream on CIFAR-10. While SDP improves performance, the overall performance falls behind SDP using
18

---

Published as a conference paper at ICLR 2023
Gaussian
Periodic-Gauue
Methods
AuuC 1
A f
AAuC 1
Awaat T
Finetune
37.78 1.28
1.226.11.25
21.99.11.53 0..94 36.97.1.199
SDP (Ours)
40.85
2.562:1.56
25.35:12.65
Table CL with no episodic memory, on Gaussian and Periodic Gaussian data stream in CIFAR-10.
memory replay by 25.66% on Gaussian data stream and 32.70% on Periodic Gaussian data stream. Closing this gap would an interesting research direction for future work.
A.12 ABLATIONS ON THE NUMBER OF REPETITIONS IN PERIODIC CONTINUAL LEARNING
Methods
==
3
Axue f
Aba
KLR
KG
Axuc
Aua 1
KRl.gg 4
KG
ER
18.511:141
64.19:15.33
22.71:2.64
25.00:1.228
59.77=0.77
63.44:1.113
25.67+1.30
26.3411.71
DER++
56.68:
63.80+4.339
22.33-2.11
15.95
58.79
64.8243.18
22.21+1.05
27.10++2.06
ER-MIR
18.89:11.04
61.8714.13
25.51=2.11
24.9911.76
59.92=1.82
63.70:22.15
24.67:1.16
26.5611.115
GDumb
44.28:10.20
39.87:10.68
40.98:2..2
11.92:10.77
0.06
42.07:10.38
37.54
12.48:10277
CLIB
64.53400.06
69.60+0.76
19.8740.91
26.47-40.10
65.41=1.14
71.5240.32
19.2441.08
SDP (Ours)
73.01:10.04
$16110.11
3.63:0.56
16.88:0083
7..81=0.05
81.4::0.25
3,71:10.63
16.16:1.441
11
5
Methods
AAuC 1
Apa 1
KLR
KGlt
Axic 1
Awa 1
K K.lla 4
KG f
ER
60.34:11.91
67.841..10
19.14:10.17
1..69
61.17=0.977
67.29:12.58
17.87.10.48
22.44:11.05
DER++
58.714-1.10
68.7743.90
18.51+0.44
26.12++095
60,70=1.07
7..41++1.83
16.47+0.53
23.45-+0.49
ER MIR
59.69:12.00
64.58.12.58
23.01:11003
23.22::1.02
60.16:1.770
67.86.111.66
18.4911.01
22.37:10.91
GDumb
43.28-40.15
42.1740.52
43.86+3.64
12,94+0.83
43.01-0.45
42.9740.91
39.88+1.90
12.87-00.83
CLIB
66.37-#0.28
72.06=1.05
17.45=11.3
26.42:10.71
66. 68.18:0
13.86:t1.38
16,52=0043
24.99-10.69
SDP (Ours)
72.74:10.09
81.54:10.30
4.20:10133
17.51:1.31
69368-0033
78.66:10.91
5.54:10.22
18.54:11.28
Comparison of Accuracy for different number of repetitions (/t) for Periodic Gaussian data stream in CIFAR-10
Table 10:
We test CL methods on 4 different number of repetition (R) for Gaussian data stream in CIFAR-10 and CIFAR-100. As seen in Table 10, SDP outperform other methods in all tested values of R, showing that our model is robust various number of repetitions.
A.13 CONTINUAL LEARNING ON GAUSSIAN MIXTURE DATA DISTRIBUTION
In the previous experiments, we used Gaussian distribution for data stream, since data as the sum of independent random variables tend to follow Gaussian distribution due to the Central Limit Theo- rem. However, a real-world data may not strictly follow Gaussian distribution, so we also consider a more complex distribution than Gaussian distribution. To test the performance of CL methods on a more complex data distribution, we use Gaussian Mixture, which is a mixture of multiple Gaus- sian distributions. We model the distribution of each class as the mixture of two Gaussians, where mean for each Gaussian is randomly sampled from [0, 1), standard deviation for each Gaussian is randomly sampled from [0, 0.2), and mixture weight for each class randomly sampled from [0, 1). Table summarizes the results for Non-Periodic and Periodic Gaussian Mixture data stream. We observe that SDP still outperforms other methods even more complex Gaussian Mixture data stream setup.
Gaussian Mixture
Periodic Gaussian Mixture
Methods
AAuC 1
Aua 1
KLR
KGR,,,
AAui
Aua t
KRL. L
KGR f
ER
59.
62.2843.77
22.0142.57
23.51++1.56
63.86=00.1
68.6744-.53
16.82411223
22.46-40.39
DER++
99.06:33.22
62,60=5.16
21.40:2.93
25.29=3.34
63.17=0.59
68.71+22.17
15.64
24.52+1..88
ER-MIR
57.95+22.77
61.01+5.14
23.3944.00
23.5742296
62.71
68.97441.58
17.3541.61
23.64++1.17
GDumb
43.13:41.71
42.42+1.94
36.6543.51
11.69-0.31
41.61=0.02
39.60:1.118
48.1343.45
12.73-+0.63
CLIB
65.98:12.24
68.81
18.66:11.29
24.07=:143
68.00=0.89
75.13:22.06
26.01:1.111 15.20
SDP (Ours)
68.85-12.80
76.22-10.67
4.9310.26
17.63+2..99
70.40=1.02
79.96:10.40
4.19:10.4
17.07+1.17
Table 11: Comparison of Accuracy for Periodic (R = 5) and Non-Periodic Gaussian Mixture data
stream in CIFAR-10
19

---

Published as a conference paper at ICLR 2023
2.5
class
class
class
class 4
2.0
frequency
1.5
1.0
0.5
0.0
0
1
3
4
time
Figure 9: Data distribution of randomly selected 4 classes in the mixed-period Gaussian Mixture data distribution setup.
We also observe in Sec. A.16 that in a real-world data, some classes may have different period with others. To model such scenario, we assign randomly assign period for each class from 1, 4, where the length of data stream 4. For distribution of each class, we use Gaussian Mixture with randomly selected two Gaussians as in the previous experiments. In summary, each class distribution is periodic two-component Gaussian mixture with randomly selected means, standard deviations, mixture weights, and periods. We visualize an example of such distribution in Fig. 9. We summarize the result on this Gaussian Mixture setup with mixed periods in Table 12. SDP outperforms other methods even on the mixed-period Gaussian Mixture setup with complex data distributions, which indicates that SDP works well various data distributions.
Multi-Periodic Gaussian Mixture
Methods
AAuc 1
Aus
K.
KGRugg 1
ER
60.2410.94
66. 16.699
18.83:20.46
26.38:1.24
DER+
59.97411.79
70.13-1081
17.18=20.44
28.35+1.12
ER-MIR
1..45++1.83
6..34++.25
1..73+1.92
27.06+1122
GDumb
42.35++1.18
40.25=1.88
37.62=3.55
11.60=1.111
CLIB
6..78++1.12
72.96:40.78
14.98-00.80
26.21::0.98
SDP (Ours)
69.38
79.20
4.92
18.8444
Table 12:
Comparison of Accuracy for Periodic Gaussian Mixture data stream with mixed period for each class.
A.14 ACCURACY TRENDS TRAINING PHASE
We visualize the accuracy trends for the tested CL methods in Fig. 10. From the trends, we can observe that SDP performs increasingly well with the training progress, since distillation becomes more important as the amount of knowledge in the past models increase. We also observe that performance in early phases tend to depend on memory management and usage, as methods that use balanced memory and train only on memory, i.e. CLIB, EMA, SDP, show much higher accuracy than methods that use reservoir sampling and experience replay, i.e. ER, DER, MIR, in the early phases. This likely because reservoir sampling is highly class-imbalanced in the early phases. As the training progresses, the reservoir memory becomes close to balanced, thus the gap between two memory management closes the later phase. However, the mid-to-later phase, the effect of distillation comes into play, and the performance of methods that use balanced memory (CLIB, EMA, SDP) deviates depending on the distillation method used, as SDP > EMA CLIB (No distillation).
20

---

Published as a conference paper at ICLR 2023
ER-MIR
CLIB
SDP
DER+
GDumb
EMA only
Accuracy
0.6
0.5
0.4
10000
20000
30000
40000
50000
samples
Figure 10: Accuracy trends for CL methods over the course of training, on CIFAR-10 Non-Periodic
Gaussian setup.
CLEAR-10
Methods
AAUC
Atast
ER
44.02-10.84
1..91
DER+
38.33-12.09
49.82=3.28
ER-MIR
44.44-11.73
56.18=1.32
GDumb
38.60:10.29
40.73=1.71
CLIB
53.49:10.75
66.91=1.27
SDP (Ours)
56.82:+:0.55
72.97=1.49
Table 13: Comparison of Accuracy for Periodic Gaussian data stream with natural domain shift using CLEAR-10.
A.15 PERIODIC CONTINUAL LEARNING WITH NATURAL DOMAIN SHIFTS
We conduct additional experiments using the recently proposed CLEAR-10 (Lin 2021) dataset that has real-world domain shifts, i.e., the visual concept drift of objects time in the data stream. It is of particular importance many real-world systems such as an e-commerce platform would experience gradual domain shifts in the streamed data for the natural temporal evolution of concepts (e.g., computers in 2020 look different from these in 2010). while being subjected to (periodically) changing data distributions. Note that in the original CLEAR-10 benchmark, the data distribution uniform and stationary. We make a Periodic Gaussian data stream with each bucket as a period and frequency each class follows Gaussian distribution within the bucket, as explained in Sec. 3. We report the results in Table 13. SDP outperforms other methods by significant margins, even though the Greedy Balancing Sampler we use for memory management is reported as not suitable for domain shifts (Mai et al., 2022). It indicates that the distillation part in SDP can effectively deals with domain shifts the data stream.
A. EXAMPLES OF WORLD PERIODIC DATA
In addition the examples mentioned in Figure 1, many search data follow periodic distribution as shown in Figure 11 and Figure 12. As we can see in Fig. 12, search frequency of seasonal fruits and home appliances, which are greatly affected by the season, follows the periodic distribution with the period of year. In the case of clothing, the period is 6 months as we can see in Figure 1. It can be seen that almost similar distribution repeated although there is slight difference in distribution for each period. In order to address these real-world distributions, we propose the periodic benchmark.
21

---

Published as a conference paper at ICLR 2023
Cardigan
Coat
20-
Search Interest
Search
42-
37.an
Jacket
Search Search Interest.
Search Interest
80-
37.jn
18jon
19:on
20 Jon
20-
22-jn
17.50n
10:00
19 Jon
20-Jon
23jon
20:an
Scarf
Sportswear
102
120
MMMMm
Search Interest
Search
40
37.jon
23:on
Figure 11: Search data with 6-month period
22

---

Published as a conference paper at ICLR 2023
192
120
Search Interest
Search Interest
60-
60-
37.
18:an
10:30
20.30
22-jn
17-jam
18:30
19
20-j
27-jnn
22:an
Fan
Grage
Search Interest
80
Search Interest
80
60-
Heating Fad
Journal Notebook
Intereet
Search Interest.
Search Interest
02-
80-
17.jn
18j0n
19:20
29.00
21-bon
22jon
17.jon
18:2n
19 Jon
20-3on
23jpn
20:0n
Pickled Cabbage
Salted Shrimp
130
90-
Search Interest
Search Interest
42-
17.jon
Socks
Sunscreen
Search Interest Interest
Interest
22-
20-
17.jen
18:jan
18.30
20:30
21-Jan
22-jan
37.jam
38:am
19:3n
20.pn
22:jan
Figure 12: Search data with 1-year period
23


Reviews:
[{"review_id": "d49zxcXhjC", "reviewer": "Reviewer_Y7gj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a novel online boundary-free CL method on the realistic boundary-free CL setting, including the newly proposed periodic setup. Moreover, new mutual information-based metric is proposed to measure the loss of past knowledge and gain of new knowledge. Comprehensive experiments have been conducted to evaluate the effectiveness of the method.", "summary_of_the_review": "I think the paper focus on an important problem, boundary-free CL, with a relatively novel solution. Therefore, I recommend weak acceptance initially.", "strength_and_weaknesses": "Strengths:\n- The topic of boundary-free CL is interesting and important.\n- The proposed metrics to measure loss in boundary-free CL is novel.\n- The data-driven balancing part is well-motivated\n\nWeakness:\n- The description of the EMA part is a bit unclear.\n- I am not very clear about why periodic Gaussian distribution is actually challenging in the CL setting. The periodic natural of the distribution naturally serves as a \"replay\" mechanism during the CL process. Therefore, shouldn't the setting be easier than the existing Gaussian schedule setting?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "auvs_nOBA1", "reviewer": "Reviewer_bdyM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work focuses on a more realistic online continual learning---learning continuously changing data distribution without explicit task boundary, which is called boundary-free setup in this paper. The main contributions can be summarized as:\n\n(1) Studying online CL with continuous data stream setup without explicit task boundary, including newly proposed periodic CL setup.\n\n(2) Proposing an online boundary-free CL method that uses scheduled transfer of past knowledge.\n\n(3) Proposing to learn to balance amount of using past and present knowledge.\n\n(4) Proposing new metrics.", "summary_of_the_review": "Marginally below the acceptance threshold.", "strength_and_weaknesses": "My main concern is about the proposed approach.\n\n(1) What is EMA? How to achieve the EMA model ($M_{\\alpha}$ or $M_{\\beta}$)? How to achieve the distillation model ($f_{SDP}(x)$)? It is necessary to describe them in details.\n\n(2) I as a little bit confused about the core ideal of proposed method. Is the distillation model used to guide training model? Please provide a detail description of your methods by understandable way, like flowchart or pseudocode.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Rze79c8bUqF", "reviewer": "Reviewer_VkR2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses the continual learning problem in a more realistic situation by considering the fact that the real-world data is online and has no explicit boundaries and its distribution shifts over time. A novel setup for online task boundary-free has been proposed which modeled the arrival time of data by periodic Gaussian distribution. Then two new evaluation metrics have been proposed based on information theory and the concept of mutual information. These metrics do not depend on old and new tasks and satisfy the online condition. To train their model, they suggest a two-term loss consisting of Cross-Entropy loss for current data samples and a Knowledge distillation Loss for past samples which combines the data by a novel method using hypo-exponential distribution called scheduled data prior (SDP) which is a weighted summation of two exponential moving average functions. The method is claimed to work on both online and task-specific setups and improve the metrics by a large margin.", "summary_of_the_review": "The paper is well written and also have a good novelty ", "strength_and_weaknesses": "Advantages:\n\n\n\nThis paper has a very strong mathematical background.\n\nNovel Setup, metrics and losses have been proposed for online boundary-free continual learning problem.\n\nThe method has been compared to state-of-the-art papers and achieved better results.\n\n\nDisadvantages:\n\n\n\nIt is preferable if a more detailed explanation is provided about the knowledge distillation term of loss function formula 9 on the page and its direct relation with formula 7 on page 5.\n\nH(Y GT) on page 4 has not been defined and it is only mentioned it is [the amount of information in GT.\n\nA_AUC and A_last in table 1 on page 6 are defined afterward on page 7", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DRGk7SBv_Pk", "reviewer": "Reviewer_46rh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies online continual learning in the boundary-free setting, i.e. we do not know when a task starts or ends (or whether it ends at all, like in the periodic setup). The paper then proposes two metrics: knowledge loss and knowledge gain, which measure the difference in performance between two time points t_1 and t_2 (i.e., how the performance improves/worsens between $t_1$ and $t_2$).\n\nTo prevent forgetting, the paper builds upon knowledge distillation strategies. To accumulate the previous weights for the teacher model, the authors propose a different schedule, which they call Scheduled Data Prior (SDP) and show that it works better than EMA. They also use a learnable coefficient to balance between the current classification loss and the knowledge distillation loss.\n\nThe ideas are tested on CIFAR 10/100 and TinyImageNet/ImageNet and the proposed approach beats the previous SotA (i.e., CLIB) by ~5-10% on them in terms of $A_\\text{AUC}$ on all the setups.\n", "summary_of_the_review": "This paper feels like a \"big bag of small contributions\", where each of the contributions could be dropped without affecting the overall paper quality much. In terms of complexity/performance tradeoff, just using EMA without the learnable loss balancing coefficient still looks better to me (i.e. it would be my way to go if I would start the project on CL right after reading the paper).\n\nThe strongest part of this paper is the scores on the leaderboard, but I **subjectively** believe that just the good leaderboard scores shouldn't be a reason for acceptance for a continual learning paper, because the modern CL benchmarks are speculative and too far from the real world anyway, and the improved performance on them does not translate into a clear practical benefit of some real-world system (in contrast to, e.g., modern object detection or segmentation). I am ready to discuss this latter statement and change my mind.", "strength_and_weaknesses": "Strengths:\n1. The overall approach is more or less simple.\n2. Authors explore more realistic setups of CL, which feels to me being a big problem.\n3. The experimental evaluation looks quite comprehensive.\n4. The writing is easy to follow.\n\nWeaknesses:\n1. While it's true that there is a lot of real-world data is periodic, I do not understand how improving on the proposed periodic benchmarks can improve the performance of some real-world system which operates on periodic data. Could you please provide me an example of some real-world system which operates on periodic data and suffers from forgetting?\n2. In terms of complexity/performance trade-off, just using EMA without learning the balancing coefficient looks better to me.\n3. The proposed metrics:\n    - are not well-motivated. It is not clear why $A_\\text{AUC}$ is bad, what important properties do KLR and KGL measure? Why should we measure KLR and KGL, they seem to always correlate with $A_\\text{AUC}$ anyway. For (2), \n    - seem to be overly-complicated. If I understood Section 4 correctly, at the end those information-theory-based metrics go down to \"average accuracy loss over some set of classes\" or \"average accuracy gain over some set of classes\". Then, why not measure these values directly? It would be both more intuitive and have better scaling (one always know the lower/upper bound for them).\n4. Learning the balancing coefficient looks tricky to me. The model can just learn to ignore rehearsing the past data completely. Could you please provide the plots of how this balancing coefficient changes? Also, how do you initialize it? Do you regularize it somehow?\n\nQuestions:\n1. Do you use any replay buffer for the knowledge distillation loss or you compute it on the currently available data?\n2. Do I get it right, that for EMA you store just a single checkpoint of the past model, and for SDP you store two checkpoints?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1tLahIrOtFW", "reviewer": "Reviewer_1ivp", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a more challenging and realistic setup for online- continual learning in which there is no explicit task boundary called a boundary-free setup. Due to the lack of task boundaries, it is difficult to find past information that needs to be preserved to identify the stability-plasticity dilemma. It proposes a scheduled transfer of previously learned knowledge to address this issue. It also suggests learning to balance the amount of using past and present knowledge. Further,  this paper proposes new metrics to measure the loss of past knowledge (i.e., forgetting) and the gain of new knowledge based on information theory.\n\n\n\n\n\n \n\n\n", "summary_of_the_review": "Overall, I found the paper interesting and conducted sufficient experiments for justifications. The paper addresses more challenging and practical issues in the existing continual learning approaches. For more details, please see the strength and weaknesses section.", "strength_and_weaknesses": "Strengths:\n1- Most of the existing continual learning approaches have proposed the task-split setup in which task boundaries are available and well exploited\nin various offline/online CL methods. However, the task-split setup is not suitable for the realistic scenario. In contrast to the prior work, this paper proposes a boundary-free task setup that is more challenging and aligned with the realistic scenario. It removes the notion of artificial task boundary; instead, the data arrival follows a certain distribution.\n \n2- The conventional approaches require the accuracy of the previous task to measure the knowledge forgetting, which is not possible in the proposed boundary-free setup. Therefore this paper presents new metrics for measuring forgetting and the ability to learn new knowledge by computing loss and gain of knowledge based on information theory.\n\n3- To determine the amount of past knowledge to be transferred in a continuous fashion, it proposes a method to leverage the previously learned knowledge by a skewed bell-shaped weighting function in a distillation framework named scheduled data prior. \n\n4- The proposed method shows significant improvement in all the benchmark datasets compared to the state-of-the-art techniques for both the setup, task boundary-free, and disjoint task split.\n\n5-  A detailed ablation study is provided to support the significance of the proposed approach.\n\n Weaknesses:\n1-  It assumes a strict assumption that the streaming data comes from the gaussian distribution, which is not practically true because real data follows a complex distribution. Is it possible to predict the task boundaries based on identifying out-of-distribution distribution between tasks? Or it would be more interesting if it covered complex distribution.\n\n2-  The extreme case for continual learning is where the data examples are available only once, unlike the replay-based approaches, where the old data may be available for future training. Applying the proposed framework to this challenging continual setup would be interesting.\n\n\n\n\n\n\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster