Improving Information Retention in Large Scale Online Continual Learning
Authors: zhipeng cai
Vladlen Koltun
Ozan Sener
Venue: Submitted to ICLR 2023 (2023)
Keywords: online continual learning
moving average
geo-localization

Abstract: Given a stream of data sampled from non-stationary distributions, online continual learning (OCL) aims to adapt efficiently to new data while retaining existing knowledge. The typical approach to address information retention (the ability to retain previous knowledge) is keeping a replay buffer of a fixed size and computing gradients using a mixture of new data and the replay buffer. Surprisingly, the recent work (Cai et al., 2021) suggests that information retention remains a problem in large scale OCL even when the replay buffer is unlimited, \emph{i.e.}, the gradients are computed using all past data. This paper focuses on this peculiarity to understand and address information retention. To pinpoint the source of this problem, we theoretically show that, given limited computation budgets at each time step, even without strict storage limit, naively applying SGD with constant or constantly decreasing learning rates fail to optimize information retention in the long term. We propose using a moving average family of methods to improve optimization for non-stationary objectives. Specifically, we design an adaptive moving average (AMA) optimizer and a moving-average-based learning rate schedule (MALR). We demonstrate the effectiveness of AMA+MALR on large scale benchmarks, including Continual Localization (CLOC), Google Landmarks and ImageNet. Code
will be released upon publication.

Under review as a conference paper at ICLR 2023
IMPROVING INFORMATION RETENTION IN LARGE SCALE ONLINE CONTINUAL LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Given stream of data sampled from non-stationary distributions, online continual learning (OCL) aims to adapt efficiently to new data while retaining existing knowledge. The typical approach to address information retention (the ability retain previous knowledge) is keeping replay buffer of a fixed size and computing gradients using mixture of new data and the replay buffer. Surprisingly. the recent work (Cai et al., 2021) suggests that information retention remains a problem in large scale OCL even when the replay buffer is unlimited, i.e., the gradients are computed using all past data. This paper focuses on this peculiarity to understand and address information retention. To pinpoint the source of this problem, we theoretically show that, given limited computation budgets at each time step, even without strict storage limit, naively applying SGD with constant or constantly decreasing learning rates fails to optimize information retention in the long term. We propose using a moving average family of methods to improve optimization for non-stationary objectives. Specifically, we design adaptive moving average learning rate schedule (MALR).
(AMA) optimizer and a moving-average-Due We demonstrate the effectiveness of AMA+MALR large-scale benchmarks, including Continual Localization (CLOC), Google Landmarks, and ImageNet. Code will be released upon publication.
INTRODUCTION
Supervised learning commonly assumes that the data is independent and identically distributed (iid.). This assumption is violated in practice when the data comes from a non-stationary distribution that evolves over time. Continual learning aims to solve this problem by designing algorithms that efficiently learn and retain knowledge over time from a data stream. Continual learning can be classified into online and offline. The offline setting (Li & Hoiem, 2017) mainly limits the storage: only a fixed amount of training data can be stored at each time step. The computation is not limited in offline continual learning: the model can be trained from scratch until convergence each step. In contrast, the online setting only allows limited amount of storage and computation at each time step.
number of metrics be used to evaluate continual learner. If the model is directly evaluated on the incoming data, the objective is learning efficacy; this evaluates the ability to efficiently adapt new data. If the model is evaluated evaluates the ability to retain existing knowledge. These two objectives are in conflict, and their tra known as the plasticity- -ttabiiiyy dilemma (McCloskey & Cohen, 1989).
historical data, the objective is information retention; this
Following recent counterintuitive results, we single out information retention in this work. A common assumption in the continual learning literature that information retention is only problem due to the storage constraint. Take replay-buffer-base based methods as exemplar. It is tacitly understood that since they cannot store the entire history, they forget past knowledge which is not stored. However, this intuition is challenged by recent empirical results. For example, Cai et al. (2021) show that the information retention problem persists even when past data is stored its entirety. We argue that, at least in part, the culprit for information loss is optimization.
Direct application of SGD to a continual data stream is problematic. Informally, consider the learning rate (i.e,, step size). It needs to decrease to zero Lan, 2013). However, this cannot be applied to a continual and non-iid. stream since infinitesimal learning rates would simply ignore the new information and fail to adapt, resulting in underfitting.
time guarantee convergence (Ghadimi &

---

Under review as a conference paper at ICLR 2023
This underfitting would worsen over time as the distribution continues to shift. We formalize this problem and further show that there is no straightforward method to control this trade-off, and this issue holds even when common adaptive learning rate heuristics are applied.
Orthogonal to continual learning, one recently proposed remedy to guarantee SGD convergence with high learning rates is using the moving average of SGD iterates (Mandt et al., 2016; Tarvainen & Valpola, 7). Informally, SGD with large learning rates bounces around the optimum. Averaging its trajectory dampens the bouncing and tracks the optimum better (Mandt et al., 2016). We apply these ideas to OCL for the first time to improve information retention.
To summarize, we theoretically analyze the behavior of SGD for OCL. Following this analysis, we propose moving average strategy to optimize information retention. Our method uses SGD with large learning rates to adapt to non-stationarity, and utilizes the average of SGD iterates for better convergence. We propose an adaptive moving average (AMA) algorithm to control the moving average weight over time. Based on the statistics of the SGD and AMA models, we further propose a moving-average-basee learning rate schedule (MALR) to better control the learning rate. Experiments on Continual Localization (CLOC) (Cai et al., 2021), Google Landmarks (Weyand et al., 2020), and ImageNet (Deng et al., 2009) demonstrate superior information retention and long-term transfer for large-scale OCL.
2 RELATED WORK
Optimization in OCL. OCL methods typically focus on improving learning efficacy. Cai et al. (2021) proposed several strategies, including adaptive learning rates, adaptive replay buffer sizes, and small batch sizes. Hu et al. (2020) proposed a new optimizer, ConGrad, which, at each time step. adaptively controls the number of online gradient descent steps (Hazan, 2019) to balance generalization and training loss reduction. Our work instead focuses on information retention and proposes new optimizer and learning rate schedule that trade off learning efficacy to improve long-term transfer. In terms of replay buffer strategies, mixed replay (Chaudhry al., 2019), originating from offline continual learning, forms a minibatch by sampling half of the data from the online stream and the other half from the history. It has been applied in OCL to optimize learning efficacy (Cai et al., 2021). Our work uses pure replay instead to optimize information retention, where minibatch is formed by sampling uniformly from all history.
Continual learning algorithms. We focus on the optimization aspect of OCI. Other aspects, such as the data integration (Aljundi et al., 2019b) and the sampling procedure of the replay buffer (Aljundi et al., 2a; Chrysakis & Moens, 2020), are complementary and orthogonal to our study. These aspects are critical for a successful OCL strategy and can potentially be used in conjuction with our optimizers. Offline continual learning (Li & Hoiem, 2017; Kirkpatrick et al., 2017) aims to improve information retention with limited storage. Unlike the online setting, SGD works in this case since the model can be retrained until convergence each time step. We refer the readers to Delange et al. (2021) for a detailed survey of offline continual learning algorithms.
Moving average in optimization. We propose new Although we are the first to apply this idea to OCI,, moving average optimizers have been widely utilized for convex (Ruppert, 1988; Polyak & Juditsky, 1992) and non-convex optimization (Izmailov et al., 2018; Maddox et al., 2019; He et al., 2020). Beyond supervised learning (Izmailov et al., 2018; Maddox et al., 2019), the moving average model has also been used as a teacher of the SGD model in superviee (Tarvainen & Valpola, 2017) and self-supervised learning (He et al., 2020). The moving average of stochastic gradients (rather than model weights) has also been widely used in ADAM-based optimizers (Kingma & Ba, 2014).
serring-vvergge-aaaaeee optimizer for OCL.
Continual learning benchmarks. We need a large-scale and realistic benchmark to evaluate OCL. For language modeling, Hu et al. (2020) created the Firehose benchmark using a large stream of Twitter posts. The task of Firehose is continual per-user tweet prediction, which is self-supervised and multi-task. For visual recognition, Lin et al. (2021) created the CLEAR benchmark by manually labeling images on a subset of YFCC100M (Thomee et al., 2016). Though the images are ordered in time, the number of labeled images is small (33K). Cai et al. (2021) proposed the continual localization (CLOC) benchmark using a subset of YFCC100M with time stamps and geographic locations. The task of CLOC is geolocalization, which is formulated image classification. CLOC
2

---

Under review as a conference paper at ICLR 2023
has significant scale (39 million images taken more than 8 years) compared to other benchmarks. Due to the large scale and natural distribution shifts, we use CLOC as the main dataset to study OCL.
PRELIMINARIES
In this section, we formalize the online continual learning problem and introduce our notation. We then discuss the dataset and the metrics we use.
Problem definition. Given the input domain X' and the label space y. online continual learning learns parametric function f(- &' + that maps an input E X' to the corresponding label
y y. At each time step E {1, ..., oo} the learner interacts with the environment as follows:
1. The environment samples the data (X,, Yr) Ty and reveals the inputs X; to the learner.
2. The learner predicts the label for each datum x X, via y = fx² 10:-1).
3. The environment reveals the true labels Y, and the learner integrates {X,, Y, into the data pool S, via S, update (X,, Y.)).
4. The learner updates the model 0, using S,-
Dataset. Due the large scale, the smooth data stream, and natural distribution shifts, we mainly use the Continual Localization (CLOC) benchmark (Cai et al., 2021) I) to study OCL. CLOC formulates image geolocalization as classification problem. It contains 39 million labeled images for training. 39 thousand images for performance evaluation, and another 2 million images for initial hyperparameter tuning. The images are ordered according to their time stamps, so that images taken earlier are seen by the model first. Images for training and performance evaluation cover the same time span and are sampled uniformly over time. Images for hyperparameter tuning are from a different time span that does not overlap with the training and evaluation data. We refer to the 39 million training images as the training data, the 39 thousand evaluation images as the evaluation data, and the 2 million images for initial hyperparameter tuning the preprocessing data.
Evaluation protocol. We evaluate OCL algorithms using the following metrics.
Learning Efficacy: For online learning applications, the model 0, is deployed for inference on data from the next time step, i.e., X,+1- In this case, we measure the ability to adapt to new data as
PLE(t) = - Σ acc((X ,,+1h,0,) t
(I)
where acc(S, 0) is the average accuracy of the model 0 on set of data S.
Information Retention: When the data for inference comes from the history, we measure the ability retain previous knowledge as
PIR(t) -cc Úf Úx 1=1 1=1
aco(((
(2)
where {X j: YF is the evaluation data from time step j. which has the same distribution as (X,, Yj} but unseen by the model. PIæ(() measures the generalization of O, to historical data. Forward Transfer: When long-term predictions are required, we measure ability to generalize from current time (t) to future time steps (t K1 to + K2, by > k1 1)
(+
H+2
Ptr(t) = acc
U j=1+h1
X
U
Y ).)) 0.)
(3)
j=
4 METHOD
We consider replay-based methods, where the common choice for continual learning is mixed replay (Chaudhry et al., 2019). The objective of mixed replay time step is
minimize 1((X,, Y.). (0,.))+++(SS..... ((..1.,)) BER"
3

---

Under review as a conference paper at ICLR 2023
where (Xe, Y,) is the data received at time t, St-1 is the historical data accumulated in the replay buffer, and V(-, 0) is the loss function. Mixed replay constructs a minibatch of training data by sampling half from the current time step t and another half from the history (from - B, to - 1 where B, decides the time range). In offine continual learning (Chaudhry et al., 2019), B, is set to t- 1 for full coverage. Cai et al. (2021) adaptively choose B, to optimize learning efficacy in OCI.
In this paper, we focus on information retention, more specifically its optimization. A reasonable objective for pure information retention is pure replay, which defined as follows at each time step t:
minimize 0E
U(S,, O).
(5)
Pure replay constructs a minibatch of training data by sampling uniformly from all historical time steps, i.e., from to t, which encourages remembering all historical knowledge. As shown in Appx. C.6, optimizing (5) (as opposed to (4)) improves both information retention and forward transfer by sacrificing learning efficacy.
4.1 OPTIMIZING INFORMATION RETENTION
When the distribution of S, remains unchanged over time (i.e., the loss function is stationary), the standard optimizer for problem (5) is Stochastic Gradient Descent (SGD). This choice is typically carried over to continual learning. At each update iteration &s (k & t if multiple updates are allowed each time step of continual learning), SGD updates the model via
0₂ f 6x- (0x-).
(6)
where 0% is the learning rate and 9b. (0x-1) 1) is the stochastic gradient of the objective at iteration k.
Although the convergence behavior of SGD is well understood for the case of stationary losses, its implications for OCL are not clear. Hence, we extend the analysis of SGD (Ghadimi & Lan, 2013) to the continual learning case. Denote the loss function at iteration & by l (0) and assume:
A1
(0) - vl.(0') || = L 0 8111, v0, 0' E Rd and k (Lipschitz smoothness).
A2
E E(gx(() [0) = v4A (0), v0 E Rd and k (Unbiased gradient estimates).
A3
[0) - (0)) 2 S p², 4 E Rd (Bounded gradient noise).
A4
(0) - (x) S Xk. 0O and & (Bounded non-stationarity).
A1 to A3 extend the standard assumptions for SGD analysis (Ghadimi Lan, 2013) from the stationary case to the non-stationary case. And A4 bounds the degree of non-stationarity between consecutive iterations. The expectations are taken over the randomness of the stochastic gradient noise. With these assumptions, we state the following theorem and defer its proof to Appx. A.2 (also see Appx. A.2 for details about how to interpret this result under limited storage):
Theorem 1. Assume Al to A4, and let on < 4
min JE
(11 T1 T T2+ TT.
where Ti (0.) 20 To Σ (20,41 1' and T3 E Lo3..) _5 Lof+1)
L, Ei a
2Σ Xi 1
Therorem bounds the minimum gradient norm achievable by SGD during OCL. A similar bound in the stationary case (Theorem 2 in Appx. A. has only the terms Ti and Tz. Hence, T' is the cost of the non-stationarity. In the stationary case (when T3- = 0), we can simply find the optimal learning rate by trading off between Tj and T2. Specifically, for a constant learning rate, T1 converges to at linear rate, but T2 is constant. Hence, the strategy for convergence is reducing learning rates to control T2 simultaneously with Ti. This result supports the effectiveness of the standard "reduce- whhen-plateuu" (RWP) learning rate heuristic in the stationary case. Since the access to the true gradient norm is not practical, RWP often uses the validation accuracy/loss as a surrogate to control the learning rate agg, where reduce ag by certain factor / once the validation accuracy/loss plateaus (Goodfellow et al., 2016; He et al., 2016).
In the continual learning setting (when T3 # 0), RWP can be less effective. For example, if Xk 2 X > 0 for a constant X. i.c. the loss function (-) changes at least at a linear rate, then T₃ can grow linearly when the learning rate is reduced to 0. Hence in OCI,, there can be cases where
4

---

Under review as a conference paper at ICLR 2023
simultaneous convergence of Ti, T2. and T3 not possible. Controlling T3 requires reasonably large learning rate, whereas using a large learning rate increases Tz. Finally, the Xx term can cause the learning rate annealing signal in RWP, i.c., the gradient norm the validation loss/accuracy, to be unstable in OCL. We argue that these are the core obstacles in application of SGD to OCI.
In summary, the major issues hindering the performance of OCL when optimized with SGD are:
Pi OCL needs large learning rates due to non-stationarity. However, convergence requires smaller learning rates.
P2 RWP allows the learning rate to be infinitesimal, which prevents the model from adapting to new data.
P3 The validation performance is not always indicative of learning. E.g. due to the distribution shift of OCL, the validation accuracy can decrease, even when the model is still improving.
To address P1, we propose an Adaptive Moving Average (AMA) optimizer in Sec. 4.2. Then, to address P2 and P3, we introduce moving-average--bass learning rate (MALR) schedule in Sec. 4.3.
4.2 ADAPTIVE MOVING AVERAGE
Our Adaptive Moving Average (AMA) optimizer extends Exponential Moving Average (EMA) (Tar- vainen & Valpola, 2017). Moving average optimizers maintain two models: the SGD model 02 and the MA model 0,4.4 At each iteration k, O₂ is first updated with (6). Then, 0.44A is updated via
g 7x 0 (1 70
(8)
where the MA weight " E [0, 1] is a hyperparameter. In EMA, "ik is a constant. In AMA, we adapt Tik over time using population-base search (Jaderberg et al., 2017) to improve the performance.
Why does AMA help information retention? Given stationary losses, SGD can converge to local optima by reducing the learning rate. In this case, the final solution found by AMA will not be very different from SGD as we validate in Appx. C.10. In the non-stationary case, SGD needs reasonably large learning rates to track the shifting optimum over time. However, the variance (T2 in (7)) of SGD increases with large learning rates, hurting convergence.
The advantage of AMA is to reduce the variance of SGD. Given stationary losses, Mandt al. (2016) have shown that SGD with constant learning rate will eventually bounce around a region centered the optimum. Averaging the SGD trajectory with uniform weights estimates the region center, i.e. the optimal solution. This observation inspires the design of AMA. Informally, SGD with large learning rates would follow and bounce around the shifting optimum in OCL. The moving average of these high-variance variance iterates tracks the shifting optimum better. As a further informal justification, consider the unfolded AMA update rule (8):
A-1
k
k
MA = (1 - 7x) O₂ + (1-7)) 7:) g
II j=1+1
7a
O,
+
78
Oo.
where 6g is the initial solution of SGD and AMA. (9) shows that the AMA model 0% M is linear combination of the SGD trajectory (Go..., }. Since 7h E [0. 1]. 0. lies inside the convex hull of (0o............... .... } The population-bar based search aims to find the best point within the convex hull by adapting "
We summarize our method in Alg. 1. To adapt 7 over time, we maintain two MA models, MA and A2, and update them using different weights, 7 MA1 and y MAa (Line 4). To decide the best model and weight, we use the validation accuracy Aceı and Acc2 (Line 5), which we define next. Once every Kw iterations, we copy the parameter of the best MA model to the other one, and adapt the MA weights (Line 7 to 10). In all experiments, we set the initial MA weight "o to 0.99, the MA weight adjustment coefficient ò to 5, and the interval Kw to adjust weights to 10000 iterations.
We continuously track validation metric for adaptation. At each time step a holdout information retention validation set SY is updated, which has the same distribution as S, To maintain Sy' online, whenever new data (X,, Y,) is received, we put random subset (5% in all experiments) into SY' and put the rest into S.. We compute the information retention validation accuracy/loss on Sy in an parallelize validation with model updates, online fashion (described in Appx. B). In this way,
5

---

Under review as a conference paper at ICLR 2023
Algorithm 1 Adaptive Moving Average (AMA)
Require: Initial model 9g. initial moving average weight yo. MA weight adjustment coefficient
8(> 0), MA weight update interval Kw.
0 AMAi +- 0g AM t- 0u. y M +- 70. MA2 t to Ace t 0, Ace2 + O, Abest
2: for & E f1, .... jo) do
Update the SGD model 0z Update OMLA, using MAi and AMA using MM 8: Copy the weights of M the other. Aceı 0, Acc2 0.
3:
4:
5: Update validation accuracy Ace; and Aco2 online
6: If Ace1 Acc₂ then ibst = 1 else lbest 2
7:
if mod Kw = then
9: Increase y & MA by a factor of ō if ibest = 1, and decrease otherwise.
10: end if
11:
end for
and choose the best MA model for inference anytime during OCL. We use the information retention validation accuracy/loss both in Algorithm 1 and later for learning rate control.
We propose several strategies to reduce the overhead of MA and population- sopulation-based search. The overhead from MA lies in 1) updating the MA model and 2) recomputing the batch normalization (BN) statistics (Toffe & Szegedy, 2015). To solve 1, we perform the MA update only every KM iterations. We set K M 10 in all experiments and do not observe a performance drop. To solve we replace BN with group normalization (GN) + weight standardization (WS) (Qiao et al., 2019), which does not store normalization statistics. We evaluate its impact Appx. C.I. The overhead from population- based search lies in the online validation operation (Line 5). Each operation requires one forward pass on a batch of validation data. To reduce the overhead, we execute Line every Kv 20 iterations. See Appx. B for a detailed version of Algorithm with the above strategies.
4.3 MOVING-AVERAGE--BAN LEARNING RATE SCHEDULE (MALR)
As discussed in (P3), monitoring only the validation accuracy not sufficient for learning rate control the effects of learning and distribution shifts counteract each other. Interestingly. AMA provides better signal for learning rate control as a by-product. Specifically, we use the difference between information retention validation loss/accuracy of the MA model 0 Aisest and the SGD model Og- We denote this difference at iteration k by dj. Intuitively, since AMA performs better than SGD given non-minimal learning rates, σ2 can be used to estimate the performance gain obtainable by learning rate reduction. To control the learning rate reduction speed, we do not reduce learning rates unless 0 plateaus, indicating that we cannot achieve better long-term performance gain. On the other hand, when the learning rate is too small, potentially incapable of adapting to new data, a will also be very small. Hence, we not reduce the learning rate unless dg exceeds a certain threshold.
In summary, MALR reduces the learning rate when the following conditions are satisfied. C1: The information retention validation performance (of the MA model) does not improve for KR iterations. (This is the original RWP condition.)
C2: σ does not increase for KR iterations. C3: ><
Computational complexity. The average computational complexity of SGD+RWP per iteration (Ky+1
CSGD+RWP (**+ Ky )er + CG + CU. where CF the complexity of one forward pass, Ky the interval do one online validation step, CG and cy are respectively the complexity for computing the gradient and updating the model. The complexity of AMA+MALR is CAMA-MALK = Kv-+3 cCF + cG + k )cu. where KM is the interval to do one MA update step. When Kv and Ky
KM are large, the overhead of AMA+MALR is small.
5 EXPERIMENTS
Implementation details. We mainly use CLOC (Cai et al., 2021) to study OCL at large scale. Similar to the setting of Cai et al. (2021), the model receives 256 images at each time step. We use the ResNet-50 architecture, a batch size of 256 and tune the initial hyperparameters on the preprocessing data. We set the initial learning rate og 0.025, the momentum of SGD 0.9, the weight decay to

---

Under review as a conference paper at ICLR 2023
(a) Information reten- (b) Log-scaled learn-(c) (c) Validation loss (d) Validation loss (c) oh over time.
tion over time. (1) ing rate over time. over time. (4)
(zoomed view). (4)
Figure I: Results on CLOC. (a): AMA+MALR outperformed all competitors. Among optimizers, AMA outperformed SGD in both RWP and MALR. (b): MALR effectively annealed the learning rate over time, outperforming RWP and CLR. (c) & (d): Under distribution shifts, the validation loss increased for constant learning rates in both long and short terms. This made RWP reduce the learning rate too fast. (e): Given a constant learning rate, 0 in MALR first increased and then plateaued. ok declined with the learning rate reduction. This trend was stable even under distribution shifts.
1.-4, 4. and the loss cross-entropy. For RWP, we reduce the learning rate by half if the information retention validation loss stops decreasing for KR = 60000 training iterations. For MALR, we set o% as the difference of the information retention validation accuracy between the MA and the SGD models, and set in C3 to 3.0%. To decouple the effect of the storage limit and better focus on studying the problem of limited computation in large scale OCI., we use replay buffer size of 40 million images in main experiments. Appx. C.4 ablates the effect of replay buffer sizes and shows that large scale OCL only requires reasonably large replay buffer (4 million images for CLOC). We use the same set of hyperparameters introduced by AMA/MALR in all experiments verify their robustness. (See Appx. C.11 for further details.) All models are trained with 8 Nvidia Geforce Rix 2080 Ti GPUs We refer the information retention validation loss/accuracy throughout the experiments as the validation loss/accuracy. The arrows (+/L) in the caption of each figure point towards the direction of improvement.
5.1 MAIN RESULT: EFFECTIVENESS OF AMA AND MALR
To study the effectiveness of AMA + MALR, we compare it against SGD + RWP (SGD is used as the base optimizer due to high performance, see Appx. C.5 for similar experiments on ADAM). To separate the effects of optimizers and learning rate schedules, we also compare against AMA + RWP, AMA + constant learning rate (CLR), the SGD model in AMA+MALR, and SGD+CLR (using training + validation data for training since no online hyperparameter adaptation is needed). We allow 80 training iterations per time step, that the number of training iterations is large enough and learning rate annealing is necessary. We evaluate information retention using (In(().
As shown in Fig. 1a, AMA+MALR outperformed all competitors. The "jumps" of the performance curves were due to the learning rate change in RWP or MALR. Note that the information retention in Fig. la was not monotonically increasing because of the distribution shifts. In terms of the optimizer, AMA was better than SGD in all learning rate schedules. Moreover, the performance gap was especially large for large learning rates, validating P1. In terms of the learning rate schedule, MALR outperformed both RWP and CLR. Specifically, the distribution shift made the validation loss increase over time for constant learning rates (Fig. Ic and Id). This phenomenon caused RWP to reduce the learning rate too fast (Fig. 1b). As result, AMA+RWP and SGD+RWP performed even worse than SGD+CLR in the long term. In contrast, MALR effectively controlled the learning rate reduction speed and the minimum learning rate with the help of the new signal σe (Fig. le), which was more robust distribution shifts than the validation loss/accuracy.
5.2 ABLATIONS
MALR extends RWP by introducing C2 and C3. To understand the effect of each condition, we compare AMA+MALR versions where C2 is removed (AMA+MALR+NO C2), C3 is removed (AMA+MALR+NO C3), and both are removed (AMA+RWP).
As shown in Fig. 2a, AMA+MALR+NO C3 failed to limit the minimum learning rate (P2), and performed worse than AMA+MALR in the long term. AMA+MALR+NO C2 reduced the learning rate too quickly (P3) at the early stage compared to AMA+MALR. With both C2 and C3 removed, AMA+RWP performed the worst in all methods. As shown in Fig. 2c, the performance of all competitors dropped when dg was too low, which happened when the learning rate was too small
7

---

Under review as a conference paper at ICLR 2023
Information Retertion (%)
10-1
AMA
AMA-RWALR
Val DVT (96)
AMA MALR
Learning
10
AAA-MALR (No C27
10
(3) AMA- MALR C2)
10
Number of Training lterations
Number Training iterations
Number of Training iterations
(a) Information retention over time. (1) (b) Learning rate over time.
(c) 0% over time.
Figure 2: Ablation for MALR. AMA+MALR outperformed all competitors. AMA+MALR+NO C2 reduced the learning rate too fast at early stages, causing P3, AMA+MALR+NO C3 did not limit the minimum learning rate, causing P2, AMA+RWP performed worst.
Information Retention (96)
MA weight
1552
EMA
EMA
1550
Number of Training Iterations
Number of Training keeatiion
(a) Information retention over time. (1)
(b) MA weight over time.
Figure 3: AMA vs EMA. AMA performed better than EMA by adapting the MA weight over time.
(Fig. 2b). Hence, 0% is an effective learning rate annealing signal for OCL. Appx. C.2 further analyzes the effect of using suboptimal learning rates. To summarize, large learning rates led poor convergence and hurt performance on data from all time ranges. On the other hand, small learning rates prevented the model from adapting new data.
AMA vs EMA. We study the effect of population- bbaed MA weight adaptation. We compare AMA with EMA and use the learning rate generated by AMA+MALR on both models to remove the effect of learning rate annealing. Due to resource limitations, we train EMA only for 700 thousand iterations. As shown in Fig. 3, AMA outperformed EMA by adapting the MA weight over time.
Further analyses in the appendix. Due to the space limit, we provide the following analyses in the appendix. Effect of batch sizes Appx. C.3: Heavy parallelization by increasing batch sizes and decreasing the number of training iterations has negative impact on all OCL metrics. Effect of training objectives, i.e., (5) vs (4), in Appx. C.6: Optimizing information retention (using (5)) is more suitable for both information retention and long-term forward transfer, at the cost of sacrificing learning efficacy. More importantly, increasing the computation budget for optimizing information retention improved all OCL metrics, whereas increasing the computation budget for optimizing learning efficacy (using (4)) hurt information retention and forward transfer.
5.2.1 EFFECTIVENESS OF AMA BEYOND CLOCC
Google Landmarks. We apply AMA+MALR to Google Landmarks v2 (Weyand et al., 2020), which contains images from different landmarks (each representing a class). We construct our continual dataset using subset of 580 thousand training images with time stamps. We refer to the constructed dataset Continual Google LandMarks (CGLM). Similar CLOC, 256 new images are revealed at each time step, and 80 training iterations are allowed per time step (see Appx. C.7 for more implementation details). As shown in Fig. 4-a, AMA performed better than SGD in CGLM. Since CGLM is small-scale, the learning rate of both RWP and MALR remained constant throughout most of the training. Hence, we only reported the performance of AMA+MALR and the SGD model. To further check this limitation, we also looked at the first 580K images of CLOC in Figs. 4-c and 4-d. Similar CGLM, the relative merit of different learning rate schedules was unclear. Hence, large-scale dataset is necessary for analyzing OCI. learning rate schedules.
ImageNet. We test our method on task-bar based benchmarks synthesized from ImageNet. Specifically, we create a 100-task continual learning problem by randomly dividing ImageNet classes, so that each
task contains data from 10 classes. During training, we allow 120 X+l training iterations per task,
256
8

---

Under review as a conference paper at ICLR 2023
100 S Saperiiin 1N)
40k
Mi
ASAA AAXL
ANS He0
260 sll vioz addoudocaaiin yalo raakinng
2000 pas 1000 ande si Trariing
2000 sodc r0d secoe
(a) Information retention CGLM (1).
(b) Learning rate over time on CGLM.
(c) Information retention on (d) Learning rate for the first the first 580K CLOOC images. 580K images of CLOC.
Figure 4: Results on the CGLM. AMA performed better than SGD. Given only 580K images, the difference of learning rate schedules are unclear for CGLM ((a) & (b)) and CLOC ((c) & (d)).
AFA. HALL
5:20 Cyslc
G2UTb
Information fatertion (8)
TaskiD
Figure 5: Image results (t). AMA+MALR out out-performed both SGD+RWP and SGD+Cyclli. The final validation accuracy is close to the supervised learning model. Due to the implicit computa- tion increase in each model update iteration, OCL methods performed worse than simple SGD+Cyclic when the memory is sufficiently large and the computation is explicitly limited.
where X is the number of images of task and 256 is the training batch size. In this setting, the total number of SGD steps of continual learner trained on all tasks is the same as the standard supervised learning model trained for 120 epochs. During evaluation, we compute the average accuracy on validation data from all revealed tasks. To demonstrate the online performance, we perform multiple evaluations during each task. Since clear task boundaries are available, we also compare against SGD+Cyclic, where the cosine learning rate schedule is used in each task. We further compare against methods designed for handling storage limitations: GDumb (Prabhu et al., 2020), ER (Aljundi et al., 2018), and MIR (Aljundi et al., 2019a). We explicitly limit the computation budget of ER, GDumb, and MIR to be similar to other methods. See Appx. C.8 for more details about the experimental setup and hyperparameter settings,
As shown in Fig. 5, AMA+MALR outperformed both SGD+RWP and SGD+Cyclic. Comparing to SGD+Cyclic, AMA+MALR had better intermediate and final performance during each task, making it more suitable for online applications. Using the same number of SGD steps, the final validation accuracy of AMA+MALR was close to supervised learning. On the other hand, ER, MIR, and GDumb performed worse than SGD++yclic since implicit computation increase in each model update step made them require more computation/tims to converge. Note that here we allow the replay buffer to store all past data. This result demonstrates that limited computation and limited storage are orthogonal problems in OCL. The are both important and require dedicated treatments. See Appx. C.9 for similar results with limited storage.
CONCLUSION
We studied the problem of improving information retention in online continual learning (OCL). We proposed to use pure replay as the objective for optimizing information retention. We provided theo- retical analysis of the convergence of SGD in OCL. We also proposed an Adaptive Moving Average (AMA) Optimizer and a Aeeeing-Averge---e based Learning Rate Schedule (MALR) to optimize the pure replay objective online. Experiments on several large-scale continual learning benchmarks demonstrate the effectiveness of AMA+MALR. In terms of limitations and future work, we used two MA models to adapt the MA weights, which increases the GPU memory footprint to store the network weights for MA models (but not the gradient information as in the SGD model). Designing more memory-effiieen techniques to adapt MA weights is an interesting future research direction. Furthermore, this work mainly focused on studying the problem of limited computation in OCL. De- veloping effective solutions to address the computation and storage aspects together is an interesting and important future direction.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 354, 2018. 9
Rahaf Aljundi, Lucas Caccia, Eugene Belilovsky, Massimo Caccia, Min Lin, Laurent Charlin, and Tinne Tuytelaars. Online continual learning with maximally interfered retrieval. arXiv preprint arXiv:/908. 04742, 2019a. 2,
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning, arXiv preprint arXiv:1903.08677 2019b. 2
Omar Besbes, Yonatan Gur, and Assaf Zeevi. Non-stationary stochastic optimization. Operations
research, 63(5):1227-1244, 2015. 13
Zhipeng Cai, Ozan Sener, and Vladlen Koltun. Online continual learning with natural distribution shifts: An empirical study with visual data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8281-8290, 2021. 1, 2, 2, 6, 16, 17
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip Torr, and Mare' Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv preprini arXiv: 10486, 2019. 2, 4, 15, 17
Aristotelis Chrysakis and Marie-Francine Moens. Online continual learning from imbalanced data. In International Conference on Machine Learning, pp. 1952-1961. PMLR, 2020.
Matthias Delange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Greg Slabaugh, and Tinne Tuytelaars. continual learning survey: Defying forgetting classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021. 2
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009,
Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization, 23(4):2341 2013. 1,4, 12
Ian Goodfellow. Yoshua Bengio, and Aaron Courville. Deep learning. MIT press. 2016. 4
Elad Hazan. Introduction online convex optimization. arXiv preprint urXiv:1909.033 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9729-9738, 2020. 2
Hexiang Hu, Ozan Sener, Fei Sha, and Vladlen Koltun. Drinking from firehose: Continual learning with web-scale natural language. arXiv preprint arXiv:2007.0933 2020. 2, 18
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. PMLR, 2015. 6
Pavel Izmailov. Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Av- eraging weights leads to wider optima and better generalization. arXiv preprint orXiv: 1803.05407, 2018. 2
Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M Czamecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, lain Dunning, Karen Simonyan, et al. Population based training of neural networks. arXiv preprint arXiv: 1.09846, 2017. 5
10

---

Under review as a conference paper at ICLR 2023
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinskar Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciencex, 114 (13):3521-3526, 2017.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-2947 2017. 1,2
Zhiqiu Lin, Jia Shi, Deepak Pathak, and Deva Ramanan. The clear benchmark: Continual learning on real-world imagery. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round? 2), 2021.
Wesley Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson A simple baseline for bayesian uncertainty in deep learning. Advances in Neural Information Processing Systems, 32: 32:13153-13164, 2019. 2
Stephan Mandt, Matthew Hoffman, and David Blei. A variational analysis stochastic gradient algorithms. In International conference on machine learning, pp. 354 363. PMLR, 2016. 2,
Michael McCloskey and Neal Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Yurii Nesterov. Introductory lectures on convex programming volume i: Basic course. Lecture notes, 3(4):5, 1998. 14
Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM journal on control and optimization, 30(4):838 855, 1992. 2
Ameya Prabhu, Philip HS Torr, and Puneet Dokania. Gdumb: A simple approach that questions our progress in continual learning. In European conference computer vision, pp. 524-540. Springer, 2020.
Siyuan Qiao, Huiyu Wang, Chenxi Liu, Wei Shen, and Alan Yuille. Micro-batch training with batch-channel normalization and weight standardization. arXiv preprint 1xiv:1903.105220 2019.
David Ruppert. Efficient estimations from a slowly convergent robbins monro process. Technical report, Cornell University Operations Research and Industrial Engineering, 1988.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. arXiv preprint arXiv: 2017.
Bart Thomee, David Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc Yfcc100m: The new data in multimedia research. Communications of the ACM, 59(2) 77, 2016.
Tobias Weyand, Andre Araujo, Bingyi Cao, and Jack Sim. Google landmarks dataset v2-a large-scale benchmark for instance-level recognition and retrieval. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2575-2584, 2020. 2, 8, 18
A CONVERGENCE ANALYSIS OF SGD
This section analyzes the convergence of SGD in both supervised learning (SL) and online continual learning (OCL). The purpose of the analysis to demonstrate the difference between the optimization problem in SL and OCI. and explain intuitively why SGD+RWP works in SL. but fail in OCL..
11

---

Under review as a conference paper at ICLR 2023
A.1 CONVERGENCE ANALYSIS IN SUPERVISED LEARNING
In supervised learning, we are given fixed set of training data, and we solve the following stationary optimization problem
minimize
1(0)-
(10)
OER
In each iteration k, SGD updates the model O2 using the stochastic gradient, from a batch of training data, via,
0₂ - 0 ar.((O=
(11)
where Eg denotes the random variable that determines the noise the stochastic gradient.
The standard assumptions (Ghadimi & Lan, 2013) for analyzing the non-convex case of supervised learning is
A5 Lipschitz Smoothness with constant L 2 0: |V((0) vi(e")) S L 1|0 00,, 0' IRd, where ||-I| is the Euclidean Norm.
A6 Unbiased gradient estimation: (0(())))) = v((0), v0 Rd.
A7 Bounded gradient noise with constant 2 0: || g E\| S p², 00 e Rdd.
Under these assumptions, we re-produce the standard convergence result of SGD the non-convex SL case:
Theorem 2. Assume A5 to A7, and let ai < +
< 2(1(On) E [10x+1)]])
Lp Σ Σラ= 03+1
min Be. (e
Σx=0 (2a La)+1)
Σj=0 (2a La}+1)
(12)
Proof. Following the proof of Theorem 2.1 until (2.11) in (Ghadimi & Lan, 2013), we have
(20 - E e. [Iv[((,)) 2(((6)) - E [(0x++)))) + T.p²
5++1
i=0
j=0
(13)
And since
&
k
(20+11 - Laz+1)) min E [Iv((())) <
Σ
(20z-1 1 (ll [|]v((0))
l8 (0......k)
==
1=
(14)
we immediately have (12) by combining (13) with (14) and dividing both sides by (2a Laz+1).
Theorem tells us that the convergence of SGD can be guaranteed by properly setting the learning I,p Ei of Σ (2a 1 Log..) as the first and the second terms at the right hand side of (12), which are of the same form as Ti and T2 of (7). To ensure that 24, goes to O, need to make sure that
rate ag. Denote T4 - [[0++))]) and T3
Σ (20 Laf_.)
lim
(2aj+1 Laz+1) co.
(15)
And to ensure that T3 goes to 0, need to ensure that
Σ 03+1
lim
0.
(16)
- La²
12

---

Under review as a conference paper at ICLR 2023
For example, can set ag O(vk) that both (15) and (16) satisfied. Theorem 2 also provides insights of why RWP is effective in SL. Given a constant learning rate aj = a, we can see that T4 ((+) after k iterations, i..., it converges with linear speed. However, the second term Lp²a is a constant over time. Hence, the gradient norm will first decrease and then converge to T3 2-La
constant given constant learning rate. And the plateau of the gradient norm is signal that we need reduce the learning rate to further improve the model. In practice, the validation loss/accuracy often used as surrogate of the gradient norm.
A.2 CONVERGENCE ANALYSIS IN OCL
We provide proof of Theorem in this section. Before the proof, we first re-state the problem, the assumptions and the theory a more detailed format.
At each iteration k, we given non-stationary objective
minimize 4.
(17)
In each iteration k., SGD updates the model 0₂ using the stochastic gradient 9a (Os-1(E)) on ls (0x0)))) from batch of training data, via,
0g + 0 a:gı (0--1E))))
(18)
where & denotes the random variable that decides the noise of the stochastic gradient.
It unsurprising that no algorithm can achieve reasonable performance if (-) change arbitrarily between consecutive iterations, since we can change the training data in a completely adversarial fasion. One reasonable assumption is that (+) does not change significantly between consecutive time steps. For example, when we want to optimize information retention, the training data pool S, does not change significantly between consecutive iterations. Given this intuition, we formalize our assumptions in the non-stationary case, with the definition of the mean operation E (which is omitted the main paper) as follows:
A1 Lipschitz Smoothness with constant L 0: (14 (4))l S L 1 0'1 V0, IRd and k.
A2 Unbiased gradient estimation: [e[m((())) = 74+ v0 e nd and k.
A3 Bounded gradient noise with constant p 2 || (0) ga (0 p², 00 Rd and k.
A4 Bounded non-stationarity with constants X Z 0: (0) (+) 00) X 0O Ed and
Intuitively, A1 to A3 extend A5 to A7 to non-stationary objectives, and A4 constraints the change of the objective function value between consecutive iterations, which similar to the analysis of SGD in the convex case (Besbes al., 2015). Note that 4 (0) is computed over the replay data, i.e., not on the discarded historical data when the replay buffer/storage limited. Hence the analysis this section naturally includes limited storage as special case. Under these assumptions, we can prove the following convergence result of SGD in the non-stationary case:
Theorem 1. Assume Al to A4, and let og < 2
min EE
<T + T2+ T3
(19)
1E kk........kk
2(1 (0.) - Be, e e (0++)0))
I,p² Σ1 03
where T
T2
and T3
Σt E) (2a La
(2 Lo -o
2 X
(20
13

---

Under review as a conference paper at ICLR 2023
Proof. With A1 and Lemma 1.2.3 of (Nesterov, 1998), we have E [++1
(20)
<e, E.....r (Oa) + (0x))(01xx(((0) 02) + L 1|00+ 0xl1
(21)
2
LEE, LE.,..... Ext,
-Ee, -E (0.)) an Ee (0x)) (0x))l²+
(22)
2
= 1 (0+)) ax+ Ee..... (1
L B (9 (0x|Ex+) v4++1 (0x) + F4++1 (0+)
(23)
EE (1 (0+)] ca+t E, 1 (0.) (0+)ll
Be, LaitiBe
- vl + (
(0+) (9 V4++1 (0x))]/2
(24)
Lo 10² [104++1(0w111)
<Ee, Se ax+1 Ee & [1 (1
(25)
2
Lag+1
LaR+1P2
=E 1l (ax+1
IB Be (0.))11))
(26)
2
where (25) is due to A3 (1 - (0x+1
p²) and A2
(E 2 (0x)² (gn+ 1 (0x(E++1) - V4++1 (0x)) = 0).
Rearranging the terms
in (26) and using A4, have
La
(ax+1
] BE
(27)
2
1 (0x+((0++))))) La
S Ee ((.+1((0)))
(28)
1P
< Reg.... (0x)) Bs (1 Xa+1 +
(29)
Summing both sides of (29) from j = 0 to k we have
(20 La Be, Be,,..,,, (0;)) 1101
/=0
k
<2(1)(00) - 14142200 + Lp²
Σ
+ Σ Xi+1-
(30)
j=0
j=0
With a similar derivation as in the proof of Theorem we immediately have (19).
Intuitively, to make T3 converge to 0, we need to set the learning rate such that
ΣS Xi+1
lim 800
= 0,
(31)
ビ== (2a
Laÿ+1)
i.e., we need Σ E)=0(20ft1 L to increase faster than Σ) Xj+1- In practice, the value of a, T3 can increase for an increasing
depends on the application. Given constant learning rate sequence of Xk- This explains why P3 can happen in practice. If X 2 X for a constant X. we cannot have a learning rate schedule that makes Ti, T2 and T3 converge to 0 at the same time. Moreover, when 0g < 1, decreasing O& will increase of T3. In the extreme case, if or, reduced to 0, T3 will increase in linear speed, causing long term failure of SGD+RWP. This explains P2, i.c., why reducing the learning rate to 0 using RWP hurts the performance of OCL. And demonstrates why we need to keep the learning rate sufficiently large, i.e., P1. Note that currently, there is no lower bound to demonstrate the tightness of (19). Hence, the analysis in this section can only be viewed as an intuition.
Interpreting Theorem 1 under limited storage In practice, both supervised learning and continual learning use limited amount of (though may be in large scale) training/replay data. In such cases,
14

---

Under review as a conference paper at ICLR 2023
the loss (0) in (17) and I(0) (10)) is only computed over the stored data. And assumptions in the theories are only w.r.t. the loss over the training/replay data. For supervised learning, each training data point can be viewed learning where not all past data can be stored, one can construct a reservoir replay buffer (Chaudhry al., 2019) online, so that at each time step each data point in the replay buffer is also random sample from the original data distribution. Hence, the bound in Theorem is directly comparable the bound in Theorem when the replay buffer size in OCL is the same as the dataset size of supervised learning. The performance of a (continual) learner over the original data distribution is the problem of generalization, which is out of the scope of convergence analysis.
random sample from the original data distribution. For continual
B FURTHER DETAILS OF AMA AND ONLINE VALIDATION
Algorithm 2
Adaptive Moving Average (AMA)
Require:
Initial model 0. the initial moving average weight no. MA weight adjustment coefficient 8(> 0). model update interval KM, validation interval Kv, MA weight update interval Kw, number of training iterations per time step p.
1: @MA 0g 0M <- 0, 7 A 70, 7 A1 8
2: Acci < 0, Ace2 <- 0, f 0, Abest f
3: for (1, 00) do
4: O% +- O2 (1.0(0x-1)
5: i mod KM = then
6:
@MAi t MA1 An
+ (1 M0 MA22 00x
7:
MM 7 MAa
8:
end if
9:
if mod Ky = then
w.Aco: tace .MA1))
n-Acestacc(BB" ,MA2)
10:
Ace; +
Ace2 +
n 4- n +1
11:
If Acc1 > Acc2 then best else ibest = 2
12:
end if
13:
if k mod Kw 0 then
Acei Acc2 0, if tbst = 1 then
15:
> Increase both MA weights but limit them to be S 1 MA1
MA + min(1, 1, 1ain(,,0,AAn s2.lA))
MAz
OMA₂ + GMAA, ibest
18:
else
Decrease both MA weights
7 MAi + MMA1 . MA2 f- y M42
19:
20:
MA, +- gild2 Mbst =
end
22:
end if
23:
end for
Alg. 2 describes the complete version of Alg. 1, with overhead reduction strategies added. Each online validation step (Line 9 to Line 12) computes the accuracy on a batch of data B sampled
from the information retention validation set SV | of the current training iteration k, where p the number of SGD updates allowed per time step. We average the computed accuracy the fly between each Kw iterations, and use the averaged accuracy as the online information retention validation accuracy. The benefit of this online validation process that the validation steps can be parallelized completely with model updates, and we can also select the best MA model for inference anytime during OCL.
C FURTHER EXPERIMENTS
C.I BN VS GN+WS
15

---

Under review as a conference paper at ICLR 2023
Information Retention (S0)
Ohh
x %
usken Number of Training Iterations
Figure 6: Information retention over time for models with different normalization layers. (T) GN+WS performed slightly better than BN in terms of information retention, but more efficient in AMA due to the removal of BN statistics.
520
Time Stmpp Exalaaiion txp
Time Stamp Fustinatio
(a) k 5.76e5. O& 1.25e-2 for AMA+MALR, (b) 4.8e8. on 1.56e3 AMA+MALR, 6.25e-3 3 for AMA+RWP, and 9.77e-5 for 9.54e-8 AMA+RWP, and 2.98e-9 for SGD+RWP. (t)
SGD+RWP.
Figure 7: Accuracy on evaluation data from different time ranges, for models at two different training iterations Comparing AMA+MALR, AMA+CLR used larger learning rate ou causing performance deterioration on all data ranges. SGD+RWP and AMA+RWP dropped the leaminng rate quickly, making the model unable to adapt to new data.
As mentioned in Sec. 4.2, we replace BN with GN+WS to remove the overhead of re-computing BN statistics in AMA. To show the effect of changing the normalization layer, compare the model trained with BN against the one trained with GN+WS. Due to resource limitations, we only train the BN model for 700 thousand iterations. As shown in Fig. GN+WS performed slightly better than BN. Combined with the efficiency, GN+WS is suitable for information retention in OCL.
C.2 SMALL LEARNING RATES STYMIE ADAPTATION TO NEW DATA
To understand the effect of using suboptimal learning rates on data from different ranges, we take the models of different methods at two distinct training iterations (A: 5.76e5 and 4. 4.8e8), and plot their accuracy on evaluation data from different time ranges. As shown in Fig. 7, AMA+CLR performed worse than AMA+MALR on all data ranges due to the use large learning rate. The performance gap larger on historical data than future data. SGD+RWP and AMA+RWP performed well only on early data and failed to improve the performance for new data when the learning rate was too low.
C.3 HEAVY PARALLELISM HURT OCL
Cai et al. (2021) have shown that increasing batch sizes for optimizing learning efficacy hurts all OCL performance metrics. We perform similar analysis study whether this phenomenon still happens when optimizing information retention. Specifically, we optimize information retention with batch sizes varying from 64 to 1024. When we increase the batch size, we reduce the number of training iterations per time step and increase the learning rate by the same factor. We allow training iteration per time step for a batch size 256. As shown in Fig. 8, increasing batch sizes for optimizing information retention also hurt all OCL performance metrics.
C.4 A REASONABLY LARGE REPLAY BUFFER IS SUFFICIENT
We now analyze the effect replay buffer sizes, by training models with replay buffer sizes 400 thousand, 4 million, and 40 million images. To optimize information retention, reservoir
16

---

Under review as a conference paper at ICLR 2023
Information Retorkannn
Loaning Effcacy
Bistiok
$6.024
Time
522
220
275
120
(a) Learning efficacy over time. (1) (b)
Information retention over time. (c) Forward transfer over time. (1)
(f)
Figure 8: Effect of batch sizes. Increasing the batch size hurt all OCL performance metrics.
Loamingg Eftracy (NN
Forward ThwardTTardtt Tranchor (6)
Information Retendion
400 Thousnd
430
Thossrr
135
Time
Time
(a) Learning efficacy over time. (1) (b) Information retention over time. (c) Forward transfer over time. (f)
(f)
Figure 9: Effect of replay buffer sizes. Though all performance metrics dropped when the replay buffer size was reduced to 400 thousand images, the performance remained similar when the replay buffer size was larger than 4 million images.
buffer (Chaudhry al., 2019) in this experiment. We allow 5 training iterations per time step for each replay buffer size.
As shown in Fig. 9, reducing the replay buffer size from 40 million to 400 thousand images signifi- cantly hurt all performance metrics. However, the performance remained similar when the replay buffer size million larger. Therefore, given fixed model and computational budget, OCL only requires a reasonably large replay buffer.
C.5 APPLY AMA TO OTHER BASE OPTIMIZERS
Besides SGD, AMA+MALR can also be combined with other base optimizers. As example, apply to ADAM and test the performance on CLOC. Specifically, we compare (ADAM-based)
AMA+MALR with ADAM+MALR and ADAM+RWP, with initial learning rate tuned to 0.001. Due to resource limitations, we only run all algorithms on partial data. As shown Fig. 10, (ADAM) AMA+MALR outperformed ADAM+RWE However, ADAM methods performed worse than SGD. We conjecture that the adaptive scaling in ADAM reduced the effective learning rate (similar to the SGD+RWP case), and leave further investigations as future works, Due to the poor performance of ADAM-based methods, we use SGD-bar based methods in the main experiments.
C.6 IMPACT OF OPTIMIZATION OBJECTIVES TO OCL
An ideal OCL algorithm should perform well for all three metries, i.e., learning efficacy (eq. (1)), information retention (eq. (2)), and forward transfer (eq, (3)). However, optimizing one objective can hurt others. To demonstrate this point, we compare the model trained optimize information retention (with AMA+MALR) and the model trained to optimize learning efficacy (with the method of Cai al. (2021)). Following Cai et al. (2021), we allow different computation budgets (1 and training iterations) per time step. To fairly compare pure replay and mixed replay, double the batch size for AMA+MALR, so that the average number of gradient descent steps per image the same for the two models. We set k and ky in the forward transfer metrie 10% and 25% of the dataset size respectively.
As shown in Fig. la, given the same budget, optimizing learning efficacy (LE Model) performed the best for learning efficacy. Optimizing information retention (IR Model) performed the best for information retention (Fig. 11b) and forward transfer (Fig. 11c). More importantly, unlike optimizing
17

---

Under review as a conference paper at ICLR 2023
Information Retention
Number of Training Iterations
Figure 10: ADAM results. (T) AMA+MALR applied ADAM also improved information retention
in OCL. However, ADAM- base- methods performed worse than SGD counterparts.
Ri Moee oodteosl Hoes Gobeegi
El iicce Statapal Ga0lep- (F Jivenn 0815013
Mode 15 SuSipp
Loominng Eammn
Emacyy
spaiien wostea
Recortor
Forward Trandter
00 (F tiven 1435.10
Time (5)
Timb (3)
Time (3)
(a) Leamiing efficacy over time. (1) (b) Information retention over time. (c) Forward transfer over time. (t)
Figure 11: Optimizing learning efficacy (LE Model) vs optimizing information retention (IR Model). Optimizing information retention hurt learning efficacy (a), but improved information retention (b) and forward transfer (c). More importantly, increasing the budget for optimizing learning efficacy hurt both information retention and forward transfer. In contrast, increasing the budget for optimizing information retention improved all OCL metrics.
information retention, where increasing the computation budget improved all OCL metries, increasing the computation budget for optimizing learning efficacy hurf information retention and forward transfer. This phenomenon is consistent with the finding in the language modeling setting (Hu et al., 2020). Hence in terms of improving long term transfer in OCL, information retention is better objective than learning efficacy.
C.7 DETAILS ON GOOGLE LANDMARKS V2
Similar to CLOC, we construct the OCL version of Google Landmarks v2 using the subset of images from Flickrs, which have time stamps that can be retrieved from flickr API. Similar to the original paper (Weyand al., 2020), we address the noisy label and class imbalance issues by using only the data from the cleaned set, and filter out the classes with fewer than 25 images. This results in 645666 images total. We further divide the dataset into 90% training data (roughly 580 thousand images) and 10% of evaluation data (roughly 65 thousand images).
In terms of syper-parametens, we use same model as the CLOC dataset. We tune other hyper- parameters on the first 20% of the training data (we still use this part of data during training because otherwise the dataset will be even smaller), resulting in the learning rate of 0. 0.135, the momentum of 0.9 for SGD, the weight decay of 1e-4, and the cross-entropy loss.
C.8 DETAILS ON IMAGENET CONTINUAL LEARNING EXPERIMENTS
We mostly follow the hyper- parameter settings of supervised learning, i.e., we use ResNet50, with batch size of 256, initial learning rate of 0. the momentum of 0.9 SGD, the weight decay of le-4. For ER, GDumb and MIR, we use the same basic hyper- parameter as SGD+Cyclic, including the optimizer, learning rates, the batch size etc. For ER, we mix 128 samples from the current task and 128 samples from replay for each model udpate step. For MIR, we select 128 examples from 256 candidates (using the $ SMJ-1 criterion) during the replay data sampling procedure, and mix them with another 128 examples from the current task to update the model. For a fair comparison, we limit the computation of each method to be the same as SGD+Cyclic. Specifically, MIR requires at least two SGD iterations in each model update step, hence, the total number of model update steps is limited to
18

---

Under review as a conference paper at ICLR 2023
into matton
A/A HALL (45+-21
Learing
Figure 12: BN vs GN+WS in ImageNet continual learning. (1) Similar as in CLOC, GN+WS also performed slightly better than the BN model on ImageNet continual learning. And its best validation accuracy in the final task (76.8%) is similar as the standard BN model trained with supervised learning.
1096-1900 PA
SOO-RAP
Sap+cycli
GOURTS
Suparvted Laarin VILKC
Inbr mirraation Roterton (3))
Figure 13: Information retention on ImageNet continual learning with small replay buffer (120K images). (f) Similar to the results in the main experiment, AMA+MALR out out uer-performed SGD+Cyclic and other OCL methods designed for handling storage limitations. Interestingly, the gap between SGD+Cyclic and ER/MIR became smaller in this case.
half of the SGD steps in SGD+Cyclic. GDumb trains model from scratch for each task, hence the number of SGD updates of GDumb and SGD+Cycllic is limit to be the same in each task. To make the final validation accuracy directly comparable to the standard ResNet50 supervised learning result, we use BN as the normalization layer in this experiment (The GN+WS model performed slightly better than the BN model). To update BN statisties on the fly, we do one extra forward pass per 10 training iterations to accumulate the BN statistics for MA models. For both RWP and AMA, we decay the learning rate by half when the corresponding conditions are satisfied. We set KR RWP and AMA to 15000, which is smaller than in CLOC, so that the effect of learning rate reduction can be clearly seen.
We also show in Fig. 12 the performance comparison between the BN model and the GN+WS model. Similar as in CLOC, GN+WS also performed slightly better than BN. And its best validation accuracy in the final task (76.8%) is similar the standard BN model trained with supervised learning. This result confirmed the generalization of our method across problems and datasets.
In the main paper, we analysed the complexity of AMA. To support our analysis, we further provide here the runtime of each algorithm to show the practical efficiency of AMA. Specifically, we provide the average runtime for each algorithm finish one training iteration. AMA+MALR used 0.264s, SGD+Cyelic used 0.245s, SGD+RWP used 0.272s, GDumb used 0.261s, MIR used 0.395s, ER used 0.327s. Note that there some noise in the runtime due to the change of environments in different experiment executions, but it is clear that AMA+MALR has similar efficiency as SGD+RWP in practice.
C.9 SMALL REPLAY BUFFER ON IMAGENET
In this section, we show the ImageNet continual learning result with limited storage. Specifically, we only allow 120K images to be stored in the replay buffer, and compare AMA+MALR against SGD+Cyelic, ER, MIR and GDumb. For AMA+MALR, SGD+Cyelic and GDumb, the replay buffer is formed using reservoir sampling. For ER and MIR, we assign half of the replay buffer for storing data from previous tasks (using reservoir sampling), and half of the replay buffer for storing data from the current task. As shown in Fig. 13, AMA+MALR still outperformed all competitors in this case. Interestingly, the performance gap between SGD+Cyelic and ER/MIR became smaller compared to the case of large replay buffers.
19

---

Under review as a conference paper at ICLR 2023
C.10
PERFORMANCE ON SUPERVISED LEARNING
Test re Acc Aoc-moo 10
Textre (92
MAX
Ezech
Spach
(a) Validation accuracy (b) Learning rate over
over time. (1)
time.
Figure 14: Performance of AMA and MALR on ImageNet for supervised learning, (1) Similar as in the OCI case, AMA performed much better than SGD when the learning rate was large. MALR adaptively reduced the learning rate, providing similar final performance as the cosine schedule.
To understand the performance of AMA + MALR supervised learning, we apply it ImageNet, and compare against SGD + cosine schedule (denoted as SGD+Cosine), which is the standard for ImageNet. To understand the effect of each component, we also show the performance of AMA+Cosine and the SGD model in AMA+MALR, To replicate the standard ResNet50 result on ImageNet, we use BN as the normalization layer, and train all models for 120 epochs. Since limiting the minimum learning rate hurts supervised learning, we only use C1 and C2 in MALR for this experiment. We reduce the learning rate by 10 times once C1 and C2 are true.
As shown in Fig. 14, though not better in terms of the final performance, AMA performed much better than SGD at early training stages, where the learning rate was large. MALR adaptively reduced the learning rate over time, providing a similar final performance as the cosine schedule.
C.I1 AMA IYPERRPARAMETER TUNING AND SENSITIVITY
As described in Sec. 4.2, we set the MA weight adjustment coefficient 8 to 5 and the interval Kw to adjust MA weights to 10000 iterations. We empirically observed that setting Kw in between 1- 1-10000, and 8 between 2- 10 does not have strong effect to the performance (i.e., the classification accuracy). Setting Kw to 10000 and to worked across problems (ImageNet, CLOC, Google Landmarks) and provides AMA similar speed as SGD in practice.
For classification tasks, we set e in C3 to 3% accuracy. This value worked for CLOC, ImageNet and CGLM (only tuned on the CLOC preprocessing data). It can be used reasonable default for other datasets as well. For tasks other than classification, one can tune by:
1. Finding the best initial learning rate on preprocessing data.
2. Using a constant learning rate schedule, with learning rate set to the best initial learning rate. Compute the performance gap between AMA and SGD models on pre-processing data, denoted as g-
3. Setting to roughly 0.3g (this is how we set our e), we observe that the performance difference is small by setting in between 0.5-0.2g.
C.12 WHY NOT SMALLER DATASETS LIKE CIFAR OR 1 TRAINING ITERATION PER TIME STEP?
Existing literature focus more on datasets with smaller scale, e.g., CIFAR, MNIST and/or with I training iteration per time step. Here we show why such experiments are not considered in this work. As shown in Fig. 15 if we allow only iteration per time step for CIFAR (5 tasks, classes for each task) and ImageNet, the learning rate for both SGD+RWP and AMA+MALR would remain constant throughout training, hence, no learning rate scheduling is needed. Also as shown in Fig. 16, even if we allow 120 iterations per time step in CIFAR, in the ImageNet experiment, SGD+RWP and AMA+MALR still used constant learning rate throughout training, as in the experiment of CGLM (Sec. 5.2.1). More importantly, the performance difference between different continual learning algorithms was much smaller in CIFAR compared to ImageNet, due to the small data stream. Hence, argue that the data stream in such cases too short for the main purpose of this work, which is to
20

---

Under review as a conference paper at ICLR 2023
Figure 15: Information retention on ImageNet (Left) and CIFAR (right) continual learning with 1 training iteration per time step. (1) We store 120K images in the replay for ImageNet and 5K for CIFAR. Due to the small number of training iterations in total, allowing training iteration per time step did not reveal the problem of SGD+RWP in the long term. Specifically, the learning rate of SGD+RWP and AMA+MALR remained constant throughout training for both CIFAR and ImageNet. And the advantage of A.A+MALR was also invisible since MA needs reasonable number of iterations to track the SGD trajectory. Hence, this experiment is not proper to validate Theorem and demonstrate the complete behavior of different algorithms large scale OCL.
JMA HTELK
Informatd ohten Rotertion 1994
Tosk
Figure 16: Information retention on CIFAR continual learning with 120 training iteration per time step, with replay buffer size of 5000 images. (f) Similar in the experiment of CGLM (Sec. 5.2.1), due to the small number of images in total, training on CIFAR does not reveal the problem of SGD+RWP in the long term with multiple training iterations per time step. The learning rate of SGD+RWP and AMA+MALR remained constant throughout training, and the performance of different algorithms also not proper to validate Theorem and demonstrate the complete behavior of different algorithms in the long term
much smaller than in the ImageNet experiment. Hence, this experiment is
reveal the problem of SGD+RWP (validating Theorem 7) and the benefit of AMA+MALR in the long term.
21


Reviews:
[{"review_id": "tLtnr-vDCU", "reviewer": "Reviewer_axbU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "\nIn Online continual Learning scenario, where one needs to deal with non-iid data stream, information retention can be a problem, even using unlimited replay buffer. In this study, applying naive Stochastic Gradient Descent and using constant or decreasing learning rate has been partly associated to the information loss in long term. The authors put forward using heuristics based on adaptive moving average to improve the optimization process, and also propose heuristics for learning rate scheduling. ", "summary_of_the_review": "In my opinion this study needs more work to be organized and to get into the shape to be published.   ", "strength_and_weaknesses": "Indeed Online continual learning is a challenging scenario and the authors opted for an interesting research topic. \n\nIn my opinion this manuscript could be reorganized in a way that it is more to the point and explain how it is differentiated to the other relevant approaches in the past.   \nIt would be interesting to discuss standard Adam and RMSProp and compare with the proposed heuristics in terms of time and performance. \nAlthough it is mentioned that the authors used a population based search to find the best adaptive point within a convex hull for \\gamma_k, I think making the effort to include the proof could indeed improve the work.\nThe overhead of the proposed approach seems to be high, how would you justify it in an online continual learning scenario. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Tbzvg0j44S", "reviewer": "Reviewer_Ndan", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studied the problem of improving information retention in online continual learning (OCL), the convergence of SGD in OCL are theoretical analyzed. Besides, an Adaptive Moving Average (AMA) Optimizer and a Moving-Average-based Learning Rate Schedule (MALR) to optimize the pure replay objective online are proposed. Experiment on several large scale continual learning benchmarks demonstrated the effectiveness of AMA+MALR.", "summary_of_the_review": "This paper is well written and studies an interesting problem, I think it is marginally above the acceptance threshold.", "strength_and_weaknesses": "Strength: \nThis paper considers an interesting problem, it is clearly presented and readable. The results of numerous experimental results are convincing. The proposed method is easy to implement. A wide range of related studies are discussed.\n\nWeaknesses: \n1.The typesetting in some places is out of order, such as equations (23), (25), and (31).\n2. In Page 4, \"A4 bounds the degree of non-stationarity between consecutive iterations\", why this assumption holds?\n3. This author should add more description about the contribution of this paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "KYVxDK3juDu", "reviewer": "Reviewer_mh9H", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces three main results:\n\n- a theoretical analysis of SGD under distribution shift to highlight limitations of current OCL approaches\n- a new OCL algorithm based on moving averages (AMA)\n- an associated heuristic to tune the learning rate also based on moving averages (MALR)\n\nother smaller contributions are a population-based approach to tune the EMA weights and empirical evaluation on multiple datasets", "summary_of_the_review": "Overall the paper falls a bit short of the acceptance threshold due to the lack of focus in the contribution, where the authors try to provide results from three sides (theoretical, algorithmic and experimental) but only achieve small gains in each. In particular:\n- The theoretical contribution is a straightforward extension of existing results, and only act as a guideline in designing the algorithm as it cannot be applied to it. This can be fixed by a more realistic analysis of AMA+MALR.\n- The algorithmic contribution is novel and based only on familiar and well established tools, but its impact is reduced by the large number of hyperparameters involved in the actual implementation.\n- The experimental results are good, but not rigorously evaluated against strong baselines and might be greatly influenced by the large number of hyperparameters.", "strength_and_weaknesses": "The strength of the papers are mainly in the simplicity of the resulting algorithm, and in the thorough exposition of the result (attempt at theoretically grounding the approach, multiple ablations, etc...)\n\nThe main weaknesses are:\n\na) the algorithm analysis is only for unlimited memory SGD. The final algorithm proposed is finite-memory and employs EMA. This is an enormous disconnect, and makes Thm. 1 more of a help to exposition in the paper than as a contribution. Section A.2 only gives a very brief and informal idea of how parts of this gap can be filled. The authors should either report a theoretical result more aligned with what the algorithm is actually doing, or clarify that Thm. 1 is mostly for illustrative purposes.\n\nb) The final algorithm seems simple but hides its complexity behind several design choice. A partial list of values to tune includes K_W, K_V, K_M, delta, and how the train/validation splits should be chosen in practice both for AMA and MALR. The conditions C1-3 also introduce extra tunable parameters, without much discussion (e.g. what happens without C3?). Only a fraction of these values are ablated in the appendix. Giving advice on how to improve this is hard, as even the obvious thing (extensive ablation study) might change greatly from dataset to dataset.\n\nc) The only baseline is SGD+RWP. This is enough to justify that the approach is superior to SGD+RWP, but many other approaches for OCL exist in the literature with varying degrees of complexity. Even a quick search yields several off-the-shelf tools to set up some baselines (e.g. https://github.com/ContinualAI/continual-learning-baselines) and at least a few of them should be included to gauge how much AMA-MALR can learn on these datasets.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "JdYYwNFJPtp", "reviewer": "Reviewer_u5yA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies various adaptive training methods and schedules in online continual learning.  ", "summary_of_the_review": "I think the paper is studying online learning rather than online continual learning. It didn't compare with any existing online learning or online continual learning methods. The adaptive learning methods and schedules proposed in the paper are not  new as they have been used in other areas. ", "strength_and_weaknesses": "Strength\nStudying different training methods is interesting in online continual learning. \n\nWeaknesses\nMy understanding of information retention is to avoid catastrophic forgetting. Why not use the standard terminology?\n\nOnline continual learning is a non-stop process, I don’t understand what you mean by convergence. In standard online continual learning, whenever a small batch of data from the stream is collected it is trained in one iteration. New tasks of completely different distributions (with new class labels) come constantly. In such a non-stationary data setting, it is unclear how you know when the convergency should happen.  \n\nYou wrote “We allow 80 training iterations per time step,” but that is not the standard online continual learning setting. My understanding is that you are doing online learning rather than online continual learning. In online continual learning, it is assumed that the system sees the data only once. Once a small batch of training data comes it is immediately trained in one iteration. Some classes as a task may come early and some classes may come later in a sequence of tasks. So, the distribution change at the task switch is drastic and the time of each such switch is unknown. I don't think that you are working under this setting, but I am not familiar with the datasets that you are using. In the standard online continual learning setting, it is unclear how your adaptive training can be applied. \n\nAgain, I think you are working on online/incremental learning rather than online continual learning. Then, you should compare with methods in online learning literature. I'm not working in that field, so I cannot give you any references off the top of my head. But I did read some papers in the area a while ago. \n\nSince your data is not commonly used in online continual learning, it is hard for me to see how your method performs against other methods. There are many online continual learning benchmark datasets. I think you should compare with those existing online continual learning baselines using the existing benchmarks. I am surprised that you did not compare with any existing online continual learning baselines, but only various variants of training methods. If no comparison, I won’t know that your information retention is better. For example, some recent systems are (you can find a lot of other baselines in these papers): \n1. Mai et al. Supervised contrastive replay: Revisiting the nearest class mean classifier in online class-incremental continual learning. CVPR workshop’2021.\n2. Bang et al. Rainbow memory: Continual learning with a memory of diverse samples. CVPR’2021.\n3. Guo et al. Online Continual Learning through Mutual Information Maximization, ICML’2022. \n\nIf you are working under a different setting, you should have compared with the existing setting and stated the differences. The existing online continual learning systems should be easily adapted for your setting by simply learning one task of all classes. Actually, in the case, existing online learning methods may do better.\n\nIt is unclear whether you are using your adaptive training schedule at each time step or in the time span of the whole training process. My guess it is the latter. \n\nThe figures in Figure 1 are too small to see. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject