Overcoming Recency Bias of Normalization Statistics in Continual Learning: Balance and Adaptation
Authors: Yilin Lyu
Liyuan Wang
Xingxing Zhang
Zicheng Sun
Hang Su
Jun Zhu
Liping Jing
Venue: NeurIPS 2023 poster (2023)
Keywords: Continual Learning
Batch Normalization
Recency Bias
Catastrophic Forgetting
TL;DR: We theoretically analyze the recency bias of normalization layers in continual learning, and propose an innovative approach to overcome this particular challenge in both training and testing phases.

Abstract: Continual learning entails learning a sequence of tasks and balancing their knowledge appropriately. With limited access to old training samples, much of the current work in deep neural networks has focused on overcoming catastrophic forgetting of old tasks in gradient-based optimization. However, the normalization layers provide an exception, as they are updated interdependently by the gradient and statistics of currently observed training samples, which require specialized strategies to mitigate recency bias. In this work, we focus on the most popular Batch Normalization (BN) and provide an in-depth theoretical analysis of its sub-optimality in continual learning. Our analysis demonstrates the dilemma between balance and adaptation of BN statistics for incremental tasks, which potentially affects training stability and generalization. Targeting on these particular challenges, we propose Adaptive Balance of BN (AdaB$^2$N), which incorporates appropriately a Bayesian-based strategy to adapt task-wise contributions and a modified momentum to balance BN statistics, corresponding to the training and testing stages. By implementing BN in a continual learning fashion, our approach achieves significant performance gains across a wide range of benchmarks, particularly for the challenging yet realistic online scenarios (e.g., up to 7.68\%, 6.86\% and 4.26\% on Split CIFAR-10, Split CIFAR-100 and Split Mini-ImageNet, respectively). Our code is available at https://github.com/lvyilin/AdaB2N.

Overcoming Recency Bias of Normalization Statisties in Continual Learning: Balance and Adaptation
Yilin Lyu't Liyuan Wang** Xingxing Zhang2t Zicheng Sun
Hang Su² Jun Zhu Liping Jing't
Beijing Key Lab of Traffic Data Analysis and Mining, Beijing Jiaotong University
Dept. of Comp. Sci. & Tech., Institute for AI, BNRist Center, THBI Lab, Ssinghua-Bosce Joint Center for ML, Tsinghua University
fyilinlyu, zichengsun, ipjing @bjtu. edu. wlyl9etsinghus org. en, xxzhangl 9938 9 com (suhangss, (teej))tsinnghie edu. on
Abstract
Continual learning entails learning a sequence of tasks and balancing their knowl- edge appropriately. With limited access old training samples, much of the current work in deep neural networks has focused on overcoming catastrophic forgetting of old tasks in gradient-based optimization. However, the normalization layers provide exception, as they are updated interdependently by the gradient and statistics of currently observed training samples, which require specialized strategies to miti- gate recency bias. In this work, we focus on the most popular Batch Normalization (BN) and provide an in-depth theoretical analysis its sub-optimality in continual learning. Our analysis demonstrates the dilemma between balance and adaptation of BN statistics for incremental tasks, which potentially affects training stability and generalization. Targeting these particular challenges, propose Adaptive Balance of BN (AdaB"N), which incorporates appropriately Bayesian-based strategy adapt task-wise contributions and a modified momentum balance BN statistics, corresponding to the training and testing stages. By implementing BN in continual learning fashion, our approach achieves significant performance gains across a wide range of benchmarks, particularly for the challenging yet realistic online scenarios (c.g., up to 7. 7.68%, 6.86% and 4.26% on Split CIFAR- 10, Split CIFAR- 100 and Split Mini-ImageNNe,, respectively). Our code is available at htttps://gtthui wtt.a//ww......llll.llo.cccccc..
Introduction
Continual learning aims to acquire, accumulate and exploit knowledge from sequentially arrived tasks. As the access to old training samples is typically limited, the acquired knowledge tends to be increasingly relevant to the more recent tasks. Such recency bias, also known catastrophic forgetting [17], is generally attributed to the nature of gradient-based optimization in deep neural networks (DNNs) [9] 30), where the learning of network parameters depends tug-of-warr over currently observed training samples. Numerous efforts have been devoted to addressing catastrophic forgetting of these learnable parameters through rectifying their gradients. However, as important component DNNs, the normalization layers employ particular updating rules that are only partially gradient-based, and their recency bias in continual learning remains be addressed.
Equua contribution. Corresponding authors.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Table 1: Comparison of normalization methods for continual learning.
Method
Statistics
Training Adaptability
Online
Offline
BN
11
Unbalanced
X
X
CN
19
Unbalanced
x
BNT 34
Balanced
X
/
TBBN 8
Balanced
X
/
AdaB2N (Ours)
Adaptively balanced
V
The normalization layers typically include affine transformation parameters and statistics parameters to normalize the internal representations, in order to accelerate convergence and stabilize training Since these two kinds of parameters are updated respectively from the gradient and statistics (i.e., the empirical mean and variance) of currently observed training samples, the normalization layers require specialized strategies to overcome recency bias in continual learning. In particular, the most popular Batch Normalization (BN) [ performs an exponential moving average (EMA) of statistics parameters for testing, which preserves historical information to some extent, and thus outperforms many other alternatives in continual learning [99 g] As the BN statistics of old tasks still decay exponentially, some recent work [99 34 enforced a balance in the contribution of individual tasks to the BN statistics, but with limited effectiveness and generality (see Table [ for conceptual comparison of their targets).
33
24
So, how to update normalization statistics for better continual learning: balance or adaptation? In this work, present in-depth theoretical analysis of the most popular BN, demonstrating its potential challenges posed by continual learning. First, a regular EMA with constant momentum cannot guarantee the balance of BN statisties for testing, which yields more recency bias more training batches of each task are observed. Although a modified EMA with normalized momentum of batch number can enforce balanced statistics, they tend to be outdated for learning each task and thus limit model adaptability. Besides, the sharp changes in task distributions can interfere with the benefit of BN in stabilization of training, which further affects continual learning.
Based on these theoretical insights, we propose Adaptive Balance of BN NAdaB²N) to improve continual learning. Specifically, we leverage Bayesian- strategy to decide appropriate way of accommodating task-wise contributions in normalized representations to stabilize training. We further incorporate an adaptive parameter to reconcile the constant and normalized momentum of EMA, in order to strike a balance of BN statistics for testing while training each task with adequate adaptability. Extensive continual learning experiments demonstrate the superiority of our approach, which provides substantial improvements especially for the challenging yet realistic online scenarios.
2 Related Work
Continual Learning: The major focus of this area is to address recency bias aka catastrophic
forgetting in DNNs 30
28 29 Representative strategies are generally designed for the learn-
able parameters with gradient- based optimization (30], such as weight regularization [13] 1 27].
experience replay 32
6 gradient projection
[I5
20 etc. In particular, experience replay of a
few old training samples has proven to be usually the most effective strategy and is applicable to
large-scale applications 30 32 BT However, its reliance on the old training samples may further
deteriorate the imbalance and overfitting of normalization statistics. Corresponding to the real-world dynamics, continual learning gives rise to many meaningful settings and particular challenges [80] For example, desirable continual learning model should perform well in both task-incremental and class- maas-incremmntall settings, which depend on whether the task identity is provided for testing. In contrast to offline scenarios of learning each task with multiple epochs, the model should also accommodate online scenarios of learning all tasks in a one-pass data stream, which is realistic but extremely more challenging.
The normalization layers have become a critical component for many representative DNN architectures, where Batch Normalization (BN) [1] is the most popular choice along with many other competitive alternatives designed for single-task learning (3 24 2 However, the normalization layers require specialized strategies to perform appropriate parameter updates in continual learning. As the EMA of statistics parameters retains a certain amount of past information, BN is naturally the dominant choice for continual learning and indeed strong bascline for overcoming
Normalization Layer:

---

recency bias in normalization layers 19 88. To further improve BN, Continual Normalization (CN) 19 removed differences in task-wise distributions and then performed BN acquire task-balanced statistics, providing benefits for online continual learning. BNT 34 and TBBN g constructed task-balanced batches to update BN, but only effective for offline continual learning. However, none of the current efforts can improve both online and offline scenarios g with the effectiveness of CN varying further with batch size in different online benchmarks. This motivates our work to provide a more in-depth investigation of BN with respect to the particular challenges of continual learning.
3 Analysis of Normalization Statistics in Continual Learning
In this section, we introduce the formulation of continual learning and representative normalization strategies, and then provide in-depth theoretical analysis to diagnose the most popular BN. Additionally, a notation table is included in Appendix A and proofs of theorems can be found in Appendix B
3.1 Notation and Preliminaries
Let's consider general setting for continual learning, where a neural network with parameters needs to learn sequence of tasks from their training sets D,,... DD in order to perform well on their test sets. The training set of task is defined as DD = {( vt.n)) (P:), where (It. Vt..)) is data-label pair of task E [1 and | denotes its amount. Since the old training sets Dia--1 = =U D; are unavailable when learning task 0 tends catastrophically forget the previously learned knowledge. An effective strategy is to save a few old training samples in a small memory buffer, denoted as M = {M& in order to recover the old data distributions when learning the current task t. We denote the m-th training batch Bn, which includes N₂ training samples from D, and N₂ training samples from M₁ for k- = 1,.... .t 1. Here N = -E N₂ = B denotes the batch size. The batch index m E (m+-1, m.). where my = LiD INJ]/N/BB E is the index of the last batch of task I, mo = O, and E the number of training epochs. We denote I' N,/N as the proportion of the current task L in Bm: where Y'm < indicates the use of replay while I'm = 1 indicates that no replay is used. Without loss of generality, we assume that the old training samples in each batch are lask-balanced i.c., N₁ = N2 = = N1-1- The test set follows the same distribution as the training set, with the task identity provided in mes-iincrementl learning (Task-IL) while not provided in class-ineremental learning (Class-IL) [25].
Inside the network, the internal representation of the L-th layer of the m-th batch is denoted as (2) Nx xD()) where ((t) and (()) represent the number of channels and feature dimensions, respectively. For the sake of notation clarity, we omit the layer and batch index hereafter when no confusion arises. Then, the operation of a normalization layer can be generally defined as:
am
a m(a)
à = 7a' + B,
(1)
Vo ((a))) ((a))
+
where the statistics parameters (i.e., the empirical mean and variance) are calculated from the corresponding functions ju(-) and o.(,,), respectively, E is a small positive constant to avoid the denominator being zero, 7 and /3 are affine transformation parameters that are learned by gradient- based optimization. Representative normalization strategies differ mainly in the ways of computing statistics parameters of internal representations, i.e., the exact forms of μ and d², detailed below.
Batch Normalization (BN) [ is the most popular choice for wide range of DNN architectures. During the training phase, BN calculates the statistics parameters internal representations for each training batch through μ NxCxD -> RC and o2 NxCxD x RC RC, where
((a)=
N ΣΣ ND
1
(nn.dd n= d=1
(a
1
ΣΣ (mn.aj-((aa)) 1=1d=1
(2)
ND
During the testing phase, make the normalization independent of other samples in the same batch, BN replaces the batch statistics of Eq. (I) with the population statisties estimated by exponential moving average (EMA). For ease of notation, here we define a convenient function S NxCxD 22xC to obtain the two kinds of statistics, where
S(-) (n()), (2))))((,))))) T.
(3)
3

---

Then we denote the statistics of internal representation dm as Sm = S(am), with the layer index omitted. Now the population statisties of BN can be written
BIS..] := BIS]a,,".. -.. _..] (1 n)B[S am-11 1] + nsm:
(4)
where 77 (0.1) is the momentum of EMA and usually set to 0.1 in practice.
There are many alternative improvements of BN designed for single-tast learning without the of EMA. Group Normalization (GN) 331 can be seen a general version, which divides the channels into groups and then normalizes the internal representations of each group. Specifically, exxcxgroups
let & RNxCxD + IR be reshaping function that groups the channels, where G is the group number and K J Then the mean and variance of GN are obtained through u IRNxCDD RN and o² RNNxxDD x B RNNGG where
p(a)= KD ((a)) (a
KD (E(a))AAA)) - u(aa))
(5)
particular, GN equivalent to Layer Normalization (LN) by putting all channels into one
group (G = 1). and Instance Normalization (IN) [24] by separating each channel as a group (G = C). In contrast to these altemnaties. Continual Normalization (CN) [19] is designed
particularly for (online) continual learning, which performs GN with y 1 and | = 0 to remove
task-wise differences and then performs BN to acquire lask-balanced statistics.
3.2 Conflict of Balance and Adaptation
Although BN has become a fundamental component for many DNN architectures, its particular recency bias in continual learning remains under addressed. To quantitatively analyze the task-wise contributions in BN statistics, we define a statistical weight (of a certain batch) of each task. Then we diagnose the recency bias of BN statistics with specified dilemma between balance and adaptation.
Definition 1 (Statistical weight of task). The statistical weight of task t w.F.t. the population statistics [[[m]] the weight wt RR+ such that we En" wt/Z. w E RR+ is the statistical weight of task of the i-th batch such that ΣT Ets En w = [[S where S == S(a,,, ((..))))))))
the normalizing constant Σ E; w/
We now give rigorously the statistical weight for EMA in order to demonstrate the recency bias: Theorem 1 (Statistical weight for EMA). Let T, Z be defined as above, and {me]T= be an increasing
sequence (i.e., my < m++1), where m. E zz+ 17: [1, mT] - (0, 1), and r [1,mT] - (0. 1). The
statistical weight of task t [1,T] w.r.t. the population statistics B[Smr] estimated by Eq. 44 is
W1 =
Σ
II (1- (1-))++
Σ
7e(1 r3)
II 5=11
(1 - (1) 12,
(6)
T-1
J=++1
i=me++1
where define 7) = 17() and r(i) for brevity.
Eq. (6) specifies the impact of each task on the EMA population statistics, which exposes a dilemma between balance and adaptation, as we demonstrate below.
Corollary 2 (Adaptation of BN statistics). Let {me}T= be defined as above. If r(()) = and 7(-) = r = 17, the statistical weight of task t E [1, T] w.r.t the population statistics B[Smy] is 7- - TT 17. When mr mT-1 = MT- MT-2 = mi. we have
Wy R (T- 1 /7, with approximation error |c| E 1 nT1
I-
Corollary shows that without the use of replay, a model dealing with evolving task distributions has strong preference for the distribution of the current task, as the statistical weights of past tasks decrease exponentially. Replay of old training samples can balance statistical weights of all seen tasks to some extent. However, it cannot eliminate the recency bias except in unrealistic case:
For simplicity, we assume zero covariance among the internal representations of different tasks.

---

Corollary 3 (Balance of BN statistics). Let {mi}r=1 be defined as above. = = n(-))
11-1--mm- ?7 1), and m 1T-1- - m = my then the statistical weight of each task t E [1, w.r.t. the population statistics _[ achieves (approximately) equal if and only if R 1/T, with approximation error le - (77
Corollary 3 shows that it is possible to balance the statistical weights by constructing task bask-balanced batch with the memory buffer. However, this will severely slow down the convergence. Besides, as the memory buffer is usually limited in size, the model can easily overfit to the preserved old training samples 4 which detrimental to generalization. Corollary ] also demonstrates the importance of keeping flexibility of 7. For instance, if we substitute n7() T/(1 + i) into Eq. (4), this leads to normalized momentum of batch number. In this case the population statistics can he considered as the cumulative moving average (CMA) [19] of the batch statistics, which can balance the statistical weights as below:
Corollary 4 (CMA). Let {m=}]=1 be defined as above. Ifr(-) rr, then the statistical weight of each lask L E [1, w.r.t the population statistics [[Smy] achieves equal if (n( = 1/(1+
However, the balanced statistical weights do not necessarily lead to better performance, as the statisties of past tasks may be outdated relative to the current model. This is empirically validated in [19]. where CMA does not outperform EMA in continual learning. Therefore, an improved strategy of updating population statistics for testing is urgently needed to reconcile balance and adaptation.
In addition to the update of population statistics during the testing phase, the current strategy of calculating batch statistics during the training phase is also sub-optimal for continual learning, as detailed in the next subsection.
3.3 Disturbed Training Stability
BN has been shown to stabilize training and improve generalization under traditional i.i.d. data setting 3 [6) However, it is in fact not clear whether BN can achieve the same benefit in continual learning, especially when the model has just switched to a new task with drastic changes in loss and parameters. We perform preliminary analysis by the following theorem:
,o,,,,,,,,, Theorem 5 (Theorem 4.1 of [21), restated). Let μ.. o² ,7,,, 2,,,a, a' be defined above. Let f. be the loss function of a BN model (i.e., using the normalized representations via Eq. (T) and L' be the identical loss function of an identical non non-BN model (i.e., a' = a). The gradient w.r.t the internal representation of the E-th layer follows that
72
V A12 S C. [CIV 6'P1- (1 (e'
C')² - (V F' d(
5 that BN can contribute reducing the Lipschitz constant of the loss and thus stabilizing the training of a single task. However, as the model underfits to the distribution of each continual learning task, the upper bound of the gradient magnitude will be loosened by the increase of { (V. L' a'} within the third term, i.e., the similarity of the normalized representation and its corresponding gradient, which potentially affects the benefit of BN. Here we present empirical results to validate the above claim. As shown in Fig. a dramatic change in loss L occurs whenever the task changes. We observe an increase of cosine similarity between the gradient and the normalized representation at this time, accompanied by dramatic change in the gradient magnitude. This clearly demonstrates the importance of properly adapting BN statistics to each continual learning task, so as to better leverage the benefit of BN in stabilization of training.
Method
Following the theoretical analysis, we propose Adaptive Balance of BN (AdaB?N) from two aspects: (1) training aspect: we apply Bayesian- based strategy to adjust inter-task weights and construct the normalized representation, allowing for learning an appropriate way of weighting task distributions to stabilize training; and (2) testing aspect: instead of using fixed momentum ry in EMA, we turn to a generalized version that balances the statistical weights of past tasks while reducing the impacts of past batches. We now present both ideas follows with pseudo-code in Algorithm
5

---

C
cos V VC
VEIF
Dj D2
BI
0.0050
0.10
my
Ours
0:0025
3-
0.0000
0.06
0.0025
40010
2000
4000
2000 Steps
Steps
Steps
(a) Loss
(b) Gradient Similarity
(c) Gradient Magnitude
Figure I:
Dynamics of the loss of training batches, gradient similarity (i.e., cosine similarity between
gradients and normalized representations) and gradient magnitude.
Algorithm Training Algorithm of Adaptive Balance of BN (AdaB2N)
Input: training dataset D.r, memory buffer M, batch size N, neural network fa... = face fult o fw(L) -.. (u f ... fwon foa), where face and fwin are the 4-th network layer and normalization layer, respectively, parameters fo wn, the objective function & in Eq. 15 and hyperparameters 17, A.
(7((0 g(6) vi(e)) Jē=1. ( \ w, initial
Output: parameters f,w
1: Initialize: 1 do,
wa, E[S]+
2: for + 1 to T do
> For each task For each epoch
3: for < 1 to E do
4: for i t to [|Dal/N] do
> For each batch
5:
B; = (X,, Y;) + Sample(Dr; (M
6:
M <- UpdateMemory (M, B,)
7:
((0) X;
8:
for E + to 1 do
b For each layer
(0)) t Saco (a(t(11)) (á(t-1))
9:
10:
Compute [[ via Eq. 13 and 17: via Eq. (16)
11:
Update [SS] via Eq. (4)
12:
Compute a d(()) via Eq. 8 and a (0) via Eq.
13:
Y, fou-- (L(
14:
4 V WoC, w t w WWC
15:
return (1,w) w)
4.1 Adaptive Balance of BN Statistics for Training
Theorem 5 demonstrates that the benefit of BN for optimization and generalization is potentially interfered continual learning. To overcome this challenge, here we propose a Bayesian- strategy for learning appropriate way of weighting task contributions in normalization statistics. We assume the statistics of batch m for certain layer follow a probability distribution S (S)amm) where the randomness originates from the neural network parameters 6, and then use the conditional expectation w.r.t. the distribution of statistics to calculate the normalized representation:
Om
B[x am
(8)
am
+E
Specifically, E[p[a,.] and E[o² [am] are the two elements of E[S]am] as described in Eq. 13
B[S)ami
am. Pm
(9)
where Pt(Stt am) is the distribution of statistics given the task identity, and (t(())m) is the likelihood of the task identity occurring in amn- We treat Po(t)a,,) as a categorical distribution over the task identity, i.c., Pm((|am) = N./N. Then, BN obtains its normalization statistics by the point estimate of the conditional expectation:
[
Σ Po am S Sm.
(10)

---

As analyzed in Section 3.3 it is critical to consider adaptability of the model parameters @ together with normalization statistics model training, as to enhance the Inspired by the theoretical works in Bayesian Inference [22 12 we propose to capture the variability of 0 through modeling the task distribution with a Bayesian principle, thus enhancing the benefit of BN for continual learning. Specifically, we assume that there random variable A
correlation.
modeling the probability distribution of tasks, where A denotes the T- simplex, i.e., Σ 72 = and Tt > 0, Vt. Applying the Bayes' rule, we have
ammm = -ΣΣ Po(S),,t, am) PolT om Pa am)s",
(I1)
where (S(STT, L, (m) and Po(4IT, 4m) are defined analogous to (S am and Po(()em) in Eq. (
Pa(rlan serves the prior captured from the model parameters 0, which is generally intractable. For ease of computation, we approximate it with Dirichlet distribution, i.e., the conjugate prior of the categorical distribution, which is parameterized by the concentration parameter & E R²+
(( IL [()))
Pa(T
II
(12)
where r(-) is the Gamma function, and e = -, th. By taking advantage of algebraic convenience of the conjugate distribution, we can estimate the conditional expectation by a closed-form expression
E|S|am] = ΣΣ Pa(rlam))P((the Sm = Σ de + N₂
S...
(13)
④ N
Intuitively, the concentration parameter & plays the role of adding pscudo-observations to the task weighting procedure, which allows the normalization statistics to adaptively balance task-wise contributions by considering the current state of 0 To optimize o, we introduce a learnable parameter z E R and obtain ó = exp(*)). It is worth noting that the dimension of ø is incremental with the number of seen tasks, so there is no need to know the total number of tasks in advance. As é is expected to capture the information of f, we propose regularization term minimize the Euclidean distance between the adaptively balanced statistics and the estimated population statistics via Eq. as follows:
L (0, v+ = [| BIS ]I?
(14)
where [[SSm]] is an improved version introduced in Section 4.2 Eq. 14 provides & with historical
information across batches by aligning the population statistics, which contributes obtaining statisties as good as joint training (detailed Section (5). From Fig. [ we observe that adding this regularization term can significantly improve the gradient similarity, leading to stabilized training dynamics. We optimize vb and 9 (with y and | inside) simultaneously, and the final objective function can be written as
min (C) 8.0
Σ cll) (0, (0,
(15)
where LOL is the loss function of continual learning (e.g.. cross-entropy loss over D, and )= for classification tasks). À is a hyperparameter to balance the two terms.
4.2 Adaptive Balance of BN Statistics for Testing
Inspired by Corollaries 2 to we propose to employ an appropriate 77 function to strike an adaptive
balance between EMA and CMA. Concretely, we design a modified momentum 17 [1, (1,rr] x (0, 1) +
(0, 1) satisfying the following recurrence relation:
174 ni Mi 11-1 (1 f)²
76-1
No : f
(16)
where rj (0, 1) the modified momentum of E.M and x E [0. 1] is an additional hyperparameter to control the balance of normalization statistics. In particular, Eq. (16) degenerates to EMA when + 1 and CMA when Æ + O. This allows us to find an inflection point for adaptive balance of statistical weights, so as to enjoy the benefits of both EMA and CMA.
5 Experiments
In this section, we first briefly introduce the experimental setups of continual learning, and then present the experimental results to validate the effecttivnness of our approach.
7

---

Table 2:
Performance of online task-incremental learning with batch size [B] final average accuracy of all seen tasks (1) with - standard deviation.
10. We report the
Split CIFAR-10
Split CIFAR-100
Split Mini-ImageNel
Method
|M|=000
[ =2000
|M|=0000
|MV|=5000
||||=0000
|M|==0000
ER-ACE w/ BN
86.34a2.35
19.17:1.21
63.83+1.94 61.21
63.62+3.14
64.60=1.73
ER-ACE w/ LN
78.01#6.78
81,5924.39
42.2940.29
44.3240.35
45.91+1.63
45.82+1.91
ER-ACE w/
84.57=1.42
86.03+1.99
49.4642.25
50.15#1.71
40,93+4.38 41.07+1.66
ER-ACE w/ GN
81.46#3.77
82.85+2.14 14
44.84+1.41
42.50+2.06
43.45+3.29
45.43=0)62
ER-ACE w/ CN
88.32#11433
90.01
61.00
63.42:0065
64.23 1.2
65. 65.1441.44 1.48
ER-ACE w/ Ours
88.74=1.77
90.84+2.01
63.88+1.58
67.01+2.90
66.78=1.91
68.03=0.41
DER+ w/ BN
87.61=1.67
90.42+1.83
65.53+1.17
66.23+0.94
61.70+2.40
61.28+1.29
DER+ w/ IN
81.35=4..25
82.80+5.48
43.24+0.86
44,42+2.90
40.05=3.67
37.6444.26
DER++ w/ IN
84.81+2.99
87.04+2.60
17.39
49.17+2.13
35.23#6.11 1..
DER+ w/ GN
82.05=1.24
83.34+3.34
13.54
45.12+1.15
38.26#3.26 1..65+1.76
DER++ w/ CN
86.92=0.89
89.75=0.76
66,20+0.38
67.39+1.88
1.70
66. 16.114:1.. 1.
DER++ Ours
90.15#2.52
91.99#0.81
70.33+0.49
71.70=0.84
66.42+3.62
69. 69.12+2.51
Split CIFAR- 10, Tack-II
Split CIFAR-10, 10, Class. II
Split CIFAR-100 100. Task-
Split CIFAR-100, Class-Il
Ours [0]
10
36
FFA on BN (%)
CN [
CN
0
PRACEE (M] 900
DRR
(M]) 500
BR-ACE
2000
DER.
Mi 2000
ER-ACE
(MI- Rd
DER
(MI Tefi
Er ACE
MM 3mn
DER
MM 230
ERAACE
IM| 2000
DER
[MI 2000
BRAACE
IM 1030
DER
[MI 5000
ER-ACE
[Mi 200)
DER-+
M4| 2300
ER-ACE Mi 5960)
DER Mi 1988
14
Figure 2: Performance improvements (i.c., A FAA) for online continual learning over different batch
sizes B and memory sizes [.M.
Benchmark: We mainly focus on the setting of online continual learning (30 which is extremely challenging but realistic application, and further consider the setting of offline continual learning. Here we evaluate three benchmarks that are widely used in both settings. Specifically, Split CIFAR- 10 [14 includes 10-class images of size 32 x 32, randomly split into 5 disjoint tasks with 2 classes each. Split CIFAR-100 [14] includes 100-class images of size 32 x 32, randomly split into 20 disjoint tasks with classes each. Split Mini-ImageeNet (26) includes 100-class images of size 84 84, randomly split into 20 disjoint tasks with 5 classes each. We consider both Task-IL IL and Class IL, depending on whether the task identity is provided at test time (25).
Baseline: Following the previous work 19. we employ ER-ACE 7 and DER++ o as the baseline approaches for continual learning, both of which rely on small memory buffer A to replay old training samples. We further consider LiDER (4) in offline scenarios, a recent approach for improving experience replay through regularizing Lipschitz continuity. We then compare our approach to three categories of normalization layers: (1) BN III. which is the most popular choice that exhibits strong continual learning performance 19 8] (2) BN's alternatives designed for single-task learning, such
as LN 2), IN (24) and GN R3 (GN can be seen as a general version of INN and IN); and (3) CN 19 which is the sfate-o-thh-are art baseline that improves BN for online continual learning.
Implementation: We follow the official implementation as many previous works 19 7 6), derived from the mammoth repository (5). Specifically, we use R8sNet-18 backbone and train all baselines with an SGD optimizer of learning rate 0.03 for Split CIFAR- 11FAR-100-100 10/-100 and 0.1 for Split Mini-ImageNNer Notably, we find the official implementation of CN 19 employs different batch sizes (either 10 or 32) for different online continual learning benchmarks. We argue that using smaller batch size is more "online" and more challenging, as fewer training samples of the current task can be accessed at a time. Therefore, we reproduce all results of online continual learning with one epoch and a small batch size of 10, and further evaluate the effect of using a larger batch size of 32. Besides, we use 50 epochs with batch size of 32 for offline continual learning. We adopt reservoir sampling save
8

---

Table 3:
Performance of online class-incremental learning with batch size [B] final average accuracy of all seen classes (1) with = standard deviation.
10. We report the
Split CIFAR-10
Split CIFAR-100
Split Mini-ImageNel
Method
|M|=000
[ =2000
|M|==000
|MV|=5000
|||==0000
|M|==000
ER-ACE w/ BN
48.86*2.87
52.25+2.55
24.55
26.07+2.78
14.41+1.31
14.53+0.58
ER-ACE w/ LN
32.29=0.37
16.64+1.92
11.5040.10
11.96#1.13
6.28&1.08
6.21
ER-ACE w/
44.16=1.98
45. 45.86+3.25
15.20+0.87
16.1540.80
5.00+0.82
5.13+1448
ER-ACE w/ GN
38.97=2.04
39. 99.60=3399
12.25+0.83
11.8640.74
5.33+0.83
6.29=0.60
ER-ACE w/ CN
50.54:4,94
53.97=1.20
23.87
27.23+0.37
15.76a0.70
15.39:1.45
ER-ACE w/ Ours
52.51=3.04
59.93=4.49
25.39=1.40
28.15+3.24
16.41=0.96
17.08=1.06
DER+ w/ BN
52.85+4.34
57.96+2.57
18.84#1.25
17.26+1.91
7.04+1.47
6.59+0.67
DER+ w/ INN
34.08=2.59
35.37=6.18 18
6.42+0.95
6.25+0.60
2.94 1.08
2.21+0.43
DER++ w/ IN
1..2
50.86+2.61
8.52+0.27
8.5810.46
1.88+0.33
2.33
DER+ w/ GN
39.54=1.47
38.48=1.13
7.4420.78
6.77+0.52
2.8040.18
2.72+0.54
DER+ w/ CN
55.28=1.06
60.25+3.39
18.8440.09
18.95+1.08 .05
10.1641.07
8.94+0.94
DER++ Ours
59.44=2.65
64.83+1.89
23.79+2.23
24.12+1.63
10.80=2.76
10.85
Split CIFAR-100, DER++, [M| 5000
Task-IL
Class-IL
24
0.0
0.001
0.01
0.1
1.0
10.0
70
Final Average Accuracy (%)
0.0
0.001 0.01 0.1
1.0
10.0
Final Average Accuracy (%)
37.3
37.5
56.2
63.4
65.5
6.8
7.0
11.3
15.
19.8
19.6
-22
0.1
61.7
71.1
68
0.1
16.5
0.4
63.7
64.3
65
67.1
71.8
m/a
0.4
17.4
17.3
17
23.9
n/a
20
66
0.7
64.3
66.5
65.6
67.6
69.7
n/a
0.7
17.6
18.6
16.2
19.7
21.8
n/a
1.0
64.5
65.3
66.0
68.8
70.2
n/a
64
1.0
17.6
17.3
16.2
20.8
22.2
n/a
18
62
16
Figure 3: Hyperparameter analysis of online task- iakkenncreeetall learning and for ablation study.
learning
old training samples, which simulates the distribution of incoming batches, and further consider ring buffer that enforces balanced number for each class (30] The memory buffer is sized differently to evaluate its impact. The group number of GN and CN is set to 32 as [9.. All results are averaged over three runs with different random seeds and task splits. Please Appendix [for more details.
Overall Performance: We first evaluate our approach for online continual learning, including the settings of online Task-IL in Table 2 and online Class-IL in Table 3 Consistent with reported results BN performs much better than alternatives designed for single-task learning (i.e., GN, LN and IN), and is moderately improved by the state-of-the-an art CN. In contrast, our approach can improve the performance of BN by a large margin (e.g., up to 7. 7.68%, 6.86% and 4.26% for online Class-Il IL on Split CIFAR-10, Split CIFAR- and Split Mini-ImageNNt respectively), and outperforms CN on wide range of memory sizes IM- When increasing the batch size [B] from to 32, which introduces more training samples at once and makes online continual learning more "offline", we observe that the benefits of CN exhibit variation in many cases, while the benefits of our approach remain consistent and become even stronger (e.g., up to 15% Class-IL of Split CIFAR- 10. see Fig. 2). Mcanwhile, the selection strategies of the memory buffer M have no substantial effect, where our approach can provide similar magnitude of improvement under either reservoir sampling or ring buffer (see Appendix D). We further present the results of offline Task-IL and Class-IL. on the hardest Split Mini-ImageNNt of the selected benchmarks (see Table 4 where CN even results in negative impacts on BN while our approach still provides strong performance gains in most cases.
19 g
Ablation Study & Analysis: We first analyze the introduced two hyperparameters in our approach, ie.. X of the regularization term for training and x of the modified momentum for testing. As shown in Fig. 3 the two hyperparameters can clearly influence the performance of continual learning, which

---

Table 4: Performance of offline task-incremental learning and class-incremental learning on Split Mini- Aani-ImaeeNet with memory size [M] = 2000.
Task-IL
Class-IL. IL
Method
ER-ACE
DER++
LiDER
ER-ACE
DER+
LiDER
BN
36.3642.33
35.91
36,96*1.92
10.85#0.54
12.69
10.99=0.74
CN
33.83=1144
35.46+1.42
36.23#2.62
10.00=0.62
12.13+0.92
9.97+0.36
Ours
37.6440.85
36.98+1.52
37.3620.79
11.05+0.10
13.07=1.30
10.96+0.67
validates the effectiveness of each component in AdaB2N, and obtain consistent improvements under wide range of combinations in different settings. Besides, we visualize the statistics parameters of normalization layers in continual learning, in order to explicitly demonstrate how our approach can improve BN (see Fig. for typical example and Appendix for all results). We take joint training (JT) of all tasks as the upper bound bascline for continual learning. As more training batches are introduced, it can be obviously seen that the BN statistics are increasingly deviated while the CN statistics are enforced to stabilize. They correspond to the biased strategies of adaptation and balance, respectively, but neither fit into the curve of JT. In contrast, approach achieves an adaptive balance normalization statistics in continual learning, matching closely the upper bound baseline.
Discussion and Conclusion
In this work, we focus on the most popular BN in continual learning and theoretically analyze its particular challenges for overcoming recency bias. Our analysis demonstrates the dilemma between balance of all tasks and adaptation to the current task, reflected in both training and testing. Following such theoretical insights, we propose Adaptive Balance of BN NAdaB*N) with Bayesian-based strategy and a modified mo- mentum implement BN in continual learn- ing fashion, which achieves strong performance gains across various scenarios. Interestingly, the robust biological neural networks address a simi- lar dilemma with combination of normalization mechanisms for neurons, synapses, layers and networks, some of which are thought to be analogous the normalization layers in DNNs 23 More in-depth analysis and modeling of these mechanisms would be a promising direction. We also expect subsequent work to further improve continual learning of other DNN components and overcome potential incompatibilities in optimization, as suggested by our work on normalization layers.
Batch Statistics
Population Statistics
10
10
6-
BN
4
Ours
0
2000
4000
2000
4000
Steps
Steps
Figure 4: The norm of BN statistics of represen- tative layer with 0.05 x variance in shaded area.
Now discuss some potential limitations and societal impacts. First, our work applies only to the DNN architectures that employ normalization layers, especially the most popular BN. Second, we need compute categorical distribution Po (t(t)mm) over the task identity, which is difficult adapt directly to the more general setting of task-agnostic continual learning. Third, the prior Po (m)) is approximated
Dirichlet distribution for ease of computation, which may not be applicable in some practical scenarios. We argue that the broader impact is not obvious at the current stage, as this work is essentially fundamental research in machine learning.
Acknowledgements
This work was partly supported by the National Natural Science Foundation of China under Grant 62176020, 62106123; the National Key Research and Development Program (2000AAA0106800) the Joint Foundation of the Ministry of Education (8091B0422) B042235); the Beijing Natural Science Foundation under Grant 1211016; the Fundamental Research Funds for the Central Universities (2019JBZ1 10); and Chinese Academy of Sciences OEII-O-202004 L. W. was also supported by Shuimu Tsinghua Scholar.
10

---

References
[1] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuyte- laars. Memory aware synapses: Learning what (not) forget. In Proceedings of the European Conference on Computer Vision, pages 139-154, 2018.
[2] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization, arXiv preprint 1r7.v:1607.06450, 2016.
[3] Nils Bjorck, Carla Gomes, Bart Selman, and Kilian Q Weinberger. Understanding batch normalization. Advances in neural information processing systems, 2018.
[4] Lorenzo Bonicelli, Matteo Boschini, Angelo Porrello, Concetto Spampinato, and Simone Calderara. On the effectiveness of lipschitz-driven rehearsal in continual learning, Advances in Neural Information Processing Systems, 188-1988-3191,, 2022.
[5] Matteo Boschini, Lorenzo Bonicelli, Pietro Buzzega, Angelo Porrello, and Simone Calderara. Class-incremental continual learning into the extended der-verse. IEEE Transactions on Puttern Analysis and Machine Intelligence, 45(5):5497-555 2022.
[6] Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: strong, simple baseline. Advances in Neural Information Processing Systems, 33: 1520-11590, 2020.
[7] Lucas Caccia, Rahaf Aljundi, Nader Asadi, Tinne Tuytelaars, Joelle Pineau, and Eugene Belilovsky. New insights on reducing abrupt representation change in online continual learning. In International Conference on Learning Representations, 2022.
[8] Sungmin Cha, Sungjun Cho, Dasol Hwang, Sunwon Hong, Moontae Lee, and Taesup Moon. Rebalancing batch normalization for exemplar-based class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20127-20136 2023.
[9] Raia Hadsell, Dushyant Rao, Andrei A Rusu, and Razvan Pascanu. Embracing change: Con- tinual learning in deep neural networks. Trends in Cognitive Sciences, 24(12):1028-1044 2020.
[10] Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning a unified classifier incrementally via rebalancing. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 831-839, 2019.
[11] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pages 448 456. PMLR, 2015.
[12] Taejong Joo, Uijung Chung, and Min-Gwan Seo. Being bayesian about categorical probability. In International conference on machine learning, pages 4950-4961. PMLR, 2020.
[13] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinske et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(1)):52213335 2017.
[14] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
[15] David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances Neural Information Processing Systems, 30, 2017.
[16] Ping Luo, Xinjiang Wang, Wenqi Shao, and Zhanglin Peng. Towards understanding regu- larization in batch normalization. In International Conference on Learning Representations, 2019.
11

---

[17] James L McClelland, Bruce L McNaughton, and Randall C O'ieellly. Why there are comple- mentary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychological review, 102(3):419, 1995.
[18] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1406-1415, 2019.
[19] Quang Pham, Chenghao Liu, and Hoi Steven. Continual normalization: Rethinking batch normalization for online continual learning. In International Conference on Learning Represen- lations, 2022.
[20] Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning. In International Conference on Learning Representations, 2021.
[21] Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch normalization help optimization? Advances in neural information processing systems, 31, 2018.
[22] Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning to quantify classification uncertainty. Advances in neural information processing systems, 31, 2018.
[23] Yang Shen, Julia Wang, and Saket Navlakha. A correspondence between normalization strategies in artificial and biological neural networks. Neural Compulation, 33(12):319--3203, 2021.
[24] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky, Instance normalization: The missing ingredient for fast stylization, arXiv preprint arXiv: 2016.
[25] Gido M van de Ven and Andreas S Tolias. Three continual learning scenarios. In NeurlPS Continual Learning Workshop, 2018
[26] Oriol Vinyals, Charles Blundell, Timothy Lillicrap. Daan Wierstra, et al. Matching networks for one shot learning. Advances in Neural Information Processing Systems, 29, 2016.
[27] Liyun Wang, Mingtian Zhang, Zhongfan Jia, Qian Li, Chenglong Bao, Kaisheng Ma, Jun Zhu, and Yi Zhong. Afec: Active forgetting of negative transfer in continual learning. Advances in Neural Information Processing Systems, 34:2237992222 223991, 2021.
[28] Liyuan Wang, Xingxing Zhang, Qian Li, Mingtian Zhang, Hang Su, Jun Zhu, and Yi Zhong. Incorporating neuro- innuro- adaptability for continual learning in artificial intelligence. arXiv preprint arXiv:2308. 14991, 2023.
[29] Liyuan Wang, Xingxing Zhang, Qian Li, Jun Zhu, and Yi Zhong. Coscl: Cooperation of small continual learners is stronger than a big one. In European Conference on Computer Vision, pages 254-271. Springer, 2022.
[30] Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. arXiv preprint -XXi:2332.000887 2023.
[31] Liyun Wang, Xingxing Zhang, Kuo Yang, Longhui Yu, Chongxuan Li, Lanqing Hong, Shifeng Zhang, Zhenguo Li, Yi Zhong, and Jun Zhu. Memory replay with data compression for continual learning. In International Conference on Learning Representations, 2022.
[32] Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 374-382, 2019.
[33] Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pages 3-19, 2018.
[34] Minghao Zhou, Quanziang Wang, Jun Shu, Qian Zhao, and Deyu Meng. Diagnosing batch normalization in class incremental learning. arXiv preprint 20X00:2202.08025, 2022.
12

---

A
Notation
Ja the notations used in this paper.
Table 5: Notations used in this paper.
Symbol
Meaning
T
Number of tasks
(Ir,s N1,s) Dr
Input samples and its label of task
Training set of task
Dia-1
Old training sets before task
M
Memory buffer Set samples from task saved in memory buffer m-th batch
Mr
B, Bm
N. [B]
Batch size
E
Number of training epochs Index of the last batch of task Proportion of task in B,,
my mm r(m)
Mm, 7(m)
Momentum of exponential moving average (EMA) for Bn Internal representation of the l-th layer of the m-th batch Normalized internal representation aym.
a. am: Om rle) d' ( dgn
à a (0)
Affined internal representation (E)
C, cle
Number of channels of arm
D. D p())
Feature dimensions of am Mean function of a normalization layer Variance function of normalization layer Statistic function of a normalization layer to obtain mean and variance Statistical weight of task t Statistical weight task within the m-th batch
o² (..)
S())
Wi
w
Sm
Batch statisties of internal representation an Batch statistics of task within internal representation Estimated population statisties w.r.t. @1 02, am
S
[[S.m] [ (1.
0ml
E dmm
Conditional expectation of batch statistics w.r.t. am
É S am
Estimated conditional expectation of batch statistics w.r.t. Om Parameters of the entire neural network Parameters the network layer Parameters of the normalization layer
0
Parameters affine transformation Parameters of adaptive balance Objective function
//
C
B Proofs
Proof of Theorem 7 Let [1, mr] x ZT- + [1.7] be function that maps the batch index to its corresponding task index. We abbreviate g(i, (m,) as g()). Then we have
É(Smrl (1 Mmr É[ + Merr Smr
(17)
May (1 P'ot
(1 Mmr E[S Mmnr P' mer may
T
Stt Smn
(18)
T
t=1
7 r5)
=
M II (1 S59
Σ
II k=j=1
(1 (
(19)
T-1
1=1
#=++1
+1
where the first part of Eq. 19 can be considered as the contribution of the "current" tasks of all batches to the running statistics, and the second part can be considered the contribution of the
13

---

"past" tasks. Now we can conclude the proof by noting that the contribution of task as the "current" tasks to the running statistics starts from 1 my and the contribution the "past" task starts from my + 1 to myr.
Proof of Corollary 2 The first result can be obtained by noting that nj is a constant and directly applying the sum formula of geometric series. When nr MT- MT-11 M -2 we have that
mi
1/ (1-m 1f(T-++)mm
TT -0mm 17
17 ()m
W/ T/ )7
(1 - (1-1)tm) m)
(1 r/Tms) 7
2
The approximation error is upper bounded by
17 ())mm))
1 1)
|d|
1) <
S 00
Z
1m-
(1 - mp 2
Proof of Corollary 3
Necessity:
Let us start by assuming that We = 11+11 By applying Eq.
(6 we have the following
172
mer
m++1
MTTT
m7
THT
II
(1 (1-m))) n3)
Σ
II (1-1H) -))
T-1
1=
j=1+1
i=
j=++1
Simplifying the equation we obtain that
r(1 - mn) (1-n) m
1i11
Σ
rT
(1 - 11) mr-i
T 1
1
i=++1
M
rT -
M++1 me
1
77) -1+
(1-1)* 17)2
(T 1)(1 m7)
(1 n7)
1
r(n 1)(1
m +r
(rT 1) - 1)(1 mh-1 +1
11(1 11) m1+1
()( 1)(1 71) m
(rT 1)(1 -m=++1-+m++)
r(1 n. mi
(T 1 1)7
r(T m 1
m m
1)77 (T
- (1
my
(T 1- -
n mp+1
(1 mi m7tit41
(7 m4
- 141 1) + (fp -
mm1111 -1
mull mp 1) + +1
T mul. (T - 1)
T
The approximation error is upper bounded by
T(T nrL (T -11 1)
|c|
< (
T. (T 1))
T Tm (T 1)
Sufficiency:
mrrTT - 1) T-1
W1+1
Σ i
II
(1 n3)
Σ
II (1 73)
(20)
j=1-1
i
+1
j=it1
14

---

Substituting 1/T into Eq. 20 yields
7
Wr+11
(1
71)
T
J=+1
11) m
(1 m-m
T
0
The approximation enror is upper bounded by
|e| T (1 17 -m---mm---00 1+))+1)
g -m,- mm-1
Proof of Corollary 4. Substituting v7(1) = 1/(1+ i) into Eq. 66
Th - ri)
W
Σ
JkTi
- 1z)) + (1
(1 - 15)
17
T
m
1
jmi+1
ma
j j=i+1
171
RVT
Ti
iz
+ MT
T - 1 + MT
1=
r + T-I
1-r
12
+ mr
We can now conclude by noting that we independent of the choice
C Implementation Details
The hyperparameter 7) is fixed to 0.1, is selected from {0.1, 0.4, 1.0], and À is selected from (0.01, 1.0, 10.0) for Split CIFAR-10 and Split CIFAR- and {0.00001, 0.0001, 0.001, 0.01} for Split Mini- ImageNet. Code available in the supplementary material and will be released upon acceptance. All experiments performed eight NVIDIA RTX A4000 GPUs. The amount of compute is easily affordable, which can be inferred from the running times given Table 6
D Extended Results
D.1 Time Complexity Analysis
We provide in Table 6 the floating point operations per step (FLOPs/step) of different normaliza- tion methods and the total running times (in seconds) under different implementations (the exact calculation of FLOPs can be found in the attached code). It can be seen that compared BN, our proposed method only slightly increases the computation, while CN almost doubles the computation because it combines both GN (without affine transformation) and BN. Note that the FLOPs only give the theoretical computation; the actual running time depends on the implementation. We measure the actual running time of two normalization implementations: 1) using the plain torch primitive and 2) using the torch. nn with cuDNN as the backend. Due to engineering difficulties, we do not currently implement
BuDNNN-opiimizer version of our method. However, can be seen from T on the plain implementation our method only slightly increases the running time, which consistent with the FL.OPs analysis. This suggests that our method has the potential achieve comparable time complexity BN through well-engineered cuDNN implementation.
15

---

Table 6: Comparison of FLOPs per step of different normalization layers and total running times (in seconds) under different implementations on Split CIFAR-100 using ER-ACE with | = 10 and
| = 2000 the baseline approach. be exploited.
BN
GN
CN
Ours
FL.PPsstep
49.20M
49.18M
86.09M
49.32M
1x
0.99x
1.75x
1.01 X
Time (Plain)
430
440
535
489
1x
1.02
1.24x
1.14x
Time (cuDNN)
324
322
376
1x
0.99
1.16x
D.2
Impacts of
To show how the statistical weights of tasks affect performance, we vary the proportion of the current task in the batch (i.c., r N)/N) different constants (with fixed N, = 10, N 10, 20, 50, ...) and conduct experiments on Split CIFAR-10 10. It can be seen from Table 7 that the performance first increases rapidly, peaks at around 1/10 (i.e., the inverse of the total number of tasks), and then stabilizes. In particular, the accuracy with unfixed is comparable with the peak, which supports our Corollary 3. Since the total number tasks is generally unavailable continual learning, using an unfixed w.r. 1/T is more practical.
Table 7: Performance comparison of different r setups on Split CIFAR- 10. T refers to the number of seen tasks that increases over time.
1/2
1/5
1/10
1/12
1/15
1/20
1/T
Task-IL ACC
23.58
55.65
61.81
62.91
61.49
60.36
61.61 24.04
58.83 23.83
Class-IL ACC
3.05
17.33
25.26
25.50
25.07
24.26
D.3 Impacts of Memory Buffer Selection Strategies
Fig. 5
shows that method obtains substantial improvements over both reservoir sampling and ring buffer, which demonstrates the robustness of A to the memory buffer selection strategy.
Split CIFAR-100,
Task-IL.
Split CIFAR-100 100, Class IL
Ours reservair CN reservair
on BN (%)
ring
ring
ER-ACE
DER
ER-ACE [A]] - 5000
DER
ER-ACE [A4] - 2000
DER
ER-ACE [M - 5000
DER
[M - 2000
JM] 2000
[M4]] - 5000
[MM] - 2000
[M 3000
Figure 5: Performance improvements (i.e., A FAA) for online continual learning over different
selection strategies of memory buffer and different memory sizes |M] (with batch size |B| 10).
D.4 Fine-tuning and Joint Training Performance
To establish a simple baseline and upper bound, we present the fine-tuning (FT) and joint training (JT) performance results of different normalization approaches Table Notably, Ours-FT consistently exhibits better performance than both BN-FT and CN-FT in the absence of replay data. On the other hand, Ours-JT does not outperform BN-JT, indicating that our performance improvements are indeed specific to continual learning.
16

---

Table Fine-tuning and joint training performance on Split CIFAR-10 with
[B]
-10.
Fine-tuning
Joint tuning
Task-IL. ACC
Class-II. ACC
Task-II. ACC
Class-II. ACC
BN
34.97+2.45
5.82+0.37
76.80=1.48
41.36*2.16
CN
34.48+0.80
4.68+0.59
76.41=0.61
41.46#0.61
Ours
36.31+1.72
5.9520.88
75.79=1.31
40.1421.74
D.5 Scalability Analysis
In order to evaluate the scalability of our proposed method, we perform an additional experiment with ImageNet-sub [10], which includes 100 classes randomly selected from ImageNet of size 224 x 224. We split them into 10 incremental phases for online continual learning (ER, B 32. [M| = 2000), and reuse the hyperparameters (A and x) of Split Mini-ImageMie the Fask-IL/Class-IL accuracy of our approach is clearly better than BN, demonstrating scalability
As shown in Table 9
larger datasets. Furthermore, we perform experiments focused on online Nomain-incrementall learning (Domain-IL) to evaluate its performance in handling more subtle distribution shifts. Specifically, employ two datasets: Permuted MNIST [3]. which applies random permutations to MNIST digit pixels create 20 subsequent tasks, and DomainNet (18), comprising 345-class images from six distinct domains.
that our approach can clearly improve batch normalization on both Permuted MNIST and DomainNet, thanks to better resolution of balance and adaptation.
Table Performance on scaling to larger datasets and handling more subtle distribution shifts.
ImageNet-sub
Permuted MNIST
DomainNet
Task-IL
Class-IL
Domain- IL
BN
72.92
16.86
64.89
7.38 65.97
CN
76.48
21.88
10.80
Ours
77.24
22.08
77.15
7.8/
D.6 Statistical Significance
We perform con-parametric paired t-test (i.e., the Wilcoxon signed-rank test) to evaluate whether A exhibits statistically significant superiority over comparable baselines in both online and offline scenarios, as well as a combination of both. The null hypothesis Hg states that the mean performance difference between A and the alternate baseline is zero for given scenario, while the alternative hypothesis H1 states a non-zero difference. It worth noting that we conduct two-tailed tests avoid making an assumption that A consistently outperforms the baseline. Our samples are collected from all the results presented in the manuscript. The resulting p-values for these tests, detailed in Table 10 demonstrate that A outperforms BN and CN statistically significantly across online, offline, and combined online and offline scenarios.
Table 10: Non-parametric paired t-test (the Wilcoxon signed-rank test) on whether A is statistically significantly better than the baselines (p < 0.05).
Online
Offline
Both
BN
CN
BN
CN
BN
CN
p-value A statistically
8.0c-13
5.2c-11
1.6c-02
1.5e-05
1.0c-14
1.0c-13
/
significantly better?
D.7
Forgetting Measure
Table I1 shows that our method is generally the lowest in terms of forgetting measure.
17

---

Table 11: Forgetting measure (4) of online task-ineremental learning with batch size B use bold, underline, and italic to indicate the first, second, and third best results respectively.
10. We
Split CIFAR-10
Split CIFAR- --0
Split Mini Mingeee
Method
|M|=000
| =2000
|M|==0000
|M|=5000
|M|=2000
|M|=5000
ER-ACE w/ BN 3.31
1.76+1.84
2.15&1.11
1.71
3.14+1.42
2.80
ER-ACE w/ LN
4.03+3.25
0.99=1.43
2.92+0.81
146=:)44
2.93+0.08
2.53+0.93
ER-ACE w/ IN 1.74
1.76#0.96
2.11#0.61
1.4100.63
3.8620.59
5.07+1.20
ER-ACE w/ GN
1.30#0.36
1.731161
0.86+0.26
3.21
4.1341998
3.07=1444
ER-ACE w/ CN
1.6740.25
0.90=0.51
2.53++0533
2.03#0.33
1.72
4.21
ER-ACE w/ Ours
1.22+0.60
0.84#0.78
00,66
1.58
2,21+2.38
1.94#0.87
DER++ w/ BN 2.35+0.73
0.22+0.03
1.2640.42
1.31
1.9740.94
2.60+0.32
DER++ w/ LN
1.1+1.05
0.65+0.22
2.02+1.28
1.881141
3.1640.70
2.95+0.61
DER+ w/
1.90=0.47
0.68#0.62
1.91
1.78+1.03
3.46+0.56
3.64#0.40
DER++ w/ GN
1.42#1.03
0.89&1.05
2.32&1.45
1.41#0.63
3.88
3.70
DER+ w/ CN
3.89+0.52
2.05#0.40
1.63+0.77
1.07=0.28
4.04+1.17
3.85
DER++ w/ Ours
1.81
0.38+0.17
0.92+0.58
0.73+0.41
3.65+1.49
1.36+0..22
D.8 Full Results for Dynamics of Normalization Statistics
We provide the norm dynamics of the batch statistics and population statistics for all normalization layer in Figs. 6 and For better illustration, each curve in the figures indicates the mean after Gaussian smoothing with kernel size of 20 and the shaded area indicates the variacar It can be seen that our method roughly tracks the joint training (JT) on many layers (e.g. 2, 5, 8, 12. 15, 17, 18, 19).
18

---

Layer
Layer
Layer
Layer
Layer
1.5
1.0-
6-
4-
0.5
4-
21
2-
2
0
5000
0
5000
5000
5000
3000
Layer
Layer
Layer
Layer
Layer
12.5
8-
10.0
6-
7.5
6
7.5
5.0
4
5
2-
2.5
2-
5000
5000
6000
5000
3000
Layer
Layer
Layer
Layer
Layer
10.0
7.5
10-
5.0
0
5000
5000
5000
5000
5000
Layer
Layer
Layer
Layer
Layer
25
200
20
15
15
10
10
10
0
5000
5000
5000
5000
3000
BN
CN
JT
Ours
Figure 6: Norm dynamics of the batch statisties of all normalization layer.
19

---

Layer
Layer
Layer
Layer
Layer
1.5
1.0
0.5
2.5
5000
0
5000
0
5000
5000
5000
Layer
Layer
Layer
Layer
Layer
12.5
10.0
5.0
3.0
2.5
0
5000
5000
5000
5000
5000
Layer
Layer
Layer
Laycr
Layor 15
10
10
15
15
10
10
10
5000
0
50000
5000
5000
5000
Layer
Layer
Layer
Layer
Layer 20
20
15
10
10-
5000
0
5000
5000
5000
5000
BN
CN
Ours
Figure 7: Norm dynamics of the population statistics of all normalization layer.
20


Reviews:
[{"review_id": "dECXx0jBib", "reviewer": "Reviewer_SBWS", "summary": "This paper studies the recency bias of the normalization statistics in continual learning and presents an theoretical analysis of the delimma of the balance and adaption in estimating the normalization statistics. \nTo address these challenges, the paper propose Adaptive Balance of BN (AdaB2N), which appropriately incorporates a Bayesian-based strategy to adapt task-wise contributions and a modified momentum to balance BN statistics.", "questions": "- Why is the \"adaptive balance of BN statistics for testing\" (Equation (16)) included in the training phase?\n- Why is the estimation in Eq. (4), which faces a dilemma between balance and adaptation, used to guide the learning of Eq. (16)? How is this to improve the estimation?\n- The performance of Eq. (14) without updating $theta$ and without using the loss function Eq.\n- In order to calculate Eq. (13), do we need to keep all $S_m^t$ (for all $m$ and $t$) and then perform the summation operation? Or do we just keep the sum of the results up to $t-1$ and then add $S_m^t$ to get the result?\n- Can the authors provide results for Corollary 3 with fixed $r$ so that we can check the impact of $r$ in the statistical estimation?", "limitations": "No societal impact.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- This paper demonstrates the adaptation and balance dilemma in normalized statistical estimation in continuous learning through theoretical analysis.\n- The method proposed in this paper is simple and effective. It only requires learning an additional vector and setting two hyperparameters to achieve satisfactory improvements.", "weaknesses": "- By theoretical analysis, the normalized statistical estimation in Eq. (4) faces a dilemma between balance and adaptation. However, Eq. (4) is used to guide the learning of conditional estimation, as shown in Eq. (14). Whether this guidance is correct or not is uncertain.\nSo it is not clear how the loss function in Eq. (14) helps to improve the normalized statistical estimation. This model parameter $\\theta$ is also updated with the loss in Eq. (14). Perhaps Eq. (14) imposes a regularization on $\\theta$ that helps to improve the performance.\n\n- The definition of $\\psi$ requires prior knowledge of the total number of tasks, which is generally not available in continuous learning.\n\n- The presentation of the paper could be improved. For example,\n  - In lines 37-38, the limitations of the existing work could be presented in more detail to help the reader better understand what the proposed improvements would look like compared to the existing work.\n  - In lines 39-46, more information about balance and adaptation and their conflicts could be included to illustrate the dilemma.\n  - The notation is complex, making it not easy to follow the theory part.\n  - In Theorem 1, $\\eta(i)$ and $r(i)$ are not explicitly defined.\n\n- The experiment can be improved.\n  - In the comparison of offline task incremental learning settings (Fig. 3), the baseline method is CN, which is specifically designed for online settings. Here two baselines [7] and [28] should be compared, which are designed for the offline setting.\n  - There is no explanation for why some of the results in Figure 4 are n\\a.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1, Q2: Why is Eq.4 used to guide Eq.14? How is this to improve...**\n\nWe would first clarify the relationship between Eqs.4, 14 and 16. As you understand it, Eq.4 (i.e., the regular EMA) faces a dilemma between balance and adaptation in continual learning (CL). To address this particular challenge, we introduce an advanced version of $\\eta$ in Eq.16 to improve the population statistics $\\hat{\\mathbb{E}}[S_m]$ of Eq.4. Then, such an improved population statistics is used to guide the learning of batch statistics $\\hat{\\mathbb{E}}[S|a_m]$ by Eq.14.\nNow we explain two specific questions as follows.\n1. Why is Eq.4 used to guide Eq.14: We suppose you might have overlooked that the population statistics in Eq.14 are in fact an improved version via Eq.16, as described in Steps 10-11 of Algorithm 1. To avoid potential misunderstanding, we will emphasize the use of Eq.16 after Eq.14.\n2. How this improves the normalized statistical estimation: First, Eq.14 provides $\\psi$ with historical information across batches by aligning the improved population statistics, which contributes to obtaining statistics as good as Joint Training. Second, the normalized representation guided by Eq.14 helps to reduce the gradient magnitude, thus smoothing and facilitating the learning of $\\theta$. The results in Fig.5 and Fig.1 support these claims, respectively. Besides, the effectiveness of Eq.14 is also demonstrated in Fig.4 and further strengthened by the experiments added in Q2 as below.\n\nWe will add more explanations to make it clearer.\n\n**W2: The definition of $\\psi$ requires...**\n\nThe task number $T$ in $\\psi$ is compatible to *the number of seen tasks*. Therefore, the definition of $\\psi$ does not require prior knowledge of the total number of tasks. We will clarify it in the final version.\n \n**W3a: In L37-38...**\n\nWe will add a table in the final version to demonstrate more clearly the limitations of existing work and the advantages of ours (see response to Reviewer rCcF's W1).\n\n**W3b: In L39-46...**\n\n*Balance* refers specifically to the balanced statistical weights of each task, i.e., each task contributes equally to the normalization statistics. In contrast, *adaptation* refers specifically to updating in a way that concentrates on adapting to the current task, corresponding to the regular EMA. They are conflicting in CL due to the lack of flexibility of existing normalization layers to address their respective shortcomings. We will add it in the final version.\n\n**W3c: The notation is complex.**\n\nWe will simply the notations and add a table to summarize them.\n\n**W3d: Define $\\eta(i)$ and $r(i)$.**\n\n$\\eta$ and $r$ are both defined as functions in L140. We will make it clearer.\n\n**W4a: Compare [7] and [28].**\n\nTBBN [7] (CVPR23) and BNT [28] (Arxiv) are two representative methods for CL of batch normalization layers. However, both of them have not released the official code, and their implementation details are not described clearly enough to ensure a fair comparison. Therefore, we only discuss them in related work rather than empirically compare with them. Note that TBBN [7] has been compared primarily with CN (ICLR21), similar to our case. We observe that TBBN outperformed CN in offline CL by a similar margin as ours. In contrast, TBBN [7] can hardly improve online CL, which is one of our major strengths as shown in Tables 1 and 2. Therefore, we believe that our approach enjoys better performance and generality. We will add it in the final version.\n\n**W4b: Explain n\\a.**\n\nWhen data distributions change in CL, the normalized representations without appropriate balance tend to greatly deviate from the estimated population statistics. If $\\lambda$ is too large (e.g., $\\lambda=10$), $\\mathcal{L}\\_{ada}$ will inappropriately outweighing $\\mathcal{L}\\_{CL}$ and thus interfere with training stability. Thus, the training loss becomes NaN and the results becomes n\\a. We will clarify it in the final version.\n\n**Q1: Why is Eq.16 included in training?**\n\nAs shown in Step 10 and Step 11 of Algorithm 1, Eq.16 is used to calculate the momentum in EMA and then obtain the (improved) population statistics by Eq.4. In other words, Eq.16 in Step 10 serves as a precursor to Step 11 for testing. To avoid potential misunderstanding, we will remove \"For testing\" from Step 11 and add more explanations.\n\n**Q3: The performance of Eq.14 without...**\n\nWe report the results with $\\lambda=0$ under the same setup as in Fig.4. Similar performance decreasing trend can be observed.\n\n|$\\kappa$|Task-IL ACC|Class-IL ACC|\n|-|-|-|\n|1.0|64.54|17.62|\n|0.7|64.35|17.59|\n|0.4|63.70|17.38|\n|0.1|61.68|16.51|\n|0.0|37.33|6.81|\n\nOptimization without updating $\\theta$ would convert Eq.15 into a bilevel (min-min) optimization problem. Our experiments show that this leads to performance degradation (e.g., -3.34% for Split CIFAR-100, |M|=5000) and almost doubles the computational cost.\n\n**Q4: In order to calculate Eq.13...**\n\nIndeed, neither of these operations is required in our method. $S_m^t$ is the statistics of samples belonging to the task $t$ within the $m$-th batch (see L136), so Eq.13 is calculated within each batch. We will make it clearer.\n\n**Q5: Provide results for Corollary 3 with fixed r...**\n\nFollowing your suggestion, we present the following results using fixed $r$ with $N_t=10$ on Split CIFAR100, where $T$ refers to *the number of seen tasks* that increases over time:\n\n|$r$|Task-IL ACC|Class-IL ACC|\n|-|-|-|\n|1|23.58|3.05|\n|1/2|55.65|17.33|\n|1/5|61.81|25.26|\n|1/10|62.91|25.50|\n|1/12|61.49|25.07|\n|1/15|60.36|24.04|\n|1/20|58.83|23.83|\n|1/T (Unfixed)|61.61|24.26|\n\nIt can be seen that the performance first increases rapidly, peaks at around $r= 1/10$ (i.e., the inverse of the total number of tasks), and then stabilizes. In particular, the accuracy with unfixed $r$ is comparable with the peak above, which supports our Corollary 3. Since the total number of tasks is generally unavailable in CL, using an unfixed $r$ w.r.t 1/T is more practical."}, {"review_id": "nTmC8UzEyk", "reviewer": "Reviewer_rCcF", "summary": "This paper studies how to update the parameters of batch normalization during continual learning. This is a problem that many researchers have mentioned, since batch statistics would be computed in a non-iid manner; however, while the problem has been acknowledged, at least in conversation, there hasn't been much work to resolve this problem. They theoretically study how to update batch norm statistics for continual learning, and they propose a new batch norm variant based on Bayesian updates. They demonstrate that on standard small-scale continual learning setups that their method increases performance significantly. ", "questions": "Why was the use of replay employed rather than studying a replay-free setting? It is very unclear if the limited memory setup makes sense vs, either a no memory setup or an unlimited memory setup (e.g., https://arxiv.org/abs/2302.01047 )\n\nWhy are only small-scale problems studied vs. studying multiple distributions on a larger dataset, e.g., ImageNet, which is also largely considered a toy dataset nowadays. Even DER when originally proposed was evaluated on ImageNet-1K.\n\nWhy were ER-ACE and DER++ studied only? \n\n", "limitations": "As far as the limitations of the work, I think this needs a lot of work. Much of the field is now working on larger scale problems, e.g., continual learning on ImageNet from various distributions for the past few years, and the use of hyperparameter sweeps for kappa and lambda significantly reduce the generality of the work since this is unrealistic for most real-world continual learning applications since this would require knowing the stream and dataset ahead of time. ", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "It is a good problem to work on and the results appear strong on the small-scale problems studied, although I have strong reservations about them given the hyperparameter sweeps.\nI appreciate the use of comparing batch sizes, rather than treating that as a hyperparameters. Likewise, I appreciate using the combined task upper bound, which many papers omit making it hard to understand the gap in performance.\n", "weaknesses": "The analysis in 3.1 is conflated with related work in terms of the content, e.g., discussing group normalization, layer normalization, etc. Likewise, the related work simply lists past work. It would strengthen the paper to better juxtapose why a new method is needed and the limitations with existing normalization methods. \n\nThe continual learning frameworks studied, i.e., Task-IL and Class-IL, is limited. It would be better to do this in a more general manner where they discuss the properties of the distribution the stream is being drawn from, as in many real-world continual learning problems the data stream would not be either of these extreme edge cases. The field is definitely moving toward thinking of the data ordering as more of robustness to different distributions rather than these specific setups, which lack any real-world applicability except as experiments regarding different stream distributions.\n\nThe use of task labels (especially at test-time), and many recent methods have largely ceased to use task labels as they have little real-world relevance. They do study Class-IL as well, but that’s more of an extreme edge case.\n\nBoth methods they employ, ER-ACE and DER++ use a small memory buffer, but addressing the problems with updating normalization layers seems best done for replay-free methods, rather than using replay-based methods. \n\nThe experiments are all on small-scale datasets and only extreme distributions are tested. I’d really like to see more natural distributions and even easy distributions evaluated (iid) with the same hyperparameters. It is unclear if kappa is being tuned to optimize results per stream, where it would be impossible to actually do that in real-world continual learning. As best I can tell (and supported by the discussion in the appendix), they are reporting the best results of a hyperparameter sweep of lambda and kappa, which is pretty unrealistic on a per-dataset and per-distribution basis. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: The related work simply lists past work...**\n\nThank you for your valuable suggestion. In the original submission, we formulate representative normalization layers especially the most popular batch normalization (BN) in line 108-128, with a conceptual analysis of their sub-optimal performance in continual learning in line 29-38. Meanwhile, we discuss several related works for continual learning of BN in line 66-78. In the following, we add a table to explicitly compare the limitations with existing normalization methods and demonstrate the advantages of ours, which will be included in the final version. We will also reorganize these discussions and add more details to make them clearer.\n\n|Method|Statistics|Training Adaptability|Online|Offline|\n|-|-|-|-|-|\n|BN (ICML15)|Unbalanced|✖|✖|✖|\n|CN (ICLR22)|Unbalanced|✔|✔|✖|\n|BNT (Arxiv22)|Balanced|✖|✖|✔|\n|TBBN (CVPR23)|Balanced|✔|✖|✔|\n|Ours|Adaptively balanced|✔|✔|✔|\n\n**W2: The continual learning frameworks studied...is limited...The field is definitely moving toward thinking of the data ordering as more of robustness to different distributions rather than these specific setups.**\n\nIn this work, we follow the continual learning frameworks widely-used in previous work, i.e., Task-IL and Class-IL. We agree that an important further work should consider the effect of different stream distributions, so as to bridge real-world continual learning problems. As an initial attempt, we perform additional experiments on Domain-IL setting, which involves explicit distribution changes in continual learning (please refer to the response to Reviewer 24XQ's W4). Our approach consistently improves BN, demonstrating its robustness to distributional changes in different scenarios.\n\n**W3: The use of task labels.**\n\nAs the response to W2, our work follows the commonly-used settings in continual learning, which provide task labels in training and/or testing, while the additional Domain-IL experiments do not require task labels at either stage. We will add a discussion for the real-world relevance of data distributions and task labels, and will explore more realistic continual learning settings in further work.\n\n\n**W4, Q1: Both methods they employ, ER-ACE and DER++ use a small memory buffer, but addressing the problems with updating normalization layers seems best done for replay-free methods, rather than using replay-based methods...Why was the use of replay employed rather than studying a replay-free setting? It is very unclear if the limited memory setup makes sense vs, either a no memory setup or an unlimited memory setup**\n\nThank you for this insightful comment. We would argue that the particular challenges of normalization layers behave **differently** in replay-based and replay-free methods. The replay-based methods rely on the replay of a few old training samples to overcome catastrophic forgetting, which are usually much fewer than the new training samples. The limited old training samples are repeatedly accessed in continual learning, which further deteriorate the imbalance and overfitting of normalization statistics.\nIn contrast, the replay-free methods usually construct task-specific parameters or regularize parameter changes to overcome catastrophic forgetting. The former hardly suffers the recency bias in normalization layers as they are generally task-specific as well. Without being affected by replay, the latter can stabilize the parameters of normalization layers in parallel with other parameters. For example, Adam-NSCL (CVPR21) used EWC to regularize batch normalization layers. \nSince replay-based methods have achieved clearly state-of-the-art performance in many continual learning scenarios, we take this direction as the primary focus, similar to many previous works for continual learning of batch normalization layers such as CN (ICLR21) and TBBN (CVPR23). \nOn the other hand, we additionally report the **fine-tuning** performance of normalization approaches in the response to Reviewer 24XQ's W3 and Q3. It can be seen that our approach still outperforms both BN and CN, indicating its generality for replay-free methods. We will add the above discussion and the reference you provided in the final version.\n\n\n**W5, Q2: The experiments are all on small-scale datasets and only extreme distributions...It is unclear if kappa is being tuned to optimize results per stream**\n\nThe benchmark datasets considered in this paper follow a representative work for continual learning of batch normalization (CN, ICLR21). To validate **scalability** of our approach and **robustness** of hyperparameters, we perform an additional experiment with ImageNet-sub, which includes 100 classes randomly selected from ImageNet of size $224 \\times 224$. We split them into 10 incremental phases for online continual learning (ER, $|B| = 32$, $|M| = 2000$), and *reuse* the hyperparameters ($\\lambda$ and $\\kappa$) of Split Mini-ImageNet. The Task-IL/Class-IL ACC of our approach (77.24% / 22.08%) is clearly better than BN (72.92% / 16.86%), demonstrating both scalability and robustness. We will add this in the final version.\n\n**Q3: Why were ER-ACE and DER++ studied only?**\n\nIn this work, we implement our approach with two representative replay-based baselines, i.e., experience replay (ER) and dark experience replay (DER), consistent with previous work for continual learning of batch normalization layers. ER-ACE and DER++ can be seen as the advanced versions of ER and DER, respectively. These methods as well as CN (ICLR21) compared in this paper are all officially implemented in the widely-used *mammoth* repository. Therefore, we implement our approach using the same repository to ensure comparison fairness. Besides, we find many more advanced replay-based methods (e.g., LiDER (already studied in Fig. 3), X-DER, CLS-ER, etc.) are also developed on the top of ER and DER, so we leave the implementation of our approach as a natural extension to them."}, {"review_id": "nyu2rQ43zr", "reviewer": "Reviewer_Qyzj", "summary": "The manuscript introduces a theoretical analysis on the forgetting in the statistics of the popular Batch Normalization layer. Based on this, it introduces a novel normalization layer, termed $AdaB^2N$, and evaluates its effectiveness on the online class-incremental and task-incremental settings.", "questions": "As I mentioned above, my only recommendation would be to include results for the Class-IL multi-epoch setting.", "limitations": "yes", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The manuscript is clearly written and provides a clear picture on the shortcomings of Batch Normalization in a continual learning setting. Furthermore, the results clearly depict a consistent improvement over a wide range of benchmarks.", "weaknesses": "I enjoyed reading the manuscript and I believe it carries no major weakness. \nMy only concern arises from the multi-epoch (offline) setting, where the authors evaluate the effectiveness of their proposal on the baselines in the Task-IL setting. Indeed, as the latter is often depicted as less challenging [1,2], I would recommend including the also results of the Class-IL in this evaluation.\n\nAdditional comments that did not influence my judgment:\n- The citation for LiDER should be aNeurIPS 2022.\n- The citation for “Three scenarios for continual learning” should be a NeurIPS Continual Learning workshop 2018.\n\n[1] Sebastian Farquhar and Yarin Gal. Towards Robust Evaluations of Continual Learning. ICMLw, 2018.\n\n[2] Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient-based sample selection for online continual learning. aNeurIPS, 2019\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback and insightful comments. Below, we provide a point-to-point response to each of the weaknesses (W) and questions (Q) and summarize the corresponding revisions in the final version.\n\n**W1: Including the also results of the Class-IL.**\n\nThank you for your valuable suggestion. We will include the results of Class-IL in the final version. In general, our AdaB2N can outperform the comparison baselines by a similar margin in both offline Task-IL and offline Class-IL. Here we present an example as below:\n\n> Setting: offline, Split Mini-ImageNet, DER++, $|B| = 32$, $|M| = 2000$\n\n| |Task-IL ACC|Class-IL ACC|\n|-|-|-|\n|BN|35.91±1.59|12.69±1.23|\n|CN|35.46±1.42|12.13±0.92|\n|AdaB2N|**36.98±1.52**|**13.07±1.30**|\n\n**W2: Bibliography format.**\n\nThank you for your valuable suggestion. We will correct them in the final version."}, {"review_id": "NfxGdYHXBu", "reviewer": "Reviewer_24XQ", "summary": "The paper presents Adaptive Balance of Batch Normalization (AdaB2N) as a method to address the sub-optimality of Batch Normalization (BN) in continual learning. The main challenge lies in updating BN layer statistics based on the currently observed training samples, leading to recency bias and limited generalization across tasks. The paper leverages a Bayesian-based strategy to determine task-wise contributions in normalized representations, and it introduces an adaptive parameter that reconciles the constant and normalized momentum of EMA, achieving a balance of BN statistics while maintaining adaptability during training.\n\n**TLDR of the review:** The paper provides a comprehensive theoretical analysis, a well-motivated problem statement, and promising experimental results. However, the paper has weaknesses such as insufficient empirical significance analysis, limited method scalability analysis, missing important baselines, and limited investigation of subtle distribution shifts. I am happy to reconsider my rating based on how the authors address these comments. ", "questions": "**Here are some suggestions to help address the weaknesses:**\n\n1.\tCan you address the concerns raised regarding statistical significance? Could you provide any explanation of the conditions under which the proposed method performs well?\n2.\tIs there a specific reason why the experiments were limited to small-scale datasets like CIFAR and Mini-ImageNet? If there are scalability limitations, please provide insights into the reasons behind them.\n3.\tCould you include the suggested baselines and compare the performance with the proposed method and other normalization techniques?\n4.\tHave you considered investigating the performance of Batch Normalization (BN) in scenarios with more subtle distribution shifts within individual tasks or in a task-free continual learning setup? \n5.\tCan you provide information about the normalization method used by JT? Please elaborate on whether the proposed normalization method is applicable in the non-continual learning scenario and discuss its potential performance compared to BN in that setting.\n", "limitations": "Limitations and potential negative societal impact are discussed (no clear negative societal impact at this stage). ", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1.\t**Comprehensive theoretical analysis (Note: I can't determine how significant these results are):** The paper offers a thorough theoretical analysis of the behavior of batch normalization in continual learning. It sheds light on the tension between balance (working effectively across tasks) and adaptation (learning new tasks), providing a solid foundation for the proposed method.\n2.\t**Well-motivated problem statement:** The problem statement presented in the paper is well-motivated, highlighting the limitations of batch normalization in continual learning.\n3.\t**Promising experimental results:** The paper provides experimental results on small-scale image datasets (CIFAR and Mini-ImageNet), which demonstrate promising outcomes.\n4.\t**Interesting analysis experiment:** The authors conducted a compelling experiment demonstrating that the model trained with the proposed method in an online continual learning (CL) setting achieves normalization statistics that closely match the statistics of a model trained on all tasks jointly, which serves as a baseline for non-continual learning.\n", "weaknesses": "*Main weaknesses that influence my rating:*\n\n1.\t**Insufficient empirical significance analysis:** The paper lacks a thorough analysis of statistical significance for the reported results. In some cases, the difference in mean results between the proposed method and competitors is not significant when considering standard deviations. For example, the online CIL results for SplitCIFAR-100/|M|=500 exhibit this issue. It is important for the authors to clarify the conditions regarding the order of stream samples under which the proposed method performs best. Additionally, to promote transparency, it would be beneficial to report the means and standard deviations for offline CL experiments, enabling an assessment of the consistency of the improvement across different seeds.\n2.\t**Limited method scalability analysis:** The experiments are primarily conducted on small-scale datasets like CIFAR and Mini-ImageNet. However, it remains unclear how well the proposed method scales to larger datasets, such as the ImageNet subset. The authors should address whether there is a specific reason for the limited scalability analysis.\n3.\t**Missing two important baselines:** Two crucial baselines are absent from the tables. Firstly, it seems that the experiments incorporate some form of replay, which might affect batch normalization (BN) statistics and mitigate recency bias. It would be valuable to include a simple fine-tuning baseline and compare the effect of different normalization methods applied to that baseline. Secondly, reporting the upper bound of no-continual learning (NoCL) baseline performance, possibly using both BN and the proposed method.\n4.\t**Limited investigation of subtle distribution shifts:** The experiments primarily focus on drastic changes occurring when switching between tasks in continual learning, primarily caused by class boundaries. However, it is essential to also explore the performance of batch normalization (BN) in handling more subtle distribution shifts within individual tasks or in a task-free continual learning setup. \n\n*Additional weaknesses that do not necessarily influence my rating:*\n\n5.\t**Lack of clarification on normalization methods:** The paper lacks clarification on the normalization used by the joint training (JT) baseline, and whether the proposed normalization method is applicable in the noCL scenario. It would be valuable to compare \"ours-JT\" (the proposed method) against \"BN-JT\" to gain insights into their relative performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback and insightful comments. Below, we provide a point-to-point response to each of the weaknesses (W) and questions (Q) and summarize the corresponding revisions in the final version.\n\n**W1, Q1: lacks a thorough analysis of statistical significance...clarify the conditions regarding the order of stream samples under which the proposed method performs best...report the means and standard deviations for offline CL experiments.**\n\nAlthough the standard deviation appears to be large relative to the difference in mean, this is basically due to the performance variance of online continual learning itself, which heavily depends on the construction of task sequence such as the class split and task order. Following your suggestion, we perform a paired t-test along each evaluation metric. We find that the p-value between AdaB2N and each baseline is below 0.01, suggesting a significant improvement in performance.\n\nRegarding your concern about the conditions under which the proposed method performs well, we provide the results for different orders in the settings you mentioned (Class-IL, Split CIFAR-100, |M|=5000, ER-ACE/DER++). It can be seen that our approach outperforms other baselines in most cases.\n\n> Setting: online, Split CIFAR-100, ER-ACE / DER++, $|B| = 10$, $|M| = 5000$, \n\n|Order|1|2|3|Class-IL ACC|\n|-|-|-|-|-|\n|BN|28.65 / 18.53|26.43 / 18.19|23.13 / 15.06|26.07±2.78 / 17.26±1.91|\n|CN|27.63 / 19.97|**26.90** / 19.01|27.16 / 17.87|27.23±0.37 / 18.95±1.05|\n|AdaB2N|**31.58** / **24.24**| 25.15 / **25.69** |**27.72** / **22.44**|**28.15±3.24** / **24.12±1.63**|\n\nFollowing your suggestion, we further report the performance of each order, together with their mean and standard deviation as follows. We can again observe consistent improvements across orders and random seeds.\n\n> Setting: offline, Split Mini-ImageNet, DER++, $|B| = 32$, $|M| = 2000$\n\n|Order|1|2|3|Task-IL ACC|\n|-|-|-|-|-|\n|BN|34.78|35.21|37.73|35.91±1.59|\n|CN|35.12|34.24|37.02|35.46±1.42|\n|AdaB2N|**36.82**|**35.55**|**38.57**|**36.98±1.52**|\n\n**W2, Q2: Limited method scalability analysis...how well the proposed method scales to larger datasets, such as the ImageNet subset.**\n\nThank you for your valuable suggestion. We evaluate our approach with the benchmark datasets (i.e., CIFAR and Mini-ImageNet) used in CN (ICLR21), which is the primary baseline considered in this work. Following your suggestion, we perform an additional experiment with ImageNet-sub, which includes 100 randomly selected classes in ImageNet of size $224 \\times 224$. We split them into 10 incremental phases with 10 classes per phase for online continual learning (ER, $|B| = 32$, $|M| = 2000$). Our approach (77.24% / 22.08%) clearly improves BN (72.92% / 16.86%) in Task-IL/Class-IL ACC, demonstrating its scalability to larger datasets.\n\n**W3, Q3: include a simple fine-tuning baseline and compare the effect of different normalization methods applied to that baseline...reporting the upper bound of no-continual learning (NoCL) baseline performance, possibly using both BN and the proposed method.**\n**W5, Q5: lacks clarification on the normalization used by the joint training (JT) baseline...whether the proposed normalization method is applicable in the noCL scenario...compare \"ours-JT\" (the proposed method) against \"BN-JT\".**\n\nThank you for your valuable suggestion. We include the fine-tuning (FT) and joint training (JT) performance of different normalization approaches as below. It can be seen that ours-FT still outperforms both BN-FT and CN-FT in the absence of replay data. On the other hand, ours-JT does not outperform BN-JT, indicating that our performance improvements are indeed **specific to continual learning**. We will add it in the final version.\n\n> Setting: online, Split CIFAR-100, **FT**, $|B| = 10$\n\n| |Task-IL ACC|Class-IL ACC|\n|-|-|-|\n|BN|34.97±2.42|5.82±0.37|\n|CN|34.48±0.88|4.68±0.59|\n|AdaB2N|36.31±1.72|5.95±0.88|\n\n> Setting: Split CIFAR-100, **JT**, $|B| = 10$\n\n| |Task-IL ACC|Class-IL ACC|\n|-|-|-|\n|BN|76.80±1.48|41.36±2.16|\n|CN|76.41±0.61|41.46±0.61|\n|AdaB2N|75.79±1.31| 40.14±1.74|\n\n**W4, Q4: the performance of batch normalization (BN) in handling more subtle distribution shifts.**\n\nThank you for your valuable suggestion. To evaluate the performance in handling more subtle distribution shifts, we perform experiments for online domain-incremental learning (Domain-IL). Our approach (77.15%/7.81%) can clearly improve batch normalization (64.89%/7.38%) on Permuted MNIST/DomainNet, thanks to a better resolution of balance and adaptation. We are now running experiments on larger-scale datasets and offline scenarios, which typically require much more time for training. We will include all results in the final version."}]

Author rebuttal:
Dear reviewers,

We express our deepest gratitude and appreciation to your constructive comments, which allow for a great improvement of our study. In response to these comments, we have added the suggested experiments, explanations and discussions, summarized as follows:

1) add a scalability analysis: scaling to ImageNet-sub;
2) verify the relevance to continual learning: adding more baselines such as replay-free (Fine Tuning) and NoCL (Joint Training);
3) investigate more challenging applications: adding online Domain-IL and offline Class-IL;
4) verify the necessity of our method: adding more ablation studies without normalization regularization in Eq.(14);
5) add empirical significance analysis: t-test.

We hope you might view this as sufficient reason to raise your score. If your have any further questions, please let us know. 

Sincerely,

Authors

Meta-review: The paper analyses the effects of adapting batch normalization statistics in continual learning. The theoretical analysis suggested a similar plasticity stability dilemma in the estimation of these statistics. The rebuttal has addressed all reviewers concerns and provided insights on the proposed approach and its stability.  I encourage the authors to incorporate the additional results and discussions.
Decision: Accept (poster)