Title: Overcoming Recency Bias of Normalization Statistics in Continual Learning: Balance and Adaptation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Continual Learning

Batch Normalization

Recency Bias

Catastrophic Forgetting
Abstract: Continual learning entails learning a sequence of tasks and balancing their knowledge appropriately. With limited access to old training samples, much of the current work in deep neural networks has focused on overcoming catastrophic forgetting of old tasks in gradient-based optimization. However, the normalization layers provide an exception, as they are updated interdependently by the gradient and statistics of currently observed training samples, which require specialized strategies to mitigate recency bias. In this work, we focus on the most popular Batch Normalization (BN) and provide an in-depth theoretical analysis of its sub-optimality in continual learning. Our analysis demonstrates the dilemma between balance and adaptation of BN statistics for incremental tasks, which potentially affects training stability and generalization. Targeting on these particular challenges, we propose Adaptive Balance of BN (AdaB$^2$N), which incorporates appropriately a Bayesian-based strategy to adapt task-wise contributions and a modified momentum to balance BN statistics, corresponding to the training and testing stages. By implementing BN in a continual learning fashion, our approach achieves significant performance gains across a wide range of benchmarks, particularly for the challenging yet realistic online scenarios (e.g., up to 7.68\%, 6.86\% and 4.26\% on Split CIFAR-10, Split CIFAR-100 and Split Mini-ImageNet, respectively). Our code is available at https://github.com/lvyilin/AdaB2N.

Overcoming Recency Bias of Normalization Statistics in Continual Learning: Balance and Adaptation

Abstract

Continual learning entails learning a sequence of tasks and balancing their knowledge appropriately. With limited access to old training samples, much of the current work in deep neural networks has focused on overcoming catastrophic forgetting of old tasks in gradient-based optimization. However, the normalization layers provide an exception, as they are updated interdependently by the gradient and statistics of currently observed training samples, which require specialized strategies to mitigate recency bias. In this work, we focus on the most popular Batch Normalization (BN) and provide an in-depth theoretical analysis of its sub-optimality in continual learning. Our analysis demonstrates the dilemma between balance and adaptation of BN statistics for incremental tasks, which potentially affects training stability and generalization. Targeting on these particular challenges, we propose Adaptive Balance of BN (AdaB 2 N), which incorporates appropriately a Bayesian-based strategy to adapt task-wise contributions and a modified momentum to balance BN statistics, corresponding to the training and testing stages. By implementing BN in a continual learning fashion, our approach achieves significant performance gains across a wide range of benchmarks, particularly for the challenging yet realistic online scenarios (e.g., up to 7.68%, 6.86% and 4.26% on Split CIFAR-10, Split CIFAR-100 and Split Mini-ImageNet, respectively). Our code is available at https://github.com/lvyilin/AdaB2N .

1 Introduction

Continual learning aims to acquire, accumulate and exploit knowledge from sequentially arrived tasks. As the access to old training samples is typically limited, the acquired knowledge tends to be increasingly relevant to the more recent tasks. Such recency bias, also known as catastrophic forgetting [17], is generally attributed to the nature of gradient-based optimization in deep neural networks (DNNs) [9, 30], where the learning of network parameters depends on a tug-of-war over currently observed training samples. Numerous efforts have been devoted to addressing catastrophic forgetting of these learnable parameters through rectifying their gradients. However, as an important component of DNNs, the normalization layers employ particular updating rules that are only partially gradient-based, and their recency bias in continual learning remains to be addressed.

The normalization layers typically include affine transformation parameters and statistics parameters to normalize the internal representations, in order to accelerate convergence and stabilize training [11, 33, 2, 24]. Since these two kinds of parameters are updated respectively from the gradient and statistics (i.e., the empirical mean and variance) of currently observed training samples, the normalization layers require specialized strategies to overcome recency bias in continual learning. In particular, the most popular Batch Normalization (BN) [11] performs an exponential moving average (EMA) of statistics parameters for testing, which preserves historical information to some extent, and thus outperforms many other alternatives in continual learning [19, 8]. As the BN statistics of old tasks still decay exponentially, some recent work [19, 8, 34] enforced a balance in the contribution of individual tasks to the BN statistics, but with limited effectiveness and generality (see Table 1 for a conceptual comparison of their targets).

So, how to update normalization statistics for better continual learning: balance or adaptation ? In this work, we present an in-depth theoretical analysis of the most popular BN, demonstrating its potential challenges posed by continual learning. First, a regular EMA with constant momentum cannot guarantee the balance of BN statistics for testing, which yields more recency bias as more training batches of each task are observed. Although a modified EMA with normalized momentum of batch number can enforce balanced statistics, they tend to be outdated for learning each task and thus limit model adaptability. Besides, the sharp changes in task distributions can interfere with the benefit of BN in stabilization of training, which further affects continual learning.

Based on these theoretical insights, we propose Adaptive Balance of BN (AdaB 2 N) to improve continual learning. Specifically, we leverage a Bayesian-based strategy to decide an appropriate way of accommodating task-wise contributions in normalized representations to stabilize training. We further incorporate an adaptive parameter to reconcile the constant and normalized momentum of EMA, in order to strike a balance of BN statistics for testing while training each task with adequate adaptability. Extensive continual learning experiments demonstrate the superiority of our approach, which provides substantial improvements especially for the challenging yet realistic online scenarios.

2 Related Work

Continual Learning: The major focus of this area is to address recency bias aka catastrophic forgetting in DNNs [17, 30, 28, 29]. Representative strategies are generally designed for the learnable parameters with gradient-based optimization [30], such as weight regularization [13, 1, 27], experience replay [32, 6], gradient projection [15, 20], etc. In particular, experience replay of a few old training samples has proven to be usually the most effective strategy and is applicable to large-scale applications [30, 32, 31]. However, its reliance on the old training samples may further deteriorate the imbalance and overfitting of normalization statistics. Corresponding to the real-world dynamics, continual learning gives rise to many meaningful settings and particular challenges [30]. For example, a desirable continual learning model should perform well in both task-incremental and class-incremental settings, which depend on whether the task identity is provided for testing. In contrast to offline scenarios of learning each task with multiple epochs, the model should also accommodate online scenarios of learning all tasks in a one-pass data stream, which is realistic but extremely more challenging.

Normalization Layer: The normalization layers have become a critical component for many representative DNN architectures, where Batch Normalization (BN) [11] is the most popular choice along with many other competitive alternatives designed for single-task learning [33, 24, 2]. However, the normalization layers require specialized strategies to perform appropriate parameter updates in continual learning. As the EMA of statistics parameters retains a certain amount of past information, BNis naturally the dominant choice for continual learning and indeed a strong baseline for overcoming

recency bias in normalization layers [19, 8]. To further improve BN, Continual Normalization (CN) [19] removed differences in task-wise distributions and then performed BN to acquire task-balanced statistics, providing benefits for online continual learning. BNT [34] and TBBN [8] constructed task-balanced batches to update BN, but only effective for offline continual learning. However, none of the current efforts can improve both online and offline scenarios [8], with the effectiveness of CN varying further with batch size in different online benchmarks. This motivates our work to provide a more in-depth investigation of BN with respect to the particular challenges of continual learning.

3 Analysis of Normalization Statistics in Continual Learning

In this section, we introduce the formulation of continual learning and representative normalization strategies, and then provide an in-depth theoretical analysis to diagnose the most popular BN. Additionally, a notation table is included in Appendix A, and proofs of theorems can be found in Appendix B.

3.1 Notation and Preliminaries

Let's consider a general setting for continual learning, where a neural network with parameters θ needs to learn a sequence of tasks from their training sets D 1 , · · · , D T in order to perform well on their test sets. The training set of task t is defined as D t = { ( x t,n , y t,n ) } |D t | n =1 , where ( x t,n , y t,n ) is a data-label pair of task t ∈ [1 , T ] and | · | denotes its amount. Since the old training sets D 1: t -1 = ⋃ t -1 i =1 D i are unavailable when learning task t , θ tends to catastrophically forget the previously learned knowledge. An effective strategy is to save a few old training samples in a small memory buffer, denoted as M = {M k } t -1 k =1 , in order to recover the old data distributions when learning the current task t . We denote the m -th training batch as B m , which includes N t training samples from D t and N k training samples from M k for k = 1 , · · · , t -1 . Here N = ∑ t k =1 N k = | B | denotes the batch size. The batch index m ∈ ( m t -1 , m t ] , where m t = ⌊|D 1: t | /N ⌋ E is the index of the last batch of task t , m 0 = 0 , and E is the number of training epochs. We denote r m = N t /N as the proportion of the current task t in B m , where r m < 1 indicates the use of replay while r m ≡ 1 indicates that no replay is used. Without loss of generality, we assume that the old training samples in each batch are task-balanced, i.e., N 1 = N 2 = · · · = N t -1 . The test set follows the same distribution as the training set, with the task identity provided in task-incremental learning (Task-IL) while not provided in class-incremental learning (Class-IL) [25].

Inside the network, the internal representation of the ℓ -th layer of the m -th batch is denoted as a ( ℓ ) m ∈ R N × C ( ℓ ) × D ( ℓ ) , where C ( ℓ ) and D ( ℓ ) represent the number of channels and feature dimensions, respectively. For the sake of notation clarity, we omit the layer and batch index hereafter when no confusion arises. Then, the operation of a normalization layer can be generally defined as:

where the statistics parameters (i.e., the empirical mean and variance) are calculated from the corresponding functions µ ( · ) and σ 2 ( · , · ) , respectively, ε is a small positive constant to avoid the denominator being zero, γ and β are affine transformation parameters that are learned by gradientbased optimization. Representative normalization strategies differ mainly in the ways of computing statistics parameters of internal representations, i.e., the exact forms of µ and σ 2 , as detailed below.

Batch Normalization (BN) [11] is the most popular choice for a wide range of DNN architectures. During the training phase, BN calculates the statistics parameters of internal representations for each training batch through µ : R N × C × D → R C and σ 2 : R N × C × D × R C → R C , where

During the testing phase, to make the normalization independent of other samples in the same batch, BN replaces the batch statistics of Eq. (1) with the population statistics estimated by exponential moving average (EMA). For ease of notation, here we define a convenient function S : R N × C × D → R 2 × C to obtain the two kinds of statistics, where

Then we denote the statistics of internal representation a ( ℓ ) m as S m := S ( a m ) , with the layer index omitted. Now the population statistics of BN can be written as

where η ∈ (0 , 1) is the momentum of EMA and usually set to 0.1 in practice.

There are many alternative improvements of BN designed for single-task learning without the use of EMA. Group Normalization (GN) [33] can be seen as a general version, which divides the channels into G groups and then normalizes the internal representations of each group. Specifically, let ξ : R N × C × D → R N × G × K × D be a reshaping function that groups the channels, where G is the group number and K = ⌊ C G ⌋ . Then the mean and variance of GN are obtained through µ : R N × C × D → R N × G and σ 2 : R N × C × D × R N × G → R N × G , where

In particular, GN is equivalent to Layer Normalization (LN) [2] by putting all channels into one group ( G = 1 ), and Instance Normalization (IN) [24] by separating each channel as a group ( G = C ). In contrast to these alternatives, Continual Normalization (CN) [19] is designed particularly for (online) continual learning , which performs GN with γ = 1 and β = 0 to remove task-wise differences and then performs BN to acquire task-balanced statistics.

3.2 Conflict of Balance and Adaptation

Although BN has become a fundamental component for many DNN architectures, its particular recency bias in continual learning remains under addressed. To quantitatively analyze the task-wise contributions in BN statistics, we define a statistical weight (of a certain batch) of each task. Then we diagnose the recency bias of BN statistics with a specified dilemma between balance and adaptation.

Definition 1 (Statistical weight of a task) . The statistical weight of task t w.r.t. the population statistics ˆ E [ S m ] is the weight w t ∈ R + such that w t = ∑ m i w t i /Z . w t i ∈ R + is the statistical weight of task t of the i -th batch such that ∑ T t =1 ∑ m i =1 w t i S t i = ˆ E [ S m ] , where S t i := S ( a i, 1 [( · ,y n ) ∈D t ] ) . 3 Z is the normalizing constant Z = ∑ T t ∑ m i w t i .

We now give rigorously the statistical weight for EMA in order to demonstrate the recency bias:

Theorem 1 (Statistical weight for EMA) . Let T, Z be defined as above, and { m t } T t =1 be an increasing sequence (i.e., m t < m t +1 ), where m t ∈ Z + , η : [1 , m T ] → (0 , 1) , and r : [1 , m T ] → (0 , 1] . The statistical weight of task t ∈ [1 , T ] w.r.t. the population statistics ˆ E [ S m T ] estimated by Eq. (4) is

where we define η i := η ( i ) and r i := r ( i ) for brevity.

Eq. (6) specifies the impact of each task on the EMA population statistics, which exposes a dilemma between balance and adaptation , as we demonstrate below.

Corollary 2 (Adaptation of BN statistics) . Let { m t } T t =1 be defined as above. If r ( · ) ≡ 1 and η ( · ) ≡ η = 1 -¯ η , the statistical weight of task t ∈ [1 , T ] w.r.t. the population statistics ˆ E [ S m T ] is w t = ¯ η mT -t -¯ η mT -t +1 1 -¯ η mT /Z . When m T -m T -1 = m T -1 -m T -2 = · · · = m 1 , we have w t ≈ ¯ η m 1 ( T -t ) /Z , with approximation error | ϵ | ≤ O ( ¯ η m 1 1 -¯ η m 1 T ) .

Corollary 2 shows that without the use of replay, a model dealing with evolving task distributions has a strong preference for the distribution of the current task, as the statistical weights of past tasks decrease exponentially. Replay of old training samples can balance statistical weights of all seen tasks to some extent. However, it cannot eliminate the recency bias except in an unrealistic case:

Corollary 3 (Balance of BN statistics) . Let { m t } T t =1 be defined as above. If r ( · ) ≡ r , η ( · ) ≡ η = 1 -¯ η , and m T -m T -1 = m T -1 -m T -2 = · · · = m 1 , then the statistical weight of each task t ∈ [1 , T ] w.r.t. the population statistics ˆ E [ S m T ] achieves (approximately) equal if and only if r ≈ 1 /T , with approximation error | ϵ | ≤ O (¯ η m 1 ) .

Corollary 3 shows that it is possible to balance the statistical weights by constructing a task-balanced batch with the memory buffer. However, this will severely slow down the convergence. Besides, as the memory buffer is usually limited in size, the model can easily overfit to the preserved old training samples [4], which is detrimental to generalization. Corollary 3 also demonstrates the importance of keeping flexibility of η . For instance, if we substitute η ( i ) = 1 / (1 + i ) into Eq. (4), this leads to a normalized momentum of batch number. In this case the population statistics can be considered as the cumulative moving average (CMA) [19] of the batch statistics, which can balance the statistical weights as below:

Corollary 4 (CMA) . Let { m t } T t =1 be defined as above. If r ( · ) ≡ r , then the statistical weight of each task t ∈ [1 , T ] w.r.t. the population statistics ˆ E [ S m T ] achieves equal if η ( i ) = 1 / (1 + i ) .

However, the balanced statistical weights do not necessarily lead to better performance, as the statistics of past tasks may be outdated relative to the current model. This is empirically validated in [19], where CMA does not outperform EMA in continual learning. Therefore, an improved strategy of updating population statistics for testing is urgently needed to reconcile balance and adaptation.

In addition to the update of population statistics during the testing phase, the current strategy of calculating batch statistics during the training phase is also sub-optimal for continual learning, as detailed in the next subsection.

3.3 Disturbed Training Stability

BNhas been shown to stabilize training and improve generalization under traditional i.i.d. data setting [3, 21, 16]. However, it is in fact not clear whether BN can achieve the same benefit in continual learning, especially when the model has just switched to a new task with drastic changes in loss and parameters. We perform a preliminary analysis by the following theorem:

Theorem 5 (Theorem 4.1 of [21], restated) . Let µ, σ 2 , γ, ℓ, a, a ′ be defined as above. Let L be the loss function of a BN model (i.e., using the normalized representations via Eq. (1) ) and L ′ be the identical loss function of an identical non-BN model (i.e., a ′ ≡ a ). The gradient w.r.t. the internal representation a of the ℓ -th layer follows that

Theorem 5 demonstrates that BN can contribute to reducing the Lipschitz constant of the loss and thus stabilizing the training of a single task. However, as the model underfits to the distribution of each continual learning task, the upper bound of the gradient magnitude will be loosened by the increase of {∇ ⊤ a L ′ · a ′ } within the third term, i.e., the similarity of the normalized representation and its corresponding gradient, which potentially affects the benefit of BN. Here we present empirical results to validate the above claim. As shown in Fig. 1, a dramatic change in loss L occurs whenever the task changes. We observe an increase of cosine similarity between the gradient and the normalized representation at this time, accompanied by a dramatic change in the gradient magnitude. This clearly demonstrates the importance of properly adapting BN statistics to each continual learning task, so as to better leverage the benefit of BN in stabilization of training.

4 Method

Following the theoretical analysis, we propose Adaptive Balance of BN (AdaB 2 N) from two aspects: (1) training aspect : we apply a Bayesian-based strategy to adjust inter-task weights and construct the normalized representation, allowing for learning an appropriate way of weighting task distributions to stabilize training; and (2) testing aspect : instead of using a fixed momentum η in EMA, we turn to a generalized version that balances the statistical weights of past tasks while reducing the impacts of past batches. We now present both ideas as follows with a pseudo-code in Algorithm 1.

0

2000

4000

Steps

3

4

5

D

1

→D

2

L

BN

CN

Ours

(a) Loss

0

2000

4000

Steps

-

0

.

0025

0

.

0000

0

.

0025

0

.

0050

cos(

∇L ′

, a

′

)

(b) Gradient Similarity

0

2000

4000

Steps

0

.

06

0

.

08

0

.

10

||∇L||

2

(c) Gradient Magnitude

Algorithm 1 Training Algorithm of Adaptive Balance of BN (AdaB 2 N)

Input : training dataset D 1: T , memory buffer M , batch size N , neural network f ϑ,ω = f ϑ ( L +1) ◦ f ω ( L ) ◦ f ϑ ( L ) · · · ◦ f ω ( ℓ ) ◦ f ϑ ( ℓ ) · · · ◦ f ω (1) ◦ f ϑ (1) , where f ϑ ( ℓ ) and f ω ( ℓ ) are the ℓ -th network layer and normalization layer, respectively, ω = { γ ( ℓ ) , β ( ℓ ) , ψ ( ℓ ) } L ℓ =1 , ϑ = { ϑ ( ℓ ) } L +1 ℓ =1 = θ \ ω , initial parameters ϑ 0 , ω 0 , the objective function L in Eq. (15), and hyperparameters ˜ η, κ, λ . Output : parameters ϑ, ω

4.1 Adaptive Balance of BN Statistics for Training

Theorem 5 demonstrates that the benefit of BN for optimization and generalization is potentially interfered in continual learning. To overcome this challenge, here we propose a Bayesian-based strategy for learning an appropriate way of weighting task contributions in normalization statistics. We assume the statistics of batch m for a certain layer follow a probability distribution S ∼ P θ ( S | a m ) , where the randomness originates from the neural network parameters θ , and then use the conditional expectation w.r.t. the distribution of statistics to calculate the normalized representation:

Specifically, E [ µ | a m ] and E [ σ 2 | a m ] are the two elements of E [ S | a m ] as described in Eq. (3):

where P θ ( S | t, a m ) is the distribution of statistics given the task identity, and P θ ( t | a m ) is the likelihood of the task identity occurring in a m . We treat P θ ( t | a m ) as a categorical distribution over the task identity, i.e., P θ ( t | a m ) = N t /N . Then, BN obtains its normalization statistics by the point estimate of the conditional expectation:

As analyzed in Section 3.3, it is critical to consider adaptability of the model parameters θ together with normalization statistics in model training, so as to enhance the gradient-representation correlation. Inspired by the theoretical works in Bayesian Inference [22, 12], we propose to capture the variability of θ through modeling the task distribution with a Bayesian principle, thus enhancing the benefit of BN for continual learning. Specifically, we assume that there is a random variable τ ∈ ∆ T -1 modeling the probability distribution of tasks, where ∆ T -1 denotes the T -1 simplex, i.e., ∑ T t τ t = 1 and τ t > 0 , ∀ t . Applying the Bayes' rule, we have

where P θ ( S | τ, t, a m ) and P θ ( t | τ, a m ) are defined analogous to P θ ( S | t, a m ) and P θ ( t | a m ) in Eq. (9). P θ ( τ | a m ) serves as the prior captured from the model parameters θ , which is generally intractable. For ease of computation, we approximate it with a Dirichlet distribution, i.e., the conjugate prior of the categorical distribution, which is parameterized by the concentration parameter ϕ ∈ R T + :

where Γ( · ) is the Gamma function, and ¯ ϕ = ∑ t ϕ t . By taking advantage of algebraic convenience of the conjugate distribution, we can estimate the conditional expectation by a closed-form expression

Intuitively, the concentration parameter ϕ plays the role of adding pseudo-observations to the task weighting procedure, which allows the normalization statistics to adaptively balance task-wise contributions by considering the current state of θ . To optimize ϕ , we introduce a learnable parameter ψ ∈ R T and obtain ϕ = exp( ψ ) . It is worth noting that the dimension of ϕ is incremental with the number of seen tasks, so there is no need to know the total number of tasks in advance. As ϕ is expected to capture the information of θ , we propose a regularization term to minimize the Euclidean distance between the adaptively balanced statistics and the estimated population statistics via Eq. (4) as follows:

where ˆ E [ S m ] is an improved version introduced in Section 4.2. Eq. (14) provides ϕ with historical information across batches by aligning the population statistics, which contributes to obtaining statistics as good as joint training (detailed in Section 5). From Fig. 1 we observe that adding this regularization term can significantly improve the gradient similarity, leading to stabilized training dynamics. We optimize ψ and θ (with γ and β inside) simultaneously, and the final objective function can be written as

where L CL is the loss function of continual learning (e.g., a cross-entropy loss over D t and {M k } t -1 k =1 for classification tasks). λ is a hyperparameter to balance the two terms.

4.2 Adaptive Balance of BN Statistics for Testing

Inspired by Corollaries 2 to 4, we propose to employ an appropriate η function to strike an adaptive balance between EMA and CMA. Concretely, we design a modified momentum η : [1 , m T ] × (0 , 1) → (0 , 1) satisfying the following recurrence relation:

where ˜ η ∈ (0 , 1) is the modified momentum of EMA and κ ∈ [0 , 1] is an additional hyperparameter to control the balance of normalization statistics. In particular, Eq. (16) degenerates to EMA when κ → 1 and to CMA when κ → 0 . This allows us to find an inflection point for adaptive balance of statistical weights, so as to enjoy the benefits of both EMA and CMA.

5 Experiments

In this section, we first briefly introduce the experimental setups of continual learning, and then present the experimental results to validate the effectiveness of our approach.

ER-ACE

|M|

= 500

DER++

|M|

= 500

ER-ACE

|M|

= 2000

DER++

|M|

= 2000

0

2

4

6

8

∆

FFA on BN (%)

Split CIFAR-10, Task-IL

Ours

|

B

|

= 10

CN

|

B

|

= 10

Ours

|

B

|

= 32

CN

|

B

|

= 32

ER-ACE

|M|

= 500

DER++

|M|

= 500

ER-ACE

|M|

= 2000

DER++

|M|

= 2000

0

5

10

15

Split CIFAR-10, Class-IL

ER-ACE

|M|

= 2000

DER++

|M|

= 2000

ER-ACE

|M|

= 5000

DER++

|M|

= 5000

-

2

0

2

4

6

Split CIFAR-100, Task-IL

ER-ACE

|M|

= 2000

DER++

|M|

= 2000

ER-ACE

|M|

= 5000

DER++

|M|

= 5000

-

2

0

2

4

6

Split CIFAR-100, Class-IL

Benchmark : We mainly focus on the setting of online continual learning [30], which is extremely challenging but realistic in application, and further consider the setting of offline continual learning. Here we evaluate three benchmarks that are widely used in both settings. Specifically, Split CIFAR-10 [14] includes 10-class images of size 32 × 32 , randomly split into 5 disjoint tasks with 2 classes each. Split CIFAR-100 [14] includes 100-class images of size 32 × 32 , randomly split into 20 disjoint tasks with 5 classes each. Split Mini-ImageNet [26] includes 100-class images of size 84 × 84 , randomly split into 20 disjoint tasks with 5 classes each. We consider both Task-IL and Class-IL, depending on whether the task identity is provided at test time [25].

Baseline : Following the previous work [19], we employ ER-ACE [7] and DER++ [6] as the baseline approaches for continual learning, both of which rely on a small memory buffer M to replay old training samples. We further consider LiDER [4] in offline scenarios, a recent approach for improving experience replay through regularizing Lipschitz continuity. We then compare our approach to three categories of normalization layers: (1) BN [11], which is the most popular choice that exhibits strong continual learning performance [19, 8]; (2) BN's alternatives designed for single-task learning, such as LN [2], IN [24] and GN [33] (GN can be seen as a general version of LN and IN); and (3) CN [19], which is the state-of-the-art baseline that improves BN for online continual learning.

Implementation : We follow the official implementation as many previous works [19, 4, 7, 6], derived from the mammoth repository [5]. Specifically, we use a ResNet-18 backbone and train all baselines with an SGD optimizer of learning rate 0.03 for Split CIFAR-10/-100 and 0.1 for Split Mini-ImageNet. Notably, we find the official implementation of CN [19] employs different batch sizes (either 10 or 32) for different online continual learning benchmarks. We argue that using a smaller batch size is more 'online' and more challenging, as fewer training samples of the current task can be accessed at a time. Therefore, we reproduce all results of online continual learning with one epoch and a small batch size of 10, and further evaluate the effect of using a larger batch size of 32. Besides, we use 50 epochs with a batch size of 32 for offline continual learning. We adopt reservoir sampling to save

0.0

0.001 0.01

0.1

1.0

10.0

λ

0.0

0.1

0.4

0.7

1.0

κ

37.3

37.5

46.6

56.2

63.4

65.5

61.7

63.3

63.0

66.1

70.5

71.1

63.7

64.3

65.3

67.1

71.8

n/a

64.3

66.5

65.6

67.6

69.7

n/a

64.5

65.3

66.0

68.8

70.2

n/a

Task-IL

0.0

0.001 0.01

0.1

1.0

10.0

λ

0.0

0.1

0.4

0.7

1.0

κ

6.8

7.0

11.3

15.1

19.8

19.6

16.5

17.2

17.5

19.0

23.2

23.8

17.4

17.3

17.7

20.2

23.9

n/a

17.6

18.6

18.2

19.7

21.8

n/a

17.6

17.3

18.2

20.8

22.2

n/a

Class-IL

64

66

68

70

Final Average Accuracy (%)

18

20

22

24

Final Average Accuracy (%)

Split CIFAR-100, DER++,

|M|

= 5000

=

62

16

old training samples, which simulates the distribution of incoming batches, and further consider ring buffer that enforces a balanced number for each class [30]. The memory buffer is sized differently to evaluate its impact. The group number of GN and CN is set to 32 as [19]. All results are averaged over three runs with different random seeds and task splits. Please see Appendix C for more details.

Overall Performance: We first evaluate our approach for online continual learning, including the settings of online Task-IL in Table 2 and online Class-IL in Table 3. Consistent with reported results [19, 8], BN performs much better than its alternatives designed for single-task learning (i.e., GN, LN and IN), and is moderately improved by the state-of-the-art CN. In contrast, our approach can improve the performance of BN by a large margin (e.g., up to 7.68%, 6.86% and 4.26% for online Class-IL on Split CIFAR-10, Split CIFAR-100 and Split Mini-ImageNet, respectively), and outperforms CN on a wide range of memory sizes |M| . When increasing the batch size | B | from 10 to 32, which introduces more training samples at once and makes online continual learning more 'offline', we observe that the benefits of CN exhibit variation in many cases, while the benefits of our approach remain consistent and become even stronger (e.g., up to 15% on Class-IL of Split CIFAR-10, see Fig. 2). Meanwhile, the selection strategies of the memory buffer M have no substantial effect, where our approach can provide a similar magnitude of improvement under either reservoir sampling or ring buffer (see Appendix D). We further present the results of offline Task-IL and Class-IL on the hardest Split Mini-ImageNet of the selected benchmarks (see Table 4), where CN even results in negative impacts on BN while our approach still provides strong performance gains in most cases.

Ablation Study & Analysis: We first analyze the introduced two hyperparameters in our approach, i.e., λ of the regularization term for training and κ of the modified momentum for testing. As shown in Fig. 3, the two hyperparameters can clearly influence the performance of continual learning, which

validates the effectiveness of each component in AdaB 2 N, and obtain consistent improvements under a wide range of combinations in different settings. Besides, we visualize the statistics parameters of normalization layers in continual learning, in order to explicitly demonstrate how our approach can improve BN (see Fig. 4 for a typical example and Appendix D for all results). We take joint training (JT) of all tasks as the upper bound baseline for continual learning. As more training batches are introduced, it can be obviously seen that the BN statistics are increasingly deviated while the CN statistics are enforced to stabilize. They correspond to the biased strategies of adaptation and balance , respectively, but neither can fit into the curve of JT. In contrast, our approach achieves an adaptive balance of normalization statistics in continual learning, matching closely the upper bound baseline.

6 Discussion and Conclusion

In this work, we focus on the most popular BN in continual learning and theoretically analyze its particular challenges for overcoming recency bias. Our analysis demonstrates the dilemma between balance of all tasks and adaptation to the current task, reflected in both training and testing. Following such theoretical insights, we propose Adaptive Balance of BN (AdaB 2 N) with a Bayesian-based strategy and a modified momentum to implement BN in a continual learning fashion, which achieves strong performance gains across various scenarios. Interestingly, the robust biological neural networks address a similar dilemma with a combination of normalization mechanisms for neurons, synapses, layers and

0

2000

4000

Steps

4

6

8

10

BN

CN

JT

Ours

0

2000

4000

Steps

2

4

6

8

10

Population Statistics

networks, some of which are thought to be analogous to the normalization layers in DNNs [23]. More in-depth analysis and modeling of these mechanisms would be a promising direction. We also expect subsequent work to further improve continual learning of other DNN components and overcome potential incompatibilities in optimization, as suggested by our work on normalization layers.

Now we discuss some potential limitations and societal impacts. First, our work applies only to the DNN architectures that employ normalization layers, especially the most popular BN. Second, we need to compute a categorical distribution P θ ( t | a m ) over the task identity, which is difficult to adapt directly to the more general setting of task-agnostic continual learning. Third, the prior P θ ( τ | a m ) is approximated as a Dirichlet distribution for ease of computation, which may not be applicable in some practical scenarios. We argue that the broader impact is not obvious at the current stage, as this work is essentially a fundamental research in machine learning.

References

Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Conference on Computer Vision , pages 139-154, 2018.

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016.

Nils Bjorck, Carla P Gomes, Bart Selman, and Kilian Q Weinberger. Understanding batch normalization. Advances in neural information processing systems , 31, 2018.

Lorenzo Bonicelli, Matteo Boschini, Angelo Porrello, Concetto Spampinato, and Simone Calderara. On the effectiveness of lipschitz-driven rehearsal in continual learning. Advances in Neural Information Processing Systems , 35:31886-31901, 2022.

Matteo Boschini, Lorenzo Bonicelli, Pietro Buzzega, Angelo Porrello, and Simone Calderara. Class-incremental continual learning into the extended der-verse. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(5):5497-5512, 2022.

Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: a strong, simple baseline. Advances in Neural Information Processing Systems , 33:15920-15930, 2020.

Lucas Caccia, Rahaf Aljundi, Nader Asadi, Tinne Tuytelaars, Joelle Pineau, and Eugene Belilovsky. New insights on reducing abrupt representation change in online continual learning. In International Conference on Learning Representations , 2022.

Sungmin Cha, Sungjun Cho, Dasol Hwang, Sunwon Hong, Moontae Lee, and Taesup Moon. Rebalancing batch normalization for exemplar-based class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 20127-20136, 2023.

Raia Hadsell, Dushyant Rao, Andrei A Rusu, and Razvan Pascanu. Embracing change: Continual learning in deep neural networks. Trends in Cognitive Sciences , 24(12):1028-1040, 2020.

Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning a unified classifier incrementally via rebalancing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 831-839, 2019.

Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning , pages 448-456. PMLR, 2015.

Taejong Joo, Uijung Chung, and Min-Gwan Seo. Being bayesian about categorical probability. In International conference on machine learning , pages 4950-4961. PMLR, 2020.

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences , 114(13):3521-3526, 2017.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.

David Lopez-Paz and Marc'Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in Neural Information Processing Systems , 30, 2017.

Ping Luo, Xinjiang Wang, Wenqi Shao, and Zhanglin Peng. Towards understanding regularization in batch normalization. In International Conference on Learning Representations , 2019.

James L McClelland, Bruce L McNaughton, and Randall C O'Reilly. Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychological review , 102(3):419, 1995.

Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision , pages 1406-1415, 2019.

Quang Pham, Chenghao Liu, and Hoi Steven. Continual normalization: Rethinking batch normalization for online continual learning. In International Conference on Learning Representations , 2022.

Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning. In International Conference on Learning Representations , 2021.

Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch normalization help optimization? Advances in neural information processing systems , 31, 2018.

Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning to quantify classification uncertainty. Advances in neural information processing systems , 31, 2018.

Yang Shen, Julia Wang, and Saket Navlakha. A correspondence between normalization strategies in artificial and biological neural networks. Neural Computation , 33(12):3179-3203, 2021.

Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022 , 2016.

Gido M van de Ven and Andreas S Tolias. Three continual learning scenarios. In NeurIPS Continual Learning Workshop , 2018.

Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in Neural Information Processing Systems , 29, 2016.

Liyuan Wang, Mingtian Zhang, Zhongfan Jia, Qian Li, Chenglong Bao, Kaisheng Ma, Jun Zhu, and Yi Zhong. Afec: Active forgetting of negative transfer in continual learning. Advances in Neural Information Processing Systems , 34:22379-22391, 2021.

Liyuan Wang, Xingxing Zhang, Qian Li, Mingtian Zhang, Hang Su, Jun Zhu, and Yi Zhong. Incorporating neuro-inspired adaptability for continual learning in artificial intelligence. arXiv preprint arXiv:2308.14991 , 2023.

Liyuan Wang, Xingxing Zhang, Qian Li, Jun Zhu, and Yi Zhong. Coscl: Cooperation of small continual learners is stronger than a big one. In European Conference on Computer Vision , pages 254-271. Springer, 2022.

Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. arXiv preprint arXiv:2302.00487 , 2023.

Liyuan Wang, Xingxing Zhang, Kuo Yang, Longhui Yu, Chongxuan Li, Lanqing Hong, Shifeng Zhang, Zhenguo Li, Yi Zhong, and Jun Zhu. Memory replay with data compression for continual learning. In International Conference on Learning Representations , 2022.

Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 374-382, 2019.

Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV) , pages 3-19, 2018.

Minghao Zhou, Quanziang Wang, Jun Shu, Qian Zhao, and Deyu Meng. Diagnosing batch normalization in class incremental learning. arXiv preprint arXiv:2202.08025 , 2022.

A Notation

Table 5 summarizes the notations used in this paper.

L

B Proofs

Proof of Theorem 1. Let g : [1 , m T ] × Z T + → [1 , T ] be a function that maps the batch index to its corresponding task index. We abbreviate g ( i, ( m t ) T t =1 ) as g ( i ) . Then we have

where the first part of Eq. (19) can be considered as the contribution of the 'current' tasks of all batches to the running statistics, and the second part can be considered as the contribution of the

'past' tasks. Now we can conclude the proof by noting that the contribution of task t as the 'current' tasks to the running statistics starts from m t -1 +1 to m t and the contribution as the 'past' task starts from m t +1 to m T .

Proof of Corollary 2. The first result can be obtained by noting that η is a constant and directly applying the sum formula of geometric series. When m T -m T -1 = m T -1 -m T -2 = · · · = m 1 , we have that

The approximation error is upper bounded by

Proof of Corollary 3. Necessity:

Let us start by assuming that w t = w t +1 . By applying Eq. (6) we have the following

Simplifying the equation we obtain that

The approximation error is upper bounded by

Sufficiency:

Substituting r = 1 /T into Eq. (20) yields

The approximation error is upper bounded by

Proof of Corollary 4. Substituting η ( i ) = 1 / (1 + i ) into Eq. (6)

We can now conclude by noting that w t is independent of the choice of t

C Implementation Details

The hyperparameter ˜ η is fixed to 0 . 1 , κ is selected from { 0 . 1 , 0 . 4 , 0 . 7 , 1 . 0 } , and λ is selected from { 0 . 01 , 0 . 1 , 1 . 0 , 10 . 0 } for Split CIFAR-10 and Split CIFAR-100 and { 0 . 00001 , 0 . 0001 , 0 . 001 , 0 . 01 } for Split Mini-ImageNet. Code is available in the supplementary material and will be released upon acceptance. All experiments are performed on eight NVIDIA RTX A4000 GPUs. The amount of compute is easily affordable, which can be inferred from the running times given in Table 6.

D Extended Results

D.1 Time Complexity Analysis

We provide in Table 6 the floating point operations per step (FLOPs/step) of different normalization methods and the total running times (in seconds) under different implementations (the exact calculation of FLOPs can be found in the attached code). It can be seen that compared to BN, our proposed method only slightly increases the computation, while CN almost doubles the computation because it combines both GN (without affine transformation) and BN. Note that the FLOPs only give the theoretical computation; the actual running time depends on the implementation. We measure the actual running time of two normalization implementations: 1) using the plain torch primitive and 2) using the torch.nn with cuDNN as the backend. Due to engineering difficulties, we do not currently implement a cuDNN-optimized version of our method. However, it can be seen from Table 6 that on the plain implementation our method only slightly increases the running time, which is consistent with the FLOPs analysis. This suggests that our method has the potential to achieve comparable time complexity as BN through a well-engineered cuDNN implementation.

.

D.2 Impacts of r

To show how the statistical weights of tasks affect performance, we vary the proportion of the current task in the batch (i.e., r = N t /N ) to different constants (with fixed N t = 10 , N = 10 , 20 , 50 , . . . ) and conduct experiments on Split CIFAR-10. It can be seen from Table 7 that the performance first increases rapidly, peaks at around r = 1 / 10 (i.e., the inverse of the total number of tasks), and then stabilizes. In particular, the accuracy with unfixed r is comparable with the peak, which supports our Corollary 3. Since the total number of tasks is generally unavailable in continual learning, using an unfixed r w.r.t 1 /T is more practical.

D.3 Impacts of Memory Buffer Selection Strategies

Fig. 5 shows that our method obtains substantial improvements over both reservoir sampling and ring buffer, which demonstrates the robustness of AdaB 2 N to the memory buffer selection strategy.

ER-ACE

= 2000

DER++

= 2000

ER-ACE

= 5000

DER++

= 5000

0

2

4

∆

FFA on BN (%)

Split CIFAR-100, Task-IL

ER-ACE

= 2000

DER++

= 2000

ER-ACE

= 5000

DER++

= 5000

0

2

4

6

Split CIFAR-100, Class-IL

Ours reservoir

CN

reservoir

Ours ring

CN

ring

|M|

|M|

|M|

|M|

|M|

|M|

|M|

|M|

D.4 Fine-tuning and Joint Training Performance

To establish a simple baseline and upper bound, we present the fine-tuning (FT) and joint training (JT) performance results of different normalization approaches in Table 8. Notably, Ours-FT consistently exhibits better performance than both BN-FT and CN-FT in the absence of replay data. On the other hand, Ours-JT does not outperform BN-JT, indicating that our performance improvements are indeed specific to continual learning.

D.5 Scalability Analysis

In order to evaluate the scalability of our proposed method, we perform an additional experiment with ImageNet-sub [10], which includes 100 classes randomly selected from ImageNet of size 224 × 224 . We split them into 10 incremental phases for online continual learning (ER, | B | = 32 , | M | = 2000 ), and reuse the hyperparameters ( λ and κ ) of Split Mini-ImageNet. As shown in Table 9, the Task-IL/Class-IL accuracy of our approach is clearly better than BN, demonstrating scalability on larger datasets. Furthermore, we perform experiments focused on online domain-incremental learning (Domain-IL) to evaluate its performance in handling more subtle distribution shifts. Specifically, we employ two datasets: Permuted MNIST [13], which applies random permutations to MNIST digit pixels to create 20 subsequent tasks, and DomainNet [18], comprising 345-class images from six distinct domains. Table 9 shows that our approach can clearly improve batch normalization on both Permuted MNIST and DomainNet, thanks to a better resolution of balance and adaptation.

D.6 Statistical Significance

We perform non-parametric paired t-test (i.e., the Wilcoxon signed-rank test) to evaluate whether AdaB 2 N exhibits statistically significant superiority over comparable baselines in both online and offline scenarios, as well as a combination of both. The null hypothesis H 0 states that the mean performance difference between AdaB 2 N and the alternate baseline is zero for a given scenario, while the alternative hypothesis H 1 states a non-zero difference. It is worth noting that we conduct two-tailed tests to avoid making an assumption that AdaB 2 N consistently outperforms the baseline. Our samples are collected from all the results presented in the manuscript. The resulting p -values for these tests, as detailed in Table 10, demonstrate that AdaB 2 N outperforms BN and CN statistically significantly across online, offline, and combined online and offline scenarios.

D.7 Forgetting Measure

Table 11 shows that our method is generally the lowest in terms of forgetting measure.

D.8 Full Results for Dynamics of Normalization Statistics

We provide the norm dynamics of the batch statistics and population statistics for all normalization layer in Figs. 6 and 7. For better illustration, each curve in the figures indicates the mean after Gaussian smoothing with a kernel size of 20 and the shaded area indicates the 0.05 × variance. It can be seen that our method roughly tracks the joint training (JT) on many layers (e.g., 2, 3, 5, 7, 8, 12, 15, 17, 18, 19).

Layer 1

Layer 2

Layer 3

Layer 4

Layer 5

8

1.5

6

8

6

1.0

6

4

4

4

0.5

4

2

2

0

5000

0

5000

0

5000

0

5000

0

5000

Layer 6

Layer 7

Layer 8

Layer 9

Layer 10

12.5

8

7

10.0

8

10.0

6

6

7.5

6

7.5

4

5.0

4

5.0

5

2.5

2

0

5000

0

5000

0

5000

0

5000

0

5000

Layer 11

Layer 12

Layer 13

Layer 14

Layer 15

15

15

10.0

10

15

7.5

6

10

10

10

8

5.0

5

5

5

0

5000

0

5000

0

5000

0

5000

0

5000

Layer 16

Layer 17

Layer 18

Layer 19

Layer 20

25

20

20

20

15

15

10

15

10

10

10

10

5

5

0

5000

0

5000

0

5000

0

5000

0

5000

BN

CN

JT

Ours

Layer 1

Layer 2

Layer 3

Layer 4

Layer 5

8

1.5

6

6

7.5

1.0

4

4

5.0

0.5

mmwmn

2

2

2.5

2

0

5000

0

5000

0

5000

0

5000

0

5000

Layer 6

Layer 7

Layer 8

Layer 9

Layer 10

12.5

8

10

7.5

10.0

6

6

7.5

4

5.0

5

5.0

2

2.5

0

5000

0

5000

0

5000

0

5000

0

5000

Layer 11

Layer 12

Layer 13

Layer 14

Layer 15

10

10

15

15

10

8

10

10

6

5

5

5

0

5000

0

5000

0

5000

0

5000

0

5000

Layer 16

Layer 17

Layer 18

Layer 19

Layer 20

20

20

20

20

15

15

10

10

10

10

10

5

5

5

0

5000

0

5000

0

5000

0

5000

0

5000

BN

CN

Ours

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents Adaptive Balance of Batch Normalization (AdaB2N) as a method to address the sub-optimality of Batch Normalization (BN) in continual learning. The main challenge lies in updating BN layer statistics based on the currently observed training samples, leading to recency bias and limited generalization across tasks. The paper leverages a Bayesian-based strategy to determine task-wise contributions in normalized representations, and it introduces an adaptive parameter that reconciles the constant and normalized momentum of EMA, achieving a balance of BN statistics while maintaining adaptability during training.\n\n**TLDR of the review:** The paper provides a comprehensive theoretical analysis, a well-motivated problem statement, and promising experimental results. However, the paper has weaknesses such as insufficient empirical significance analysis, limited method scalability analysis, missing important baselines, and limited investigation of subtle distribution shifts. I am happy to reconsider my rating based on how the authors address these comments.\n\nSTRENGTHS:\n1.\t**Comprehensive theoretical analysis (Note: I can't determine how significant these results are):** The paper offers a thorough theoretical analysis of the behavior of batch normalization in continual learning. It sheds light on the tension between balance (working effectively across tasks) and adaptation (learning new tasks), providing a solid foundation for the proposed method.\n2.\t**Well-motivated problem statement:** The problem statement presented in the paper is well-motivated, highlighting the limitations of batch normalization in continual learning.\n3.\t**Promising experimental results:** The paper provides experimental results on small-scale image datasets (CIFAR and Mini-ImageNet), which demonstrate promising outcomes.\n4.\t**Interesting analysis experiment:** The authors conducted a compelling experiment demonstrating that the model trained with the proposed method in an online continual learning (CL) setting achieves normalization statistics that closely match the statistics of a model trained on all tasks jointly, which serves as a baseline for non-continual learning.\n\nWEAKNESSES:\n*Main weaknesses that influence my rating:*\n\n1.\t**Insufficient empirical significance analysis:** The paper lacks a thorough analysis of statistical significance for the reported results. In some cases, the difference in mean results between the proposed method and competitors is not significant when considering standard deviations. For example, the online CIL results for SplitCIFAR-100/|M|=500 exhibit this issue. It is important for the authors to clarify the conditions regarding the order of stream samples under which the proposed method performs best. Additionally, to promote transparency, it would be beneficial to report the means and standard deviations for offline CL experiments, enabling an assessment of the consistency of the improvement across different seeds.\n2.\t**Limited method scalability analysis:** The experiments are primarily conducted on small-scale datasets like CIFAR and Mini-ImageNet. However, it remains unclear how well the proposed method scales to larger datasets, such as the ImageNet subset. The authors should address whether there is a specific reason for the limited scalability analysis.\n3.\t**Missing two important baselines:** Two crucial baselines are absent from the tables. Firstly, it seems that the experiments incorporate some form of replay, which might affect batch normalization (BN) statistics and mitigate recency bias. It would be valuable to include a simple fine-tuning baseline and compare the effect of different normalization methods applied to that baseline. Secondly, reporting the upper bound of no-continual learning (NoCL) baseline performance, possibly using both BN and the proposed method.\n4.\t**Limited investigation of subtle distribution shifts:** The experiments primarily focus on drastic changes occurring when switching between tasks in continual learning, primarily caused by class boundaries. However, it is essential to also explore the performance of batch normalization (BN) in handling more subtle distribution shifts within individual tasks or in a task-free continual learning setup. \n\n*Additional weaknesses that do not necessarily influence my rating:*\n\n5.\t**Lack of clarification on normalization methods:** The paper lacks clarification on the normalization used by the joint training (JT) baseline, and whether the proposed normalization method is applicable in the noCL scenario. It would be valuable to compare \"ours-JT\" (the proposed method) against \"BN-JT\" to gain insights into their relative performance.\n\nQUESTIONS:\n**Here are some suggestions to help address the weaknesses:**\n\n1.\tCan you address the concerns raised regarding statistical significance? Could you provide any explanation of the conditions under which the proposed method performs well?\n2.\tIs there a specific reason why the experiments were limited to small-scale datasets like CIFAR and Mini-ImageNet? If there are scalability limitations, please provide insights into the reasons behind them.\n3.\tCould you include the suggested baselines and compare the performance with the proposed method and other normalization techniques?\n4.\tHave you considered investigating the performance of Batch Normalization (BN) in scenarios with more subtle distribution shifts within individual tasks or in a task-free continual learning setup? \n5.\tCan you provide information about the normalization method used by JT? Please elaborate on whether the proposed normalization method is applicable in the non-continual learning scenario and discuss its potential performance compared to BN in that setting.\n\nLIMITATIONS:\nLimitations and potential negative societal impact are discussed (no clear negative societal impact at this stage).", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-28"}

comments: SUMMARY:
The manuscript introduces a theoretical analysis on the forgetting in the statistics of the popular Batch Normalization layer. Based on this, it introduces a novel normalization layer, termed $AdaB^2N$, and evaluates its effectiveness on the online class-incremental and task-incremental settings.

STRENGTHS:
The manuscript is clearly written and provides a clear picture on the shortcomings of Batch Normalization in a continual learning setting. Furthermore, the results clearly depict a consistent improvement over a wide range of benchmarks.

WEAKNESSES:
I enjoyed reading the manuscript and I believe it carries no major weakness. 
My only concern arises from the multi-epoch (offline) setting, where the authors evaluate the effectiveness of their proposal on the baselines in the Task-IL setting. Indeed, as the latter is often depicted as less challenging [1,2], I would recommend including the also results of the Class-IL in this evaluation.

Additional comments that did not influence my judgment:
- The citation for LiDER should be aNeurIPS 2022.
- The citation for “Three scenarios for continual learning” should be a NeurIPS Continual Learning workshop 2018.

[1] Sebastian Farquhar and Yarin Gal. Towards Robust Evaluations of Continual Learning. ICMLw, 2018.

[2] Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient-based sample selection for online continual learning. aNeurIPS, 2019

QUESTIONS:
As I mentioned above, my only recommendation would be to include results for the Class-IL multi-epoch setting.

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies how to update the parameters of batch normalization during continual learning. This is a problem that many researchers have mentioned, since batch statistics would be computed in a non-iid manner; however, while the problem has been acknowledged, at least in conversation, there hasn't been much work to resolve this problem. They theoretically study how to update batch norm statistics for continual learning, and they propose a new batch norm variant based on Bayesian updates. They demonstrate that on standard small-scale continual learning setups that their method increases performance significantly.\n\nSTRENGTHS:\nIt is a good problem to work on and the results appear strong on the small-scale problems studied, although I have strong reservations about them given the hyperparameter sweeps.\nI appreciate the use of comparing batch sizes, rather than treating that as a hyperparameters. Likewise, I appreciate using the combined task upper bound, which many papers omit making it hard to understand the gap in performance.\n\nWEAKNESSES:\nThe analysis in 3.1 is conflated with related work in terms of the content, e.g., discussing group normalization, layer normalization, etc. Likewise, the related work simply lists past work. It would strengthen the paper to better juxtapose why a new method is needed and the limitations with existing normalization methods. \n\nThe continual learning frameworks studied, i.e., Task-IL and Class-IL, is limited. It would be better to do this in a more general manner where they discuss the properties of the distribution the stream is being drawn from, as in many real-world continual learning problems the data stream would not be either of these extreme edge cases. The field is definitely moving toward thinking of the data ordering as more of robustness to different distributions rather than these specific setups, which lack any real-world applicability except as experiments regarding different stream distributions.\n\nThe use of task labels (especially at test-time), and many recent methods have largely ceased to use task labels as they have little real-world relevance. They do study Class-IL as well, but that’s more of an extreme edge case.\n\nBoth methods they employ, ER-ACE and DER++ use a small memory buffer, but addressing the problems with updating normalization layers seems best done for replay-free methods, rather than using replay-based methods. \n\nThe experiments are all on small-scale datasets and only extreme distributions are tested. I’d really like to see more natural distributions and even easy distributions evaluated (iid) with the same hyperparameters. It is unclear if kappa is being tuned to optimize results per stream, where it would be impossible to actually do that in real-world continual learning. As best I can tell (and supported by the discussion in the appendix), they are reporting the best results of a hyperparameter sweep of lambda and kappa, which is pretty unrealistic on a per-dataset and per-distribution basis.\n\nQUESTIONS:\nWhy was the use of replay employed rather than studying a replay-free setting? It is very unclear if the limited memory setup makes sense vs, either a no memory setup or an unlimited memory setup (e.g., https://arxiv.org/abs/2302.01047 )\n\nWhy are only small-scale problems studied vs. studying multiple distributions on a larger dataset, e.g., ImageNet, which is also largely considered a toy dataset nowadays. Even DER when originally proposed was evaluated on ImageNet-1K.\n\nWhy were ER-ACE and DER++ studied only?\n\nLIMITATIONS:\nAs far as the limitations of the work, I think this needs a lot of work. Much of the field is now working on larger scale problems, e.g., continual learning on ImageNet from various distributions for the past few years, and the use of hyperparameter sweeps for kappa and lambda significantly reduce the generality of the work since this is unrealistic for most real-world continual learning applications since this would require knowing the stream and dataset ahead of time.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the recency bias of the normalization statistics in continual learning and presents an theoretical analysis of the delimma of the balance and adaption in estimating the normalization statistics. \nTo address these challenges, the paper propose Adaptive Balance of BN (AdaB2N), which appropriately incorporates a Bayesian-based strategy to adapt task-wise contributions and a modified momentum to balance BN statistics.\n\nSTRENGTHS:\n- This paper demonstrates the adaptation and balance dilemma in normalized statistical estimation in continuous learning through theoretical analysis.\n- The method proposed in this paper is simple and effective. It only requires learning an additional vector and setting two hyperparameters to achieve satisfactory improvements.\n\nWEAKNESSES:\n- By theoretical analysis, the normalized statistical estimation in Eq. (4) faces a dilemma between balance and adaptation. However, Eq. (4) is used to guide the learning of conditional estimation, as shown in Eq. (14). Whether this guidance is correct or not is uncertain.\nSo it is not clear how the loss function in Eq. (14) helps to improve the normalized statistical estimation. This model parameter $\\theta$ is also updated with the loss in Eq. (14). Perhaps Eq. (14) imposes a regularization on $\\theta$ that helps to improve the performance.\n\n- The definition of $\\psi$ requires prior knowledge of the total number of tasks, which is generally not available in continuous learning.\n\n- The presentation of the paper could be improved. For example,\n  - In lines 37-38, the limitations of the existing work could be presented in more detail to help the reader better understand what the proposed improvements would look like compared to the existing work.\n  - In lines 39-46, more information about balance and adaptation and their conflicts could be included to illustrate the dilemma.\n  - The notation is complex, making it not easy to follow the theory part.\n  - In Theorem 1, $\\eta(i)$ and $r(i)$ are not explicitly defined.\n\n- The experiment can be improved.\n  - In the comparison of offline task incremental learning settings (Fig. 3), the baseline method is CN, which is specifically designed for online settings. Here two baselines [7] and [28] should be compared, which are designed for the offline setting.\n  - There is no explanation for why some of the results in Figure 4 are n\\a.\n\nQUESTIONS:\n- Why is the \"adaptive balance of BN statistics for testing\" (Equation (16)) included in the training phase?\n- Why is the estimation in Eq. (4), which faces a dilemma between balance and adaptation, used to guide the learning of Eq. (16)? How is this to improve the estimation?\n- The performance of Eq. (14) without updating $theta$ and without using the loss function Eq.\n- In order to calculate Eq. (13), do we need to keep all $S_m^t$ (for all $m$ and $t$) and then perform the summation operation? Or do we just keep the sum of the results up to $t-1$ and then add $S_m^t$ to get the result?\n- Can the authors provide results for Corollary 3 with fixed $r$ so that we can check the impact of $r$ in the statistical estimation?\n\nLIMITATIONS:\nNo societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-08"}