Year: 2021

Abstract:
Existing continual learning methods use Batch Normalization (BN) to facilitate training and improve generalization across tasks. However, the non-i.i.d and nonstationary nature of continual learning data, especially in the online setting, amplify the discrepancy between training and testing in BN and hinder the performance of older tasks. In this work, we study the cross-task normalization effect of BN in online continual learning where BN normalizes the testing data using moments biased towards the current task, resulting in higher catastrophic forgetting. This limitation motivates us to propose a simple yet effective method that we call Continual Normalization (CN) to facilitate training similar to BN while mitigating its negative effect. Extensive experiments on different continual learning algorithms and online scenarios show that CN is a direct replacement for BN and can provide substantial performance improvements. Our implementation will be made publicly available upon acceptance.

heading: 1 INTRODUCTION
text: Continual learning (Ring, 1997) is a promising approach towards building learning systems with human-like capabilities. Unlike traditional learning paradigms, continual learning methods observe a stream of tasks and simultaneously perform well on all tasks with limited access to previous data. Therefore, they have to achieve a good trade-off between retaining old knowledge (French, 1999) and acquiring new skills, which is referred to as the stability-plasticity dilemma (Abraham & Robins, 2005). Continual learning has been a challenging research problem, especially for deep neural networks because of their ubiquity and promising results on many applications (LeCun et al., 2015; Parisi et al., 2019).
While most previous works focus on developing strategies to alleviate catastrophic forgetting and facilitating knowledge transfer (Parisi et al., 2019), scant attention has been paid to the backbone they used. In standard backbone networks such as ResNets (He et al., 2016), it is natural to use Batch Normalization (BN) (Ioffe & Szegedy, 2015), which has enabled the deep learning community to make substantial progress in many applications (Huang et al., 2020). Although recent efforts have shown promising results in training deep networks without BN in a single task learning (Brock et al., 2021), we argue that BN has a huge impact on continual learning. Particularly, when using an episodic memory, BN improves knowledge sharing across tasks by allowing data of previous tasks to contribute to the normalization of current samples and vice versa. Unfortunately, in this work, we have explored a negative effect of BN that hinders its performance on older tasks and thus increases catastrophic forgetting. Explicitly, unlike the standard classification problems, data in continual learning arrived sequentially, which are not independent and identically distributed (noni.i.d). Moreover, especially in the online setting (Lopez-Paz & Ranzato, 2017), the data distribution changes over time, which is also highly non-stationary. Together, such properties make the BN’s running statistics heavily biased towards the current task. Consequently, during inference, the model normalizes previous tasks’ data using the moments of the current task, which we refer to as the “cross-task normalization effect”.
Although using an episodic memory can alleviate cross-task normalization in BN, it is not possible to fully mitigate this effect without storing all past data, which is against the continual learning
purposes. On the other hand, spatial normalization layers such as GN (Wu & He, 2018) do not perform cross-task normalization because they normalize each feature individually along the spatial dimensions. As we will empirically verify in Section 6.1, although such layers suffer less forgetting than BN, they do not learn individual tasks as well as BN because they lack the knowledge transfer mechanism via normalizing along the mini-batch dimension. This result suggests that a continual learning normalization layer should balance normalizing along the mini-batch and spatial dimensions to achieve a good trade-off between knowledge transfer and alleviating forgetting. Such properties are crucial for continual learning, especially in the online setting (Lopez-Paz et al., 2017; Riemer et al., 2019), but not satisfied by existing normalization layers. Consequently, we propose Continual Normalization (CN), a novel normalization layer that takes into account the spatial information when normalizing along the mini-batch dimension. Therefore, our CN enjoys the benefits of BN while alleviating the cross-task normalization effect. Lastly, our study is orthogonal to the continual learning literature, and the proposed CN can be easily integrated into any existing methods to improve their performances across different online protocols.
In summary, we make the following contributions:
• We study the benefits of BN and its cross-task normalization effect in continual learning. • We identify the desiderata of a normalization layer for continual learning and propose CN,
a novel normalization layer that improves the performance of existing continual learning methods.
• Through extensive experiments on different online continual learning benchmarks, we demonstrate that CN is a plug-in replacement for BN and offers promising improvements across different metrics with limited computational overhead.

heading: 2 NOTATIONS AND PRELIMINARIES
text: This section provides the necessary background of continual learning and normalization layers, including the batch normalization (BN) layer and its variants.

heading: 3 CONTINUAL LEARNING PROTOCOL
text: We focus on the online continual learning protocol (Lopez-Paz & Ranzato, 2017) over a data stream of T tasks. At each step, the model only observes training samples drawn from the current training task data and has to perform well on the testing data of all observed tasks so far. Moreover, data comes strictly in a streaming manner, resulting in a single epoch training setting. Throughout this work, we conduct experiments on different continual learning protocols, ranging from the taskincremental (Lopez-Paz & Ranzato, 2017), class-incremental (Rebuffi et al., 2017), to the task-free scenarios (Aljundi et al., 2019b). We will describe each scenario in more details in Section 6.

heading: 3.1 NOTATIONS
text: We focus on the image recognition problem and denote an image as x ∈ RW×H×C , where W , H , C are the image width, height, and the number of channels respectively. A convolutional neural network (CNN) with parameter θ is denoted as fθ(·). A feature map of a mini-batch B is defined as a ∈ RB×C×W×H . Finally. each image x is associated with a label y ∈ {1, 2 . . . , Y } and a task identifier t ∈ {1, 2, . . . , T}.

heading: 3.2 NORMALIZATION LAYERS
text: Normalization layers are essential in training deep neural networks (Huang et al., 2020). In general, a normalization layer takes a mini-batch of feature maps a = (a1, . . . ,aB) as input and perform the Z-normalization as:
a′ = γ ( a− µ√ σ2 + ) + β, (1)
where µ and σ2 are the mean and variance calculated from the features in B, and is a small constant added to avoid division by zero. The affine transformation’s parameters γ, β ∈ RC are
|C|-dimensional vectors learned by backpropagation to retain the layer’s representation capacity. For brevity, we will use “moments” to refer to both the mean µ and the variance σ2.
Batch Normalization BN is one of the first normalization layers that found success in a wide range of deep learning applications (Ioffe & Szegedy, 2015; Santurkar et al., 2018; Bjorck et al., 2018). During training, BN calculates the moments across the mini-batch dimension as:
µBN = 1
BHW B∑ b=1 W∑ w=1 H∑ H=1 abcwh, σ 2 BN =
1
BHW B∑ b=1 W∑ w=1 H∑ H=1 (abcwh − µBN )2. (2)
At test time, it is important for BN to be able to make predictions with only one data sample to make the prediction deterministic. As a result, BN replaces the mini-batch mean and variance in Eq. (2) by an estimate of the global values obtained during training as:
µ← µ+ η(µB − µ), σ ← σ + η(σB − σ), (3) where η is the exponential running average’s momentum, which were set to the standard value of 0.1 in our experiments.
Spatial Normalization Layers The discrepancy between training and testing in BN can be problematic when the training mini-batch size is small (Ioffe, 2017), or when the testing data distribution differs from the training distributions. In such scenarios, the running estimate of the mean and variance obtained during training are a poor estimate of the moments to normalize the testing data. Therefore, there have been tremendous efforts in developing alternatives to BN to address these challenges. One notable approach is normalizing along the spatial dimensions of each sample independently, which completely avoid the negative effect from having a biased global moments. In the following, we discuss popular examples of such layers.
Group Normalization (GN) (Wu & He, 2018) When the mini-batch size is small, BN’s running estimates of the global moments become inaccurate. Therefore, GN proposes to normalize each feature individually by dividing each channel into groups and then normalize each group separately. Moreover, GN does not normalize along the batch dimension, thus performs the same computation during training and testing. Computationally, GN first divides the channels C into G groups as:
a′bgkhw ← abchw, where k = b∗c C
G , (4)
Then, for each group g, the features are normalized with the moments calculated as:
µ (g) GN =
1
m K∑ k=1 W∑ w=1 H∑ h=1 a′bgkhw, σ (g) GN 2 = 1 m K∑ k=1 W∑ w=1 H∑ h=1 (a′bgkhw − µ (g) GN ) 2 (5)
GN has shown comparable performance to BN with large mini-batch sizes (e.g. 32 or more), while significantly outperformed BN with small mini-batch sizes (e.g. one or two). Notably, when putting all channels into a single group (setting G = 1), GN is equivalent to Layer Normalization (LN) (Ba et al., 2016), which normalizes the whole layer of each sample. On the other extreme, when separating each channel as a group (setting G = C), GN becomes Instance Normalization (IN) (Ulyanov et al., 2016), which normalizes the spatial dimension in each channel of each feature. Figure 1 provides an illustration of BN, GN, IN, and the proposed CN, which we will discuss in Section 5.

heading: 4 BATCH NORMALIZATION IN CONTINUAL LEARNING

heading: 4.1 THE BENEFITS OF NORMALIZATION LAYERS IN CONTINUAL LEARNING
text: We argue that BN is helpful for forward transfer in two aspects. First, BN makes the optimization landscape smoother (Santurkar et al., 2018), which allows the optimization of deep neural networks to converge faster and better (Bjorck et al., 2018). In continual learning, BN enables the model to learn individual tasks better than the no-normalization method. Second, with the episodic memory, BN uses data of the current task and previous tasks (in the memory) to update its running moments. Therefore, BN further facilitates forward knowledge transfer during experience replay: current task data is normalized using moments calculated from both current and previous samples. Compared to BN, spatial normalization layers (such as GN) lack the ability to facilitate forward transfer via normalizing using moments calculated from data of both old and current tasks. We will empirically verify the benefits of different normalization layers to continual learning in Section 6.1.

heading: 4.2 THE CROSS-TASK NORMALIZATION EFFECT
text: Recall that BN maintains a running estimate of the global moments to normalize the testing data. In the standard learning with a single task where data are i.i.d, one can expect such estimates can well-characterize the true, global moments, and can be used to normalize the testing data. However, online continual learning data is non-i.i.d and highly non stationary. Therefore, BN’s estimation of the global moments is heavily biased towards the current task because the recent mini-batches only contain that task’s data. As a result, during inference, when evaluating the older tasks, BN normalizes previous tasks’ data using the current task’s moments, which we refer to as the crosstask normalization effect.
We consider a toy experiment on the permuted MNIST (pMNIST) benchmark (Lopez-Paz & Ranzato, 2017) to explore the cross-task normalization effect in BN. We construct a sequence of five task, each has 2,000 training and 10,000 testing samples, and a multilayer perceptron(MLP) backbone configured as Input(784)-FC1(100)-BN1(100)-FC2(100)-BN(100)-Softmax(10), where the number inside the parentheses indicates the output dimension of that layer. We consider the Single and ER strategies for this experiment, where the Single strategy is the naive method that trains continuously without any memory or regularization. For each method, we implement an optimal BN variant (denoted by the suffix -BN∗) that calculates the global moments using all training data of all tasks before testing. Compared to BN, BN∗ has the same network parameters and only differs in the BN’s moments used to normalize the testing data. We emphasize that although BN∗ is unrealistic, it sheds light on how cross-task normalization affects the performance of CL algorithms. We report the averaged accuracy at the end of training ACC(↑) (Lopez-Paz et al., 2017), forgetting measure FM(↓) (Chaudhry et al., 2018), and learning accuracy LA(↑) (Riemer et al., 2019) in this experiment. Table 1 reports the evaluation metrics of this experiment. Besides the standard metrics, we also report the moments’ difference magnitudes between the standard BN and the global BN variant. We observe that the gap between two BN variants is significant without the episodic memory. When using the episodic memory, ER can reduce the gap because the pMNIST is quite simple and even the ER strategy can achieve close performances to the offline model (Chaudhry et al., 2019b). Moreover, it is important to note that training with BN on imbalanced data also affects the model’s learning dynamics, resulting in a suboptimal parameter. Nevertheless, having an unbiased estimate of the global moments can greatly improve overall accuracy given the same model parameters. Moreover,
this improvement is attributed to reducing forgetting rather than facilitating transfer: BN∗ has lower FM but almost similar LA compared to the traditional BN. In addition, we observe that the hidden features’ variance becomes inaccurate compared to the global variance as we go to deeper layers (∆(1)σ2 < ∆ (2) σ2 ). This result agrees with recent finding (Ramasesh et al., 2021) that deeper layers are more responsible for causing forgetting because their hidden representation deviates from the model trained on all tasks. Overall, these results show that normalizing older tasks using the current task’s moments causes higher forgetting, which we refer to as the “cross-task normalization effect”.

heading: 4.3 DESIDERATA FOR CONTINUAL LEARNING NORMALIZATION LAYERS
text: While BN facilitates continual learning, it also amplifies catastrophic forgetting by causing the crosstask normalization effect. To retain the BN’s benefits while alleviating its drawbacks, we argue that an ideal normalization layer for continual learning should be adaptive by incorporating each feature’s statistics into its normalization. Being adaptive can mitigate the cross-task normalization effect because each sample is now normalized differently at test time instead of being normalized by a set of biased moments. In literature, adaptive normalization layers have shown promising results when the mini-batch sizes are small (Wu & He, 2018), or when the number of training data is extremely limited (Bronskill et al., 2020). In such cases, normalizing along the spatial dimensions of each feature can alleviate the negative effect from an inaccurate estimate of the global moments. Inspired by this observation, we propose the desiderata for a continual learning normalization layer:
• Facilitates the performance of deep networks by improving knowledge sharing within and across-task (when the episodic memory is used), thus increasing the performance of all tasks (ACC in our work);
• Is adaptive at test time: each data sample should be normalized differently. Moreover, each data sample should contribute to the normalized feature’s statistic, which reduces catastrophic forgetting;
• Does not require additional input at test time such as the episodic memory, or the task identifier.
To simultaneously facilitate training and mitigating the cross-task normalization effect, a normalization layer has to balance between both across mini-batch normalization and within-sample normalization. As we will show in Section 6.1, BN can facilitate training by normalizing along the mini-batch dimension; however, it is not adaptive at test time and suffers from the cross-task normalization effect. On the other hand, GN is fully adaptive, but it does not facilitate training compared to BN. Therefore, it is imperative to balance both aspects to improve continual learning performance. Lastly, we expect a continual learning normalization layer can be a direct replacement for BN, thus, it should not require additional information to work, especially at test time.

heading: 5 CONTINUAL NORMALIZATION (CN)
text: CN works by first performing a spatial normalization on the feature map, which we choose to be group normalization. Then, the group-normalized features are further normalized by a batch normalization layer. Formally, given the input feature map a, we denote BN1,0 and GN1,0 as the batch normalization and group normalization layers without the affine transformation parameters1, CN obtains the normalized features aCN as:
aGN ← GN1,0(a); aCN ← γBN1,0(aGN) + β. (6)
In the first step, our GN component does not use the affine transformation to make the intermediate feature BN1,0(aGN) well-normalized across the mini-batch and spatial dimensions. Moreover, performing GN first allows the spatial-normalized features to contribute to the BN’s running statistic, which further reduce the cross-task normalization effect. An illustration of CN is given in Figure 1.
By its design, one can see that CN satisfies the desiderata of a continual learning normalization layer. Particularly, CN balances between facilitating training and alleviating the cross-task normalizing effect by normalizing the input feature across mini-batch and individually. Therefore, CN is adaptive
1which is equivalent to setting γ = 1 and β = 0
at test time and produces well-normalized features in the mini-batch and spatial dimensions, which strikes a great balance between BN and other instance-based normalizers. Lastly, CN uses the same input as conventional normalization layers and does not introduce extra learnable parameters that can be prone to catastrophic forgetting.
We now discuss why CN is a more suitable normalization layer for continual learning than recent advanced normalization layers. SwitchNorm (Luo et al., 2018) proposed to combine the moments from three normalization layers: BN, IN, and LN to normalize the feature map (Eq. 13). However, the blending weightsw andw′ make the output feature not well-normalized in any dimensions since such weights are smaller than one, which scales down the moments. Moreover, the choice of IN and LN may not be helpful for image recognition problems. Similar to SN, TaskNorm (Bronskill et al., 2020) combines the moments from BN and IN by a blending factor, which is learned for each task. As a result, TaskNorm also suffers in that its outputs are not well-normalized. Moreover, TaskNorm addresses the meta learning problem requires knowing the task identifier at test time, which violates our third criterion of requiring additional information compared to BN and our CN.

heading: 6 EXPERIMENT
text: We evaluate the proposed CN’s performance compared to a suite of normalization layers with a focus on the online continual learning settings where it is more challenging to obtain a good global moments for BN. Our goal is to evaluate the following hypotheses: (i) BN can facilitate knowledge transfer better than spatial normalization layers such as GN and SN (ii) Spatial normalization layers have lower forgetting than BN; (iii) CN can improve over other normalization layers by reducing catastrophic forgetting and facilitating knowledge transfer; (iv) CN is a direct replacement of BN without additional parameters and minimal computational overhead. Due to space constraints, we briefly mention the setting before each experiment and provide the full details in Appendix E.

heading: 6.1 ONLINE TASK-INCREMENTAL CONTINUAL LEARNING
text: Setting We first consider the standard online, task-incremental continual learning setting (LopezPaz & Ranzato, 2017) on the Split CIFAR-100 and Split Mini IMN benchmarks. We follow the standard setting in Chaudhry et al. (2019a) to split the original CIFAR100 (Krizhevsky & Hinton,
2009) or Mini IMN (Vinyals et al., 2016) datasets into a sequence of 20 tasks, three of which are used for hyper-parameter cross-validation, and the remaining 17 tasks are used for continual learning.
We consider two continual learning strategies: (i) Experience Replay (ER) (Chaudhry et al., 2019b); and (ii) Dark Experience Replay++ (DER++) (Buzzega et al., 2020). Besides the vanilla ER, we also consider DER++, a recent improved ER variants, to demonstrate that our proposed CN can work well across different ER-based strategies. All methods use a standard ResNet 18 backbone (He et al., 2016) (not pre-trained) and are optimized over one epoch with batch size 10 using the SGD optimizer. For each continual learning strategies, we compare our proposed CN with five competing normalization layers: (i) BN (Ioffe & Szegedy, 2015); (ii) Batch Renormalization (BRN) (Ioffe, 2017); (iii) IN (Ulyanov et al., 2016); (iv) GN (Wu & He, 2018); and (v) SN (Luo et al., 2018). We cross-validate and set the number of groups to be G = 32 for our CN and GN in this experiment.
Results Table 2 reports the evaluation metrics of different normalization layers on the Split CIFAR-100 and Split Mini IMN benchmarks. We consider the averaged accuracy at the end of training ACC(↑) (Lopez-Paz et al., 2017), forgetting measure FM(↓) (Chaudhry et al., 2018), and learning accuracy LA(↑) (Riemer et al., 2019), details are provided in Appendix B. Clearly, IN does not perform well because it is not designed for image recognition problems. Compared to adaptive normalization methods such as GN and SN, BN suffers from more catastrophic forgetting (higher FM(↓) ) but at the same time can transfer knowledge better across tasks (higher LA(↑) ). Moreover, BRN performs worse than BN since it does not address the biased estimate of the global moments, which makes normalizing with the global moments during training ineffective. Overall, the results show that although traditional adaptive methods such as GN and SN do not suffer from the cross-task normalization effect and enjoy lower FM(↓) values, they lack the ability to facilitate knowledge transfer across tasks, which results in lower LA(↑) . Moreover, BN can facilitate knowledge sharing across tasks, but it suffers more from forgetting because of the cross-task normalization effect. Across all benchmarks, our CN comes out as a clear winner by achieving the best overall performance (ACC(↑) ). This result shows that CN can strike a great balance between reducing catastrophic forgetting and facilitating knowledge transfer to improve continual learning.
Complexity Comparison We study the complexity of different normalization layers and reporting the training time on the Split CIFAR100 benchmark in Tabe 3. Both GN and our CN have minimal computational overhead compared to BN, while SN suffers from additional computational cost from calculating and normalizing with different sets of moments.

heading: 6.2 ONLINE CLASS-INCREMENTAL CONTINUAL LEARNING
text: Setting We consider the online task incremental (Task-IL) and class incremental (Class-IL) learning problems on the Split-CIFAR10 (Split CIFAR-10) and Split-Tiny-ImageNet (Split Tiny IMN) benchmarks. We follow the same experiment setups as Buzzega et al. (2020) except the number of training epochs, which we set to one. All experiments uses the DER++ strategy (Buzzega et al., 2020) on a ResNet 18 (He et al., 2016) backbone (not pre-trained) trained with data augmentation using the SGD optimizer. We consider three different total episodic memory sizes of 500, 2560, and 5120, and focus on comparing BN with our proposed CN in this experiment.
Result Table 4 reports the ACC and FM metrics on the Split CIFAR-10 and Split Tiny IMN benchmarks under both the Task-IL and Class-IL settings. For CN, we consider two configurations with the number of groups beingG = 8 andG = 32. In most cases, our CN outperforms the standard BN on both metrics, with a more significant gap on larger memory sizes of 2560 and 5120. Interestingly, BN is highly unstable in the Split CIFAR-10, Class-IL benchmark with high standard deviations. In contrast, both CN variants show better and more stable results, especially in the more challenging Class-IL setting with a small memory size (indicated by small standard deviation values). We also report the evolution of ACC in Figure 2. Both CN variants consistently outperform BN throughout training, with only one exception at the second task on the Split Tiny IMN. CN (G = 32) shows more stable and better performances than BN and CN(G = 8).

heading: 6.3 LONG-TAILED ONLINE CONTINUAL LEARNING
text: We now evaluate the normalization layers on the challenging task-free, long-tailed continual learning setting (Kim et al., 2020), which is more challenging and realistic since real-world data usually follow long-tailed distributions. We consider the PRS strategy (Kim et al., 2020) and the COCOseq and NUS-WIDEseq benchmarks, which consists of four and six tasks, respectively. Unlike the previous benchmarks, images in the COCOseq and NUS-WIDEseq benchmarks can have multiple labels, resulting in a long-tailed distribution over each task’s image label. Following Kim et al. (2020), we report the average overall F1 (O-F1), per-class F1 (C-F1), and the mean average precision (mAP) at the end of training and their corresponding forgetting measures (FM). We also report each metric over the minority classes (<200 samples), moderate classes (200-900 samples), majority classes (>900 samples), and all classes. Empirically, we found that smaller groups helped in the long-tailed setting because the moments were calculated over more channels, reducing the dominants of head classes. Therefore, we use G = 4 groups in this experiment.
We replicate PRS with BN to compare with our CN and report the results in Table 5 and Table 6. We observe consistent improvements over BN from only changing the normalization layers, especially in reducing FM(↓) across all classes.

heading: 6.4 DISCUSSION OF THE RESULTS
text: Our experiments have shown promising results for CN being a potential replacement for BN in online continual learning. While the results are generally consistent, there are a few scenarios where CN does not perform as good as other baselines. First, from the task-incremental experiment in Table 2, DER++ with CN achieved lower FM compared to GN. The reason could be from the DER++’s soft-label loss, which together with GN, overemphasizes on reducing FM and achieved lower FM. On the other hand, CN has to balance between reducing FM and improving LA. Second,
training with data augmentation in the online setting could induce high variations across different runs. From Table 4 and Table 11 (Appendix E.7), we can see that most methods have high standard deviations on the Split CIFAR-10 benchmark, especially with small memory size (500). In such scenarios, there could be insignificant differences between the first and second best methods. It is also worth noting that we only highlight the best averaged results in all our experiments for consistency. Lastly, on the NUS-WIDEseq benchmark, CN has lower evaluation metrics on minority classes compared to BN. One possible reason for this discrepancy is the noisy nature of the NUSWIDE dataset, including the background diversity and huge number of labels per image, which could highly impact the performance of moderate and tail classes.

heading: 7 RELATED WORK

heading: 7.1 CONTINUAL LEARNING
text: Continual learning aims at developing models that learn continuously over a stream of tasks. In literature, extensive efforts have been devoted to develop better continual learning algorithms, ranging from the dynamic architectures (Rusu et al., 2016; Serra et al., 2018; von Oswald et al., 2020; Yoon et al., 2018; Li et al., 2019; Xu & Zhu, 2018), experience replay (Lin, 1992; Chaudhry et al., 2019b; Aljundi et al., 2019a; Riemer et al., 2019; Rolnick et al., 2019; Wu et al., 2019), regularization (Kirkpatrick et al., 2017; Zenke et al., 2017; Aljundi et al., 2018; Ritter et al., 2018) and representation learning approaches (Rebuffi et al., 2017). The main goal is preventing the catastrophic forgetting of older tasks while leveraging the learned knowledge to facilitate the learning of incoming tasks.
Our work takes an orthogonal approach by studying the benefits and drawbacks of BN, which is commonly used in the backbone of most continual learning methods.

heading: 7.2 NORMALIZATION LAYERS
text: Normalizing the network’s intermediate features has been shown to be important in the success of training deep networks in many applications (Huang et al., 2020). The most popular method is the
Batch Normalization (Ioffe & Szegedy, 2015), which has been widely used until this day. Despite its success, BN suffers from two main drawbacks: i) the normalization at training and testing are different, which is problematic when training and testing data follow different distributions; and ii) BN suffers from poor performance with small mini-batch sizes. As a result, there has been a tremendous effort to address these weaknesses such as spatial normalization layers (Ulyanov et al., 2016; Ba et al., 2016; Wu & He, 2018) or only normalizing the the global moments (Ioffe, 2017). Another promising approach is augmenting BN with another spatial normalization layer by blending the moments (Luo et al., 2018; Bronskill et al., 2020) or the normalized features (Nam & Kim, 2018; Pan et al., 2018; Yan et al., 2019) to reduce the inaccurate estimates of the global moments.
Existing works have demonstrated that having a specific normalization layer tailored towards each learning problem is beneficial. However, the continual learning community stills mostly adopt the standard BN in their backbone network. To the best of our knowledge, this is the first work proposing a dedicated continual learning normalization layer.

heading: 8 CONCLUSION
text: In this paper, we investigate the potentials and limitations of BN in online continual learning, which is a common component in most existing methods but has not been actively studied. We showed that while BN can facilitate knowledge transfer by normalizing along the mini-batch dimension, the cross-task normalization effect hinders older tasks’ performance and increases catastrophic forgetting. This limitation motivated us to propose CN, a novel normalization layer especially designed for online continual learning settings. Our extensive experiments corroborate our findings and demonstrate the efficacy of CN over other normalization strategies. Particularly, we showed that CN is a plug-in replacement for BN and can offer significant improvements on different evaluation metrics across different online settings with minimal computational overhead.

heading: ETHIC STATEMENT
text: No human subjects were involved during the research and developments of this work. All of our experiments were conducted on the standard benchmarks in the lab-based, controlled environment. Thus, due to the abstract nature of this work, it has minimal concerns regarding issues such as discrimination/bias/fairness, privacy, etc.

heading: REPRODUCIBILITY STATEMENT
text: In this paper, we conduct the experiments five times and report the mean and standard deviation values to alleviate the randomness of the starting seed. In Appendix E, we provide the full details of our experimental settings. In Appendix F, we provide the pseudo code to implement the proposed method. Lastly, in the supplementary materials, we provide our implementation of CN on the existing DER++ framework (Buzzega et al., 2020).

heading: A APPENDIX ORGANIZATION
text: The Appendix of this work is organized as follows. In Appendix B we detail the evaluation metrics used in our experiments. We provide the details of additional normalization layers that we compared in our experiments in Appendix C. In Appendix D, we discuss how our results related to existing studies. Next, Appendix E provides additional details regarding our experiments, including the information of the benchmarks and additional ablation studies. Lastly, we conclude this work with the implementation in Appendix F.

heading: B EVALUATION METRICS
text: For a comprehensive evaluation, we use three standard metrics to measure the model’s performance: Average Accuracy (Lopez-Paz & Ranzato, 2017), Forgetting Measure (Chaudhry et al., 2018), and Learning Accuracy (Riemer et al., 2019). Let ai,j be the model’s accuracy evaluated on the test set of task Tj after it finished learning the task Ti. The aforementioned metrics are defined as:
• Average Accuracy (higher is better):
ACC(↑) = 1 T T∑ i=1 aT,i. (7)
• Forgetting Measure (lower is better):
FM(↓) = 1 T − 1 T−1∑ j=1 max l∈{1,...T−1} al,j − aT,j . (8)
• Learning Accuracy (higher is better):
LA(↑) = 1 T T∑ i=1 ai,i. (9)
The Averaged Accuracy (ACC) measures the model’s overall performance across all tasks and is a common metric to compare among different methods. Forgetting Measure (FM) measures the model’s forgetting as the averaged performance degrades of old tasks. Finally, Learning Accuracy (LA) measures the model’s ability to acquire new knowledge. Note that in the task free setting, the task identifiers are not given to the model at any time and are only used to measure the evaluation metrics.

heading: C ADDITIONAL NORMALIZATION LAYERS
text: This section covers the additional normalization layers used in our experiments.

heading: C.1 LAYER NORMALIZATION (LN)
text: Layer Normalization Ba et al. (2016) was proposed to address the discrepancy between training and testing in BN by normalization along the spatial dimension of the input feature. Particularly, LN computes the moments to normalize the activation as:
µLN = 1
CWH C∑ c=1 W∑ w=1 H∑ h=1 abcwh
σ2LN = 1
CWH C∑ c=1 W∑ w=1 H∑ h=1 (abcwh − µLN )2 (10)
C.2 INSTANCE NORMALIZATION (IN)
Similar to LN, Instance Normalization Ulyanov et al. (2016) also normalizes along the spatial dimension. However, IN normalizes each channel separately instead of jointly as in LN. Specifically, IN computes the moments to normalize as:
µIN = 1
WH W∑ w=1 H∑ h=1 abcwh
σ2IN = 1
WH W∑ w=1 H∑ h=1 (abcwh − µLN )2 (11)

heading: C.3 BATCH RENORMALIZATION (BRN)
text: Batch Renormalization Ioffe (2017) was proposed to alleviate the non-i.i.d or small batches in the training data. BRN proposed to normalize training data using the running moments estimated so far by the following re-parameterization trick:
aBRN = γ
( r ( a− µBN σBN + ) + d ) + β (12)
where
r =stop grad (
clip[1/rmax,rmax] ( σBN σr )) d =stop grad ( clip[−dmax,dmax] ( µBN − µr
σr
)) ,
where stop grad denotes a gradient blocking operation where its arguments are treated as constant during training and the gradient is not back-propagated through them.

heading: C.4 SWITCHABLE NORMALIZATION (SN)
text: While LN and IN addressed some limitations of BN, their success are quite limited to certain applications. For example, LR is suitable for recurrent neural networks and language applications (Ba et al., 2016; Xu et al., 2019), while IN is widely used in image stylization and image generation (Ulyanov et al., 2016; 2017). Beyond their dedicated applications, BN still performs superior to LR and IN. Therefore, SN (Luo et al., 2018) is designed to take advantage of both LN and IN while enjoying BN’s strong performance. Let Ω = {BN,LN,IN} , SN combines the moments of all three normalization layers and performs the following normalization:
a′ = γ
( a− ∑ k∈Ωwµk√
w′σ2k +
) + β, (13)
where w and w′ are the learned blending factors to combine the three moments and are learned by backpropagation.

heading: D EXISTING REMEDIES FOR THE CROSS-TASK NORMALIZATION EFFECT
text: From Table 1, we can see that the performance gap between BN and BN∗ is much smaller in ER compared to the Single method. Since ER employs an episodic memory, its moments are estimated from both the current data and a small subset of previous memory data. However, there still exists a gap compared to the optimal BN’s moments, as shown by large ∆µ and ∆σ2 values. In literature, there exist efforts to such as herding (Rebuffi et al., 2017) and coreset (Nguyen et al., 2018) to maintain a smaller subset of samples that produces the same feature mean (first moment) as the original data. However, empirically they provide insignificant improvements over the random sampling strategies (Javed & Shafait, 2018; Chaudhry et al., 2019a). We can explain this result by the internal covariate shift property in BN: a coreset selected to match the whole dataset’s feature mean is corresponding to the network parameters producing such features. After learning a new task, the network parameters are changed to obtain new knowledge and become incompatible with the coreset constructed using the previous parameters. Unfortunately, we cannot fully avoid this negative effect without storing all previous data. With limited episodic memory, lost samples will never contribute to the global moments estimated by the network parameters trained on newer tasks.

heading: E ADDITIONAL EXPERIMENTS

heading: E.1 DATASET SUMMARY
text: Table 7 summaries the datasets used throughout our experiments.

heading: E.2 EXPERIMENT SETTINGS
text: This section details the experiment settings in our work.
Toy pMNIST Benchmark In the toy pMNIST benchmark, we construct a sequence of five task by applying a random but fixed permutation on the original MNIST LeCun et al. (1998) dataset to create a task. Each task consists of 1,000 training samples and 10,000 testing samples as the original MNIST dataset. In the pre-processing step, we normalize the pixel value by dividing its value by 255.0. Both the Single and Experience Replay (ER) methods were trained using the Stochastic Gradient Descent (SGD) optimizer with mini-batch size 10 over one epoch with learning rate 0.03. This benchmark follows the “single-head” setting Chaudhry et al. (2018), thus we only maintain a single classifier for all tasks and task-identifiers are not provided in both training and testing.
Online Task-IL Continual Learning Benchmarks (Split CIFAR100 and Split Mini IMN) In the data pre-processing step, we normalize the pixel density to [0−1] range and resize the image size to 3×32×32 and 3×84×84 for the Split CIFAR100 and Split minIMN benchmarks, respectively. No other data pre-processing steps are performed. All methods are optimized by SGD with minibatch size 10 over one epoch. The episodic memory is implemented as a Ring buffer Chaudhry et al. (2019a) with 50 samples per task.
Online Task-IL and Class-IL Continual Learning Benchmarks (Split CIFAR-10 and Split Tiny IMN) We follow the protocol as the DER++ work (Buzzega et al., 2020) except the number of epochs, which we set to one. Particularly, training is performed with data augmentation on both the data stream and the memory samples (random crops, horizontal flips). We follow the configurations provided by the authors, including the hyper-parameters of mini-batch size, the reservoir sampling memory management strategy, learning rates, etc. For the memory size of 2560, which were not conducted in the original paper, we use the same setting as the case of 5120 memory slots.
Online Task-Free Long-Tailed Continual Learning (COCOseq and NUS-WIDEseq) We follow the same settings as the original work Kim et al. (2020) in our experiments. Particularly, we train each method using the Adam optimizer Kingma & Ba (2014) with default hyper-parameters (β1 = 0.9, β2 = 0.999, = 1e − 4), mini-batch size 10 over one epoch. The episodic memory is implemented as a reservoir buffer using the PRS (Kim et al., 2020) strategy with total 2,000 slots.

heading: E.3 COMPARING AMONG CN VARIANTS
text: Recall that our CN design starts with GN without the affine parameters and follows up with BN. We consider an alternative setting of CN, denoted by BN-GN, by performing BN and then GN, in which the affine parameters are added at the GN layer. For each setting, we create a “tied-weight” variant where the affine parameters are also used in the first normalization step. We consider the split Mini IMN with ER and the COCOseq benchmark with PRS and report the result in Table 8. Due to space constraints, we only report the mean results over five runs. For the COCOseq benchmark, we only report the metrics in the Overall category. All variants are performing comparable to our original design on the Split Mini IMN benchmarks in terms of ACC. However, they have higher FM, indicating that normalizing features before BN is beneficial and sharing weights in the first normalizing steps is not helpful. However, on the challenging COCOseq benchmark, CN’s original
design performs much better than other variants across all metrics. Moreover, adding the affine parameters to the first normalization step (tied weight) reduces the normalization properties of the features and reduces overall performance.

heading: E.4 CHANGING RATE OF THE MOMENTS IN BATCH NORMALIZATION
text: We now investigate the changing rate of the moments in BN. We consider the toy MNIST experiment as introduced in Section 4.2 with the naive finetuning model (Single). Throughout training, we measure the difference between the running moments of BN and the optimal moments from BN∗ estimated at the end of each task. Figure 3 plots the changing of each moment from the two-layers MLP backbone throughout the learning of five pMNIST tasks. Notably, only the mean of the second layer remains stable throughout learning. All the other moments, especially from the first layer, quickly diverge as the model observes more tasks. Moreover, the discrepancy in the moments of the first layer causes a difference between the first hidden features, which cascades through the network depth and results in inaccurate final prediction in BN compared to BN∗.

heading: E.5 ADDITIONAL RESULTS OF BN∗
text: We further investigate the performance of BN∗ on the Split-CIFAR100 benchmark and report and report the result in Table 9. Since BN∗ requires calculating the true moments from all observed data, we cannot scale it to the standard benchmark of 17 tasks. Table 9 only consider a sequence of four tasks, which is the maximum memory capacity that our GPU allows. Interestingly, we observe that
GN achieves low FM in this experiment, suggesting that GN does not suffer much from catastrophic forgetting early on during training, or when the sequence of tasks is short. As there are more tasks, we expect the gap among different methods to be more significant as demonstrated throughout our work.

heading: E.6 DECOUPLING THE BENEFITS OF EXPERIENCE REPLAY FROM CN
text: We investigate the benefit of CN while decoupling from the episodic memory for experience replay by considering the naive finetuning method (Single) with different normalization layers and report the results in Table 10.
On both Split CIFAR-100 and Split mini IMN benchmarks, we observe consistent improvements of CN over BN in the final performance (1% on Split CIFAR-100 and 2% on Split mini IMN). This improvement comes from the reduction of catastrophic forgetting (around 0.7% reduction in FM) and the improvements of forward transfer (from 1 to 2% increases in LA). We also observe that GN achieved competitive performance on the Split CIFAR-100 benchmark thanks to its ability to alleviate the cross-task normalization effects compared to other methods that have a BN component. However, GN’s performance falls short on the more challenging Split mini IMN benchmark because it cannot learn individual task well.
Please note that this naive strategy is extremely challenging to train in continual learning because it does not have any mechanism to prevent catastrophic forgetting or support forward transfer. The difference in the first layer normalization layer could cascade through the network depth and result in a huge difference in the final prediction when using BN versus BN∗.

heading: E.7 ADDITIONAL RESULTS OF DER++
text: In Table 11 we results of additional normalization layers of DER++ on the Split CIFAR-10 and Split Tiny IMN benchmarks. This is the full version of the results presented in Section 6.2 and Table 4. Overall, the results are consistent with the current ones in that GN and SN are generally more resistant to catastrophic forgetting (lower FM), but they lack the ability to forward transfer knowledge, resulting in lower overall performance (ACC) compared to our CN, and sometimes BN. The only notable difference is that IN achieved the lowest FM in the Class-IL Tiny IMN benchmark. By inspecting the results, we found that since this benchmark is more challenging, IN struggled to learn individual tasks and only achieved slightly better final ACC than random guessing. Therefore,
IN had significantly lower FM because it did not accumulate much knowledge during training, which is similar to the no-normalization method.

heading: E.8 STANDARD DEVIATIONS OF THE COCOSEQ AND NUS-WIDESEQ BENCHMARKS
text: This section provides the standard deviation (stdev) of results for the long-tailed online continual learning experiments with the COCOseq and NUS-WIDEseq benchmarks in Section 6.3. Table 12 shows the stdev values of the evaluation metrics while Table 13 shows the stdev values of the FM associating with each evaluation metric.
In most cases, our CN has comparable stdev values with BN on both benchmarks. While these values are relatively small on COCOseq (mostly smaller than 4%), the stdev values are higher on the NUS-WIDEseq benchmark, especially for the FM. This phenomenon could happen because the NUS-WIDE dataset is highly noisy, which greatly impacts the continual learning benchmark. Nevertheless, both BN and our CN have similar FM values for most evaluation metrics in this scenario.

heading: F CONTINUAL NORMALIZATION IMPLEMENTATION
text: In the following, we provide the CN’s implementation based on Pytorch (Paszke et al., 2017).
class CN(_BatchNorm): def __init__(self, num_features, eps = 1e-5, G = 32, momentum):
super(_CN, self).__init__(num_features, eps, momentum) self.G = G
def forward(self, input): out_gn = F.group_norm(input, self.G, None, None, self.eps) out = F.batch_norm(out_gn, self.running_mean, self.running_var,
self.weight, self.bias, self.training, self.momentum, self.eps)
return out

Review summaries:
This paper investigates the problem of cross-task normalization in a non-stationary online continual learning(CL) setting. It argues that batch normalization (BN) is important to CL; however, BN in the current form introduces a bias towards current task, leading to catastrophic forgetting. The paper presents a continual normalization(CN) layer to reduce this bias. In particular, it combines spatial and batch normalization into one layer that is suitable for CL. The authors conducted experiments to evaluate the performance of the proposed method. 

The paper considers the problem of activation normalization in neural networks in the context of training on non-i.i.d. data in continual learning. The authors showcase the issue of the "cross-task normalization effect", where the data from a given task is normalized by statistics biased towards another (latest) task. Based on this phenomenon, they show that commonly used normalization schemes, such as Batch Normalization or Group Normalization, suffer from certain problems in this setting (high forgetting, low transfer). They propose the Continual Normalization which combines Group Normalization with Batch Normalization to obtain better results and show the improvements empirically on multiple CL datasets.

The paper studies the role of normalization layers in continual learning. The main argument is that vanilla BatchNorm is not very suitable since the statistics of data change across tasks. To alleviate this, the authors propose the CntinualNorm layer which is essentially the combination of GroupNorm followed by BatchNorm. The experiments compare different normalization schemes across various CL benchmarks.

This paper proposes a novel normalization layer called Continual Normalization for continual learning. In the paper, the authors point out the problem of global moment bias of Batch Norm. To address the problem, the authors combine the advantages of Group Norm and Batch Norm together to improve sharing while reduce forgetting. Comprehensive experiments on various benchmarks and continual settings demonstrate the effectiveness of the proposed method.