Continual Normalization: Rethinking Batch Normalization for Online Continual Learning
Authors: Quang Pham
Chenghao Liu
Steven HOI
Venue: ICLR 2022 Poster (2022)
Keywords: Continual Learning
Batch Normalization

Abstract: Existing continual learning methods use Batch Normalization (BN) to facilitate training and improve generalization across tasks. However, the non-i.i.d and non-stationary nature of continual learning data, especially in the online setting, amplify the discrepancy between training and testing in BN and hinder the performance of older tasks. In this work, we study the cross-task normalization effect of BN in online continual learning where BN normalizes the testing data using moments biased towards the current task, resulting in higher catastrophic forgetting. This limitation motivates us to propose a simple yet effective method that we call Continual Normalization (CN) to facilitate training similar to BN while mitigating its negative effect. Extensive experiments on different continual learning algorithms and online scenarios show that CN is a direct replacement for BN and can provide substantial performance improvements. Our implementation will be made publicly available upon acceptance.

Published as a conference paper at ICLR 2022
CONTINUAL NORMALIZATION: RETHINKING BATCH NORMALIZATION FOR ONLINE CONTINUAL LEARNING
Quang Pham', Chenghao Liu', Steven C.H. Hoi 1.2 Singapore Management University
hqpham. 2017 7 78smu. sgu.sg
2 Salesforce Research Asia
{chenghao. 1iu, shoilesalessorce.ccm
ABSTRACT
Existing continual learning methods use Batch Normalization (BN) to facilitate training and improve generalization across tasks. However, the non-i.i.d and non- stationary nature of continual learning data, especially in the online setting, am- plify the discrepancy between training and testing in BN and hinder the perfor- mance of older tasks. In this work, we study the cross-task normalization effect of BN in online continual learning where BN normalizes the testing data using mo- ments biased towards the current task, resulting in higher catastrophic forgetting. This limitation motivates us to propose a simple yet effective method that we call Continual Normalization (CN) to facilitate training similar to BN while mitigat- ing its negative effect. Extensive experiments on different continual learning algo- rithms and online scenarios show that CN is direct replacement for BN and provide substantial performance improvements. Our implementation is available at /ttttps://github. com/ http://guuuugggeeeeae Cooquang/Contineee Continuat Normalizar ion
INTRODUCTION
Continual learning (Ring 1997) is a promising approach towards building learning systems with human-like capabilities. Unlike traditional learning paradigms, continual learning methods observe a stream tasks and simultaneously perform well on all tasks with limited access to previous data. Therefore, they have to achieve good trade-off between retaining old knowledge (French, 1999) and acquiring new skills, which is referred to as the & Robins 2005). Continual learning has been challenging research problem, especially for deep neural networks because of their ubiquity and promising results on many applications (LeCun al., 2013 Parisi et al. 2019).
dilemma (Abraham
While most previous works focus on developing strategies to alleviate catastrophic forgetting and facilitating knowledge transfer (Parisi et al. 2019), scant attention has been paid to the backbone they used. In standard backbone networks such as ResNets (He et al. 2016), it is natural to use Batch Normalization (BN) (loffe & Szegedy 2015), which has enabled the deep learning community to make substantial progress in many applications (Huang et al. (0200.. Although recent efforts have shown promising results in training deep networks without BN in single task learning (Brock et al. 202T), we argue that BN has a huge impact on continual learning. Particularly, when using an episodic memory, BN improves knowledge sharing across tasks by allowing data of previous tasks to contribute to the normalization of current samples and vice versa. Unfortunately, in this work, we have explored a negative effect of BN that hinders its performance on older tasks and thus increases catastrophic forgetting. Explicitly, unlike the standard classification problems, data in continual learning arrived sequentially, which are not independent and identically distributed (non- i.i.d). Moreover, especially in the online setting (Lopez-Paz & Ranzato 2017), the data distribution changes over time, which is also highly non-stationary. Together, such properties make the BN's running statistics heavily biased towards the current task. Consequently, during inference, the model normalizes previous tasks' data using the moments of the current task, which we refer to as the "cross-task normalization effect".

---

Published as a conference paper at ICLR 2022
Batch Norm
Instance Norm
Group Norm
Continual Norm
H. NW
M
AM
MLW
H4
Figure 1: An illustration of different normalization methods using cube diagrams derived from Wu & He (2018). Each cube represents feature map tensor with N the batch axis, C as the channel axis, and (H.W) as the channel axes. Pixels in blue are normalized by the same moments calculated from different samples while pixels in orange normalized by the same moments calculated from within sample. Best viewed in colors.
Although using an episodic memory can alleviate cross task normalization in BN, is not possi- ble to fully mitigate this effect without storing all past data, which is against the continual learning purposes. On the other hand. spatial normalization layers such as GN (Wu & He 2018 do not perform cross-task normalization because they normalize each feature individually along the spatial dimensions. we will empirically verify in Section [5.]] although such layers suffer less forget- ting than BN, they do not learn individual tasks well as BN because they lack the knowledge transfer mechanism via normalizing along the mini-batch dimension. This result suggests that continual learning normalization layer should balance normalizing along the mini-bate and spatial dimensions to achieve a good trade-off between knowledge transfer and alleviating forgetting. Such properties are crucial for continual learning, especially in the online setting (Lopez-Paz al. 2017 Riemer et al., 2019), but not satisfied by existing normalization layers. Consequently, we propose Continual Normalization (CN), novel normalization layer that takes into account the spatial infor- mation when normalizing along the mini-bus batch dimension. Therefore, our CN enjoys the benefits of BN while alleviating the cross-tass normalization effect. Lastly, our study is orthogonal the continual learning literature, and the proposed CN can be easily integrated into any existing methods improve their performances across different online protocols.
In summary, we make the following contributions. First, we study the benefits of BN and its cross- task normalization effect continual learning. Then, we identify the desiderata of a normalization layer for continual learning and propose CN, novel normalization layer that improves the perfor- mance of existing continual learning methods. Lastly, we conduct extensive experiments to validate the benefits and drawbacks of BN, as well as the improvements of CN over BN.
2 NOTATIONS AND PRELIMINARIES
This section provides the necessary background of continual learning and normalization layers.
2.1 NOTATIONS
We focus on the image recognition problem and denote image as Wx where W. H. C' are the image width, height, and the number of channels respectively. convolutional neural network (CNN) with parameter 0 is denoted as fa(-). A feature map of mini-batch B is defined as a E R BBCxx xH. Finally. each image æ is associated with label y E 21,2.... .2....Y Y) and task
identifier E 21, T), that is optionally revealed to the model, depending the protocol.
2.2 CONTINUAL LEARNING
Continual learning aims at developing models that learn continuously over a stream of tasks. In literature, extensive efforts have been devoted to develop better continual learning algorithms, rang-
ing from the dynamic architectures (Rusu et al. 2016 Serra et al. 2018 von Oswald et al. 2020 Yoon et al. 2018 Li et al. 2019: Xu & Zhu 2018), experience replay (Lin) 1992] Chaudhry et al. AAlubiiAljundi et al. 9a Riemer et al. 2019 Rolnick et al. 2019 Wu u et al. 2019). regulariza- al. 2018 Ritter al. 2018). 8. to fast and
tion (Kirkpatrick et al. 2017 Zenke et al. Aljunddi
slow frameworks (Pham et al.
2020 2021). ). Our work takes an orthogonal approach by studying the benefits and drawbacks of BN, which commonly used in most continual learning methods.
2

---

Published as a conference paper at ICLR 2022
Continual Learning Protocol We focus on the online continual learning protocol (Lopez- Paz & Ranzato 2017) over T tasks. At each step, the model only observes training samples drawn from the current training task data and has to perform well on the testing data of all observed tasks so far. Moreover, data comes strictly in a streaming manner, resulting in a single epoch training setting. Throughout this work, we conduct experiments from the task-incremental (Lopez-Paz & Ranzato 2017), class-incremental (Rebuffi et al. 2017 to the task-free scenarios (Aljundi et al. 2019b).
different continual learning protocols, ranging
2.3 NORMALIZATION LAYERS
Normalization layers are essential in training deep neural networks (Huang et al.) 2020). Ex- isting works have demonstrated that having specific normalization layer tailored towards each learning problem is beneficial. However, the continual learning community stills mostly adopt the standard BN in their backbone network, and lack a systematic study regarding normalization lay- ers (Lomonaco al.] 2020)) To the best of our knowledge, this is the first work proposing a dedicated continual learning normalization layer.
In general, normalization layer takes mini-batch of feature maps (a)...., ..)) as input and perform the Z-normalization as:
+ + (
a' = a'=
where μa and o² are the mean and variance calculated from the features in B, and a small constant added avoid division by zero. The affine transformation's parameters 7, 3 RC ICI-dimensional vectors learned by backpropagation to retain the layer's representation capacity. For brevity, we will use "moments" refer both the mean μl and the variance o².
Batch Normalization BN is one of the first normalization layers that found success in a wide range of deep learning applications (loffe & Szegedy) 22033; SSanturkar al.] 2018) Bjorck et al. 2018). During training, BN calculates the moments across the mini-batch dimension as:
B
W
H
B
W
H
PBN BHW
ΣΣ b
Obouh:
TBN
ΣΣΣ w= H=1
(Obewh - MBN)?
(2)
BHW
w= H=1
hmm
At test time, it is important for BN to be able to make predictions with only one data sample to make the prediction deterministic. As result, BN replaces the mini-batch mean and variance in Eq. ( by an estimate of the global values obtained during training as:
/ - μ + nuu - p), a + o + ((
(3)
where 77 is the exponential running average's 's momentum, which were set 0.1.
Spatial Normalization Layers The discrepancy between training and testing in BN can be prob- lematic when the training mini-batch size is small (Toffe 2017), when the testing data distribution differs from the training distributions. In such scenarios, the running estimate of the mean and variance obtained during training are poor estimate of the moments to normalize the testing data. Therefore, there have been tremendous efforts in developing alternatives to BN to address these challenges. One notable approach is normalizing along the spatial dimensions of each sample inde- pendently, which completely avoid the negative effect from having a biased global moments. In the following, we discuss popular examples of such layers.
Group Normalization (GN) (Wu & He 2018) GN proposes to normalize each feature individu- ally by dividing each channel into groups and then normalize each group separately. Moreover, GN does not normalize along the batch dimension, thus performs the same computation during training and testing. Computationally, GN first divides the channels C' into G groups as:
abghto abelow, where A = T
(4)
Then, for each group the features are normalized with the moments calculated as:
K
W
H
W N
(g) MGN
1
ΣΣΣ
Lightur
(u) 2 OGN
(g) (abghhu MGN)
(5)
h=1
3

---

Published as a conference paper at ICLR 2022
Method
ACC
FM
LA
A
A
(2)
A
(1)
(2)
o²
Single-BN Single-BN"
72.81
18.65
87.74
10.85
0.91
3.69
7.74
75.92
15.13
88.24
ER-BN
80.66
9.34
88.23
3.56
0.46
1.41
3.16
ER-BN"
81.75
8.51
88.46
Table 1: Evaluation metrics on the PMNIST benchmark. The magnitude of the differences are calculated as A = ||ا =BNN-1l1 where w E (1,,02) and k E {1, denotes the first or second BN layer. Bold indicates the best scores.
GN has shown comparable performance to BN with large mini-batch sizes (c.g. 32 or more), while significantly outperformed BN with small mini-batch sizes (e.g. all channels into a single group (setting G = 1), GN is equivalent to Layer Normalization (LN) (Ba et al. (016)). which normalizes the whole layer of each sample. On the other extreme, when separat- ing each channel as group (setting G = C), GN becomes Instance Normalization (IN) (Ulyanov al., 2016), which normalizes the spatial dimension in each channel of each feature. Figure
two). Notably, when putting
provides an illustration of BN, GN, IN, and the proposed CN. which we will discuss in Section
3 BATCH NORMALIZATION IN CONTINUAL LEARNING
3.1 THE BENEFITS OF NORMALIZATION LAYERS IN CONTINUAL LEARNING
We argue that BN is helpful for forward transfer in two aspects. First, BN makes the optimization landscape smoother (Santurkar et al. 2018), which allows the optimization of deep neural networks to converge faster and better (Bjorck et al, 2018). In continual learning, BN enables the model to leam individual tasks better than the no-normalization method. Second, with the episodic memory, BN uses data of the current task and previous tasks (in the memory) to update its running moments. Therefore, BN further facilitates forward knowledge transfer during experience replay: current task data is normalized using moments calculated from both current and previous samples. Compared BN. spatial normalization layers (such as GN) lack the ability to facilitate forward transfer via normalizing using moments calculated from data of both old and current tasks. We will empirically verify the benefits of different normalization layers to continual learning Section 5.1
3.2 THE CROSS- NORMALIZATION EFFECT
Recall that BN maintains running estimate of the global moments normalize the testing data. In the standard learning with a single task where data are i.i.d, one can expect such estimates can well-characterize the true, global moments, and can be used to normalize the testing data. However, online continual learning data is non-i.idd and highly non stationary. Therefore, BN's estimation of the global moments is heavily biased towards the current task because the recent mini-batchin only contain that task's data. As result, during inference, when evaluating the older tasks, BN normalizes previous tasks' data using the current task's moments, which we refer to as the cross- task normalization effect.
We consider a toy experiment on the permuted MNIST (pMNIST) benchmark (Lopez-Paz & Ran- zato 2017 explore the cross-task normalization effect in BN. We construct a sequence of five task, each has 2,000 training and 10,000 testing samples, and a multilayer perceptron(MLP) backbone configured as
(10)), where the number inside the parentheses indicates the output dimension of that layer. We consider the Single and ER strategies for this experiment, where the Single strategy the naive method that trains continuously without any memory or regularization. For each method, we implement an aptimal-at-test-ting BN variant (denoted by the suffix -BN*) that calculates the global moments using all training data of all tasks before testing. Compared to BN, BN has the same parameters and only differs in the moments used to normalize the testing data. We emphasize that although BN is unrealistic, it sheds light on how cross-task normalization affects the performance of CL algorithms. We report the eraged accuracy at the end of training ACC(t) (Lopez-Paz et (017). forgetting measure FM(J) (Chaudhry et al. 2018), and learning accuracy LA(() (Riemer al. 2019) in this experiment.
(1000))
4

---

Published as a conference paper at ICLR 2022
Table reports the evaluation metries of this experiment. Besides the standard metrics, we also report the moments' difference magnitudes between the standard BN and the global BN variant. We observe that the gap between two BN variants is significant without the episodic memory. When using the episodic memory, ER can reduce the gap because the PMNIST is quite simple and even the ER strategy can achieve close performances to the offline model (Chaudhry et al. (2019h). Moreover, it is important to note that training with BN on imbalanced data also affects the model's is learning dynamics, resulting in suboptimal parameter. Nevertheless, having an unbiased estimate of the global moments can greatly improve overall accuracy given the same model parameters. Moreover, this improvement is attributed to reducing forgetting rather than facilitating transfer: BN has lower FM but almost similar LA compared to the traditional BN. In addition, we observe that the hidden features' variance becomes inaccurate compared to the global variance as we go to deeper layers (A < x² (2)) This result agrees with recent finding (Ramasesh et al. 2021 that deeper layers are more responsible for causing forgetting because their hidden representation deviates from the model trained on all tasks. Overall, these results show that normalizing older tasks using the current task's s moments causes higher forgetting, which we refer to as the "cross-task normalization effect".
3.3 DESIDERATA FOR CONTINUAL LEARNING NORMALIZATION LAYERS
While BN facilitates continual learning, it also amplifies catastrophic forgetting by causing the cross- task normalization effect. To retain the BN's benefits while alleviating its drawbacks, we argue that an ideal normalization layer for continual learning should be adaptive by incorporating each feature's 's statistics into its normalization. Being adaptive can mitigate the cross-task normalization effect because each sample is now normalized differently test time instead of being normalized by set of biased moments. In literature, adaptive normalization layers have shown promising results when the mini-batch sizes are small (Wu He 2018), or when the number of training data extremely limited (Bronskill et al.] (020). In such cases, normalizing along the spatial dimensions of each feature can alleviate the negative effect from an inaceurate estimate of the global moments. Inspired by this observation, we propose the desiderata for a continual learning normalization layer:
Facilitates the performance of deep networks by improving knowledge sharing within and across-task (when the episodic memory is used), thus increasing the performance of all tasks (ACC in our work);
Is adaptive at test time: each data sample should be normalized differently. Moreover, each data sample should contribute to its normalized feature's statistic, thus reducing catas- trophic forgetting;
Does not require additional input at test time such as the episodic memory, or the task identifier.
To simultancously facilitate training and mitigating the cross-task normalization effect, normal- ization layer has to balance between both across mini-batch normalization and sathin-sample nor- malization. As we will show in Section 5. BN can facilitate training by normalizing along the mini-batch dimension; however, it is not adaptive at test time and suffers from the cross-task nor- malization effect. On the other hand, GN is fully adaptive, but it does not facilitate training compared to BN. Therefore, is imperative to balance both aspects to improve continual learning performance. Lastly, we expect continual learning normalization layer can be direct replacement for BN, thus, it should not require additional information to work, especially at test time.
4 CONTINUAL NORMALIZATION (CN)
CN works by first performing a spatial normalization on the feature map, which we choose to be group normalization. Then, the sroup-normalized features are further normalized by batch nor- malization layer. Formally, given the input feature map a, we denote BN1,0 and GN1, as the batch
normalization and group normalization layers without the affine transformation parameters" CN obtains the normalized features acN
aGN + GN): aCN + 7BN,,,NNN,.. + B.
(6)
which is equivalent to setting 7 and B
5

---

Published as a conference paper at ICLR 2022
Table Evaluation metries of different normalization layers on the Split CIFARI00 and Split Mini IMN benchmarks. Bold indicates the best averaged scores, suffix indicates non-adaptive method
ER
Split CIFARI00
Split Mini IMN
Norm. Layer
ACC(T)
FM(()
LA(1)
ACC(t)
FM(J)
LA(1)
NoNL
55.87
4.46:10.48
57.26-0.64
47.40
3.17::0.9
45.31
BN
64.97+1.09
9.24
71.56=0.75
59.09#1.74
8,57+1.52
65.244:..2
BRN
13.47
8.43
69.83-22.22
54.55
58.53+ 6.66
IN
59.17
11.47-+0.92
69.40-10.33
48.74411.98
1..28
6..88
GN
63.42:10.22
7.39#1.24
68.03=0.19
55.65+2.92
8.31=1000
59.25=0.72
SN
64.79-0.88
7.92:0.64
71.10=0.51
56.84+1.37
10.11=1.46
64.09=1.53
CN(ours)
0.84
7.29
0..2
64.28
8.08
70.90
DER+
Split CIFARI00
Split Mini IMN
Norm. Layer
ACC(T)
FM())
LA(T)
ACC(t)
FM(J)
LA(1)
NoNL
07.1410.46
4.46:10.48
57.26:10.64
47.1843.20
2.77
45.01
BN
66.50+2.52
8.58
73.78+1.02
61.08
6.90
66.10
BRN
6.22
4.28
73.30+0008
57.37+
6.66
66.53++1.66
IN
61.18=0.96
10.59:0.77
71.00=0.57
54.05=1.226
11.82.+1.22
65.03+1.69
GN
66.58=0.27
5.70110.6
69.63=1.12
60.50
6.17
13.10
SN
67.17=0.23
6.01+0.15
72.13+0.23
57.73++1.97
8.92+1.84
63.87++064
CN(ours)
0.51
6.48+0081
0..49
66.29+
6.47+1.46
71.75:10.68
In the first step, GN component does not use the affine transformation make the intermediate feature BN ((ax) well-normalized across the mini-batch and spatial dimensions. Moreover, per- forming GN first allows the spatial-normalizer features contribute to the BN's running statistic, which further reduce the cross-task normalization effect. An illustration of CN is given in Figure]
By its design, one can see that CN satisfies the desiderata of continual learning normalization layer. Particularly, CN balances between facilitating training and alleviating the cross-task normalizing effect by normalizing the input feature across mini- batch and individually. Therefore, CN is adaptive at test time and produces normalzeed features in the mini- mini-batch and spatial dimensions, which strikes great balance between BN and other instance- baaaad normalizers. Lastly, CN uses the same input conventional normalization layers and does not introduce extra learnable parameters that can be prone to catastrophic forgetting.
We now discuss why CN is more suitable normalization layer for continual learning than recent advanced normalization layers. SwitchNorm (Luo et al. 2018) proposed to combine the moments from three normalization layers: BN, IN, and LN normalize the feature map (Eq. 13 However, the blending weights w and w' make the output feature not well-normalized in any dimensions since such weights smaller than one, which scales down the moments. Moreover, the choice of IN and LN may not be helpful for image recognition problems. Similar to SN, TaskNorm (Bronskill et al., 2020) combines the moments from BN and IN by blending factor, which is learned for each task. As result, TaskNorm also suffers in that its outputs are not vell-normalized. Moreover, TaskNorm addresses the meta learning problem requires knowing the task identifier our third criterion of requiring additional information compared to BN and our CN.
time, which violates
EXPERIMENT
We evaluate the proposed CN's performance compared to a suite of normalization layers with a focus on the online continual learning settings where it is more challenging to obtain good global moments for BN. Our goal is to evaluate the following hypotheses: (i) BN can facilitate knowledge transfer better than spatial normalization layers such as GN and SN (ii) Spatial normalization layers have lower forgetting than BN; (iii) CN can improve over other normalization layers by reducing catastrophic forgetting and facilitating knowledge transfer; (iv) CN is direct replacement of BN without additional parameters and minimal computational overhead. Due to space constraints, we briefly mention the setting before each experiment and provide the full details in Appendix D)

---

Published as a conference paper at ICLR 2022
5.1 ONLINE TASK-INCREMENTAL CONTINUAL LEARNING
Setting We first consider the standard online, task- insremnntetttlll continual learning setting Lopezz Paz & Ranzato 2017) on the Split CIFAR-100 100 and Split Mini IMN benchmarks. We follow the standard setting in Chaudhry et al. (2099a) to split the original CIFAR 100 (Krizhevsky & Hinton 2009) or Mini IMN [Vinyals et 2016) datasets into a sequence of 20 tasks, three of which are used for thyper-paramette cross validation,, and the remaining 17 tasks are used for continual learning,
We consider two continual learning strategies: (i) Experience Replay (ER) (Chaudhry et al. (019b); and (ii) Dark Experience Replay++ (DER++) (Buzzega et al. 2020). Besides the vanilla ER, we also consider DER++ a recent improved ER variants, to demonstrate that our proposed CN can work well across different ER-based strategies. All methods use standard ResNet 18 backbone (He ct al., 2016) (not pre-trained) and are optimized over one epoch with batch size 10 using the SGD optimizer. For each continual learning strategies, we compare our proposed CN with five competing normalization layers: (i) BN (Toffe & Szegedyl (013)); (ii) Batch Renormalization (BRN) (loffe)
2017] (iii) IN (Ulyanov al. 2016); (iv) GN (Wu & He 2018); and (v) SN (Luo et al. 2018). We
cross-validate and set the number of groups to be G = 32 for our CN and GN in this experiment.
Results Table 2 reports the evaluation metrics of different normalization layers on the Split CIFAR-100 and Split Mini IMN benchmarks. We consider the averaged accuracy at the end of training ACC(t) (Lopez-Paz et al. (017), forgetting measure FM(J) (Chaudhry et al. 2018). and learning accuracy LA(T) (Riemer et al., 2019), details are provided in Appendix [A] Clearly, IN does not perform well because it is not designed for image recognition problems. Compared to adaptive normalization methods such as GN and SN, BN suffers from more catastrophic forgetting (higher FM(J) ) but at the same time can transfer knowledge better across tasks (higher LA(1) Moreover, BRN performs worse than BN since it does not address the biased estimate of the global moments, which makes normalizing with the global moments during training ineffective. Overall, the results show that although traditional adaptive methods such as GN and SN do not suffer from the tross-task normalization effect and enjoy lower FM()) values, they lack the ability to facilitate knowledge transfer across tasks, which results in lower LA(T). Moreover, BN can facilitate knowl- edge sharing across tasks, but it suffers more from forgetting because of the cross-task normalization effect. Across all benchmarks, our CN comes out as clear winner by achieving the best overall performance (ACC(1)) ). This result shows that CN can strike a great balance between reducing catastrophic forgetting and facilitating knowledge transfer to improve continual learning.
Complexity Comparison We study the com- plexity of different normalization layers and reporting the training time on the Split CI- FARI00 benchmark in Tabe 3 Both GN and our CN have minimal computational overhead compared BN, while SN suffers from addi- tional computational cost from calculating and normalizing with different sets of moments.
Table 3: Running time ER on the Split CI- FAR100 benchmarks of different normalization layers. A% indicates the percentage increases of training time over BN
BN
GN
SN
CN
Time (s) A %
1583 0%
1607
2036
1642
1.51%
28.61% %
3.72%
5.2 ONLINE CLASS-INCREMENTAL CONTINUAL LEARNING
Setting We consider the online task incremental (Task-IL) and class incremental (Class-II learn- ing problems on the Split-CIFAR10 (Split CIFAR-10) and Split-Tiny-maggeNet (Split Tiny IMN) benchmarks. We follow the same experiment setups as Buzzega al. (2020) except the number of training epochs, which we set to one. All experiments uses the DER++ strategy (Buzzega et al.) 2020) ResNet 18 (He et al. 2016) backbone (not pre-trained) trained with data augmentation using the SGD optimizer. We consider three different total episodic memory sizes of 500, 2560, and 5 5120, and focus on comparing BN with our proposed CN in this experiment.
Result reports the ACC and FM metrics on the Split CIFAR-10 and Split Tiny IMN bench- marks under both the Task-IL and Class-IL settings. For CN, we consider two configurations with the number of groups being G = 8 and G 32. In most cases, CN outperforms the standard BN on both metrics, with a more significant gap on larger memory sizes of 2560 and 5120. Interestingly, BN is highly unstable in the Split CIFAR-10, Class-II. benchmark with high standard deviations. In
7

---

Published as a conference paper at ICLR 2022
Table 4: Evaluation metries of DER++ with different normalization layers on the Split CIFAR-10 and Split Tiny IMN benchmarks. Parentheses indicates the number of groups in CN. Bold indicates the best averaged scores
Split CIFAR-10
Split Tiny IMN
Buffer Method
Class-IL
Task-IL
Class-IL
Task-IL
ACC(1)
FM())
ACC(T)
FM(J)
ACC(1)
FM())
ACC(1)
FM(()
BN
48.914.5
33.615.88
82.6.2.3
3.2.19
6.710.1
38.1100.5
40.2.10.9
6.511.0
500
CN (G 8)
48.910.3
27.9.5.00
84.7.0.5
2.2.1.33
7.310.9
37.5.1..3
42.2.12.1
4,9.2.2
CN (G = 32)
51.7.1199
28.2.14.0
86.2.1.2
2.3.1.3
6.5.0.77
40.111.7
40.6.11.
6.8.1.99
BN
52.3.:4.6
29.711..1
8.66
0.9
11.2.:2.3
36.0.12.00
50.811.7
2.8.1.22
2560
CN (G = 8)
53.71:3.
25.5
87.3-2.7
1.641.66
10.74:1.3
38.04:1.2
52.6::1.2
1.5.1.55
CN (G = 32)
27.3-1-2.0
21.6:1:.66
8..4-1.1
1.7.1.11
11.9:00.3
36.71:12
51.5+10.2
2.8:10.9
BN
52.0+
20.7++10.3
85.6+3.3
2.0::.55
11.24:2.7
36.84-200
52.2+1177 2.6
5120
CN (G = 8)
5..1+4.00
24.0++4.0
87.1-288
0.7
12.240.66
34,64-2.6
53.1
3.1+1.99
CN (G 32)
57.9+4.1
22,2++100
0.9
1.3+0.9
12,2+-0,2
35.6+
54.9+1.5
1.5
50
ACC
40
CN(G=37) CN(G-B)
CNIG-321
CNIG-8)
BN
BN
2
3
2
4
7
9
10
Task
Task
(a) Split CIFAR -10, Task-IL
(b) Split Tiny IMN, Task-IL
Figure 2: The evolution of ACC(f) on observed tasks so far on the Split CIFAR- and Split Tiny IMN benchmarks, Task-IL screnario with DER++ and memory size of 5120 samples.
contrast, both CN variants show better and more stable results, especially in the more challenging Class-IL setting with a small memory size (indicated by small standard deviation values). We also report the evolution of ACC in Figure 2 Both CN variants consistently outperform BN throughout training, with only one exception at the second task on the Split Tiny IMN. CN (G 32) shows more stable and better performances than BN and CN(G' 8).
5.3 LONG-TAILES TAILED ONLINE CONTINUAL LEARNING
We now evaluate the normalization layers on the challenging task-free, long-tailed continual learn- ing setting (Kim et al. 2020), which is more challenging and realistic since real-world data usually follow long-tailed distributions. We consider the PRS strategy (Kim et al. (2000) and the COCOseq and NUS-W NUS-WIDESee WSDEe benchmarks. which consists of four and six tasks, respectively. Unlike the previ- ous benchmarks, images the COCOseg and NUS WIDEseq benchmarks can have multiple labels, resulting in loig-tailed distribution over each task's 's image label. Following Kim et al. (020), we report the average overall F1 (O-F1), per-class F1 (C-F1), and the mean average precision (mAP) at the end of training and their corresponding forgetting measures (FM). We also report each metric over the minority classes (<200 samples), moderate classes (200-900 samples), majority classes (>900 samples), and all classes. Empirically, we found that smaller groups helped in the long-tailed setting because the moments were calculated over more channels, reducing the dominants of head classes. Therefore, use G 4 groups in this experiment.
We replicate PRS with BN to compare with our CN and report the results Table and Table We observe consistent improvements over BN from only changing the normalization layers, especially in reducing FM()) across all classes.
8

---

Published as a conference paper at ICLR 2022
Table Evaluation metrics of the PRS strategy on the COCOseg and NUS-WIDEseq benchmarks. We report the mean performance over five runs. Bold indicates the best averaged scores
Majority
Moderate
Minority
Overall
COCOseq
C-F1
O-F1
mAP
C-F1
O-F1
mAP
C-F1
O-F1
mAP
C-F1
O-F1
mAP
BN
64.2
58.3
66.2
51.2
48.1
55.7
31.6
31.4
38.1
51.8
48.6
54.9
CN(ours)
64.8
58.5
66.8
52.5
49.2
55.7
35.7
35.5
38.4
53.1
49.8
55.1
Majority
Moderate
Minority
Overall
NUS-WIDEseq
C-F1
O-FI
mAP
C-FI
O-FI
mAP
C-F1
O-FI
mAP
C-F1
O-FI
mAP
BN
24.3
16.1
21.2
16.2
16.5
20.9
28.5
28.2
32.3
23.4
20.9
25.7
CN(ours)
25.0
17.2
22.7
17.1
17.4
21.5
27.3
27.0
31.0
23.5
21.3
25.9
Table 6: Forgetting measure (FM(() ) of each metric from the PRS strategy on the COCOseq and NUS-WIDESSE benchmarks, lower is better. We report the mean performance five runs. Bold indicates the best averaged scores
Majority
Moderate
Minority
Overall
COCOseq
C-F1
O-F1
mAP
C-F1
O-F1
mAP
C-F1
O-F1
mAP
C-F1
O-F1
mAP
BN
23.5
22.8
8.4
30.0
30.2
9.4
36.2
35.7
13.2
29.7
29.5
9.7
CN(ours)
23.5
23.1
6.5
26.9
26.9
7.4
26.8
26.5
12.2
25.6
25.7
8.0
Majority
Moderate
Minority
Overall
NUS-WIDEseq
C-F1
O-F1
mAP
C-F1
O-F1
mAP
C-F1
O-F1
mAP
C-F1
O-F1
mAP
BN
54.6
50.7
12.5
62.2
61.9
15.5
52.5
52.4
12.2
57.6
55.7
11.2
CN(ours)
52.7
48.7
8.6
58.8
58.0
10.3
51.2
50.6
11.6
57.4
55.5
8.1
5.4 DISCUSSION OF THE RESULTS
Our experiments have shown promising results for CN being potential replacement for BN in line continual learning. While the results are generally consistent, there are a few scenarios where CN does perform as good other baselines. First, from the task-incremental experiment in Ta- 2 DER++ with CN achieved lower FM compared to GN. The reason could be from the DER++'s loft-label loss, which together with GN, overemphasizes on reducing FM and achieved lower FM. On the other hand, CN has to balance between reducing FM and improving LA. Second, training with data augmentation in the online setting could induce high variations across different runs. Ta- ble 4 shows that most methods have high standard deviations on the Split CIFAR- 10 benchmark, especially with small memory sizes. In such scenarios, there could be insignificant differences be- tween the first and second best methods. Also, on the evaluation metrics on minority classes than BN. One possible reason is the noisy nature of the NUS- WIDE dataset, including the background diversity and huge number of labels per image, which could highly impact the tail classes' performance. Lastly, CN introduces an additional (number of groups), which needs to be cross-validated for optimal performance.
benchmark, CN has lower
CONCLUSION
In this paper, we investigate the potentials and limitations of BN in online continual learning, which common component in most existing methods but has not been actively studied. We showed that while BN can facilitate knowledge transfer by normalizing along the mini-batch dimension, the cross-task normalization effect hinders older tasks' performance and increases catastrophic forget- ting. This limitation motivated to propose CN, novel normalization layer especially designed for online continual learning settings. Our extensive experiments corroborate our findings and demon- strate the efficacy of CN over other normalization strategies. Particularly. showed that CN is plug-in in replacement for BN and can offer significant improvements on different evaluation metrics across different online settings with minimal computational overhead.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENT
The first author is supported by the SMU PGR scholarship. We thank the anonymous Reviewers for the constructive feedback during the submission of this work.
ETHIC STATEMENT
No human subjects were involved during the research and developments of this work. All of our experiments were conducted on the standard benchmarks in the lab-based, controlled environment. Thus, due to the abstract nature of this work, it has minimal concerns regarding issues such as discrimination//iis/aiiiinss privacy, etc.
REPRODUCIBILITY STATEMENT
In this paper, we conduct the experiments five times and report the mean and standard deviation values to alleviate the randomness of the starting seed. In Appendix D we provide the full details of our experimental settings.
REFERENCES
Wickliffe C Abraham and Anthony Robins. Memory retention-th the synaptic stability versus plastic- ity dilemma. Trends neurosciences, 28(2):73-78, 2005.
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Confer- ence on Computer Vision (ECCV), pp. 139-154, 2018.
Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia. Online continual learning with maximal interfered retrieval. In Advances 1889-11860, 2019a.
in Neural Information Processing Systems,
Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars. Trsk-free continual learning. In Pro- ceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 11254-11263, 2019b.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.064550 2016.
Johan Bjorck, Carla Gomes, Bart Selman, and Kilian Q Weinberger. Understanding batch normal- ization. In Proceedings of the 32nd International Conference Neural Information Processing Systems, pp. 7705-7716, 2018.
Andrew Brock, Soham De, and Samuel L Smith. Characterizing signal propagation to close the performance gap in unnormalized resnets. arXiv preprint arXiv:2101.08992 2021.
John Bronskill, Jonathan Gordon, James Requeima, Sebastian Nowozin, and Richard Turner. Tas- knorm: Rethinking batch normalization for meta-learning. In International Conference Ma- chine Learning, pp. 1153-1164. PMLR, 2020.
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: a strong, simple bascline. In 34th Conference on Neural Information Processing Systems (NeurIPS 2020), 2020.
Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. Proceedings of the European Conference Computer Vision (ECCV), pp. 532-547, 2018.
Arslan Chaudhry, Mare' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. International Conference on Learning Representations (ICLR), 2019a.
10

---

Published as a conference paper at ICLR 2022
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan. Puneet K Dokania, Philip HS Torr and Marc' Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv preprint arXiv:1902.10488 2019b.
Robert French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3(4):128-135, 1999.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Lei Huang, Jie Qin, Yi Zhou, Fan Zhu, Li Liu, and Ling Shao. Normalization techniques in training dnns: Methodology, analysis and application. arXiv preprint arXiv:2009.12836 2020.
Sergey loffe. Batch renormalization: towards reducing minibatch dependence in batch normalized models. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 1942-19500 2017.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 148-456. PMLR, 2015.
Khurram Javed and Faisal Shafait. Revisiting distillation and incremental classifier learning. In Asian conference computer vision, pp. 3-17. Springer, 2018.
Chris Dongjoo Kim, Jinseo Jeong, and Gunhee Kim. Imbalanced continual learning with partition- ing reservoir sampling. In European Conference Computer Vision, pp. 411-428, Springer, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.69880 2014.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska. et al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 2017.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical report, Citeseer, 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (6(11):2278-222 1998.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521((553):436 444, 2015.
Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to grow: continual structure learning framework for overcoming catastrophic forgetting. In International Conference on Machine Learning, pp. 3925-3934, 2019.
Long-Ji Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine learning, 8(3-4):293-311, 1992.
Vincenzo Lomonaco, Davide Maltoni, and Lorenzo Pellegrini. Rehearsal-free continual learning over small non-iid batches. In CVPR Workshops. pp. 989-998, 2020.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, pp. 6467-6476, 2017.
David Lopez-Paz, Robert Nishihara, Soumith Chintala, Bernhard Scholkopf, and Léon Bottou. Dis- covering causal signals in images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6979-6987, 2017.
11

---

Published as a conference paper at ICLR 2022
Ping Luo, Jiamin Ren, Zhanglin Peng. Ruimao Zhang, and Jingyu Li. Differentiable learning-tor normalize via switchable normalization. International Conference on Learning Representa- fions, 2018.
Cuong V Nguyen, Yingzhen Li, Thang D Bui, and Richard Turnnr. Variational continual learning. International Conference on Learning Representations (ICLR), 2018.
German I Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 2019.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS-W, W, 2017.
Quang Pham, Chenghao Liu, Doyen Sahoo, et al. Contextual transformation networks for online
continual learning. In International Conference Learning Representations, 2020.
Quang Pham, Chenghao Liu, and Steven Hoi. Dualnet: Continual learning, fast and slow. Advances in Neural Information Processing Systems, 34, 2021.
Vinay V Ramasesh, Ethan Dyer, and Maithra Raghu, Anatomy of catastrophic forgetting: Hidden representations and task semantics. International Conference on Learning Representations, 2021.
Sylvestre- Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interfer- ence. International Conference on Learning Representations (ICLR), 2019.
Mark Ring. Child: first step towards continual learning. Machine Learning, 28(1):77-104 1997.
Hippolyt Ritter, Aleksandar Botev, and David Barber. Online structured laplace approximations for overcoming catastrophic forgetting. Advances in Neural Information Processing Systems, pp- 3738-3748. 2018
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. In Advances in Neural Information Processing Systems, pp. 348- 358. 2019.
Andrei Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint 1.0vv:6006.4400 2016.
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch nor- malization help optimization? In Proceedings of the 32nd International Conference Neural Information Processing Systems, pp.
2018
Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention the task. In Proceedings of the 35th International Conference on Machine Learning-Vont Volume 80, pp. 4548-4557. JMLR. org, 2018.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing in- gredient for fast stylization. arXiv preprint arXiv: 607.0802 2016.
Dmitry Ulyanov, Andrea Vedaldi, and Vietor Lempitsky. Improved texture networks: Maximizing quality and diversity in feed-forward stylization and texture synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6924-6932, 2017.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in neural information processing systems, pp. 3630-3638, 2016.
12

---

Published as a conference paper at ICLR 2022
Johannes von Oswald, Christian Henning, João Sacramento, and Benjamin F Grewe. Continual learning with hypernetworks. International Conference on Learning Representations (ICLR), 2020.
Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 374 382, 2019.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pp. 3-19, 2018.
Jingjing Xu, Xu Sun, Zhiyuan Zhang, Guangxiang Zhao, and Junyang Lin. Understanding and improving layer normalization. arXiv preprint
2019.
Ju Xu and Zhanxing Zhu. Reinforced continual learning. In Advances in Neural Information Pro- cessing Systems, pp. 899-908, 2018
Jachong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. International Conference on Learning Representations (ICLR), 2018.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In Proceedings of the 34th International Conference on Machine Learning-Vonn Volume 70, pp. 3987- 3995. JMLR. org, 2017.
APPENDIX ORGANIZATION
The Appendix of this work is organized as follows. In Appendix A we detail the evaluation metrics used in our experiments. We provide the details of additional normalization layers that we compared in our experiments in Appendix B In Appendix C we discuss how our results related to existing studies. Next, Appendix [D] provides additional details regarding our experiments, including the information of the benchmarks and additional ablation studies. Lastly, conclude this work with the implementation Appendix
A EVALUATION METRICS
For comprehensive evaluation,
three standard metrics measure the model's performance: Average Accuracy (Lopez-Paz & Ranzato 2017). Forgetting Measure (Chaudhry et al. 2018), and Learning Accuracy (Riemer al.] 2019).. Let a., be the model's 's accuracy evaluated the test set of task 7j after it finished learning the task T. The aforementioned metrics are defined as:
Average Accuracy (higher is better):
1 ACC(t) T
ari
1=1
Forgetting Measure (lower better):
max FM(J) T (1(....T-))
aljj - QT.
(8)
Learning Accuracy (higher is better):
1 LA(t)
T Σ Oisi-
(9)
The Averaged Accuracy (ACC) measures the model's overall performance across all tasks and is common metric to compare among different methods. Forgetting Measure (FM) measures the model's 's forgetting as the averaged performance degrades of old tasks. Finally, Learning Accuracy (LA) measures the model's is ability to acquire knowledge. Note that in the task free setting, the task identifiers are not given to the model at any time and are only used to measure the evaluation metrics.
13

---

Published as a conference paper at ICLR 2022
B ADDITIONAL NORMALIZATION LAYERS
This section covers the additional normalization layers used our experiments.
B.1 LAYER NORMALIZATION (LN)
Layer Normalization Ba et al. (016) was proposed to address the discrepancy between training and testing in BN by normalization along the spatial dimension of the input feature. Particularly, LN computes the moments to normalize the activation as:
C
W
H
1
ULN CWH
ahownh
h=1
LN
CWH
abowh [LLN
(10)
w=1k=l
B.2 INSTANCE NORMALIZATION (IN)
Similar to LN, Instance Normalization Ulyanov et al. (016)) also normalizes along the spatial di- mension. However, IN normalizes each channel separately instead of jointly as in LN. Specifically, IN computes the moments to normalize as:
W
PIN
Obeanh
WH
h=1
W
H
i
Odowc MLLN )
(11)
WH
w=1 hm1
B.3 BATCH RENORMALIZATION (BRN)
Batch Renormalization loffe 2017) was proposed to alleviate the non-i.i.d or small batches in the training data. BRN proposed normalize training data using the running moments estimated so far by the following re-parameterization trick:
QBRN 7
77
a PBN
)
+d)
(12)
OBN T +e
where
JBN
r-stop-grad
clip
Or
HBN f4,
=stop-grad clipp-dmar
Or
where stop-grad denotes gradient blocking operation where arguments are treated as con- stant during training and the gradient is not back-propagated through them.
B.4 SWITCHABLE NORMALIZATION (SN)
While LN and IN addressed some limitations of BN, their success are quite limited to certain applica- tions. For example, LR is suitable for recurrent neural networks and language applications (Ba et al.) 2016 et al. 2019), while IN widely used in image stylization and image generation (Ulyanov al. 2016: 2017). Beyond their dedicated applications, BN still performs superior to LR and IN. Therefore, SN (Luo et al. 2018 is designed to take advantage of both LN and IN while enjoying BN's strong performance. Tet 52 (BN,LN,IN) SN combines the moments of all three normal- ization layers and performs the following normalization:
a - Σken wiih
a' r
B,
(13)
V +
14

---

Published as a conference paper at ICLR 2022
Table
Dataset summary
#task
img size
#training imgs
#testing imgs
#classes
Section
PMNIST
28 x88
10,000
50,000
10
Section 3.2
Split CIFAR10
5
3x32x32
50,000
10,000
10
Section 5.2
Split CIFAR100
20
3x84x84
50,000
10,000
100
Section 5.
Split Mni IMN
20
3x84x84
50,000
10,000
100
Section 5.
Split Tiny IMN
10
3x64x64
100,000
10,000
200
Section 5.2
COCOseg
4
3x224x224
35,072
6,346
70
Section 5.3
NUS- WIDEseq
6
3x224x224
48,724
2,367
49
Section
where w and w' are the learned blending factors to combine the three moments and are learned by backpropagation.
C EXISTING REMEDIES FOR THE CROSS-TASK NORMALIZATION EFFECT
From Table we can see that the performance gap between BN and BN is much smaller in ER compared to the Single method. Since ER employs an episodic memory, its moments estimated from both the current data and small subset of previous memory data. However, there still exists a gap compared the optimal BN's moments, as shown by large Aμ and A values. In litera- ture, there exist efforts to such as herding (Rebuffi et al. (017) and coreset (Nguyen et al., 2018) to maintain a smaller subset of samples that produces the same feature mean (first moment) as the original data. However, empirically they provide insignificant improvements over the random sam- pling strategies (Javed Shafait 2018 Chaudhry al. 2019ah. We can explain this result by the internal covariate shift property in BN: coreset selected match the whole dataset's 's feature mean corresponding to the network parameters producing such features. After learning a new task, the network parameters changed obtain new knowledge and become incompatible with the coreset constructed using the previous parameters. Unfortunately. we cannot fully avoid this negative effect without storing all previous data. With limited episodic memory, lost samples will never contribute to the global moments estimated by the network parameters trained on newer tasks.
D ADDITIONAL EXPERIMENTS
D.1 DATASET SUMMARY
Table summaries the datasets used throughout our experiments.
D.2 EXPERIMENT SETTINGS
This section details the experiment settings in our work.
Toy PMNIST Benchmark In the toy PMNIST benchmark, construct sequence of five task by applying random but fixed permutation on the original MNIST LeCun et al. (1998) dataset to create task. Each task consists of 1,000 training samples and 10,000 testing samples the original MNIST dataset. In the pre-processing step, we normalize the pixel value by dividing its value by 255.0. Both the Single and Experience Replay (ER) methods were trained using the Stochastic Gradient Descent (SGD) optimizer with mini- batch size 10 over one epoch with learning rate 0.03. This benchmark follows the "single-head" setting [Chaudhry al. (2018), thus we only maintain single classifier for all tasks and task-identifiers are not provided in both training and testing.
Online Task-IL Continual Learning Benchmarks (Split CIFAR100 and Split Mini IMN) In the data pre-processing step, we normalize the pixel density to [0- (0-1) range and resize the image size to x 32 x 32 and x 84 x 84 for the Split CIFAR100 and Split minIMN benchmarks, respectively. No other data pre- processein steps performed. All methods optimized by SGD with mini- batch size 10 over one epoch. The episodic memory implemented as a Ring buffer Chaudhry et al. (019a) with 50 samples per task.
15

---

Published as a conference paper at ICLR 2022
Table 8: Evaluation metries of CN variants on the Split Mini IMN and COCOseq benchmarks (overall classes). We report the mean results over five runs. Bold indicates highest score, "tw" indicates tied weight in the affine parameters
Split Mini IMN
COCOseq
CN Variant
ACC
FM
LA
C-F1
O-F1
mAP
GN + (Nroriinall
64.28
8.08
70.90
53.1
49.8
55.1
BN +GN
62.63
9.92
71,43
51.5
47.9
53.8
GN +BN+t
62.17
9.21
70.17
51.1
47.7
53.3
BN +N+W
62.23
8.20
69.30
50.0
47.4
52.5
Online Task-IL and Class-IL Continual Learning Benchmarks (Split CIFAR-10 10 and Split Tiny IMN) We follow the protocol as the DER++ work (Buzzega al. 2020) except the number of epochs, which we set to one. Particularly, training is performed with data augmentation on both the data stream and the memory samples (random crops, horizontal flips). We follow the configurations provided by the authors, including the hyper-parameters of mini-batch size, the reservoir sampling memory management strategy, learning rates, etc. For the memory size of 2560, which were not conducted in the original paper, we use the same setting the case of 5120 memory slots.
Online Task-Free Long-Tailed Continual Learning (COCOseg and NUS-WIDEseg) We fol- low the same settings as the original work Kim et al. (2020) in our experiments. Particularly, we train each method using the Adam optimizer Kingma & Ba (014) with default hyper-parameters (31 = 0.9, 32 = 0.999, = Le 4). mini-batch size 10 over one epoch. The episodic memory is implemented reservoir buffer using the PRS (Kim al. 2020 strategy with total 2,000 slots.
D.3 COMPARING AMONG CN VARIANTS
Recall that our CN design starts with GN without the affine parameters and follows up with BN. We consider an alternative setting of CN, denoted by BN-GN, by performing BN and then GN, in which the affine parameters are added at the GN layer. For each setting, we create "tied-weight" variant where the affine parameters are also used in the first normalization step. We consider the split Mini IMN with ER and the COCOseq benchmark with PRS and report the result in Table 3 Due to space constraints, we only report the mean results over five runs. For the COCOseq benchmark, only report the metrics in the Overall category. All variants original design on the Split Mini IMN benchmarks in terms of ACC. However, they have higher FM, indicating that normalizing features before BN is beneficial and sharing weights in the first normalizing steps is not helpful. However, on the challenging COCOseq benchmark, CN's original design performs much better than other variants across all metries. Moreover, adding the affine parameters to the first normalization step (tied weight) reduces the normalization properties of the features and reduces overall performance.
performing comparable to
D.4 CHANGING RATE OF THE MOMENTS IN BATCH NORMALIZATION
We now investigate the changing rate of the moments in BN. We consider the toy MNIST experiment as introduced Section 3.2 with the naive finetuning model (Single). Throughout training, we measure the difference between the running moments of BN and the optimal moments from BN estimated at the end of each task. Figure plots the changing each moment from the two-layers MLP backbone throughout the learning of five PMNIST tasks. Notably, only the mean of the second layer remains stable throughout learning. All the other moments, especially from the first layer, quickly diverge the model observes more tasks. Moreover, the discrepancy in the moments of the first layer causes a difference between the first hidden features, which cascades through the network depth and results in inaccurate final prediction in BN compared to BN
16

---

Published as a conference paper at ICLR 2022
Changes of moments throughout training
Mean Layer
STDEV Layer
Mean Layer
STDEV Layer
10.000
8.000
6.000
4,000
2.000
2
4
Task
Figure 3: Changes of moments (LI distance y axis) between BN and BN throughout training on
2-layers MLP backbone. Standard deviation values omitted due small values (e.g. < 0.1)
Table 9: Comparison with BN* on the 4-tasks Split CIFAR-100 benchmarks. Bold indicates the best averaged scores
Split CIFAR-100 (4 tasks)
ER
ACC
FM
LA
BN
60.14:3..47
6.21 1.99
63.98-12.38
BN
61.38
5.87+11.77
6.701
GN
55.96:12.43
2.23:10.9
53.25-2..44
CN
62.1810.56
5.66110.76
64.9411.68
D.5 ADDITIONAL RESULTS OF BN*
We further investigate the performance of BN on the Split-CIFAR100 benchmark and report and report the result in Ta Since BN requires calculating the true moments from all observed data, we cannot scale to the standard benchmark of 17 tasks. Jable consider sequence of four tasks, which is the maximum memory capacity that our GPU allows. Interestingly, we observe that GN achieves low FM this experiment, suggesting that GN does not suffer much from catastrophic forgetting early during training, when the sequence of tasks is short. As there more tasks, we expect the gap among different methods to be more significant as demonstrated throughout our work.
D.6 DECOUPLING THE BENEFITS OF EXPERIENCE REPLAY FROM CN
We investigate the benefit of CN while decoupling from the episodic memory for experience replay by considering the naive finetuning method (Single) with different normalization layers and report the results in Table [0
On both Split CIFAR-100 and Split mini IMN benchmarks, we observe consistent improvements of CN over BN in the final performance (1% Split CIFAR-100 and 2% on Split mini IMN). This improvement comes from the reduction of catastrophic forgetting (around 0.7% reduction FM) and the improvements of forward transfer (from 1 to 2% increases in LA). We also observe that GN achieved competitive performance on the Split CIFAR- 100 benchmark thanks to its ability to alleviate the cross-task normalization effects compared to other methods that have BN component.
17

---

Published as a conference paper at ICLR 2022
Table 10: Evaluation metrics of the naive single strategy on the task-aware Split CIFAR-100 and Split mini IMN benchmarks. Bold indicates the best averaged scores
Split CIFAR-100
Split mini IMN
Single
ACC
FM
LA
ACC
FM
LA
BN
33.47::..24
33.93+2.23
65.7412.22
32.26::2.42
29.76:10.98
57.20
BRN
28.74:12.53
31.78:13.06
58.02::1.94
25.60:10.93
25.77=2.00
49.43::1.77
GN
36.66:+2.32
17.40++2.33
52.62+1.611
28.05=1.177
12.97
39.60=
CN
34.52+2.62
33.27++1.23
66.37:11.00
34.23++1.37
29.05
61.11+1.661
Table 11: Evaluation metrics of DER+ with different normalization layers on the Split CIFAR-10 and Split Tiny IMN benchmarks. Parentheses indicates the number of groups in CN. Bold indicates the best averaged scores. This is the full version of Table 4
Split CIFAR-10
Split Tiny IMN
Buffer Method
Class-IL
Task-IL
Class-IL
Task-IL
ACC(f)
FM(()
ACC(t)
FM(()
ACC(1)
FM(L)
ACC(t)
FM(()
48.9.14.5
33.6115.8
82.6.1.33
3.2.1.99
6.7.1.11
38.1100.5
40.2.10.9
6.5.1.0
IN
35.9+2.0
45.013.6
78881.6
1.840.9
2.2:0.55
16.9+0.9
20.6=00.
1.7
GN
46.8+5.1
11.7
82.155.2
1.212
7.240.4
36.0+2.4
41.1=0.55
4.5
500
SN
42.3#3.9
32.3111.6
80.6#3.4
2.6+2.0
4.5+1.0
25.0+2.4
33.6=1.3
2.5
CN (G=8)
48.9+0.3
27.9+.5.0
84.7+-0.5
2.2.0.33
7.3+0.9
37.5+2.3
42.24:2.1
4.9+2.22
CN (G=32)
51.7++1.9
28.2+44.0
86.2+2.2
2.0
6.5+0.7
40.1+1.7
40.6+1144
6.8+199
52.314.6
29.716.1
86.611.6
0.910.7
11.24223
36.01:200
50.8+1.7
2.21.2
IN
36.14-2.8
37.84888
79.1+1.1
0.++1.
2.4+0.4
19.2+1.5
25.4+1.7
0.6+0.5
GN
53.3+33.5
28.8+332
87.3+2.4
0,6
11.0+1..
35.5+1.2
50.9+3.0
1.9+1.6
2560
SN
42.3+33.9
30.8+10.4
80.0#6.44
4.7+5.2
4.4+1.4
27.3+229
37.6+3.0
3.5
CN (G=8)
53.7.33.4
25.5.14.7
87.312.7
1.6.1.6
10.711.3
38.0.1.22
52.61122
1.510.5
(G=32)
57.3.1200
21.611..6
88.411.1
1.711.1
11.910.3
36.711.2
51.5.102
2.8.0.9
BN
52.0+7.8
26.7++10.3
85.6+3.3
2.011.5
11.2+2.7
36.8+2.0
52.2+1.7
2.6
IN
32.2.14.5
41.5413.6
77.213.8
2.112.33
2.3.1..6
18.0
25.5
1.611.9
GN
48.6.12.6
32.244.8
86.510.8
0.5
9.241.8
36,811.6
48.0.16.0
4.9.6.33
5120
SN
42.918.9
36.24-8.3
81.142.4
3.6.1..4
4.341..6
23.34:19
37.913.0
2.74199
CN (G=8)
54.1#4.0
24.0=14.0
87.12288
0.7
12.2:0.6
34.6#2.6
53.1=18
3.1
(G=32)
57.914.1
22.21110
88.310.9
1.310.9
12.210.2
35.611..
54.911.5 1.5111
However, GN's performance falls short on the more challenging Split mini IMN benchmark because it cannot learn individual task well.
Please note that this naive strategy is extremely challenging to train in continual learning because it does not have any mechanism to prevent catastrophic forgetting or support forward transfer. The difference the first layer normalization layer could cascade through the network depth and result huge difference in the final prediction when using BN versus BN
D.7 ADDITIONAL RESULTS OF DER++
In Table II we results of additional normalization layers of DER++ on the Split CIFAR-10 and Split Tiny IMN benchmarks. This is the full version of the results presented in Section 5.2 and Table Overall, the results are consistent with the current ones in that GN and SN are generally more resistant to catastrophic forgetting (lower FM), but they lack the ability to forward transfer knowledge, resulting in lower overall performance (ACC) compared to our CN, and sometimes BN. The only notable difference is that IN achieved the lowest FM in the Class-IL Tiny IMN benchmark. By inspecting the results, we found that since this benchmark is more challenging, IN struggled to learn individual tasks and only achieved slightly better final ACC than random guessing. Therefore, IN had significantly lower FM because it did not accumulate much knowledge during training, which similar the no-normalization method.
18

---

Published as a conference paper at ICLR 2022
Table 12: Effects of different moving average strategies on the Split CIFAR-100 benchmark, with ER+BN
Moving Average
Split CIFAR-100
ACC
FM
LA
CMA
N/A
35.86=10.90
20.24::..0
45.53:13.65
17 = 0.01
62.90:12.03
6.32+12.2
1..
17 0.05
63.61#1.11
7.18
68.43:+0.87
EMA
7 = 0.1
64.97+11009
9.24+1.99
71.56:+0.75
1/ = 0.5
62.4342..65
8.54
68.8
11 = 0.9
61.46:42.36
9.03:+2.38
16.81+t.49
D.8 ANALYSIS OF THE MOVING AVERAGE IN BN
We now explore how the moving average affects BN. First, we consider the cumulative moving average (CMA) strategy which keeps the standard average of the statistics for the first n mini-batches of data as:
21
22
+
ÆCMA
(14)
where TCMA is the CMA moving average statistic, D, is the mini-batch statistic.
Next, we consider the standard exponential moving average strategy (EMA). For consistency with the Pytorch framework, we refer to the update of the running statistics in BN as
nnew = (1 7) x á + x 2t,
(15)
where é is the current running statistic, and æ; is the current mini-batch statistic. WIth this formula- tion, lower 7) values assign higher weights to older statistics, while higher v values will focus more on recent statistics.
We report the results of CMA versus different EMA's 7) configurations on the Split CIFAR-100 benchmark with ER under the task-aware setting in Table 12 From the results, we draw the follow- ing conclusions:
The standard CMA strategy yields undesirable performance and is worse than EMA. The reason is that when using CMA, the network parameters at the carly stages of training con- tribute equally to CMA moments compared to the latest parameter, which is problematic because the early parameters are not well-trained and do not observe the newer tasks' sam- ples. When using the exponential running average (the standard practice), the contributions of early parameters to the running moments degrades exponentially compared to the recent, well-trained parameters.
For BN with EMA, low values of π 0.01 changing rate, which makes the EMA statistics more stable and results less forgetting (indicated by lower FM). However, these settings cannot take advantage of the statistics from the recent model's 's parameters and result in lower forward transfer (lower LA), which results in worse overall performance (lower ACC).
0.05 slow the running statistics'
On the other hand, with > 0.1. the running statistics can be too sensitive to the noisy updates from mini-batches, resulting in worse performance than the standard setting of ") = 0.1
also worth noting that any improvements with the EMA moving average can also benefit our CN via adapting its BN component accordingly. It is also interesting to note that our BN strategy equivalent to not estimating a running averaged but calculating these statistics at the time of evaluation using the most recent model.
D.9 STANDARD DEVIATIONS OF THE COCOSEO AND NUS- WIDESEQ BENCHMARKS
This section provides the standard deviation (stdev) of results for the long-tailed online continual learning experiments with the COCOseg and
benchmarks in Section 5.3 Table 13
19

---

Published as a conference paper at ICLR 2022
Table 13: Standard deviations of the evaluation metrics of the PRS strategy on the COCOseq and NUS- NUS-WIDEseq benchmarks
Majority
Moderate
Minority
Overall
COCOseq
C-F1
O-F1
mAP
C-F1
O-F1
mAP
C-F1
O-F1
mAP
C-F1
O-F1
mAP
BN
0.77
1.33
1.04
0.98
1.28
1.04
2.35
2.33
1.78
0.89
1.24
1.07
CN(ours)
1.20
1.97
1.12
0.61
1.27
0.66
1.73
1.53
0.59
0.64
1.22
0.63
Majority
Moderate
Minority
Overall
NUS-WIDEseq
C-F1
O-FI
mAP
C-FI
O-FI
mAP
C-F1
O-FI
mAP
C-F1
O-FI
mAP
BN
1.01
1.97
0.91
1.72
1.77
0.68
1.95
1.78
1.44
0.47
0.27
0.55
CN(ours)
2.38
1.59
1.33
2.78
3.06
0.97
1.72
1.05
0.58
1.26
1.40
0.34
Standard deviations of the forgetting measure (FM()) ) of each metric from the PRS strategy on the COCOseq and NUS-WIDEseq benchmarks, lower is better
Table 14:
Majority
Moderate
Minority
Overall
COCOseq
C-F1
O-FI
mAP
C-F1
O-F1
mAP
C-F1
O-FI
mAP
C-FI
O-FI
mAP
BN
3.77
4.60
0.44
3.11
2.80
1.63
4.63
4.32
2.34
2.63
2.93
1.29
CN(ours)
3.31
3.62
2.53
1.46
1.54
1.72
5.72
6.32
0.80
1.26
1.34
1.34
Majority
Moderate
Minority
Overall
JUS-WIDEseq
C-F1
O-F1
mAP
C-FI
O-FI
mAP
C-FI
O-FI
mAP
C-FI
O-FI
mAP
BN
7.35
8.19
1.49
6.60
7.03
1.35
13.34
13.24
4.14
4.71
5.11
1.08
CN(ours)
7.59
6.60
4.16
8.89
9.14
3.77
7.35
7.15
7.42
2.00
2.38
3.08
shows the stdev values the evaluation metrics while Table 14 shows the stdev values of the FM associating with each evaluation metric.
In most cases, our CN has comparable stdev values with BN on both benchmarks. While these values are relatively small on COCOseq (mostly smaller than 4%), the stdev values are higher on the NUS-WE WIDEseq benchmark, especially for the FM. This phenomenon could happen because the NUS-WIDE dataset highly noisy, which greatly impacts the continual learning benchmark. Nevertheless, both BN and our CN have similar FM values for most evaluation metrics in this scenario.
E CONTINUAL NORMALIZATION IMPLEMENTATION
In the following, provide the CN's implementation based on Pytorch (Paszke et al. 2017).
class CN (_BatchNorm)
def _init_ (self, num num_features, eps = 1e-5, G 32, momentum) super C_N,, self) . self.G
.affe__.iin init (num features, eps, momentum)
def forward(self, input):
out_gn = F w..rrop_noommmf (input, self.G, None, None, self.eps) out F. b.batch_norm (out_gn, remf..ruminggmmnn relffraaning_varr weighteight self.bias, self.training self momentum, self.eps) return out
20


Reviews:
[{"review_id": "s6J_M-eOiR7", "reviewer": "Reviewer_BLEs", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper focuses on an important problem of activation normalization which to my knowledge has not yet been properly explored in the continual learning setting. While a lot of effort has been directed towards proposing better algorithms on top of existing network architectures, not that much research has been done in the direction of fixing the architectures themselves and in particular fixing specific parts of the architectures, such as the normalization schemes.\n- The practicality and relevance of the proposed technique are high. Continual Normalization is very easy to use and adapt to existing architectures, which makes the paper useful for the community. \n- The improvements offered by the method are mostly consistent between different tested settings, with few exceptions.\n- The empirical evaluation provided in the paper is extensive and authors consider multiple important perspectives, such as the computational complexity of CN (the running time in Table 3), how it behaves in different CL settings (Task-IL, Class IL), and how an \"oracle method\" performs (BN*).\n\nWeaknesses:\n- Although the paper provides an empirical investigation of the issue of normalization in a continual learning setting, it would benefit from a more in-depth study of how different normalization schemes affect the results. Currently, the paper mostly shows that the proposed method works in practice and offers some intuitions to justify their good performance (BN being useful for forward transfer, GN being useful for reducing forgetting), but in my opinion, there is much more to understand here. Why is BN so useful for forward transfer? Why do the \"forgetful\" properties of BN vanish after applying GN? How fast do the BN statistics change after each task? I feel like questions like those have not yet been answered satisfactorily.\n- Although I find the empirical evaluation overall good, there are several points that could be improved or discussed more thoroughly.\n  - I think the \"oracle\" baseline from Table 1, which re-calculates statistics from all training data, is very interesting. However, it is not used in the subsequent experiments, where, in my opinion, it could be helpful in understanding the bigger picture. Additionally, I wanted to ask - do you consider this baseline to be the upper bound of what can be achieved with normalization?\n  - In Table 3 you consider many different normalization schemes, but after that, you only show results with BN and CN. I think that understanding how different normalization schemes impact the results in continual learning is an important part of this work and should be included in more experiments (e.g. by putting the extended versions of the results in the appendix).\n  - I don't think the paper properly acknowledges certain shortcomings of CN in the experimental section. I think that the overall results are satisfactory, but at the same time, there should be a discussion of the limitations of the method. The paper doesn't really discuss the fact that CN's results are worse for the minority classes of the NUS-WIDEseq dataset, or the fact that sometimes the difference of the mean result of CN and BN is not much higher than the difference of standard deviation, posing a question about the significance of the results. To reiterate, I don't think the results overall are bad, but I would suggest discussing the limitations more openly.\n  - The paper does not provide the standard deviation of the results for the case of the COCOSeq benchmark. The authors write that \"Due to space constraints, we only report the mean results over five runs.\", but as the appendix does not have space constraints, it would be useful to add this information.\n  - The paper doesn't consider the impact of the hyperparameter $\\eta$ from Equation (3), which controls how fast the running statistics are changing. Since it directly impacts the moments, it possibly could have a high impact on the cross-task normalization effect and the results in the CL settings. Unfortunately, this is not discussed. In fact, if I'm not mistaken, the paper doesn't even state what value of $\\eta$ is used in the experiments. Another possible option would be to consider calculating a standard mean instead of a running mean.\n\nMinor comments:\n- Section 2 should probably be removed or merged with Section 3 (only two sentences, referring to contents of Section 3)\n- Section 4, second sentence: \"For convenient\" -> \"For convenience\"\n- It would be interesting to consider the non-online continual learning setting (i.e. multiple epochs).", "summary_of_the_paper": "The paper considers the problem of activation normalization in neural networks in the context of training on non-i.i.d. data in continual learning. The authors showcase the issue of the \"cross-task normalization effect\", where the data from a given task is normalized by statistics biased towards another (latest) task. Based on this phenomenon, they show that commonly used normalization schemes, such as Batch Normalization or Group Normalization, suffer from certain problems in this setting (high forgetting, low transfer). They propose the Continual Normalization which combines Group Normalization with Batch Normalization to obtain better results and show the improvements empirically on multiple CL datasets.", "summary_of_the_review": "The research questions asked in this paper are very interesting and go in an orthogonal direction to most of the CL research, which could be beneficial for the CL community. The proposed solution was evaluated empirically with satisfactory results and it's very easy to implement, making it a useful tool for CL researchers and practitioners. On the other hand, the paper would benefit from a more thorough investigation of the problem and fixing certain issues in terms of empirical evaluation. As such, I would say that at the moment it is marginally above the acceptance threshold.\n\n**EDIT after the discussion period:** During the rebuttal process the authors provided important clarifications and introduced significant improvements to the paper. As such, I have decided to increase my score to 8 (accept, good paper). ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "P6OS-a464v0", "reviewer": "Reviewer_9JHA", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strong Points\n\n* The paper takes one of the most import issues in continual learning: non-stationary online CL setting. For me, the problem itself is real and practical.\n\n* The proposed approach is reasonable and addresses the problem of cross-task normalization in CL setting.\n\n* The proposed adaptive normalization layers can be a plug-in replacement of BN.\n\n* The authors made extensive comparison between CN and prior work such as BN, IN, and GN.\n\n* Overall, the paper is well written. In particular, the Related Work section has a nice flow and puts the proposed method into context. Despite the method having limited novelty, the method has been well motivated by pointing out the limitations in SOTA methods.\n\n* The authors provide code for reproducing the results in the paper.\n\nWeak Points\n\n* The proposed adaptive normalization layer is a straight-forward combination of spatial and batch normalization. So the novelty is limited.\n\n* Table 2, in DER++ setting, CN's FM score (for catastrophic forgetting) is not as good as GN's. Please explain the reason for that.\n", "summary_of_the_paper": "This paper investigates the problem of cross-task normalization in a non-stationary online continual learning(CL) setting. It argues that batch normalization (BN) is important to CL; however, BN in the current form introduces a bias towards current task, leading to catastrophic forgetting. The paper presents a continual normalization(CN) layer to reduce this bias. In particular, it combines spatial and batch normalization into one layer that is suitable for CL. The authors conducted experiments to evaluate the performance of the proposed method.\n", "summary_of_the_review": "Overall, I vote for marginally accepting. I like the idea of cross-task normalization and handling it by the proposed adaptive normalization layer. My major concern is about the limited novelty of the paper and the performance of CN on catastrophic forgetting (see weakness above). Hopefully the authors can address my concern in the rebuttal period.\n\n[After rebuttal]\nAfter reading the other reviewers' comments and the authors' rebuttal, I confirm my rating.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "D9qIaKlgl4b", "reviewer": "Reviewer_8iRv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n\n1- I like how the paper is structured. It is well-motivated and well-written.   \n2- The arguments are sound, supported by experimental design.\n\n\n### Weaknesses \n\n1- I expected more details in the earlier sections of the paper regarding the effect of normalization schemes on the role of batchnorm. For instance, in Table 1, it would be interesting to see what would the performance metrics be if we don't use BatchNorm at all. So, generally, I suggest extending Section 4.1 with more emphasis on the performance when we use normalization versus when we don't use normalization.\n\n2- It would have been more interesting to disentangle the role of normalization from replay buffers. Table 1 suggests the benefit of BN* diminishes when we are using the replay method, and Table 2 doesn't report the results on the Naive Finetuning (or Single in the paper). So, if possible, I encourage the authors to add more results (for MNIST and CIFAR) for the Naive(Single) method in the appendix. \n\n3- While the arguments about normalization layers are sound and intuitive, the reasons behind the benefit/drawbacks of different normalization layers are still not studied beyond the fact that the statistics change. I believe it is important to study \"how\" these changes in statistics change performance. For instance, how would normalization schemes impact the forward/backward transfer?\n\n\n-----------\n### Update After Discussion Period\n\nI believe the authors have addressed my previous comments and I increase my initial score.", "summary_of_the_paper": "The paper studies the role of normalization layers in continual learning. The main argument is that vanilla BatchNorm is not very suitable since the statistics of data change across tasks. To alleviate this, the authors propose the CntinualNorm layer which is essentially the combination of GroupNorm followed by BatchNorm. The experiments compare different normalization schemes across various CL benchmarks.", "summary_of_the_review": "I believe the paper is good and intuitive, however needs some improvements that can improve its contributions.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5ZnklMadQN1", "reviewer": "Reviewer_9jXz", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The paper is well-written and easy to understand.\n2. The idea of Continual Normalization is novel under the continual learning setting.\n3. Multiple continual learning settings, including task-incremental, class-incremental and long-tailed continual learning, make the experiments part comprehensive.\n\n\nWeaknesses and Questions:\n1. Section 2 is weird, maybe section 3 should be section 2.1?\n2. In (6), the Batch Norm operation still exists, so the cross task normalization effect is not directly addressed. The intuition of why an \"adaptive normalization\" at test time could reduce forgetting (or the cross task normalization effect) should be elaborated better.\n3. It's nice to see the method worked with replay based method (Table 2, 4). However, I still wonder the performance of CN when there is no replay involved. As replay in some sense already alleviates the cross-task normalization effect, it would be better to decouple the influence of CN and replay.\n4. (minor) According to the paper, the method can combine the merits of both BN and GN under the continual learning setting. But the method itself is not limited to continual learning problem. Any intuition about how CN would perform under the i.i.d. supervised learning setting?", "summary_of_the_paper": "This paper proposes a novel normalization layer called Continual Normalization for continual learning. In the paper, the authors point out the problem of global moment bias of Batch Norm. To address the problem, the authors combine the advantages of Group Norm and Batch Norm together to improve sharing while reduce forgetting. Comprehensive experiments on various benchmarks and continual settings demonstrate the effectiveness of the proposed method.", "summary_of_the_review": "The proposed Continual Normalization is novel, however, the reasoning and intuition behind the idea should be elaborated better. Moreover, additional experiments without replay-based method can better demonstrate the effectiveness of the method.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper sheds light on issues with BN in continual learning and proposes a quite simple, which is a strength, solution to fix it. 

The Authors first draw attention to the fact that using recalculated moments boosts performance and reduces forgetting, which serves as an argument that at least partially BN contributes to catastrophic forgetting in continual learning. Given that BN remains quite important in certain application areas such as vision, it is a strong motivation for the paper.

The experiments are thorough and clearly show that CN is a practically relevant alternative to BN in continual learning.

One weakness of the paper is that the method is poorly motivated, and relatedly, it has quite limited novelty. CN combines the strengths and weaknesses of BN and GN. Hence, it is not clear why it outperforms both, given that it still has the issue of BN that normalization statistics might become outdated. This is one of the weaknesses pointed out by 9jXz who recommended rejecting the paper. It would be also nice to compare to Mode Normalization https://openreview.net/forum?id=HyN-M2Rctm. 

Other papers have suggested changing normalization for sequential learning. Changing batch normalization (to batch renormalization) was investigated in [1] in the context of continual learning. Relatedly, [2] proposes TaskNorm for meta-learning.

Despite these issues, it is a solid contribution and it is my pleasure to recommend acceptance. In the camera-ready, please describe more clearly the design principles behind CN.

[1] Rehearsal-Free Continual Learning over Small Non-I.I.D. Batches, https://openaccess.thecvf.com/content_CVPRW_2020/papers/w15/Lomonaco_Rehearsal-Free_Continual_Learning_Over_Small_Non-I.I.D._Batches_CVPRW_2020_paper.pdf

[2] TaskNorm: Rethinking Batch Normalization for Meta-Learning, https://arxiv.org/abs/2003.03284
Decision: Accept (Poster)