Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper studies the role of normalization layers in continual learning. The main argument is that vanilla BatchNorm is not very suitable since the statistics of data change across tasks. To alleviate this, the authors propose the CntinualNorm layer which is essentially the combination of GroupNorm followed by BatchNorm. The experiments compare different normalization schemes across various CL benchmarks.
main_review: ### Strengths

1- I like how the paper is structured. It is well-motivated and well-written.   
2- The arguments are sound, supported by experimental design.


### Weaknesses 

1- I expected more details in the earlier sections of the paper regarding the effect of normalization schemes on the role of batchnorm. For instance, in Table 1, it would be interesting to see what would the performance metrics be if we don't use BatchNorm at all. So, generally, I suggest extending Section 4.1 with more emphasis on the performance when we use normalization versus when we don't use normalization.

2- It would have been more interesting to disentangle the role of normalization from replay buffers. Table 1 suggests the benefit of BN* diminishes when we are using the replay method, and Table 2 doesn't report the results on the Naive Finetuning (or Single in the paper). So, if possible, I encourage the authors to add more results (for MNIST and CIFAR) for the Naive(Single) method in the appendix. 

3- While the arguments about normalization layers are sound and intuitive, the reasons behind the benefit/drawbacks of different normalization layers are still not studied beyond the fact that the statistics change. I believe it is important to study "how" these changes in statistics change performance. For instance, how would normalization schemes impact the forward/backward transfer?


-----------
### Update After Discussion Period

I believe the authors have addressed my previous comments and I increase my initial score.
summary_of_the_review: I believe the paper is good and intuitive, however needs some improvements that can improve its contributions.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper considers the problem of activation normalization in neural networks in the context of training on non-i.i.d. data in continual learning. The authors showcase the issue of the "cross-task normalization effect", where the data from a given task is normalized by statistics biased towards another (latest) task. Based on this phenomenon, they show that commonly used normalization schemes, such as Batch Normalization or Group Normalization, suffer from certain problems in this setting (high forgetting, low transfer). They propose the Continual Normalization which combines Group Normalization with Batch Normalization to obtain better results and show the improvements empirically on multiple CL datasets.
main_review: Strengths:
- The paper focuses on an important problem of activation normalization which to my knowledge has not yet been properly explored in the continual learning setting. While a lot of effort has been directed towards proposing better algorithms on top of existing network architectures, not that much research has been done in the direction of fixing the architectures themselves and in particular fixing specific parts of the architectures, such as the normalization schemes.
- The practicality and relevance of the proposed technique are high. Continual Normalization is very easy to use and adapt to existing architectures, which makes the paper useful for the community. 
- The improvements offered by the method are mostly consistent between different tested settings, with few exceptions.
- The empirical evaluation provided in the paper is extensive and authors consider multiple important perspectives, such as the computational complexity of CN (the running time in Table 3), how it behaves in different CL settings (Task-IL, Class IL), and how an "oracle method" performs (BN*).

Weaknesses:
- Although the paper provides an empirical investigation of the issue of normalization in a continual learning setting, it would benefit from a more in-depth study of how different normalization schemes affect the results. Currently, the paper mostly shows that the proposed method works in practice and offers some intuitions to justify their good performance (BN being useful for forward transfer, GN being useful for reducing forgetting), but in my opinion, there is much more to understand here. Why is BN so useful for forward transfer? Why do the "forgetful" properties of BN vanish after applying GN? How fast do the BN statistics change after each task? I feel like questions like those have not yet been answered satisfactorily.
- Although I find the empirical evaluation overall good, there are several points that could be improved or discussed more thoroughly.
  - I think the "oracle" baseline from Table 1, which re-calculates statistics from all training data, is very interesting. However, it is not used in the subsequent experiments, where, in my opinion, it could be helpful in understanding the bigger picture. Additionally, I wanted to ask - do you consider this baseline to be the upper bound of what can be achieved with normalization?
  - In Table 3 you consider many different normalization schemes, but after that, you only show results with BN and CN. I think that understanding how different normalization schemes impact the results in continual learning is an important part of this work and should be included in more experiments (e.g. by putting the extended versions of the results in the appendix).
  - I don't think the paper properly acknowledges certain shortcomings of CN in the experimental section. I think that the overall results are satisfactory, but at the same time, there should be a discussion of the limitations of the method. The paper doesn't really discuss the fact that CN's results are worse for the minority classes of the NUS-WIDEseq dataset, or the fact that sometimes the difference of the mean result of CN and BN is not much higher than the difference of standard deviation, posing a question about the significance of the results. To reiterate, I don't think the results overall are bad, but I would suggest discussing the limitations more openly.
  - The paper does not provide the standard deviation of the results for the case of the COCOSeq benchmark. The authors write that "Due to space constraints, we only report the mean results over five runs.", but as the appendix does not have space constraints, it would be useful to add this information.
  - The paper doesn't consider the impact of the hyperparameter $\eta$ from Equation (3), which controls how fast the running statistics are changing. Since it directly impacts the moments, it possibly could have a high impact on the cross-task normalization effect and the results in the CL settings. Unfortunately, this is not discussed. In fact, if I'm not mistaken, the paper doesn't even state what value of $\eta$ is used in the experiments. Another possible option would be to consider calculating a standard mean instead of a running mean.

Minor comments:
- Section 2 should probably be removed or merged with Section 3 (only two sentences, referring to contents of Section 3)
- Section 4, second sentence: "For convenient" -> "For convenience"
- It would be interesting to consider the non-online continual learning setting (i.e. multiple epochs).
summary_of_the_review: The research questions asked in this paper are very interesting and go in an orthogonal direction to most of the CL research, which could be beneficial for the CL community. The proposed solution was evaluated empirically with satisfactory results and it's very easy to implement, making it a useful tool for CL researchers and practitioners. On the other hand, the paper would benefit from a more thorough investigation of the problem and fixing certain issues in terms of empirical evaluation. As such, I would say that at the moment it is marginally above the acceptance threshold.

**EDIT after the discussion period:** During the rebuttal process the authors provided important clarifications and introduced significant improvements to the paper. As such, I have decided to increase my score to 8 (accept, good paper).
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel normalization layer called Continual Normalization for continual learning. In the paper, the authors point out the problem of global moment bias of Batch Norm. To address the problem, the authors combine the advantages of Group Norm and Batch Norm together to improve sharing while reduce forgetting. Comprehensive experiments on various benchmarks and continual settings demonstrate the effectiveness of the proposed method.
main_review: Strengths:
1. The paper is well-written and easy to understand.
2. The idea of Continual Normalization is novel under the continual learning setting.
3. Multiple continual learning settings, including task-incremental, class-incremental and long-tailed continual learning, make the experiments part comprehensive.


Weaknesses and Questions:
1. Section 2 is weird, maybe section 3 should be section 2.1?
2. In (6), the Batch Norm operation still exists, so the cross task normalization effect is not directly addressed. The intuition of why an "adaptive normalization" at test time could reduce forgetting (or the cross task normalization effect) should be elaborated better.
3. It's nice to see the method worked with replay based method (Table 2, 4). However, I still wonder the performance of CN when there is no replay involved. As replay in some sense already alleviates the cross-task normalization effect, it would be better to decouple the influence of CN and replay.
4. (minor) According to the paper, the method can combine the merits of both BN and GN under the continual learning setting. But the method itself is not limited to continual learning problem. Any intuition about how CN would perform under the i.i.d. supervised learning setting?
summary_of_the_review: The proposed Continual Normalization is novel, however, the reasoning and intuition behind the idea should be elaborated better. Moreover, additional experiments without replay-based method can better demonstrate the effectiveness of the method.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper investigates the problem of cross-task normalization in a non-stationary online continual learning(CL) setting. It argues that batch normalization (BN) is important to CL; however, BN in the current form introduces a bias towards current task, leading to catastrophic forgetting. The paper presents a continual normalization(CN) layer to reduce this bias. In particular, it combines spatial and batch normalization into one layer that is suitable for CL. The authors conducted experiments to evaluate the performance of the proposed method.

main_review: Strong Points

* The paper takes one of the most import issues in continual learning: non-stationary online CL setting. For me, the problem itself is real and practical.

* The proposed approach is reasonable and addresses the problem of cross-task normalization in CL setting.

* The proposed adaptive normalization layers can be a plug-in replacement of BN.

* The authors made extensive comparison between CN and prior work such as BN, IN, and GN.

* Overall, the paper is well written. In particular, the Related Work section has a nice flow and puts the proposed method into context. Despite the method having limited novelty, the method has been well motivated by pointing out the limitations in SOTA methods.

* The authors provide code for reproducing the results in the paper.

Weak Points

* The proposed adaptive normalization layer is a straight-forward combination of spatial and batch normalization. So the novelty is limited.

* Table 2, in DER++ setting, CN's FM score (for catastrophic forgetting) is not as good as GN's. Please explain the reason for that.

summary_of_the_review: Overall, I vote for marginally accepting. I like the idea of cross-task normalization and handling it by the proposed adaptive normalization layer. My major concern is about the limited novelty of the paper and the performance of CN on catastrophic forgetting (see weakness above). Hopefully the authors can address my concern in the rebuttal period.

[After rebuttal]
After reading the other reviewers' comments and the authors' rebuttal, I confirm my rating.
review_title: null

Meta-review: The paper sheds light on issues with BN in continual learning and proposes a quite simple, which is a strength, solution to fix it. 

The Authors first draw attention to the fact that using recalculated moments boosts performance and reduces forgetting, which serves as an argument that at least partially BN contributes to catastrophic forgetting in continual learning. Given that BN remains quite important in certain application areas such as vision, it is a strong motivation for the paper.

The experiments are thorough and clearly show that CN is a practically relevant alternative to BN in continual learning.

One weakness of the paper is that the method is poorly motivated, and relatedly, it has quite limited novelty. CN combines the strengths and weaknesses of BN and GN. Hence, it is not clear why it outperforms both, given that it still has the issue of BN that normalization statistics might become outdated. This is one of the weaknesses pointed out by 9jXz who recommended rejecting the paper. It would be also nice to compare to Mode Normalization https://openreview.net/forum?id=HyN-M2Rctm. 

Other papers have suggested changing normalization for sequential learning. Changing batch normalization (to batch renormalization) was investigated in [1] in the context of continual learning. Relatedly, [2] proposes TaskNorm for meta-learning.

Despite these issues, it is a solid contribution and it is my pleasure to recommend acceptance. In the camera-ready, please describe more clearly the design principles behind CN.

[1] Rehearsal-Free Continual Learning over Small Non-I.I.D. Batches, https://openaccess.thecvf.com/content_CVPRW_2020/papers/w15/Lomonaco_Rehearsal-Free_Continual_Learning_Over_Small_Non-I.I.D._Batches_CVPRW_2020_paper.pdf

[2] TaskNorm: Rethinking Batch Normalization for Meta-Learning, https://arxiv.org/abs/2003.03284
Decision: Accept (Poster)