Architecture Matters in Continual Learning
Authors: Seyed Iman Mirzadeh
Arslan Chaudhry
Dong Yin
Timothy Nguyen
Razvan Pascanu
Dilan Gorur
Mehrdad Farajtabar
Venue: Submitted to ICLR 2023 (2023)
Keywords: Continual Learning
Catastrophic Forgetting
Neural Network Architecture

Abstract: A large body of research in continual learning is devoted to overcoming the catastrophic forgetting of neural networks by designing new algorithms that are robust to the distribution shifts. However, the majority of these works are strictly focused on the "algorithmic" part of continual learning for a "fixed neural network architecture", and the implications of using different architectures are not clearly understood. The few existing continual learning methods that expand the model also assume a fixed architecture and develop algorithms that can efficiently use the model throughout the learning experience. In contrast, in this work, we build on existing works that study continual learning from a neural network's architecture perspective and provide new insights into how the architecture choice, for the same learning algorithm, can impact stability-plasticity trade-off resulting in markedly different continual learning performance. We empirically analyze the impact of various architectural components providing best practices and recommendations that can improve the continual learning performance irrespective of the learning algorithm.

Under review as a conference paper at ICLR 2023
ARCHITECTURE MATTERS IN CONTINUAL LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
A large body of research in continual learning is devoted to overcoming the catas- trophic forgetting of neural networks by designing new algorithms that are robust to the distribution shifts. However, the majority of these works are strictly focused on the algorithmic part of continual learning for fixed neural network architecture, and the implications of using different architectures are not clearly understood. The few existing continual learning methods that expand the model also assume a fixed architecture and develop algorithms that can efficiently use the model throughout the learning experience. In contrast, in this work, we build on existing works that study continual learning from a neural network's ' architecture perspective and pro- vide new insights into how the architecture choice, for the same learning algorithm, can impact stability cabblity-plasstiiity trade-off resulting in markedly different continual learning performance. We empirically analyze the impact of various architectural components providing best practices and recommendations that can improve the continual learning performance irrespective of the learning algorithm.
INTRODUCTION
Continual learning (CL) (Ring 1995 Thrun 1995) is a branch of machine learning where the model exposed to a sequence of tasks with the hope of exploiting existing knowledge to adapt quickly to new tasks. The research in continual learning has seen surge in the past few years with the explicit focus of developing algorithms that can alleviate catastrophic forgetting (McCloskey & Cohen 1989) whereby the model abruptly forgets the information of the past when trained new tasks.
While most of the research in continual learning is focused on developing learning algorithms, that can perform better than naive the best of our knowledge, is not explicitly studied in any of the existing works. Even the class of parameter isolation expansion-bon only have a cursory focus on the model architecture insofar that they assume a specific architecture and develop algorithm operating on the architecture. Orthogonal to this direction for designing algorithms, our motivation is that the inductive biases induced by different architectural components could be important for continual learning irrespective of the learning algorithm. Therefore, we seek to characterize the implications of different architectural choices in continual learning.
on stream of data, the role of model architecture, to
methods, for example (Rusu et al 2016 Yoon et al. 2018).
To motivate our study, consider ResNet- 18 model (He et al. 2016) on Split CIFAR- 100, where CIFAR- 100 dataset (Krizhevsky et al.) 2009) is split into 20 dis disjoint sets prevalent architecture and benchmark the existing continual learning works. Fig. la shows that explicitly designed CL algorithms, EWC (Kirkpatrick et al. 2017) (a parameter regularization-basee method) and experience replay (Riemer et al.] 2018) (a memory- based CL algorithm) indeed improve upon the naive fine- tuning. However, similar better performance can be obtained this benchmark by simply removing the global average pooling layer from ResNet- 18 and performing the naive fine-tuning. This clearly demonstrates the need for better understanding of network architectures in the context of continual learning where the architectural choices are not solely based on the performance of a single task but on a trade-off between the learning of new and previous tasks. Similar observation, though in more limited scenarios have been previously studied, for example Mirzadeh et al. (2022) looks the role of layer width, while Ramasesh al. (2022) focuses on the scale of the model. We build on these works, extending the analysis to architecture choices as well understanding particular components typically used like batch norm. It is also useful to note that these observations do not imply that the algorithmic improvements are not important. In fact, we show in Appendix B that one can achieve even better performance by combining our architectural findings with specially designed continual learning algorithms.

---

Under review as a conference paper at ICLR 2023
10
2103 eia Fretu
28 (106)
(146-155
Final Auvenge Aocurayy
Fiel Aveage Exegeting
Aosme" Resuney
R100-10-10
22/12
50
M/N6-10-2
A37-522
MMRM 20.02
Chilo
+
00
35
Learned
Paroms
Params (M)
(b)
(C)
Figure 1: Split CIFAR-100: (a) While compared to naive fine-tuning, continual learning algorithms such as EWC and ER improve the performance, a simple modification to the architecture (removing global average pooling (GAP) layer) match the performance of ER with a replay size of 1000 examples. (b) and (c) Different architectures lead to very different continual learning performance levels in terms of accuracy and forgetting. This work will investigate the reasons behind these gaps and provide insights into improving architectures.
To understand the implications of architectural decisions in continual learning, we thoroughly study different architectures including MLPs, CNNs, ResNets, Wide-ResNets and Visual Transformers. Our experiments suggest that different components of these architectures can have different effects on the relevant continual learning metrics- namely average accuracy, forgetting, and learning accuracy (cf. Sec. 2.1 to the extent that vanilla fine-tuning with modified components can achieve similar of better performance than specifically designed CL methods on a given base architecture without significantly increasing the parameters count.
Contributions. We summarize our main contributions as follows:
We compare both the learning and retention capabilities of popular architectures. We study the role of individual architectural decisions (e.g., width and depth, batch normalization, skip-connections, and pooling layers) and how they can impact the continual learning performance.
We show that, in some cases, simply modifying the architecture can achieve similar or better performance compared specifically designed CL algorithms (on top of a base architecture).
In addition to the standard CL benchmarks, Rotated MNIST and Split CIFAR- 100, we report results the large-scale Split ImageNet- 1K benchmark, which is rarely used in the CL literature, to make sure our results hold in more complex settings.
Inspired by our findings, we provide practical suggestions that are computationally cheap and can improve the performance of various architectures in continual learning.
Limitations. We emphasize that our main focus is to illustrate the significance of architectural decisions in continual learning. We do not claim that this work covers all the possible permutations of architectural components and different continual learning scenarios. Consequently, the majority of our experiments are focused on the task-incremental setup with popular architectures. However, our results in Appendix [B55 for the class-incremental setup confirm our results for the task-incremental setup. Moreover, the secondary aim of this work research on the architectural side of continual learning. That is why we focus on the breadth rather than depth of some topics. Finally, while there are a limited number of works in the literature that study the role of architecture in continual learning, in Sec. [ will discuss why those works solely focus on specific topies while this work draws a comprehensive and general picture. We believe our work provides many interesting directions that require deeper analysis beyond the scope of this paper but can significantly improve our understanding of continual learning.
stepping-stone that encourages further
2
COMPARING ARCHITECTURES
2.1
EXPERIMENTAL SETUP
Here, for brevity, we explain our experimental setup but postpone more detailed information (e.g.. hyper-parameters, details of architectures, etc.) to Appendix A
Benchmarks. We use three continual learning benchmarks for our experiments. The Split CIFAR- 100 includes 20 tasks where each task has the data of classes (disjoint), and we train on each task for 10 epochs. The Split ImageNet-Ii includes 10 tasks where each task includes 100 classes of

---

Under review as a conference paper at ICLR 2023
ImageNet- K and we train on each task for 60 epochs. Finally, for few experiments, we use the small Rotated MNIST benchmark with 5 tasks where the first task is the standard MNIST dataset, and each of the subsequent tasks adds 22.5 degrees of rotation to the images of the previous task. We note that the Split CIFAR- -000 and Split ImageNet-IK I benchmarks use multi-head classification layer, while the MNIST benchmark uses a single-head classification layer. Thus, Split CIFAR-I00 100 and Split ImageNet-I1 IK belong to the so-called task incremental learning setting, whereas Rotated MNIST belongs to domain incremental learning (Hsu et al. 2018). Finally, for Split CIFAR-100 and Split ImageNet-1K K benchmarks, we randomly shuffle the labels in each run, for 5 runs, ensure that the results are not biased towards a specific dataset ordering.
Architectures. We denote each architecture with descriptor. MLP-N represents fully connected networks with hidden layers of width N. Convolutions neural networks (CNN) are represented by CNNxN where N is the multiplier of the number of channels in each layer. Unless otherwise stated, the CNNs have only convolutional layers (with a stride of 2), followed by a dense feed-forward layer for classification. For the CIFAR-100 experiments, we use three convolutional layers, and for the ImageNet- IK experiments, we use six convolutional layers. Moreover, whenever we add pooling layers, we change the convolutional layer strides to I to keep the dimension of features the same. The standard ResNet (He et al. (0116) of depth D is denoted by ResNet-D and Nete (WRN) (Zagoruyko & Komodakis (016) are denoted by WRN-D-N where D and N are the depths and widths, respectively. Finally, we also use the recently proposed Vision Transformers (ViT) (Doso- vitskiy et al. 202T). For the ImageNet-I IK experiments, we follow the naming convention in the original paper (Dosovitskly et al. 2021). However, for the Split CIFAR- experiments, use smaller versions of ViT's where VIT N/M stands for a 4-layer vision transformer with the hidden size of N and MLP size of M. For each architecture, we search over a large grid of hyper-parameters and report the best results. Further, we average the results over 5 different random initializations, for the corresponding best syper-parameters. and report the average and standard deviations.
Metries. We are interested in comparing different architectures from two aspects: (1) how well an architecture can learn a new task i.e. their learning ability and (2) how well an architecture can preserve the previous knowledge i.e. their retention ability. For the former, we record average accuracy, learning accuracy, and joint/ multi-task accuracy, while for the latter we measure the average forgetting of the model. We now define these metrics.
(1) Average Accuracy E [0, 100] (the higher the better): The average validation accuracy after the model has been continually trained for T tasks is defined as: AT = 7 Σ aT, where, az is the validation accuracy on the dataset of task after the model finished learning task t.
(2) Learning Accuracy E [o, 100] (the higher the better): The accuracy for each task directly after it learned. The learning accuracy provides a good representation of the plasticity of model and can be calculated using: LAT = + E ag Note that for both Split CIFAR- 100 and ImageNet- IK bench- marks, since tasks include images with disjoint labels, the standard deviation of this metric can be high. (3) Joint Accuracy E [o. 100] (the higher the better): The accuracy of the model when trained on the data of all tasks together.
(4) Average Forgetting E [-100, 100] (the lower the better): The average forgetting is calculated as the difference between the peak accuracy and the final accuracy of each task, after the continual learning experience is finished. For a continual learning benchmark with T tasks, it is defined as: (ar,i @T,;).
F
T ET max.E
2.2 RESULTS
We first compare different architectures on the Split CIFAR- 100 and Split ImageNet- IK benchmarks. While this section broadly focuses on the learning and retention capabilities of different architectures, explanation
the explanations behind the performance gaps across different architectures and the analysis of different architectural components is given in the next section.
Tab. I lists the performance of different architectures on Split CIFAR-100 benchmark. One can make several observations from the table. First, very simple CNNs, which are not state-of-the-art (SOTA) in the single image classification tasks, significantly outperform both the ResNets, WRNs, and ViTs (all SOTA architectures in image classification) in terms of average accuracy and forgetting. This observation holds true for various sizes of widths and depths in all the architectures. A similar overall trend where, for a given parameter count, simple CNNs outperform other architectures, can also be seen in Fig. [b] anddn[[e This shows that architectures that are cross-validated for a single task in an IID fashion are not necessarily optimal for continual learning settings.
3

---

Under review as a conference paper at ICLR 2023
Table 1: Split CIFAR-100 the learning and re- Table 2: Split ImageNet-IK3 the learning and re- tention capabilities vary significantly across tention capabilities can vary significantly across different architectures.
different architectures.
Model
Params (M)
Average Accuracy
Average Forgetting
Learning Accuracy
Model
Params
Average Accuracy
Average Forgetting
Learning Accuracy
(M)
CNN x1
0.3
62.2 11.35
12.6 11.14
74.1
CNN x3
63.3 10.68
5.4 #0.93
71.6 =231
CNN
66.3
CNN x6
24.2
66.7 +0662
3.9 #0.66
70.1 =3.21
CNN x4
2.3
68.1 =0.5
8.7 =0.21
7642
CNN x12
72.4
67.8 1.04
2.8 4:077
70.3 =2.2
CNN x8
7.5
69.9 +0062
8.0 +0.71
77.5 +6.78
CNN x16
26.9
76.8 +0.76
4.7 =0.84
81.0 #6.97
ResNet-34
62.7 +0.53
17.3
ResNet-50
25.5
66.1 +0.69
19.0 +0.67
83.3 =157
ResNer-18
11.2
45.0
36.8 11.0
74.9 #3.98
ResNet-101
44.5
64.1 #0.72
18.9 +1.32
81.1 =2.9
ResNet-34
21.3
44.8 1:2.44
39,9 #2.28
72.6 16.34
ResNet-50
23.6
56.2 +0.88
9.5 =0.38
WRN-50-2
63.2 1:661
21.7
85.8 1.65
ResNer-101
42.6
56.8 +1.62
9.2 +0089
65.7455.2
ViT-Base
86.1
58.3 +0.65
15.9 +1.11
72.8 -222
WRN-10-2
50.5
36.5 #2.74
VIT-Large
307.4
60.7 #1.31
10.6 #1.1
73.2 =2.12
WRN-10-10
7.7
56.8 +2203
31.7 1..34
86.7 #4.94
WRN-16-2
0.7
4-2.8
41.4 -144
82.4 +6.09
WRN-16-10
17.3
+1.47
37.6 #2.22
86.9 43.96
WRN-28-2
5.9
46.6 4-2.77
39.5 +-229
82.5 +6.26
WRN-28-
36.7
49.3 12.02
12.56 35.8
82.5 16.26
VIT-512/1024
51.7 14
+1.
75.5
VIT 546
30.7
60.4 +1.56
12.2 L112
67.4 45.557
CNN xB
VIT 512/1024
85
CNN x12
VTT LargL
WRN-50-2
ResNee
Average Accuracy
60
Average Accuracy
80
70
70
60
65
50
5
10
15
200
10
Tasks Learned
Tasks Learned
(a) Split CIFAR- 100
(b) Split ImageNet-
Figure 2: Evolution of average accuracy for various architectures on (a) Split CIFAR-100: CNNs have smaller forgetting than other architectures while WideResNets have the highest learning accuracy, and (b) Split ImageNet- IK WideResNets and ResNets have higher learning accuracy than CNNs and ViTs. However, the latter has smaller forgetting.
Second, mere increase the parameters count, within across the architectures, does not nec- essarily translate into the performance increase in continual learning. For instance, ResNet- 18 and ResNet-34 have roughly the same performance despite almost twice the number of parameters the latter. Similarly, WRN-10-10 outperforms WRN-16-1 10 and WRN 28- 10, despite having significantly less number of parameters. Note that we do not draw a general principle that overparamettization is not helpful in continual learning. In fact, in some cases, it indeed is helpful can be in the peross-the-boarc performance improvement when ResNet-34 is compared with ResNet-50 or when the WRN-10-2 is compared to the WRN-10-10 10. In the next section, will analyze when the overparametrization can help the performance in continual learning.
Finally, explicitly comparing the learning and retention capabilities of different architectures, one see from the table that ResNets and WRNs have a higher learning accuracy suggesting that they are better learning terms of retention, CNNs and ViTs are much better, as evidenced by their lower forgetting numbers. This is further demonstrated in Fig. 2a (CIFAR- 100) and Fig. 2b (ImageNet-IK), where ResNets and WRNs learn each individual task much better resulting in higher average accuracy for the first few tasks. However, as the number of tasks increases, CNNs outperforms the other architectures due to their smaller forgetting, eventually translating into overall flatter average accuracy curve.
task. This also explains their frequent in single task settings. However, in
trend similar to CIFAR-100 can also seen the ImageNet- IK benchmark as shown in Table 2 However, the performance difference, as measured by the average accuracy, between CNNs and other architectures is smaller compared to that of CIFAR-100. We believe that this due to the very high learning accuracy of other architectures compared to CNNs on this benchmark, and hence their

---

Under review as a conference paper at ICLR 2023
Table 3: Role of width and depth: increasing the number of parameters (by increasing width) reduces the forgetting and hence increases the average accuracy. However, increasing the depth does not necessarily improve the performance, and thus, it is essential to distinguish between scaling the models by making them deeper and wider.
Params (M)
Average Accuracy
Average Forgetting
Learning Accuracy
Benchmark
Model
Depth
Rot MNIST
MLP-128
0.1
70.8 +0068
31.5 +0092
96.0 =0,90
Rot MNIST
MI.P-128
0.2
68.9 +1.07
35.4
97.3 +0.76
Rol MNIST
MLP-256
0.3
71.1 10.43
31,4 10.48
96.1 10.82
Rot MNIST
MLP-256
0.7
70.4 10.61
32.1 10.75
96.3 10.77
Rot MNIST
MLP-512
0.7
72.6 #0.27
29.6 10.36
20.73
CIFAR-100
CNN x4
2.3
68.1 -0.5
8.7 10.21
76.4 16.92
CIFAR-100
CNN x4
5.4
62,9 10.86
12.4 1166
77.7 15,49
CIFAR-100
CNN x8
7.5
69.9 #0.62
8.0 =0.71
77.5 16.78
CIFAR-100
CNN x8
19.9
66.5 1.01
10.7 #1.9
74.78
CIFAR-100
VIT 512/1024
4.6
16.4
15,9 +0.95
68.1 #7.15
CIFAR-100
VIT 512/1024
8.8
51.7=1.4
21.9 113
71.4 #5.52
frequent use in the single task settings, resulting in an improved final average accuracy. The average forgetting of CNNs is still much smaller than other architectures.
Overall, from both tables, we conclude that ResNets and WRNs have better learning abilities, whereas CNNs and ViTs have better retention abilities. In our experiments, simple CNNs achieve the best trade-off between learning and retention. In the next section, we study the individual components of these architectures to better understand their performance gap with respect to each other.
ROLE OF ARCHITECTURE COMPONENTS
We now study the individual components in various architectures to understand how they impact continual learning performance. We start by generic structural properties all architectures such as width and depth (ef. Sec. [.1], Sel.B.1), and show that as the width increases, the forgetting decreases. In Sec. 3.2 we study the impact of batch normalization and observe that it can significantly improve the learning accuracy in continual learning. Then, in Sec. 3.3 we that adding skip connections (or shortcuts) to CNNs does not necessarily improve the CL performance whereas pooling layers (cf. Sec. 3.4 and Sec. 3.5) have significant impact on learning accuracy and forgetting. Moreover, briefly study the impact of attention heads in ViTs in Sec. 3.6) Finally, based on the observations make in the aforementioned sections, in Sec. 4
provide summary of modifications that can improve various architectures on both Split CIFAR TEFA-1104 100 and ImageNet-I - benchmarks
3.1 WIDTH AND DEPTH
Tab. shows that across all architectures, peerpaaramttiizatoon through increasing width is helpful in improving the continual learning performance as evidenced by lower forgetting and higher average accuracy numbers. For MLP, when the width is increased from 128 512, the performance in all met- ries improves. However, for both MLP-128 and MLP-256 when the depth is increased from 2 to 8 the average accuracy is reduced, and the average forgetting is increased with a marginal gain in learning accuracy. Finally, note that MI.P-256 with 8 layers has roughly the same number of parameters the MLP-512 with 2 layers. However, the wider MI.P-512 has better continual learning performance.
A similar analysis for ResNets and Wete is demonstrated in Tab. ResNet-50 and ResNet- 101 are four times wider than ResNet-15 and ResNet-34, and from the table, it can be seen that this width translates into drastic improvements in average accuracy and forgetting. Similarly, ResNet-34 and ResNet-100 that increasing the depth is not helpful in this case either. Finally, wider WRN-10-10, WRN-16-10 10, and WRN-28-11 10 outperform the narrower 10-2 WRN-10-10, WRN-28-10, 10, respectively. Whereas if we fix the width, increasing the depth is not helpful. Overall, can that overparametrization through width is helpful in continual learning, whereas a similar claim cannot be made for the depth.
101 are the deeper versions of ResNet- 18 and ResNet-50, respectively. We can observe
We note that Mirzadeh et al. (2022) studied the same width vs. depth trade-off in the task-incremental setting and demonstrated that overparametrization through width induces lazy-training regime during training and makes the gradients of different tasks more orthogonal and sparser, ultimately resulting in
In this section, we duplicate some of the results across tables to improve readability.
5

---

Under review as a conference paper at ICLR 2023
Table 4: Role of various components for the Split CIFAR-100 benchmark: While adding skip
connections does not have significant impact on the performance, batch normalization and max polling can significantly improve the learning accuracy of CNNs.
Params (M)
Average Accuracy
Average Forgetting
Learning Accuracy
Joint Accuracy
Model
CNN x4
2.3
68.1 +0.5
8.7 #0.21
76.4 +6.92
73.4 +:0.9
CNN x4 Skip
2.4
68.2 +0.56
8,9 +0.72
77.07
73.8 +:047
CNN x4 BN
74.0 #0.06
8.1 10.35
81.7 16.68
80.2 10.16
CNN x4 AvgPool
68.5 10.6
80.57
76.6
73.6 +0.83
CNN x4 MaxPool
74.4 +0.34
9.347
83.3 46.1
79.9 +0.53
CNN x4 All
2.4
77.7 10.77
6,5 10.58
16.31
10.77
CNN x8
7.5
69.9 +0.62
+.07
+6678
74.1 +0.83
CNN x8 Skip
7.8
70.7 =0.31
6.8 #0.91
77.1 16.87
74.4 +0.35
CNN x8 + BN
7.5
76.1 +0.3
5..1
81.7 #6.83
80.5 +0.27
CNN x8 + AvgPool
7.5
71.2 +0.5
8.3 +0.35
79.0
74.0 +1002
CNN x8 + MaxPool
7.5
77.2 10.53
7.1 10.33
84.0
80.6 +0.35
CNN x8 + All
7.8
78.1 11.5
5.7 #0.36
83.3 16.27
81.9 #0.51
CNN x16
26.9
76.8 +0.76
4.7 +0.84
81.0
79.1 +0.86
CNN x16 + All
27.9
78.9 +0.27
4.5 +0.36
82.9 +6.48
82.1 +0.46
better continual learning performance. Our results verify their conclusions on more benchmarks and architectures. Moreover, we report the benefits of width in class-incremental setup in Appendix [B.5]
Conclusion. Overall, in continual learning setups, over-parameeerizaaion thorough width can increase the accuracy and decrease the forgetting across all architectures. Given the same computing budget, wider and shallower networks can outperform thinner and deeper networks.
3.2 BATCH NORMALIZATION
Batch Normalization (BN) (loffe & Szegedy 2015) is normalization scheme that shown to increase the convergence speed of the network due to its optimization and generalization benefits (Santurkar al. 2018 Bjorck et al. 2018). Another advantage of the BN layer its ability reduce the covariate shift problem that is specifically relevant for continual learning where the data distribution may change from one task to the next. There are relatively few works that have studied the BN in the context of continual learning. Mirzadeh et al. (2020) analyzed the BN in continual learning through the generalization lens. Concurrently to this work, [hhm et al. (2022) study the normalization schemes in continual learning and show that BN enables improved learning of each task. Additionally, the authors showed that in the presence of a replay buffer of previous tasks, BN facilitates a better knowledge transfer compared to other normalization schemes such as Group Normalization (Wu & He 2018).
Intuitively, however, one might think that since due to evolving data distribution the BN statistics changing across tasks and the statistics of each task are not kept, the BN should contribute to an increased forgetting. This is not the case in some of the experiments that we conducted. Similar to the results in Mirzadeh et al. (2020); Pham et al. (2022). we found the BN to facilitate the learning accuracy in Split CIFAR- 100 and split ImageNet- IK (ef. Tab. 4 and Tab. 7
We believe that this could be due relatively unchanging BN statistics across tasks in these datasets. To verify this, in Appendix B.2 we plot the BN statistics of the first layer of CNNx4 4 on the Split CIFAR-100 dataset, and show that the BN statistics are stable throughout the continual learning experience. However, if this hypothesis were to be true, the converse statisties change lot across tasks, such as Permuted MNIST should hurt the continual learning performance. In Appendix B.3] we plot the BN statistics of the first layer of MIP-112 on Permuted MNIST. It can seen from the figure that indeed the BN statistics are changing in this benchmark. As consequence, adding BN to this benchmark significantly hurt the performance, evidenced by the increased forgetting in Tab. 9
benchmark where the BN
Conclusion.
The effect of the batchnorm layer is data-dependent. In the setups where the input distribution relatively stays stable, such as Split CIFAR-100 or Split ImageNet-IK. IK, the BN layer improves the continual learning performance by increasing the learning capability of the models. However, for setups where the input distribution changes significantly across tasks, such as Permuted MNIST, the BN layer can hurt the performance by increasing the forgetting.

---

Under review as a conference paper at ICLR 2023
3.3 SKIP CONNECTIONS
Skip connections (Cho et al. 2011) originally proposed for convolutional models by He et al. (2016) are crucial in the widely used ResNet architecture. They are also used in many other architectures such as transformers (Vaswani et al., 2017). Many works have been done to explain why skip connections are useful: Hardt & Ma 2016) show that skip connection tends to eliminate spurious local optima: Bartlett et al. (20 8b) study the function expressivity of residual architectures; Bartlett ct al. (2018a) show that gradient descent provably learns lincar transforms in ResNets: JJsttzebski ResNetts,assttzbski et al. 2017) show that skip connections tend to iteratively refine the learned representations.
However, these works mainly focus on learning a single task. In continual learning problems, due to the presence of distribution shift, it is unclear whether these benefits of skip connections still have a significant impact on model performance, such as forgetting and average accuracy over tasks. We empirically study the impact of skip connection continual learning problems. Interestingly, as illustrated in Tab. 4 adding skip connections to plain CNNs does not change the performance significantly, and the results are very close (within the standard deviation) of each other.
Conclusion. The skip connection does not show significant positive or negative impact on the model performance in our benchmarks.
3.4 POOLING LAYERS
Pooling layers were the mainstay of the improved performance of CNNs before ResNets. Pooling layers not only add local translation invariances, which help in applications like object classifica- tion (Krizhevsky et al. 2017; Dai et al., 202T), but also reduce the spatial resolution of the network features, resulting in the reduction of computational cost. Since one family of the architectures that we study are all convolutional CNNs, we revisit the role of pooling layers in these architectures in a continual learning setup.
We compare the network without pooling, CNNxN against those that have pooling layers "CNNxN +AvgPool' or 'CNNxN +MaxPool")) in Tab. To keep the feature dimensions fixed, we set the convolutional stride from 2 to I when pooling is used. We make the following observations from the table. First, the average pooling (+AvgPool) does not have any significant impact on the continual learning metrics. Second, max pooling (+MaxPooll improves the learning capability of the network significantly, as measured by the improved learning accuracy. Third, in terms of retention, pooling layers do not have a significant impact as measured by similar forgetting. All in all, max pooling achieves the best average accuracy, owing to its superior learning capability.
The ability of max pooling to achieve better performance in a continual learning setting can be attributed a well-known empirical observation by |Sprinneebberg et al. (2015), where is shown that max pooling with stride outperforms a CNN with stride 2 and no pooling, Further, we believe that max pooling might have extracted the low-level features, such as edges, better, resulting in improved learning in a dataset like CIFAR- 100 that consists of natural images. There is some evidence in the literature that max pooling provides sparser features and precise localization (Zhou et al., 2016)) This could have transferred over to the continual learning setup that we considered.
Conclusion. We believe the ability of max pooling for extracting low-level features, makes them a suitable choice for increasing the learning capability of models, especially on datasets consisting of natural images. Moreover, the intrinsic sparsity induced by max pooling in network may be helpful in continual learning scenarios. However, it remains interesting future direction to further study the gains brought by max pooling in both the standard and continual learning setups.
3.5 GLOBAL POOLING LAYERS
Global average pooling (GAP) layers are typically used in convolutional networks just before the final classification layer to reduce the number of parameters in the classifier. The consequence of adding a GAP layer is to reduce the width of the final classifier. It is argued in Sec. Sec. that wider networks forget less. Consequently, the architectures with GAP layer can suffer from increased forgetting. Tab, 5Jempricallyy demonstrate this intuition. From the table it can be seen that applying the GAP layer significantly increases the forgetting resulting in a lower average accuracy. In the previous section, we already demonstrated that average pooling does not result in a performance decrease as long as the spatial feature dimensions are the same. To demonstrate that there is nothing inherent to the GAP layer, and it is just consequence of a reduced width of the final classifier, we construct another baseline by multiplying the number of channels in the last convolutional layer by 16 and then apply the GAP, This network is denoted as "CNN x4 (16x)" in Tab. 5 and it has the same
7

---

Under review as a conference paper at ICLR 2023
Table 5: Role of Global Average Pooling (GAP) for Split CIFAR-100: related to our arguments in Sec. 3.T] adding GAP CNNs significantly increases the forgetting, Later, we show that removing GAP from ResNets can also significantly reduce forgetting as well.
Model
Params (M)
Pre-Clusiifiation
Average Accuracy
Average Forgetting
Learning Accuracy
Joint Accuracy
Width
CNN
2.3
8192
68.1 1:0.5
8.7 #0.21
76.4 16.92
73.4 1:0.99
CNN + GAP
1.5
512
60.1 +:0.43
14.3 +0.8
66.1 47.76
76,9 =0081
CNN x4 (16x) + GAP
32.3
8192
73.6 +:039
5.2 #0.66
75.6 44.77
77.9 +0.37
CNN x8 CNN x8 + GAP
7.5
16384
69.9 ++0.62
8.0 +00.1
77.5 +66788
74.1 +0.83
6.1
1024
63.1 +2.0
14.7 1.68
70.1 #7.18
78.3 +0.97
CNN x16 CNN x16 1116+GAI GAP
26.9
32768
76.8 +-0.76
4.7 +0.84
81.0 +6697
74.6 +0086
23.8
2048
66.3 +:0.2
12.2 20.65
72.3 +6.02
78.9 +0.27
classifier width as that of the network without GAP. can be seen this architecture has considerably smaller forgetting showing GAP affects continual learning through the width of the final classifier.
Conclusion. Global pooling layers, while being computationally beneficial, reduce the dimensionality of the final features resulting in low continual learning performance. A practical workaround could be to use pooling layers with smaller filter sizes. Inspired by this observation, in Sec. 4 we show that the performance of ResNets in continual learning can be significantly improved by either removing the GAP layer or using the smaller average pooling window rather than the GAP.
3.6 ATTENTION HEADS
The attention mechanism is a prominent component in the transformer-based architectures Vaswani et al.) (2017) that have had great success in natural language processing and, more recently, computer vision tasks, For the latter, the heads of vision transformers (ViTs) shown attend both local and global features in the image Dosovitskiy et al. (2021).
Table 6: Role of attention heads: for the Split CIFAR- 100 benchmark, increasing the num- ber of attention heads (while fixing the total width), does not impact the performance sig- nificantly.
Model
Heads
Params (M)
Accuracy
Average
Forgetting
Average
Leaming Accuracy
For our experiments, we double the number of heads while fixing the width to ensure that any change in
VIT 512/1024
8.5
50.9 29.25
23.8 #1.3
VIE 512/1024
8.5
$1.7 11.
219 21.3
71.4
VIT 1024/1536
30.7
57.4
21.9
66.0 =5.99
results not due to the increased dimension of repre- sentations. In Tab. 6 it can be seen that even doubling the number of heads in ViTs only marginally helps in increasing the learning accuracy and lowering the forgetting. This suggests that increasing the number of attention heads may not an efficient approach for improving continual learning performance in ViTs. We, however, note that consistent with other observations in the literature (Paul & Chen 2022). ViTs show promising robustness against distributional shifts as evidenced by lower forgetting numbers, for the same amount of parameters, on the Split CIFAR-100 benchmark (ef. Fig. [cc..
VIT 1024/1536
30.7
1.56
12. +1.13
67.4 -5.57
Conclusion. While increasing the number of attention heads can improve the performance, the gain is not substantial. We believe increasing the hidden dimension of ViTs is better approach for improving the performance our computational budget allows for it.
4 PRACTICAL SUGGESTIONS FOR IMPROVING ARCHITECTURES
We now provide practical suggestions to improve the performance of architectures, that are derived from results in Sec. 3 For CNNs, we add BatchNormalization and MaxPooling, they both are shown to improve the learning ability of the model and skip connections that make the optimization problem easier. Tab. 7 shows the results on both CIFAR-100 and ImageNet- IK. It can be seen from the table that adding these components significantly improves the CNNs performance, as evidenced by improvement over almost all the metrics, including average accuracy. In addition, motivated by Sec. 3.I. we can use wider CNNs that have less forgetting and higher accuracy. For ResNets, we either remove the GAP, as is the case with CIFAR-100, or locally average the features in the penultimate layer by a 4x4 filter, as is the case with Image The reason for not fully removing the average pooling from the ImageNet- K the resultant large increase in the number of parameters in the classifier layer. It can be seen from Tab. that fully or partially removing the GAP, CIFAR- 100 or ImageNet- IK, respectively, highly improves the retention capability of ResNets, indicated by their lower forgetting.
8

---

Under review as a conference paper at ICLR 2023
Table 7: Improving CNNs and ResNets architectures on both Split CIFAR-100 and ImageNet-
Params (M)
Average Accuracy
Average Forgetting
Learning Accuracy
Model
Benchmark
CNN x4
CIFAR-100
2.3
68.1 10.55
8.7 +0.21
76.4 16.92
CNN x4 + BN + MaxPool + Skip
CIFAR-100 100
1.5
77.7 1:0.7
6,5 +0.58
83.7 16,31
CNN x8
CIFAR-100
7.5
69.9 +0.62
8.0 -0.71
77.5 16.78
CNN x8 + BN + MaxPool + Skip
CIFAR-100
6.1
78.1 +1.15
5.7 #0.36
83.3 #6.27
CNN x3
ImageNet-1K
9.1
63.3 +0.68
5.8 10.93
71.6 1:2.31
CNN x3 BN + MaxPool + Skip
ImageNet-1K
9.1
66.4 10.47
10.3
74.7 1:21
CNN x6
ImageNet-1K
24.2
66.7 +0.62
3,9 #0.86
70.1 +3.21
CNN x6 BN + MaxPool + Skip
ImageNet-IK
24.3
72.1 40.41
4.0 -0.22
75.7 =2.57
R
CIFAR-100
11.2
45.0 +0.63
36.8 +1.08
74.9 +:3.98
ResNet-11 w/o GAP
CIFAR-100
11.9
67.4 +0.76
11.2 1199
74.2 4:479
ResNet-55
CIFAR-I00
23.6
56.2 +0.88
9.5 10.38
67.8 15009
ResNet-50 w/o GAP
CIFAR-100 100
26.7
71.4 10029
6.6 10.12
73.0 15.18
ResNet-34
ImageNet-11
21.8
62.7 +0.53
19.0 +0.67
80.4 +2.57
ResNet-34 w 4x4 AvgPool
ImageNet-Ill I
23.3
66.0 40.24
4.2 10116
70.2 +3.87
ResNet-50
ImageNet-Il IK
25.5
66.1 +0.69
17.3 #0.58
83.3 1.57
5e w 4x4 AvgPool
ImageNet- IK
31.7
67.2 10.13
3,5 10.35
72.8 13.27
5 RELATED WORK
Algorithms. Perhaps the most related side of algorithms our work is the parameter-isslattio methods where different parts of the model are devoted to specific task (Yoon et al. 2018) Mallya & Lazebnikk 2018:) Wortsman al., 2020). Often, these algorithms start with fixed-size model, and the algorithm aims to allocate a subset of the model for each task such that only a specific part of the model is responsible for the knowledge for that task. For instance, PackNet (Mallya & Lazebnik 2018) uses iterative pruning to free space for future tasks, while (Wortsman et al. 2020) propose to fix with a randomly initialized and over- peee-prraeterized network to find a binary mask for each task. Nevertheless, even though the focus of these methods is the model and its parameters, the main objective is to have an algorithm that can use the model as efficiently as possible. Consequently, the significance of the architecture is often overlooked. We believe our work is orthogonal to the algorithmic side of continual learning research as any of the methods mentioned above can use a different architecture for their proposed algorithm, we have shown in Fig. Ta and Tab. X
Architectures. There have been some efforts on applying architecture search to continual learning (Xu & Zhu, 2018; Huang et al.) 2019 Gao et al.) 20200 Li et al.] 2019 Lee al.) 2021).
However, when it comes to architecture search, the current focus of continual learning literature mainly on deciding how efficiently share or expand the model by delegating these decisions to the controller. For instance. [Lee et al. (202T) use a ResNet as the base model and propose a search algorithm that decides whether the parameters should be reused or new parameters are needed for the new task, and the implications of architectural decisions have not been discussed. We believe future works that focus on the architecture search can benefit from our work by designing better search spaces that include various components that we have shown are important in continual learning. Finally, beyond architecture search methods, there are several works that focus on the non-al -oo-itgoii side of CL. Mirzadeh et al. (2022) study the impact of width and depth forgetting and show that as width increases, the forgetting will decrease. Our work extends their analysis for larger- lcrger-scale ImageNet- IK benchmark. more architectures, and more setups. Recently, Ramasesh et al. (2022) shows that in the pre-training setups, the scale of model and dataset can improve CL performance. In this work, however, we study the learning and retention capabilities of various architectures when models are trained from scratch. However, we show that while scaling the model can be helpful, the impact of architectural decisions is more significant. For instance, shown in Tab. I and Tab 2b the best performing architectures are not the ones with highest number of parameters.
DISCUSSION
In this work, we have studied the role of architecture in continual learning. Through extensive experiments on a variety of benchmarks, we have shown that different neural network architectures have different learning and retention capabilities, and a slight change in the architecture can result in significant change in performance in continual learning setting. While our work has limitations regarding the studied domains, we believe our results show promising directions for future works and add novel insights over the existing works in this space.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Peter Bartlett, Dave Helmbold, and Philip Long. Gradient descent with identity initialization efficiendl learns positive definite linear transformations by deep residual networks. In International Conference on Machine Learning, pp. 521-530. PMLR, 201 8a.
Peter I. Bartlett, Steven N Evans, and Philip M Long. Representing smooth functions as compositions of near-identity functions with implications for deep network optimization. arXiv preprint arXiv: 1804.05012, 2018b.
Nils Bjorek, Carla P Gomes, Bart Selman, and Kilian Q Weinberger. Understanding batch normalization. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula. Adam Paszke. Jake VanderPlas, Skye Wanderman- Milne, and Qiao Zhang. JAX: composable transfornations of Python+NumPy programs, 2018. BtLL:ttp:///tt http:://gthub. com/ google/
KyungHyun Cho. Tapani Raiko, and Alexander Ilin. Enhanced gradient and adaptive learning rate for training restricted boltzmann machines. In Proceedings of the 28th International Conference Machine Learning. ICML, pp. 105-112. Omnipress, 2011.
Zihang Dai, Hanxiao Liu, Quoe V. Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems, NewrIPS, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations, ICLR, 2021, 2021.
Qiang Gao. Zhipeng Luo, and Diego Klabjan. Efficient architecture search for continual learning. CoRR. abs/2006.04027, 2020.
Ian Goodfellow, Mehdi Mirza, Da, Aaron Courville, and Yoshua Bengio. An empirical investigation of catastrophic forgeting in gradient-basse neural networks. 2nd International Conference on Learning Representations, ICL.R, 2014.
Moritz Hardt and Tengyu Ma. Identity matters deep learning. arXiv preprint arXiv:161 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Tom Hennigan, Trevor Cai, Tamara Norman, and Igor Babuschkin. Haiku: Sonnet for JAX, 2020. URI. http: thub. com/deepmi nd/ām-haiku
Matteo Hessel, David Budden, Fabio Viola, Mihaela Rosca, Eren Sezener, and Tom Hennigan. Oplax: compos- able gradient transformation and optimisation, in jax!, 2020. URL http:://tthuu.b...///// com/ deepmind//
optax
Yen-Chang Hsu, Yen-Cheng Liu, Anita Ramasamy, and Zsolt Kira. Re-evaluating continual leamning scenarios: A calegorization and case for strong baselines. arXiv preprint arXiv:1810.1 12488, 2018
Shenyang Huang, Vincent François- Lavet, and Guillaume Rabusseau. Neural architecture search for class- incremental learning. CoRR, abs/1909 htts//999.06666. 2019.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning, ICMI. 2015, volume 37 of JMLR Workshop and Conference Proceedings, pp. 448-456. JMLR.org, 2015.
Stanislaw Jastrzebski, Devansh Arpit, Nicolas Ballas, Vikas Verma, Tong Che, and Yoshua Bengio. Residual connections encourage iterative inference. arXiv preprint a1Xiv://10.04773, 2017.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Bart Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13)335213355 -3526, 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. Commun, ACM, 60(6), 2017.
10

---

Under review as a conference paper at ICLR 2023
Seungwon Lee, Sima Behpour, and Eric Faton. Sharing less is more: Lifelong learning in deep networks with selective layer transfer. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Leurning Research, pp. 6065 6075. PMLR, 18-24 Jul 2021.
Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting. In Proceedings of the 36th International Con- ference Machine Learning, ICMI. 2019, volume 97 of Proceedings of Machine Learning Research, pp. 3925-3934, 2019.
Arun Mallya and Svellana Lazebnik. Packnet: Adding multiple tasks to single network by iterative pruning. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, pp. 7765-7773. Computer Vision Foundation/ / IEEE Computer Society, 2018.
Michael McCloskey and Neal J. Cohen, Catastrophic interference in connectionist networks: The sequential learning prohlem. Psychology of Learning and Motivation, 24:109-165, 1989.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Razvan Pascanu, and Hassan Ghasemzadeh. Understanding the role of training regimes in continual leamnnn. In Advances in Neural Information Processing Systems, volume 33, pp. 7308-7320, 2020.
Seyed Iman Mirzadeh, Arslan Chaudhry, Dong Yin, Huiyi Hu, Razvan Pascanu, Dilan Gorur, and Mehrdad Farajtabar. Wide neural networks forget less catastrophically. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 15699- 15717, 2022.
Sayak Paul and Pin- Yu Chen. Vision transformers are robust learners. AAAI, 2022.
Quang Pham, Chenghao Liu, and Steven HOL Continual normalization: Rethinking batch normalization for online continual learning. In International Conference on Learning Representations, 2022.
Vinay Venkatesh Ramasesh, Aitor Lewkowycz, and Ethan Dyer. Effect of scale catastrophic forgetting in neural networks. In International Conference on Learning Representations, 2022.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. International Conference on Learning Representations, 2018.
Mark B. Ring. Continual learning in reinforcement environments. PhD thesis, University of Texas at Austin, TX, USA, 1995.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv: 1606 04671, 2016.
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does hatch normalization help optimization? In Advances in Neural Information Processing Systems, volume 31, Curran Associates, Inc., 2018.
Jost Tobias Springenberg. Alexey Dosovitskiy, Thomas Brox, and Martin A. Riedmiller. Striving for simplicity: The all convolutional net. CoRR, aas//abss...eee 142.688666 2015.
Sebastian Thrun. A lifelong learning perspective for mobile robot control. In Intelligent robots and systems, pp. 201-214, Elsevier, 1995.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neurad Information Processing Systems, pp. 5998-6008, 2017.
Mitchell Wortsman. Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, and Ali Farhadi. Supermasks superposition. In Advances Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems, NewrlPS, 2020.
Yuxin Wu and Kaiming He. Group normalization. In Computer Vision ECCV 2018 15th European Conference, Munich, volume 1 11217 of Lecture Notes in Computer Science, pp. 3-19. Springer, 2018
Ju Xu and Zhanxing Zhu, Reinforced continual learning. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems, NeurIPS, pp. 907-916, 2018.
Jachong Yoon, Eunho Yang, Jungtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In Sixth International Conference on Learning Representations. ICLR, 2018

---

Under review as a conference paper at ICLR 2023
Sergey Zagoruyko and Nikos Komodakis, Wide residual networks. Proceedings of the British Machine Vision Conference 2016, BMVC 2016, York, UK, September 19-22, 2016. BMVA Press, 2016.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Pp- 2921-2929, 2016.

---

Under review as a conference paper at ICLR 2023
APPENDIX OVERVIEW
In this document, we provide additional details regarding the main work. More specifically:
First, in Sec. A we cover the details regarding our experimental setup, design choices, architecture details, and syper-parametest
Second, in Sec. B we provide additional experiments and results and more detailed version of some of the experiments.
A EXPERIMENTAL SETUP DETAILS
A.1 ARCHITECTURES
A.1.1 CNNS
Unless otherwise stated, the CNN models this work solely include the convolutional layers, followed by a single feed-forward layer for classification. All CNNs use the ReLU activation function and use 3x3 kernels and a stride of 2.
For the CIFAR-100 experiments, the base CNN contains 3 convolutional layers with 32, 64, and 128 channels respectively, and CNN xN refers to the base CNN model where the number of channels in each layer is multiplied by N. Hence, CNV x4 refers 3 convolutional layers with 128, 256, and 512 channels, followed by a feed-forward classification layer. In Split ImageNet- IK experiments, the base CNN contains convolutional layers with 64 channels for the first four layers and 128 channels for the last two layers. Similar to the setup for the Split CIFAR- 100 experiments, all models have a single-layer forward layer for classification and use kernel size of with stride of 2.
When we use pooling layers in CNNs(e.g.. Sec. 3.4), to keep the dimension of features the same with the CNNs that don't i have pooling layers, we use stride of for convolutional layers. In other words, every convolutional layer leads to a reduced feature map by factor of 2: either the convolution has stride (e.g., CNNs in Tab. [T or else it has a stride followed by pooling layer (e.g., CNN+(Avg/Max)Pod in Tab. 4 and Tab. Finally, for the experiments with kippcconecctions added, we add skip-connections (with projections) for the 3-layer CNNs on Split CIFARIO0 that adds shorteuts from layer to output of layer 2 and layer 2 to the output of layer 3.
1.1.2 RESNETS
The ResNets we use in the ImageNet experiment are the standard models, and we do not modify them. However, the ResNets in the CIFAR-100 experiments use 3 kernels with a stride of 1, rather than the 7 X 7 kernels with stride of 2. This is a common practice for the ResNet models for low-dimensional CIFAR images since it does not reduce the dimension of input significantly. Other than this modification for 100 experiments, the rest of the ResNet architecture kept the same.
A. 1.3 WIDERESNETS
The WideResNets (WRN) models on both CIFAR- 100 and ImageNet benchmarks follow the original implementation of WRN models. For all experiments, we use dropout factor of 0.1 as we empirically observed increasing the dropout does not improve the performance significantly.
1.1.4 VISION TRANSFORMERS
The Vision Transformer (ViT) models in our ImageNet experiment follow the same architecture as the original vision transformers. Similar the original ViT models, we use the patch size of 16 for both VIT models in our ImageNet- K experiments.
However, since the original vision transformer paper does not provide the details for the best practices for the CIFAR benchmark, we used smaller versions of the ViTs to match the training cost of other models. In those experiments, VIT 5/2-1024 refers to ViT model with layers, with width of 512 and MI.P size of 1024. Similarly, VIT 1024-1538 has width of 1024 with the MLP hidden size of 1536. All models use the patch size of 4 (i.e., 64 patches for CIFAR images), but we empirically observed increasing the patch size to 8 does not impact the results significantly.
13

---

Under review as a conference paper at ICLR 2023
A.2 HYPERPARAMETERS
In this section, report the hyper-pane paper-parameterss used for experiments. We include the chosen Hyper-parameter for each architecture in parentheses.
A.2.1 ROTATED MNIST
We follow the setting in (Mirzadeh al. 2022 for our MNIST experiments in Sec. 3.1
learning rate: [0.001, 0,01 IMGP), 0.95, 0.2]
momentum: [0.0 (MLP), 0.8]
weight decay: [0.0 (MLP), 0.0001]
batch aize: [16, 32, 54 (MLP)
A.2.2 SPLIT CIFAR-100
We the following grid of aeper-parameters for the CIFAR-100 experiments:
learning rate: [0.001, 0.005, 0.31 (ViT 1024, ReaNet 30/101), 0.05 (CNNs, ResNet 18/34, Vi7 512, WANS), 0.13 learning rate decay: [1.0 (Reatet 50/101), 0.8 (CNN, ViTs, ResNet 18/34, WRNe) momentum: [0.0 (CNN, Vits, ReaNet 50/101), 0.6 (Restet 13/34, WRNs)]
weight decay: [0.0 (CNNS, vits), 0.0001 (ResNets, WHNS)]
botcch size: [8 (ONNs, ResMet1 8/34),
(MRNs) 64 |ReaNet 50/101, VIT 5121, 128 1VIT (024))
We note that the learning rate decay is applied after each task.
A.2.3 SPLIT IMAGENET-IK
Due computation budget, we smaller smaller grid for the ImageNet- IK experiments. However, we make sure that the grid is diverse enough to cover various family of architectures.
learning
[0.005, 0,01, 0.05, 0.1 (A11 modeis)
learning rate decay: [1.0, 0.75 (A11 models]]
momentum: [0.0, 0.8 modelst
weight decay: [0.0 (CNNs), 0.0001 (8egNets, KRN, viTs)
batch size: [64 (A) modela), 256]
A.3 TRAINING INFRASTRUCTURE
For each run of the MNIST and CIFAR-100 experiments, we have used NVIDIA V-100 GPU, For the ImageNet-1K K experiments, have used 32 TPU v2 or v3. Moreover, have used JaxBradbury et al. 2018). Haiku Hennigan et al. 22020), and Optax Hessel et al. (2020) for the implementation.
B ADDITIONAL RESULTS
B.1 ALGORITHMS vs. ARCHITECTURES
In Fig. la we have provided the results for the ResNet-18 model. Here, provide the average accuracy and average forgetting for various models and algorithms on split CIFAR-100 with 20 tasks.
Table 8: Different Algorithms and Architectures
Algorithm
Parameters
Architecture
Average Accuracy
Average Forgetting
Finetune
11.2
ResNet-18
45.0 +0,63
36.8 +-1.08
11.2
AGEM (Mem = 1000)
11.2
ResNet-18
61.8 +0.45
22.9 +-1.59
1000)
11.2
Finetune
11.9
ResNet-18 18 (w/o GAP)
67.4 40.76
11.2
(Mem 1000)
11,9
ResNet. GAP)
ER (Mem 1000)
11.9
ResNet-18 18 (wro GAP)
74.4 +0033
4.6 +0.54
Finetune
2.3
CNN x4
68.1 :10.0
8.7 10.21
ER (Mem 1000)
2.3
CNN x4
74.4 #0.27
2.4 2..12.12
We remind the reader that the aim of our work not to undermine the importance of continual learning algorithms. On the contrary, we appreciate the recent algorithmic improvements in the continual learning literature. The aim of this work is to show that the role of architecture significant, and good architecture can complement good algorithm in continual learning.
14

---

Under review as a conference paper at ICLR 2023
B.2 BATCHNORM STATISTICS ON SPLIT CIFAR-100
We show the first layer's BN statistics for CNNx4 in Fig. using kernel density estimation with Gaussian kernel. As illustrated, the batch statistics and learned BN parameters do not change significantly across different tasks. Here, for simplicity, we have shown only four tasks from the beginning, middle, and late of the learning experience. Moreover, we focus on the first layer's statistics, which is the first layer of the model that operates the data.
running_mmm
bn1 running, var
tack
task
task 30
1ak115
task
task
bn1 weight
bn1 bias
10
task 10
tack 15
Tade
(c)
(d)
Figure 3: BN statistics for the first layer of CNNx4 Split CIFAR-100: the statistics do change significantly throughout the continual learning experience.
B.3 BATCHNORM STATISTICS ON PERMUTED MNIST
In Sec. 3.2] have discussed that if the batch statistics change significantly across tasks, batch normalization can increase the forgetting. To this end, we design an experiment where we train two MLP-128 networks (with and without BN) the Permuted MNIST benchmark(Goodfelll et al. 2014) with five tasks. While Permuted MNIST is not very realistic benchmark, it fits our requirements for synthetic distribution shift (i.e., shuffling pixels).
While Tab. 9 demonstrates the benefit of adding BN (i.c., improving learning accuracy), we can observe significant increase average forgetting as well. To investigate this more, we visualize the BN statistics in Fig. 4 where we can see compared to Fig. the statistics change more, confirming our hypothesis in Sec. 3.2
Table 9: Permuted MNIST: The MLP with BN has slightly higher learning accuracy, but significantly higher forgetting as well.
Model
Average Accuracy
Average Forgetting
Learning Accuracy
MLPP128
86.8 +0.95
10.9 +00.88
95.5 -0.33
MLP-128+BB
73.2 +0.82
32.5 #0.72
97.8 =0.45
B.4 MORE NUMBER OF EPOCHS ON SPLIT CIFAR-100
While in our main text, we have used 10 epochs for Split CIFAR- 100, here in Tab, [0 we show that the main conclusions hold even when we train the models longer. More specifically, we can see that wider models perform better in terms of learning and retention capabilities, batch normalization and max pooling improve the learning ability of the models, and removing the global average pooling can improve the performance of ResNets.
15

---

Under review as a conference paper at ICLR 2023
bn1 running,
bn1 running
task
aak
task
aask
Lask
62
813
6.
(b)
weight
bnl bias
1aak4
task
600
<098
Figure 4: BN statistics for the first layer of MLP-128 Permuted MNIST: the statistics change more compared to Fig.
Interestingly, we can see that simple MLPs (with two hidden layers), perform surprisingly well and demonstrate a comparable performance in terms of forgetting with CNNs. However, as it is well known in the single task setup, they are not great learners of image data, and their learning accuracy is often much smaller than their convolutional models. But, it is worth mentioning that these simple MLPs still outperform the commonly used ResNets (with GAP layer) in this task iask-incremental benchmark, which is another strong evidence of the significance of architecture in continual learning.
Table 10: Comparing the impact of components in with two different settings: The components that are helpful in the short training time (c.g.. removing GAP layers, adding pooling or batch norm layers), are also beneficial when the training time is longer.
Epochs = 10
Epochs 50
Params (M)
Average Accuracy
Average Forgetting
Learning
Average Accuracy
Average Forgetting
Learning
Model
Accuracy
Accuracy
ResNet-18
11.2
45,0 1 0.63
36,8 1 1.08
74,9 3.98
37.1 0.59
48.9 1 1.35
82.4 4,83
ResNet-18 w/o GAP
11.9
67.4 + 0.76
11.2 + 1.98
74,2 + 4.79
66.1 0.44
17.3 + 1.43
80.2 4.77
ResNet-50
23.6
56.2 + 0.88
9.5 0.38
67.8 + 5.09
53.4 0.29
14.5 + 0.64
75.3 5.65
ResNect-50 w/o GAP
26.7
71.4 0.29
6.6 - 0.12
73.0 5.18
71.2 0.18
7.3 0.22
76.5 1.5=
MLP-512
1.9
53.9 + 0.99
14.0 + 1.41
63.2 + 6.35
53.7 = 1.06
17.2 1.51
64.7 : 5.77
MLP-512 BN
58.2 + 0.55
12.8 + 0.71
69.5 4.11
57.0 0.43
17.6 = 0.47
72.0
MLP-1024
4.3
57.11 0.4
9.0.1. 1.11
65.31 2.42
$7.51 1.18
13.61 1.36
73.41 3.41
MI.P- 1024
4.3
62.0+ 0.66
8.3+ 0.74
71.1+ 2.48
61.5+ 0.5
13.5+ 1.37
72.8+ 2.68
CNN x4
2.3
68.1
8.71
76.4 L 6.92
62.6 0.4
14..1...
75.2 1 6.25
CNN x4 BN
2.3
74.0 0.56
8.1
81.7 6.68
68.9 0.93
13.8 + 0.68
80.7 + 5.83
CNN x4 Maxpool
74.4
9.44
83.3 6.1
69.3 0.79
13.5 0.85
81.9 5.47
CNN x8
7.5
69.9
8.0
77.5 6.78
64.3 0.82
1.01
78.8 6.61
CNN BN
76.1 06.10.3
0.16
81.7 1 6.83
71.7 0.79
11.5 1 0.85
82.4 6.18
CNN Maxpool
7.5
77.2 0.53
0.33
84.0
2.25
12.9 1.07
84.4 5.06
16

---

Under review as a conference paper at ICLR 2023
B.5 PRELIMINARY RESULTS ON CLASS- INCREMENTAL. LEARNING
While exploring all scenarios with all architectures is beyond the scope of this work, proof of concept, in Tab. II we report the results (over five runs) on Split CIFAR- 10 with 5 tasks (5 epochs per task) in the cass-incremental scenario Most importantly,
observe:
1. The architectures (e.g., CNN vs ResNet) play significant role as we can see by various metrics.
2. Architectural decisions (e.g., removing GAP, adding MaxPooling) are beneficial and they can significantly increase the learning and/or retention capabilities of models.
Table 11: The class-incremental scenario 5-split CIFAR-10: 10: Similar to the task-incremental scenario, we can observe the significant impact of architectural changes on continual learning performance.
Params (M)
Average Accuracy
Average
Learning
Model
Forgetting
Accuracy
CNNx4
1.6
62.0+(1.44)
74.4+(1.11) 15.6++0..55)
CNNx8
6.2
66.5=(1.23)
12.1=(0.5)
75.22:0.83)
CNNx4 MaxPool
1.6
69.1=(0.45)
15.9=(1.07)
82.02(1.09)
CNNx8 MaxPool
6.2
(2.51(0.18)
(2.8.1(0.91)
84.31(0091)
ResNet-18
11.1
62.11:(.22
37.7=(1.08)
92.2110.68)
ResNet- 18 w/o GAP
11.2
(0.64)
(7.5++0.92)
81.5+(1.57)
VIT 512/1024
8.5
64.31((011
20.5.((107)
83.1110.8)
Overall, while our small-scale experiment is limited to specific setup, still, the conclusions are in line with our results on task-incremental setup.
17


Reviews:
[{"review_id": "uniVsvEo5d", "reviewer": "Reviewer_b2F4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper empirically looks at how changing the neural network architecture can impact continual learning performance. For example, if changing width/depth of the network, or adding BatchNorm layers, or adding skip connections, changes how well the network can learn continually. Many experiments are performed on Split CIFAR-100 and Split ImageNet-1k, which is a large-scale benchmark. Recommendations are made based off empirical results. ", "summary_of_the_review": "Unfortunately points 6, 7, and 8 in weaknesses are very important for me and therefore I recommend reject. I am not convinced that these results will hold for continual learning algorithms that are not fine-tuning. I also need to see and compare all results with Joint accuracy. Therefore I do not agree with many of the claims made in the abstract. ", "strength_and_weaknesses": "**Strengths**\n\n1. Many experiments are run, including on ImageNet-1k, which is large-scale for continual learning.\n\n2. The paper is well-written: it reads well and the paper structure is good. \n\n3. I am not aware of many other papers that look at how changing the architecture impacts continual learning performance (aside from the papers cited in the Introduction of this paper). \n\n4. I liked the style and content of the 'Limitations' paragraph in the Introduction.\n\n5. I like the result of Figure 1a: ResNet-18 without GAP layer but with fine-tuning outperforms EWC (and ER with 1000 samples) on standard ResNet-18. \n\n\n**Weaknesses (especially first 3 points)**\n\n6. This paper (almost) exclusively uses fine-tuning as the 'continual learning' algorithm to test different architectures. This is a problem because I am not sure if results with fine-tuning translate to results with other algorithms like EWC, ER, or more recent algorithms. As a random representative example: perhaps, using EWC with skip connections improves performance considerably over EWC without skip connections? Other algorithms are considered in Figure 1a and Appendix B1, but I believe such results need to be throughout the entire paper. In Appendix B1, the authors say that 'a good architecture can complement a good algorithm in continual learning': please convince me that this is indeed the case! \n\n7. The authors draw a lot of conclusions from changing an architecture choice and looking at continual learning metrics. For example, with skip connections and pooling layers, they say that there is not much improvement in continual learning. However, I find this misleading as the joint accuracy on this benchmark also does not improve significantly. Previous works (cited in the paper) argue that these architecture changes are important for image classification / joint accuracy in general, but this is clearly not the case with the benchmark used. So I cannot tell if these architecture changes are helpful or not in continual learning: ideally, they would help in joint accuracy before drawing conclusions about continual learning. For MaxPool, the joint accuracy increases by a similar amount as the average accuracy. (I note that the authors do not claim anything specific in the text, but I view these results as not very helpful/useful when it comes to architecture design for continual learning.) \n\n8. Following up from this point, the 'joint accuracy' baseline is missing in many of the tables/results. I want to *always* see if joint accuracy increases with an architecture change, and how this change correlates with the continual learning average accuracy change. Else, any benefits in changing the architecture could just be due to the joint accuracy increasing, which says the importance of this architecture change to the specific datasets, but not to the ability of a network to perform a good stability-plasticity trade-off in general. \n\n9. Section 3.1 is about width and depth of neural networks, which was already studied in Mirzadeh et al. There is no new insight or takeaway in this paper. I agree that this paper's results verify the same conclusions on more benchmarks and architectures, however, this is not too significant by itself. \n\n10. Removing GAP in Table 5 improves results, but also drastically increased number of parameters. This is an interesting result but can the authors dig deeper? For example, CNNx8 has worse average accuracy than CNNx4 (16x) + GAP, despite having twice the pre-classification width. The results do not seem conclusive enough to me that GAP only reduces performance because it reduces the last-layer width. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ZMJGhIju79", "reviewer": "Reviewer_y8Pr", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work looks at empirical analysis of the effect of architecture choice and components on continual learning performance and draws a few insights on best practices for architecture selection.", "summary_of_the_review": "Effect of architecture choice on continual learning is an important area of research. A more thorough analysis needs to be done especially connecting the architecture sensitivity to the classes of continual learning approaches to make this work actionable and improve the novelty aspect which currently is less due to similarity with other sensitivity analysis works", "strength_and_weaknesses": "Strengths:\n\n•\tArchitecture choice is a key component of the modeling process and can have a considerable effect on the continual learning performance. This is an important area of research that needs to be considered. \n\n•\tThe experiments within the considered architectures have been done thoroughly with multiple random initializations, and corresponding best hyperparameters. The datasets are also well suited for the analysis did in this work.\n\n•\tIt is interesting to see the effect of various architecture components such as width, depth, batchnorm, skip connections, pooling layers and attention heads.\n\nLimitations:\n\n•\tOnly Task-incremental has been primarily studied and not the class-incremental learning setting which is a harder problem\n\n•\tThe effect of width and depth have already been studied in Mirzadeh et al. 2022, and effect of normalizing schemes has been \nstudied in Pham et al 2022, effect of distribution shift is partially studied in Paul and Chen 2022, with these having similar conclusions as this work.\n\n•\tEffect of skip connections is not fully developed. For example, skip connections can be applied at different spans of layers. It is not clear what is used in this work.\n\n•\tThe explanation of why max pooling helps is speculative, visualizing the activations might be helpful instead of speculating that low-level feature might have been effectively learned and helped generalization.\n\n•\tMajority of the work looks at naïve training in the continual learning setting but the effect of the architecture choice on various classes of continual learning is not fully developed (table 8). I was expecting to see how for example the architecture choices effect the experience-replay methods etc., which will be more useful for future algorithm development. The approaches considered in table 8 are also not the state of the art. If the state-of-the-art approaches are not sensitive to architecture choice then this work will not be useful.\n\n•\tNot clear how the hyperparameter search is done differently for continual learning compared to the single-task learning. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4RmNjFCs25U", "reviewer": "Reviewer_4mDG", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the architecture choices in the context of countinual learning (CL). It \n\n1) compares different popular architectures\n2) study component choices for these architectures, such as width/depth, BN, skip, pooling etc.\n3) draw conclusions from the study to make pratical suggestions.\n\nThe experiments are on Rotated MNIST, CIFAR-100 Split and ImageNet-1k Split.\n", "summary_of_the_review": "Overall the paper is well-written. The studies done is novel and very informative. In particular, authors draw conclusions that have good empirical evidence and also point out things that do not have enough evidence to support. The suggestions given are also pratically useful for other researchers. I recommend accept.", "strength_and_weaknesses": "Strength:\n\n1. Well-organized goals and supportive experiments. It is a pleasure to read this paper because all of the factors that the author wish to study are well organized, with direct experiments to support the conclusions. Experiments are particularly supportive because 1) the metrics chosen (average forgetting, average accuracy, learning accuracy) are meaningful for CL; 2) direct controlled experiments to study each individual factor.\n\n2. Great pratical suggestions summarized for researchers to be able to use the conclusions in this paper. For example, from the paper, it is clear that \"ResNets and WRNs have better learning abilities, whereas CNNs and ViTs have better retention abilities. Simple CNNs achieve the best trade-off between learning and retention.\"\n\n3. This architecture study can complement algorithm study. From Appendix B.1. It shows briefly that CL algorithms can further improve from the arhictecture that is chosen by this paper.\n\n\n\nWeakness:\nNot exactly weakness, but I am curious of the conclusions with other more recent architectures, and other tasks. The authors do not need to provide these in the rebuttal. If the evaluation pipeline code can be released publicly, it would be even more impactful (other researchers can evaluate those new architectures and contribute). \nAnother thing is that how CL algorithm and architecture impact each other is also an interesting future work, from a pratical view.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject