Designing Less Forgetful Networks for Continual Learning
Authors: Nicholas I-Hsien Kuo
Mehrtash Harandi
Nicolas Fourrier
Gabriela Ferraro
Christian Walder
Hanna Suominen
Venue: ICLR 2022 Submitted (2022)
Keywords: Continual Learning

Abstract: Neural networks usually excel in learning a single task. Their weights are plastic and help them to learn quickly, but these weights are also known to be unstable. Hence, they may experience catastrophic forgetting and lose the ability to solve past tasks when assimilating information to solve a new task. Existing methods have mostly attempted to address this problem through external constraints. Replay shows the backbone network externally stored memories; regularisation imposes additional learning objectives; and dynamic architecture often introduces more parameters to host new knowledge. In contrast, we look for internal means to create less forgetful networks. This paper demonstrates that two simple architectural modifications -- Masked Highway Connection and Layer-Wise Normalisation -- can drastically reduce the forgetfulness in a backbone network. When naively employed to sequentially learn over multiple tasks, our modified backbones were as competitive as those unmodified backbones with continual learning techniques applied. Furthermore, our proposed architectural modifications were compatible with most if not all continual learning archetypes and therefore helped those respective techniques in achieving new state of the art.

Under review as a conference paper at ICLR 2022
DESIGNING LESS FORGETFUL NETWORKS FOR CONTINUAL LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Neural networks usually excel in learning single task. Their weights are plas- tic and help them to learn quickly, but these weights are also known to be un- stable. Hence, they may experience catastrophic forgetting and lose the ability to solve past tasks when assimilating information to solve a new task. Existing methods have mostly attempted to address this problem through external con- straints. Replay shows the backbone network externally stored memories; reg- ularisation imposes additional learning objectives; and dynamic architecture of- ten introduces more parameters host new knowledge. In contrast, we look for internal means to create less forgetful networks. This paper demonstrates that two simple architectural modifications Masked Highway Connection and Layer-Wiss Wise Normalisation - can drastically reduce the forgetfulness in a back- bone network. When naïvely employed sequentially learn over multiple tasks, our modified backbones were as competitive as those unmodified backbones with continual learning techniques applied. Furthermore, our proposed architectural modifications were compatible with most if not all continual learning archetypes and therefore helped those respective techniques in achieving new state of the art!.
INTRODUCTION
Continual learning (McCloskey & Cohen, 1989) is difficult experimental setup for most neural networks. It places a single learning agent in a dynamic environment where it must learn to adapt from stream of tasks make new predictions. This setup means that the underlying data are not independent and identically distributed (non-iid). Thus when we use conventional gradient descent methods (Rumelhart al., 1986) to fine-tune a pre-trained network for new task, its weights will be modified to satisfy the learning objectives of the new task. As result, the re- re-parameterised network will experience catastrophic forgetting (McCloskey & Cohen, 1989; French, 1999) and abruptly lose the ability to solve a previously learnt task.
forgetnnn
Recently, continual learning has been of increasing interest. Most of the latest studies can be roughly categorised the three archetypes of replay, regularisation, and dynamic architecture. Replay techniques such as gradient episodic memory (Lopez-Paz & Ranzato, 2017) and experience re- play (Chaudhry et al., 2019) externally store handful of data per task. Then upon sequentially learning a new task, they re-expose copies of the stored data to the backbone network along those data of the new task. Regularisation methods like learning without forgetting (Li & Hoiem, 2017) and elastic weight consolidation (Kirkpatrick et al., 2017), on the other hand, impose the backbone network with secondary learning objectives to learn robust mappings that are less prone para- metric corruptions. Furthermore, the dynamic architectural approaches of dynamically expandable networks (Yoon al., 2018) and hard attention to the task (Serra al., 2018) either introduce more parameters to host new knowledge or explicitly control the backbone synaptic connections to perform inference on a subset of pre-selected weights. Though the three archetypes of continual learning methods may appear to be ideologically different, they all seek external means to retain learnt knowledge. contrast, this paper will introduce internal modifications to mitigate forgetting.
That is, we aim to create extremely plastic yet stable network architecture that can withstand catastrophic forgetting even when is naively implemented (simply fine-tuned) for sequential learn- ing over multiple tasks. Our study inspired by the recent work of Kuo et al. (202lb). In that
TThe codes will be made publicly available once the paper is published.

---

Under review as a conference paper at ICLR 2022
-Highwayy Gated Unkk
Hiihwwy 1. Operator
Norrooaato
Dab
MMIC Mask
:LWN
MLP Classifier
HCN
(c) MHC
(4) LIWN
Figure 1: Different Classifier Network Designs
paper, the authors compartmentalised their backbone network. They employed fixed feature ex- tractor and only fine-tuned their classifiers. Unlike the conventional muitiple layer perceptron (MLP) classifier of Figure ((a) used in most studies (such as Mai al. (2020)), their classifier which they called highway connection networks (HCNs) had the peculiar design shown in Figure 1(b). An HCN included a two-layer perceptron with highway connection (Srivastava et al., 2015) followed by softmax layer. We used red denote fully connected layers, yellow for softmax, and blue for the HCN gated unit. They demonstrated that HCNs were more robust against forgetting than MLPs; and even showed that naïvely sequential learning with HCNs could often outperform MLP classi- fiers with continual learning techniques applied. Moreover, they found that highway connections were specifically required and could not be replaced with residual connections (He et al., 2016).
In this paper, we introduce two novel modifications to HCNs. Our new ideas draw inspiration from the stability-plastiiitty dilemma (Mermillod et al., 2013). Stability is the ability to retain learnt knowledge upon parametric corruption (Sun et al., 2021); whereas plasticity refers to the ease of assimilating new information. Our study focuses on designing new architectures in the forward pass which indirectly but positively affect the updates in the backward pass. This kind of approach was previously investigated in weight normalisation (Salimans & Kingma, 2016). In that paper, the authors found that by reparameterising the weight vectors, they could influence the gradient covariance matrix by making it closer an identity matrix thereby hastening network configuration.
We propose the Masked Highway Connection (MHC) and Layer-Wise Normalisation (LWN) shown in Figures 1(c) and 1(d) respectively. While similar techniques exist for general machine learning; their applications for continual learning are not well studied. After discussing how HCN mitigates forgetting (Section 3), we will introduce our MHC and show that HCNs can be made even more robust via self-identifying important weights that are worth fine-tuning through masking mechanism (Section 4). Then, we will alternatively characterise forgetting as shifts that occur in previous distributions (Ebrahimi et al., 2020); and that the LWN can reduce such shifts (Section 5).
Our experiments (Section 6) indicated that both MHC and LWN were robust against forgetting and greatly outperformed their HCN and MIP baselines. In addition, both designs were compatible with the continual learning archetypes of replay, regularisation, and dynamic architecture; and they were able to help those respective techniques to achieve new state of the art. Moreover, MHC and LWN could also be jointly implemented. Together, they often produced even more robust results and this was because they each addressed different aspect of catastrophic forgetting. In order suggest the best practice for MHC and LWN for continual learning, we also tested our novel classifiers over several different settings. Importantly, we found that LWN should not be incautiously substituted with butch normalisation (loffe & Szegedy, 2015).
2 THE CONTINUAL LEARNING FRAMEWORK
As stated Lopez-Paz & Ranzato (2017), the continual learning setup exposes a base learner F to continuum of data with locally, but not globally, iid tasks where
(xt, w1, ((t.wivii) vi)...,, (x) Wi, V......, (#% wn. vo)
(1)
Each task corresponds dataset (æ) vL) Dw, with & samples = 1,..., Φ. Once the learner F observes data from the j-th task, full access an earlier dataset w; where < j is prohibited to consolidate learnt knowledge. The performance is tracked using matrix 2 1)*+1)x> where each column represents a distinct task. When F is initialised, its accuracy over all tasks is recorded in row o (1mm))) After learning the i-th task, the new performances updated in row 11+4))(11)))
2

---

Under review as a conference paper at ICLR 2022
3 HIGHWAY CONNECTIONS IMPLICITLY RE-SCALE GRADIENTS
intermediate layer Tyy in an MI.P can be represented as
م (aL,, 1) WL,)
(2)
where the layer output az, is transformed from the layer input a over the non-linear function H with weight Wt,, In addition, say there are I' layers then the network output is atr y. To fine-tune W by we use gradient descent with the derivative of a cost C with respect to the parameters
Rel: Kno al. (202
ac
ac
DaL
daL, OW L,
OWL,
dAlC
Rel: Srivastava et al. (2015)
Below, we discuss how gradient vanishing and catastrophic forgetting are closely related (Rel:). Deep MLPs are known to suffer gradient vanishing (Bengio et al., 1994). When the layers increase but the magnitudes of the individual Jacobian terms daii
small (i.e., E [0. 1)), the term
delic+1) diminishes to 0 (and so does JW and no effective update made to WL,,
<=Y daLc
Highway connection (Srivastava al., 2015) is well-known for its ability withstand gradient vanishing. It reformulates an intermediate layer as
k1, (1 TT,) kL, TL H (kbr- W (,) with
gate Tr, T(kky QI.))
(5)
and the element-wise dot product @). The colour-coded gated unit Tr,, has its weight Qr, and its values lie the range of [0, 1). This new formulation has the Jacobian term
ūkL, Oki
if T = if T =
(6)
WL,).
By introducing the identity matrix II to the Jacobian (and thus the [.] term of Equation (3)), highway connections prevent the vanishing of gradients and thus facilitate the training of deep networks.
In Kuo et al. (2021b). the authors noted a second property unique to highway connections. While the (aL, 1) W.,) a highway connection had the OWL,
BAL.
(-) term of Equation (3) of an MLP was BW by
alternative form of 0k LL, 3.46 (kL We,) which included the gated unit. Since each BW L. entry in Tr,, lied in [0, 1]. highway connections thus implicitly scaled down the magnitudes of the gradients used in gradient descent. This was desirable for continual learning because less changes were made pre-trained weight W.,; and HCNs were hence less susceptible to forgetting.
oW.,
4 IMPROVING THE HCN ARCHITECTURE
Though highway connection is effective against catastrophic forgetting, its best practices and short- comings remain unclear. this section, we review the different roles that the gated unit L., plays in the forward pass (inference) and in the backward pass (update). Furthermore, we will show that there exists an interesting structural connection between the HCN gated unit Ty,, and the derivative 8 (k1,, WL,) which can be exploited to make the network even more robust to forgetting. OWL,
Overleaf in the top half of Figure 2, we show that HCN performs a point-to-poot gradient reduction. On the left, we superimpose dotted box on top of HCN. The inference and update that occur within this box are illustrated on the right. Without loss of generality, the scenario the figure is for 3 hidden dimensions and that each Tr., entry scales down its corresponding feature of the fully connected layer output ki,, during inference. Note that the entries of kL, are shown in different shades of red; and they correspond specific rows in the weight W L,, Likewise for update, 0.3 (KL (kL
highlight the rows of
in different shades green; and we show how each row of
8WL, this derivative is scaled down, in point-to-point fashion, by the entries of the gated unit T L...
3

---

The wolf Figure presents the inner workings of our Masked Highway Connecton (MHC) classifier. We add binary Mask to HCN and change the formulation from Equations (4) and (5) to Under review as a conference paper at ICLR 2022
HCN
Inference:
Tr
WI, L, KL
T kg,,
1 +
|
Update:
STOTET
01
Tz
Barr n: Wr.)
Ti
8,20 (kr Wr.)
oW
OW
Inference:
Mask
T7.
WL, L, kg
Mask
kL,
MHC
k
Update:
MHC
8. Wx.)
MHC
(kr, Wa,)
2W,
8W,,
Figure 2: Differences in the Inferences and Updates of the HCN and MHC Classifiers
But does the entirety of weight W L., need to be updated at all times? Two prior studies, the lottery ticket hypothesis (L.TH) (Frankle Carbin, 2019) and the uncertainty-regularivee continual learning (UCL) (Ahn et al., 2019), suggested not. In network pruning, L.TH found that only small subset of the dense network needed to be updated to achieve competitive performances. Whereas UCL showed that continual learning could be achieved by regularising specific nodes in the weight that deviated from determeerdn distribution. They both considered the structural relationship among nodes in the base learner. A similar approach can be applied to HCN.
k
L.,, kL,
GL,, 3 G. (kL WL,) with
masked gate Gr., = Mask ④ Tz, Mask @ T(kE Qz
The Mask keeps the largest neural values in Tı, with the top-X function. As shown in Fig- 2, when TT, R3 and that X = 1, the MHC masked gate GL, nullifies the two with weaker activation values. The mask also nullifies specific updates during optimisation. In our exper- iments in Section 6, we will demonstrate that this minimalistic update can reduce excessive weight modification thereby improving network stability for the sequential learning setup.
5 FORGETTING, MEASURED THROUGH FEATURE SHIFTS
The gating mechanism in HCN showed that Kuo et al. (2021b) regarded forgetting as the amount of change that occurred in the pre-trained weights. However, both Ebrahimi et al. (2020) and Ahn et al. (2019) quantified forgetting through alternative means. Both studies employed Bayesian neural networks (BNNs) (Mackay, 1992) for continual learning. The former conceptualised forgetting as shifts that occurred the distributions of weights; and the latter showed that BNNs could become stable by regularising the network with changes in the standard deviations of the weight matrix.
Thus in addition to MHC, we further propose Layer- Wise Normalisation (LWN) to mitigate shifts in the classifier. However, address internal covariate shifts (ICSs) (Shimodaira, 2000) instead of weight distribution shifts. ICS refers to the change in network activation distributions due to parametric changes during training; and will show that forgetting can also be mitigated at the feature level. An LWN with hidden dimension H changes Equation 2 to
aj., -HL, from ai H Aflalaa (aL WL,) with (9)
normalised layer aL., o
H
H
mean:
PL.,
a Loon
and standard deviation: OL,
(a 2nn, - |L,,), (10)
H
H 1=
n=
4

---

we re-applied BMNIST and collected the new activation a for each ξ-th update. We then plotted Under review as a conference paper at ICLR 2022
-24
Eerstion
(a) MLP with 100
(b) LWN with 100
Figure 3: Longitudinal Changes in Activation Values After Task Changing
In order to demonstrate the effectiveness of LWNs over MLPs, we prepared a toy example which first trained both classifiers with MNIST (LeCun et al., 1998) and then with FashionMNIST (Xiao al., 2017). Both classifiers had the hidden dimensions H = 100 with the two layer designs shown in Figures ((d) and ((a) respectively. Both datasets contained grey-scale images on 28 28 pixel boxes with 60K and 10K images for training and testing. After pre- training both classifiers on MNIST, we externally stored batch of MNIST images BMNIST and applied them on the classifiers to collect the output activations of the second layer a (0)
L2 When we fine-tuned the classifiers for MNSSi
the changes in activation in Figures 3(a) and 3(b). See Appendix for more details in the setup.
The changes in activation were defined as
Diffact = abs
LE PANNSTT ( (na
(0) L2,m - a aL2..
(11)
and it measured the magnitudes of the expected differences the changes of the layer two activation (over samples b in BMNIST). The deeper the colours the larger the differences. Each row represented the changes in a unique feature a after the classifier received updates to solve FashionMNIST. Each row generally became deeper and this showed that ICS occurred during re-parameterriatiion and that the learnt mapping for MNIST images BMNIST had diverged. After fine-tuning, we found that the divergence was Diff 86.44 in MLP and Diff 69.03 in LWN. Forgetting was hence 20%(- - [1 - 69.03/ 86.44] x 100%) less severe in LWN than in MLP. This was because that the differences in LWN features became smaller after redistributing the activation along the feature dimension. LWN also scaled well with larger dimensionality; see Appendix B for details.
Unlike the well-known layer normalisation (Ba et al., 2016), our LWN did not have learnable param- eters to reshape the post-normalised activation distribution. This was because that during our exper- iments, we found that these features were not necessary. Also, found that LWN was specifically required to mitigate forgetting; and it could not be replaced by the popular batch normalisation (Ioffe & Szegedy, 2015). This will be further discussed in Section 6.3.
EXPERIMENTS AND RESULTS
We compared six classifier networks with four continual learning setups over three datasets. The main results were summarised in Tables and 2, with more results in Tables 4 - 12 in the appendices.
The Datasets
We tested image permutation (Kirkpatrick et al., 2017) on MNIST (LeCun et al., 1998), and in- cremental classes (Rebuffi et al., 2017) on Cifar100 (Krizhevsky, 2009) and CUB200 (Wah et al., 2011). MNIST was previously mentioned in Section 5. Cifar100 included 100 classes of 32 x 32 coloured images: with 50K for training and 10K for testing. Whereas CUB200 contained 5, 99994 training and 5, 794 test images over 200 coloured bird species mostly 500 x 500 pixel boxes.
The Tasks
Permuted MNIST (Perm-MNIST) was derived by first reshaping MNIST images as vectors of 784(= 28 x 28) pixels, then by applying a unique permutation on all vectors of that task. On the other hand, incremental C0 (Ine-Cifarl0) sequentially introducing a fixed set of classes per task drawn without replacement from the datasets. There were 20 tasks for Perm-MNIST For 20 classes per task. Likewise for Inc- CUB-C0,00 we introduced 10, 20, and 40 classes per task. (The Inc scenarios thus had 20, 10, and 5 respective tasks.) We mainly followed Douillard et al. (2020)
and incremental CUB200 (Inc-CUB200) were generated via
we tested three scenarios with 5. 10. and
5

---

Under review as a conference paper at ICLR 2022
to process the datasets" and setup
learning (Van de Ven & Tolias, 2019).
The Backbone Networks
For Perm-MNIST, our backbone networks were simply two-layer MLPs, HCNs, and MHCs. We tested MHCs with a range of different settings - we used top-X to keep between the top 10% to the top 90% of the gated unit activation. In addition, we embedded LWN in MLPs, in HCNs, and in MHCs. All of the normalisation were applied specifically to the input of layer two. Interested readers could find elaborated illustrations in Figure 7 of Appendix C.
For Inc-Cifar 100 and Inc-CUB200, our backhone networks were compartmentalised as fixed pre- trained feature extractors with sequential learning classifiers. The feature extractors for Inc-Cifar were ResNet18: 8s (He et al., 2016) pre-trained on ImageNet (Deng et al., 2009); and those for Inc- CUB200 were ResNeXt50s (Xie et al., 2017) pre-trained on ImageNet. We replaced the final layer of the feature extractors with our classifiers. Our classifiers here were also two-layer MLPs, HCNs, and MHCs. The MHC again kept top-X with X E [10%, 90%]; however, we embedded the LWNs differently. The normalisation were applied in two different places first on the input of layer one (extracted by the feature extractors), and then on the input of layer two. Again, interested readers could find elaborated illustrations in Figure 8 of Appendix C.
Additional Training Detials
For Perm-MNIST and Inc-Cifarlee 100, our classifiers had hidden dimensions H 100; and it was H = 400 for Inc-CUB200. The Perm-MNIST backbones observed 5K images on Task I then IK images for all subsequent tasks. Likewise for Inc-Cifar100 it was 5 epochs on Task 1 then 1 epoch for the rest. However, the backbones observed 5 epochs for all Inc- Inc-CUB200 tasks. We used batch size B 10, and updated all classifiers with SGD (Rumelhart al., 1986) with learning rate 0.01.
The Continual Learning Setups
We denoted singular the scenario when the backbones were naïvely and sequentially trained (simply fine-tuned) over all tasks. Then, we tested the regularisation-baat elastic weight consoli- dation (EWC) (Kirkpatrick et al.. 2017), the memory-based experience replay (ER) (Chaudhry al., 2019), and the dynamic architectural-bast
hard attention to the task (HAT) (Serra et al., 2018)3. Some secondary experiments could be found in Appendix F.
Metrics
We used the two metrics below evaluate the sequential learning performances 1 + Ant,ii
Average accuracy
(ACC)
and
Final Accuracy of Task 1 (FAI) A.11.11
Both metrics were based on matrix af of Section 2. The classifier plasticity could be indicated through ACC, and FAI could reflect the classifier stability. All units were in percentage accuracy, the higher the better; and all results were reported as the 95% confidence intervals over 10 seeds.
6.1 RESULTS ON NAÎVELY SEQUENTIAL LEARNING
We summarised a subset of results for the singular setup overleaf in Table 1. It included all results for Perm-MNIST, Inc-Cifar 100 with 20 classes per task, and Inc-CUB200 with 40 classes per task. The baseline results for MLPs and HCNs were shown in grey, and we highlighted our results that were better than the baselines in cyan. This table reflected four important remarks.
The first remark was that, our classifier designs could outperform the baselines with large margin if they were configured properly. However the second remark showed that, the best practice of MHC varied for each task. For Perm-MNIST, the best performce in MHC was recorded when we kept the top X 70% of gated unit activation; but for Inf-rr 100 and Inc-CUB000, it was when only the top X = 10% activation was kept. This was potentially due to the contextual complexity of the datasets; such that it was possible learn with less information from the contextually rich Cifar 100 and CUB200. The third remark was that LWN greatly improved both the plasticity (ACC score) and stability (FA1 score) for all classifiers. While was well-known that normalisation could improve network plasticity (Ba et al., 2016), our work provided empirical evidence that normalisation along the feature dimension was beneficial for mitigating forgetting. In addition, the fourth remark was
s.p.pppp py in Douillard et al. (2020)'s official repository. We set the regularisation coefficient À = 3 for EWC; stored 50 images per task for ER; and applied HAT only on the first layer of classifier synapses.
Refer to inc ea rn/1 /

---

Under review as a conference paper at ICLR 2022
Table 1: Subset of Results on the Naive Sequential Learning of Classifiers
Task
Ours
Classifier
ACC (%)
FA1 (%)
Term-MNIST
MLP
66.4 23.1
67.9 11.9
HCN
72.0 1 15.5
78.7 1: 8.4
4
MHC (90%)
72.1 + 16.1
79.1 = 7.7
MHC (70%)
72.4 15.1
79.5 7.3
MHC (50%)
72.1 14.7
79.3 1: 7.4
4
MHC (30%)
71.4 14.5
78.4
V
MHC (10%)
15.1
76.1 8.4
4
LWN in MLP
68.4 + 23.5
71.3 + 11,5
4
LWN in HCN
74.6 14.2
80.5 - 7.5
LWN MHCC (70%)
75.3 13.1
8.4
Inc- Cifar 100
MLP
41.7 18.1
32.1 + 5.9
(inc. 20 classes)
HCN
46.0 12.1
41.8 + 3.8
MHC (90%)
46.1 11.9
41.2 +
MHC (70%)
46,2 12.1
40.8 + 5.9
7
MHC (50%)
11.4
40.8 + 4.6
MHC (30%)
46.7 10.8
MHC (10%)
46.9 10.7
42.0 6.1
IWN in MLP
45.3 16.0
37.0 + 8.1
LWN HCN
48.0 10.8
42.7 4.2
IWN in MHC (10%)
48.4 + 9.6
44.2 + 4.5
Inc- CUB200
MLP
55.0 + 19.7
45.3 + 11.1
(inc. 40 classes)
HCN
57.4 1. 15.3
51.7 1 6.4
MHC (90%)
56.8 15.4
51.2 7.7
V
MHC (70%)
55.2 - 17.1
48.3 + 7.6
4
MHC (50%)
56.8 15.2
51.4 + 5.8
7
MHC (30%)
57.5 15.7
53.6 + 6.7
V MHC (10%)
60.8 1. 14.0
57.6 1 4.6
LWN MLP
71.7 19.1
61.3 9.5
LWN in HCN
75.6
16.3
66.0 = 5.7
V LWN in MHC (10%)
76.2 1 15.3
67.3 1: 5.3
that MHC and LWN could be made more powerful when they were jointly implemented. This was not particularly surprising because both techniques addressed forgetting from different perspectives. MHCs considered forgetting the modifications made the pre-trained weights (see Section 4); whereas LWNs were effective in minimising the feature shifts in a learnt mapping (see Section 5).
We made more results for the singular setup available in Appendix D. We refer to Table for additional results on Inc-Cifart 100; and to Table 5 for additional results on Inc-CUB200.
6.2 RESULTS WITH CONTINUAL LEARNING TECHNIQUES APPLIED
After the singular setup, we tested all classifiers with EWC, ER, and HAT applied. Our results were summarised in Table overleaf. Due to space constraints, we selected the best performing LWN MHC results. Interested readers could find the results for all other backbone network designs in Tables 12 in Appendix D. We found three additional important remarks from the experiments.
The first remark was that our LWN in MHCs were compatible with all of EWC, ER, and HAT. Since these three techniques were from the most well-known continual learning archetypes, we expected our LWN MHCs to be compatible with most if not all existing continual learning techniques. Second, our LWN in MHCs achieved better results than the baselines. Hence, they helped the afore- mentioned techniques in reaching new state-of -ftte--rrt-ttttttt status. The third remark was that, the majority of our naively sequential learning LWN in MHCs from Table I (both ACC and FAI) achieved better results than the baseline classifiers with continual learning techniques applied. Hence, we demon- strated the importance of employing the correct classifier design to mitigate forgetting.
6.3 BATCH NORMALISATION IS NOT GOOD FOR CONTINUAL LEARNING
In this section, we tested batch normalisation (BN) (loffe & Szegedy, 2015) in MLP. The results for Perm-MNIST in Table 3 showed that BN was detrimental for the backbone networks.

---

Under review as a conference paper at ICLR 2022
Table A Subset of Results with Continual Learning Techniques Applied
Task
Ours
CL. Tech.
Classifier
ACC (%)
FA1 (%)
Perm-MNIST
EWC
MLP
68.9 1. 18.3
76.5 1: 7.0
EWC
HCN
T3.7 + 14.4
82.3 82.3 + 5.7
V
EWC
LWN in MHC (70%)
76.5 + 13.3
84.6 = 6.2
Perm-MNIST
ER
MLP
72.1 + 12.5
80.3 + 2.8
HCN
73.3 + 15.6
V
ER
LWN in MHC (70%)
76.0 + 14.3
85.9 + 1.9
Perm-Mi MONM-NIITT
HAT
MLP
67.2 + 20.6
72.3 + 18.5
HAT
HCN
71.6 + 15.1
79.2 - 6.0
HAT
LWN in MIC (70%)
13.5
83.2 1 4.0
Inc-Cifar100
EWC
MLP
43.3 15.2
36.1 +
(inc. 20 classes)
EWC
HCN
47.0 1: 10.6
44.0 1 4.9
EWC
LWN in MHC (10%)
49.2 = 8.6
46.9 = 3.2
Inc-Cifar100
ER
MLP
41.7 17.0
37.1
(inc. 20 classes)
HCN
42.1 16.2
38.5 +
ER
LWN in MHC (10%)
45.1 14.4
41.6 1 4.8
Inc-Citar100
HAT
MLP
43.4 1 13.0
38.0 1 6.0
(inc. 20 classes)
HAT
HCN
44.2 + 13.1
HAT
LWN in MHC (10%)
17.3 1 9.1
44.9 1 4.8
Inc Inc-CUB200
EWC
MLP
56.0 + 17.8
46.8 13.5
(inc. 40 classes)
EWC
HCN
58.8 +: 13.7
51.8 + 6.6
EWC
LWN in MHC (10%)
77.4 14.0
69.7
Inc-CUB200
ER
MLP
51.9 + 20.1
47.6 +
(inc. 40 classes)
ER
HCN
52.0 + 20.2
48.2 + 9.0
ER
LWN in MHC (10%)
71.8 1: 20.0
63.1 + 8.6
Inc-CUB200
HAT
MLP
55.7 + 16.1
59.6 + 6.1
(inc. 40 classes)
HAT
HCN
56.4 14.1
62.1
HAT
LWN in MHC (10%)
78.8 1: 7.7
77.5 + 4.6
BN MLP
(b) LWN MLP
Figure 4: Longitudinal Changes in Activation Values After Task Changing
In order to understand the reason why BN was disadvan- tageous for continual learning, we repeated the experi- ments in Section where we first trained BN MLP for MNIST and then fine-tuned for FaShionMNIST. We illus- trated the results in Figure 4. This figure was formatted
Table 3: Results Perm-MNIST
Classifier
ACC (%)
FA1 (%)
MLP
66.4
67.9
LWN MLP
68.4
71.3
BN in MLP
44.3
38.3
following the style of Figure 3. BN in MLP had much deeper colours than LWN in MLP. In addition, the divergence was Diffaa = 97.66 in BN MLP and forgetting was hence 41%(= [1 69.03 02.03/97.660 67666] 100%) more severe than LWN in MLP.
The deep colours first appeared after the BN in MLP started to observe data from FashionMNIST.. This indicated that the learnt parameters (i.e., the scale and the shifi) in BN required readjustments as a response to the change in source data distribution. This was expected since BN normalised neural network values along the batch dimension rather than the feature dimension. That is, while BN could decrease the ICS within a single task, it could not decrease the ICS occurring between tasks. These results showed that LWN was specifically required and that it should not be incautiously replaced with an alternative normalisation.
8

---

Under review as a conference paper at ICLR 2022
6.4 SEQUENTIAL LEARNING IN NOISY ENVIRONMENT
We further concerned the realistic scenario of continual learning under the presence of noise. The noise injection (NI) (Kuo et al., 2021a) scenario randomly shuffled a por- tion of pixels of PerS-MNIST images. The lowest setting injected 10% noise while the highest setting injected 50% noise. Refer to the details of this setup in Appendix E.
uift geea
"Oth (9ow 30%
We naïvely sequentially trained the classifiers and plotted the results in Figure 5. Both HCNs and LWN in MHCs were able to achieve better results than MLPs on plas- ticity (ACC). In addition, our novel LWN in MHCs also outperformed HCNs in stability (FAI). We attributed this to LWN in MHC's s minimalistic updates for reducing the impact of input corruption optimisation.
Figure 5: Perm-MNIST with Noise
RELATED WORK
Our results highlight the architectural importance for continual learning. The components in the forward pass not only impact plasticity, but also influence the backward pass and thus stability. From this perspective, continual learning shares many important research questions with network pruning and neural architectural search (NAS).
originally proposed to reduce both the computational and memory cost of neural networks. Han et al. (2015) found that large networks could perform with no loss in accuracy with only 10% of its total parameters. Zhou et (2019) further demonstrated that neural networks could still perform competitively when the weights were reset but with their original signs kept. This implied that the architecture had direct influence on the topology of the solution space (Li et al., 2018) and potentially also the capacity of the minima on the loss landscape (Hochreiter & Schmidhuber, 1997). This in turn decided how easy was to not forget. These insights echoed with some findings in NAS. NAS considered network building as learning problem (Zoph & Le, 2017). was recently extended to continual learning in Mundt et al. (2021); and some linear classifiers with random weights were found to perform on par with fully trained deep counterparts.
Network pruning
Besides network pruning and NAS, many work in the lottery ticket hypothesis (LTH) (Frankle & Carbin, 2019) also highlighted the importance of controlled and minimalistic updates. Gohil et al. (2019) found that when configured correctly, small and sparsified subnetwork within a dense net could be generalised across tasks. Additional studies the structural relations between subnetworks and feature distributions were also investigated in Ayinde et al. (2019) and Kuo et al. (2021a).
8 CONCLUSION
This paper introduced Masked Highway Connections (MHCs) and Layer- Wise Normalisations (L.NNs) as classifiers to mitigate forgetting for backbone networks that sequentially learn over sev- eral tasks. Unlike the reply, regularisation, and dynamic architectural continual learning archetypes, our approach did not add external constraints to prevent forgetting. Instead, we focused on internal modifications that were beneficial for the classifier design. To elaborate, our methods were able to stop forgetting without the externally stored data of replay and were thus memory efficient. Simi- larly, our methods were different to dynamic architectures and did not introduce new modules upon learning new task. Hence, our methods were also computationally efficient.
We demonstrated that forgetting could be prevented at both the weight level and at the feature level. MHC achieved the former via minimalistic updates to lower the impact of weight corruption. While LWN achieved the latter through feature redistribution retain previously learnt mappings.
MHCs and LWNs were able to outperform their baselines. In addition, they were compatible with techniques of all continual learning archetypes hence helping those techniques in achieving new state of the art. Moreover, LWN in MHCs were very stable even when they were the naively implemented; they often achieved better results than baseline classifiers with continual learning techniques applied.

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
This paper investigated simple yet effective classifier designs to prevent catastrophic forgetting in the continual learning setup. All of the experiments conducted in this paper were still on the nicely crafted datasets of MNIST, Cifar100, and CUB200. Thus, the content of this work does not present any foreseeable societal consequences.
REPRODUCIBILITY STATEMENT
All of the important hyper- peeerppaametticc settings were described in text. Interested readers should refer to The Backhone Networks and Additional Training Details in Section 6 for more details. In addition, the authors of this manuscript promise to make the codes of this paper publicly available when the paper is published.
REFERENCES
Hongjoon Ahn, Sungmin Cha, Donggyu Lee, and Taesup Moon. Uncertainty-Based Continual Learning with Adaptive Regularisation. In the Advances in Neural Information Processing Systems, 2019.
Babajide O Ayinde, Tamer Inane, and Jacek M Zurada. Redundant Feature Pruning for Acceler- ated Inference in Deep Neural Networks. Neural Networks, 2019.
Jimmy Ba, J. Kiros, and Geoffrey E. Hinton. Layer Normalisation. In the Neural Information Processing Systems Deep Learning Symposium, 2016.
Yoshua Bengio, Patrice Simard, and Paolo Frasconi. Learning Long-term Dependencies with
Gradient Descent Difficult. IEEE Transactions on Neural Networks, 1994.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Mare' Aurelio Ranzato. On Tiny Episodic Memories in Con- tinual Learning. In the International Conference on Machine Learning Workshop on Multi-Task and Lifelong Reinforcement Learning, 2019.
Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A Large- Scale Hierarchical Image Database. In the IEEE Conference on Computer Vision and Patlern Recognition, 2009.
Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. Podnet:
Pooled Outputs Distillation for Small-Tasks Incremental Learning. In the European Confer- ence on Computer Vision 2020.
Sayna Ebrahimi, Mohamed Elhoseiny, Trevor Darrell, and Marcus Rohrbach. Uncertainty Continual Learning with Bayesian Neural Networks. In the International Conference on Learning Representations, 2020.
Jonathan Frankle and Michael Carbin. The Lottery Ticket Hypothesis: Finding Sparse, Train-
able Neural Networks. In the International Conference on Learning Representations, 2019.
Robert M French. Catastrophic Forgetting in Connectionist Networks. Trends in Cognitive Sciences, 1999.
Varun Gohil, S. Deepak Narayanan, and Atishay Jain. One Ticket to Win Them All: Generalising Lottery Ticket Initialisations Across Datasets and Optimisers. In the Advances in Neural Information Processing Systems Reproducibility Challenge, 2019.
Song Han, Jeff Pool, John Tran, and William Dally. Learning Both Weights and Connections for Efficient Neural Network. the Advances in Neural Information Processing Systems, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In the IEEE Conference Computer Vision and Pattern Recognition, 2016.
10

---

Under review as a conference paper at ICLR 2022
Sepp Hochreiter and Jürgen Schmidhuber. Flat Minima. Neural Computation, 1997.
Sergey loffe and Christian Szegedy. Batch Normalisation: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In the International Conference on Machine Learning, 2015.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiaoo Ramalho, Agnieszka Bubbska-Barwinska. Demis Hass- abis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming Catastrophic For- getting in Neural Networks. In the Proceedings of the National Academy of Sciences, pp. 3521- 3526, 2017.
Alex Krizhevsky. Learning Multiple Layers of Features from Tiny Images. In Tech Report of the University of Toronto, 2009.
Nicholas IH Kuo, Mehrtash Harandi, Nicolas Fourrier, Christian Walder, Gabriela Ferraro, and Hanna Suominen. Learning Continually Learn Rapidly from Few and Noisy Data. In the Metannearning and Co-Hosted Competition of the AAAI Conference on Artificial Intelligence, 2021a.
Nicholas IH Kuo, Mchrtash Harandi, Nicolas Fourrier, Christian Walder, Gabriela Ferraro, and Hanna Suominen. Plastic and Stable Gated Classifiers for Continual Learning. In the IEEE Conference on Computer Vision and Pattern Recognition Workshop on Continual Learning in Computer Vision, 2021b.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-Based Learning Ap- plied Document Recognition. Proceedings of the IEEE, 1998.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualising the Loss Landscape of Neural Nets. the Advances in Neural Information Processing Systems, 2018.
Shuang Li, Yilun Du, Gido M Van de Ven, and Igor Mordatch. Energy-Based Models for Contin- ual Learning. arXiv preprint arXiv:2011.122166 2020.
Zhizhong Li and Derek Hoiem. Learning without Forgetting, In the IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
David Lopez-Paz and Marc' Aurelio Ranzato. Gradient Episodic Memory for Continual Learn- ing. In the Advances Neural Information Processing Systems, 2017.
David John Cameron Mackay. Bayesian Methods for Adaptive Models. Doctoral Dissertation, California Institute of Technology, 1992.
Zheda Mai, Hyunwoo Kim, Jihwan Jeong, and Scott Sanner. Batch-level Experience Replay with Review for Continual Learning. In the IEEE Conference on Computer Vision and Pattern Recognition Workshops of Continual Leanrning in Vision, 2020.
Michael McCloskey and Neal J Cohen. Catastrophic Interference in Connectionist Networks:
The Sequential Learning Problem. In Psychology of Learning and Motivation. 1989.
Martial Mermillod, Aurélia Bugaiska, and Patrick Bonin. The vestigating the Continuum from Catastrophic Forgetting to Age-Limited Learning Effects. Frontiers Psychology, 2013.
Dilemma: In-
Martin Mundt, Iuliaa Pliushch, and Visvanathan Ramesh. Neural Architecture Search of Deep Priors: Towards Continual Learning without Catastrophie Interference. In the IEEE Confer- on Computer Vision and Pattern Recognition Workshop Continual Learning in Computer Vision, 2021.
Siivestrr-Alvisee Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. iCaRL: Incremental Classifier and Representation Learning. the IEEE conference on Computer Vision and Pattern Recognition, 2017.
11

---

Under review as a conference paper at ICLR 2022
David E Rumelhart, Geoftrey E Hinton, and Ronald Williams. Learning Representations by Pacc--rroggg Errors. Nature, 1986.
Tim Salimans and Durk P Kingma. Weight Normalisation: A Simple Reparameterisation to Ac- celerate Training of Deep Neural Networks. In the Advances in Neural Information Processing Systems, 2016.
Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming Catastrophic Forgetting with Hard Attention to the Task. the International Conference on Machine Learning, 2018.
Hidetoshi Shimodaira. Improving Predictive Inference under Covariate Shift by Weighting the Log-Likelihood Function. Journal of Statistical Planning and Inference, 2000.
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway Networks. In the Workshop of International Conference on Machine Learning, 2015.
Xu Sun, Zhiyuan Zhang, Xuancheng Ren, Ruixuan Luo, and Liangyou Li. Exploring the Vulnera- bility of Deep Neural Networks: A Study of Parameter Corruption. In the AAAI Conference on Artificial Intelligence, 2021.
Gido M Van de Ven and Andreas S Tolias. Three Scenarios for Continual Learning. arXiv preprint 17X1v:190..07734, 2019.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The Caltech- UCSD Birds-200-2011 Dataset. California Institute of Technology. 2011.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: A Novel Image Dataset for Benchmarking Machine Learning Algorithms. arXiv preprint (tt.iv:1708.077:::: 2017.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated Residual Transformations for Deep Neural Networks. In the IEEE Conference Computer Vision and Pattern Recognition, 2017.
Jachong Yoon, Eunho Yang, Jeongtac Lee, and Sung Ju Hwang. Lifelong Learning with Dynam- ically Expandable Networks. In the International Conference on Learning Representations, 2018.
Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. Deconstructing Lottery Tickets: Ze- ros, Signs, and the Supermask. In the Advances in Neural Information Processing Systems, 2019.
Barret Zoph and Quoe V Le. Neural Architecture Search with Reinforcement Learning. In the International Conference on Learning Representations, 2017.
12

---

Under review as a conference paper at ICLR 2022
A DETAILS ON THE EXPERIMENTAL SETUP OF SECTION 5
This section aims to provide more details on the experimental setup for the toy experiment in Section The purpose of the experiment is compare the changes in the learnt mapping between an MLP classifier and an LWN in MLP classifier when we change from training them on MNIST (LeCun al., 1998) to MaSST (Xiao et al., 2017).
The design of the baseline MLP classifier looked like that in Figure ((a), and similarly the LWN in MLP classifier took the shape of that in Figure I(d). Both classifiers had 2 layers with in- put dimension 784, hidden dimension 100, and output dimension 10; and they were trained with SGD (Rumelhart al., 1986) with learning rate 0.01. As shown in the figure, the normalisation in LWN in MLP was applied at the input of layer two (equivalently, the output of layer one).
The two classifiers were first trained on MNIST. They were exposed 5K copies MNIST images in batches of size 10. Prior to input, each image was flattened After learning from the copies, we externally stored a subset of MNIST images size 50 (i.e., | 50). Then, applied the saved images BMNIST to the pre-trained classifiers to collect the layer two activation 2 (0)
vector of size 784(=28 x 28).
We proceeded to fine-tune the classifiers on 10K images of FashionMNIST. Similar to the previous setup, each image was flattened as vectors of 784 pixels and that the batch size was 10. There were hence 1K updates. After each step of update, we re-applied BMNIST to the fine-tuned classifiers and (8)
collected the new activation values at layer two L We then recorded the longitudinal changes in
the learnt mapping for MNIST as abs [
(aa La.9
(g)
- a (0) )) and plot the results in Figure 3.
This experiment was later re-complied in Section 6.3. The BN in MLP classifier also had the nor- malisation applied at the input of layer two.
B LWN SCALES WELL WITH LARGE HIDDEN DIMENSIONS
(a) LWN with H 100
(b) LWN with H
50
Figure 6: Longitudinal Changes in Activation Values After Task Changing
Following the experimental setup as described in Appendix A, we further compared LWNs with 100 and with 50 hidden dimensions. As shown in Figure 6, there were more patches of deeper colours in an LWN with 50 hidden dimensions than there were in an LWN with 100 hidden dimensions. After fine-tuning both classifiers, we found that an LWN with 100 hidden dimensions had a divergence score of Diff 69.03 while LWN with 50 hidden dimensions had Diff 37.34 After weighting equally across the number of dimensions, it was Diff = 0.69( (= 69.03/100 per dimension for the former and Diff 0.75(= 37 37.34/50) per dimension for the latter. Forgetting was hence 8.00%(= [1 0.69/0.75] x 100%) less severe in the former setting. This was because that LWN exploited the hidden dimension in neural networks the higher the dimensionality, the more evenly the features could be redistributed.
13

---

Under review as a conference paper at ICLR 2022
C FIGURATIVE CLARIFICATIONS ON THE CLASSIHIESS IN SECTION 6
(a) MLP Classifler
(b) HCN
[c] MHC
MHC
Figure 7: Different Classifier Network Designs for Perm-MNIST
Classifler
Feature Extractor
Navandos
(a) Backbone Network
(bb LIWN in MLP
(c) LIWN HCN
Figure 8: Different Classifier Network Designs for Incremental Classes
In Figure illustrated the different classifier designs that used for the Perm-MNIST exper- iments. As mentioned previously, the backbone for the Perm-MNIST experiments were solely the classifiers and that feature extractor was involved. In addition, all LWN were applied at the input of layer two.
Likewise, we presented our different classifier designs for Inc-Cifar 00 and Inc- me-CUB200 overleaf in Figure 8. There were two main differences. First, the backbone networks of the incremental classes were compartmentalised as fixed pre-trained feature extractors and sequential learning classifiers. Second, the normalisation from LWN were applied at two places instead of one. In addition to the usual normalisation for the input of layer two, we added LWN to the input of layer one. That is, normalisation applied directly after the feature extractors pre- processed the input data.
D MORE RESULTS FROM THE EXPERIMENTS
Due to space constraint, we were unable to put all experimental results in the main text. Hence we only showed the most important results, but stored the rest of the performances in this section.
In Table 4, we provided additional results on Inc-Cifarl00 100 under the naive sequential learning setup. The results were for the scenarios with class increments of size and size 10. Similarly, Table 5 provided additional results on Inc-CUB200 under the naïve sequential learning setup with class incremental size 10 and size 20.
Then Tables to 12, we listed more results when the classifiers were trained with continual learning techniques applied. The aditional results in this section provided the extra metric scores for
14

---

Under review as a conference paper at ICLR 2022
MHC, LWN in MLP, and LWN in HCN for all scenarios. In addition, also listed those results for Inc- Pee-Cifar100 with incremental class sizes and 10; and similarly Inc-CUB200 with incremental class sizes 10 and 20.
Table More Results on the Naive Sequential Learning of Classifiers
Task
Ours
Classifier
ACC (%)
FAT (%6
Inc- Cifar100
MLP
63.0 38.2
53.6 16.0
(inc. 5 classes)
HCN
66.9 30.4
61.6 11.0
MHC (90%)
66.3 = 31.5
61.7 12.8
MHC (70%)
65.9 33.0
62.1 13.4
7
MHC (50%)
67.1 + 30.6
62.8 + 11.8
V
MHC (30%)
66.7 = 31.5
62.8 11.1
MHC (10%)
66.9 33.2
65.0 + 7.7
4
IWN in MLP
66.4 + 33.2
55.8 + TO.0
4
LWN in HCN
66.9 - 33.5
62.4 5.2
LWN MHC (10%)
67.8 31.2
63.9 9.3
Inc-Cifarl0h
MLP
49.9 20.5
40.0 12.0
(inc. 10 classes)
HCN
54.6 1 15.8
47.9 1 8.0
MHC (90%)
54.1 16.6
41.7
MHC (70%)
54.4 -1 16.3
48.3 1 8.8
4
MHC (50%)
55.1 15.5
49.2 - 10.4
MHC (30%)
55.4 15.4
48.6 11.2
7
MHC (10%)
55.5 1 15.0
47.6 11.7
4
LWN in MLP
54.1 18.5
44.8 + 6.9
V
LWN HIN
15.6
48.8
10.0
LWN in MHC (10%)
57.3 14.2
52.4188 8.3
Table 5: More Results the Naïve Sequential Learning of Classifiers
Task
Ours
Classifier
ACC (%)
FA1 (%)
Inc-C Inc-CU2200
MLP
57.7 1. 46.0
36.7 1 20.7
(inc. 10 classes)
ICCN
60.9 1 44.8
45.1
16.5
V
MHC (90%)
60.4 + 44.0
43.8
15.2
V
MHC (70%)
58.9 1 45.7
44.0 1: 20.8
MHC (50%)
58.9 + 44.4
44.3 21.3
V
MHC (30%)
58.9 1 45.9
47.7 11.1
V
MHC (10%)
62.2 1 42.2
52.4 1 16.7
LWN MLP
72.0 + 38.3
62.4 + 18.4
V
LWN HCN
74.6 1 34.7
71.1 1 10.8
V
LWN in MHC
74.6 + 35.4
70.4 + 13.1
Inc CUB200
MLP
55.6 + 25.1
50.2 16.3
(ine. 20 classes)
HCN
59.1 26.5
55.1 + 11.1
V
MHC (90%)
58.5 + 27.0
54.3 - 11.7
MHC (70%)
56.5 27.00
50.4 1: 10.2
MHC (50%)
56.0 2611
51.9 + 9.7
V
MHC (30%)
57.9 26.9
53.4 + 12.0
MHC (10%)
40.5
56.1 12.6
V
LWN MLP
71.2
64.9 = 11.4
V
LWN in HCN
73.8 + 19.7
68.1 = 11.2
LWN MHC (10%)
73.5 + 19.0
68.4 6.0
15

---

Under review as a conference paper at ICLR 2022
Table 6: More Results with Continual Learning Techniques Applied
Task
Ours
CL. Tech.
Classifier
ACC (%)
FA1 (%)
Perm-MNIST
EWC
MLP
68.9 1 18.3
76.5 7.0
EWC
HCN
73.7 + 14.4
82.3 5.7
V
EWC
MHC (70%)
73.7 + 14.4
82.8
4.6
V
EWC
LWN in MLP
70.8 1 18.8
79.3 7.4
EWC
LWN HCN
76.5 + 13.1
84.2 5.8
EWC
LWN in MHCC (70%)
76.5 1 13.3
84.6 1 6.2
Perm-MNIST
ER
MLP
72.1 + 12.5
80.3 2.8
ER
HCN
+ 15.6
83.9 2.1
ER
MHC (70%)
73.0 + 16.6
81.2 2.2
ER
LWN MLP
74.4 + 12.2
82.5 3.0
ER
LWN HCN
76.6 + 13.6
85.9
ER
LWN in MHC (70%)
76.0 14.3
85.9
1.9
Perm-MNIST
HAT
MLP
67.2 1 20.6
72.3 1 18.5
HAT
HCN
71.6 1 15.1
79.2 6.0
HAT
MHC (70%)
70.5 + 15.7
78.8 6.6
HAT
LWN in MLP
71.5 1: 17.0
77.6 1 12.3
HAT
LWN HON
75.7 13.2
83.4 3.8
HAT
LWN MHC (70%)
75.3 E 13.5
83.2 = 4.0
Table 7: More Results with Continual Learning Techniques Applied
Task
Ours
CL Tech.
Classifier
ACC (%)
FA1 (%)
Inc Cifar100
EWC
MLP
65.5 33.4
62.3
13.8
(inc. 5 classes)
EWC
HCN
68.0 + 27.9
66.3 + 9.7
EWC
MHC (10%)
68.0 30.0
69.1 5.0
EWC
LWN in MLP
68.9 27.7
62.8 + 6.9
EWC
LWN in HCN
69.0 + 28.4
67.0 + 6.4
EWC
LWN in MHC (10%)
69.8 26.1
69.9 = 5.7
Ine-Cifarl00
MLP
69.5 23.2
67.1 + 6.4
(inc. 5 classes)
HCN
70.0 = 23.7
67.2 - 6.5
ER
MHC (10%)
70.0 1. 23.4
68.0 1 6.4
LWN MLP
71.0 23.2
67.5 + 4.8
ER
LWN in HCN
70.8 23.0
67.0 1 3.4
ER
LWN In MHC (10%)
71.1 1. 23.4
67.7 1: 6.2
Inc C00
HAT
MLP
65.8 30.3
60.4 + 12.3
(inc. 5 classes)
HCN
65.9 33.2
63.3 + 7.7
HAT
MHC (10%)
68.3 + 27.9
66.8 + 8.3
HAT
LWN in MLP
66.6 = 31.6
61.5 - 9.3
LWN in HCN
66.3 34.2
65.4 + 5.3
HAT
LWN in MHC (10%)
67.3 + 30.9
64.9 1 7.9
16

---

Under review as a conference paper at ICLR 2022
Table 8: More Results with Continual Learning Techniques Applied
Task
Ours
CL Tech.
Classifier
ACC (%)
FAT (%)
Inc-Cifar100
EWC
MLP
52.1 17.1
48.7 1 11.3
(inc. classes)
EWC
HCN
55.4 14.2
61.088.33
EWC
MHC (10%)
56.2 + 13.1
51.3 + 9.8
EWC
LWN in MLP
55.4 1: 15.7
51.5 1 7.1
EWC
LWN in HCN
57.3 13.4
53.5 8.9
EWC
LWN in MHC (10%)
57.9 1: 12.3
57.1 1 7.4
Inc-Cifar100
ER
ML.P
51.9 + 13.8
52.6 + 5.9
(inc. classes)
HCN
52.9 13.4
53.2
ER
MHC (10%)
53.3 + 13.2
53.9 5.8
ER
LWN in MLP
54.7 = 13.5
54.9 - 4.6
LWN HCN
55.4 12.8
54.4 7.3
ER
LWN in MHC (10%)
55.6 12.7
54.3 4.1
Inc-Cifar100
HAT
MLP
52.7 1 17.3
43.5 1 13.6
(inc. 10 classes)
HAT
HCN
55.3 14.5
46.1 1 11.8
HAT
MHC (10%)
55.8 = 13.4
52.0 + 12.7
HAT
LWN in MLP
54.1 17.2
45.9 1 8.7
HAT
LWN HCN
56.0 14.9
50.8 + 10.0
HAT
LWN in MHC (10%)
55.8 14.6
50.9 + 12.4
Table 9: More Results with Continual Learning Techniques Applied
Task
Ours
CL Tech.
Classifier
ACC (%)
FA1 (%)
Inc Cifar100
EWC
MLP
43.3 15.2
36.1
(inc. 20 classes)
EWC
HCN
47.0 + 10.6
44.0 + 4.9
EWC
MHC (10%)
47.5
44.3 5.6
EWC
LWN in MIP
46.4 13.9
40.4
EWC
LWN in HCN
48.8 + 9.3
45.2 + 2.9
EWC
LWN in MHC (10%)
49.2 = 8.6
46.9 = 3.2
Inc- mnc-Cifar100
MLP
41.7 17.0
37.1 5.5
(ine, 20 classes)
HCN
42.1 16.2
38.5 5.2
ER
MHC (10%)
42.4 1. 15.5
39.0 1 4.0
LWN MLP
43.8 15.6
39.4
ER
LWN in HCN
44.9 1 15.1
40.7 1 4.3
ER
LWN in MIC (10%)
45. 1..11:0 14.4
41.6 1 4.8
Mie-Cifar100
HAT
MLP
43.4 13.0
38.0 6.0
(ine. classes)
HCN
44.2 13.1
38.6 + 8.3
HAT
MHC (10%)
45.6 9.9
43.1 + 5.3
HAT
LWN in MLP
11.8
41.5 5.8
LWN in HCN
47.0 10.5
43.5 7.6
HAT
LWN in MHC (10%)
47.3
44.9 1 4.8
17

---

Under review as a conference paper at ICLR 2022
Table 10: More Results with Continual Learning Techniques Applied
Task
Ours
CL Tech.
Classifier
ACC (%)
FAT (%)
Inc-C CUB200
EWC
MLP
57.7 46.3
43.3 1: 25.1
(inc. classes)
EWC
HCN
63.0 42.2
49.8 + 14.1
EWC
MHC (10%)
63.2 + 40.9
57.6 + 19.7
EWC
LWN in MLP
73.6 37.5
70.2 1 14.4
EWC
LWN in HCN
77.4 31.5
76.3 + 13.3
EWC
LWN in MHC (10%)
76.9 1 31.8
77.0 1 12.2
Inc-CUBB000
ER
ML.P
71.3 + 36.2
76.3 + 9.1
(inc. classes)
HCN
71.7 35.5
78.5
ER
MHC (10%)
72.7 36.0
79.6 + 6.4
ER
LWN in MLP
76.1 32.1
7..2
LWN HCN
78.2 30.3
82.5
ER
LWN in MHC (10%)
77.9 + 30.7
82.6 + 6.4
Inc-CUB200
HAT
MLP
60.9 1: 43.0
48.8 1 25.8
(inc. 10 classes)
HAT
HCN
62.3 11 42.1
55.8 1 18.5
HAT
MHC (10%)
66.5 - 39.0
66.1 + 13.5
HAT
LWN in MLP
75.2 1 33.8
82.2 1 17.8
HAT
LWN HCN
72.0 + 40.9
81.4 + 12.9
HAT
LWN in MHC (10%)
74.0 36.1
81.2 + 9.4
Table 11: More Results with Continual Learning Techniques Applied
Task
Ours
CL Tech.
Classifier
ACC (%)
FA1 (%)
Inc- CUB200
EWC
MLP
57.8 24.2
53.4 1 14.7
(inc. 20 classes)
EWC
HCN
61.1 23.3
55.7 + 12.4
EWC
MHC (10%)
62.8 - 23.7
60.3 + 14.2
EWC
LWN in MIP
74.0 18.2
69.5 +
EWC
LWN in HCN
76.7 17.1
73.0 + 10.3
EWC
LWN in MHC (10%)
76.3 = 17.0
72.7 + 10.0
Inc-CUB200 CUB200
ER
MLP
60.3 24.4
6.89
(inc. classes)
ER
HCN
62.2 23.7
6.58
ER
MHC (10%)
63.0 1: 24.3
68.4 6.1
LWN MLP
67.5 23.2
70.4
ER
LWN in HCN
72.0 21.1
74.5 1 6.8
ER
LWN in MHC (10%)
72.7 1. 20.1
75.0 1 8.1
Inc- CUBCCU2200
HAT
MLP
58.6 24.5
65.4 : 10.8
(inc. classes)
HCN
58.1 26.7
65.3 + 13.5
HAT
MHC (10%)
61.3 + 24.3
7..4
HAT
LWN in MLP
75.0 22.8
81.1 = 7.2
LWN in HCN
75.6 18.7
77.2 +
HAT
LWN in MHC (10%)
75.5 + 18.6
79.0 + 10.7
18

---

Under review as a conference paper at ICLR 2022
Table 12: More Results with Continual Learning Techniques Applied
Task
Ours
CL. Tech.
Classifier
ACC (%)
FAT (%)
Inc Inc-CUB200
EWC
MLP
56.0 1 17.8
46.8 1 13.5
(inc. 40 classes)
EWC
HCN
58.8 13.7
51.8 +
V
EWC
MHC (10%)
62.3 + 11.0
57.4 4.9
V
EWC
LWN in MLP
72.7 1 17.7
62.4 1 8.8
EWC LWN in HCN
76.8 + 14.3
68.4
V
EWC LWN in MIC (10%)
77.4 1. 14.0
69.7 1 7.7
Inc-CUB200
ER
MLP
51.9 + 20.1
47.6 + 4.5
(inc. 40 classes)
HCN
52.0 + 20.2
48.2
ER
MHC (10%)
53.4 + 20.5
49.6 + 8.4
ER
LWN in MLP
64.7 = 25.3
53.9
LWN in HCN
70.0 + 21.4
60.6
ER
LWN in MHC (10%)
71.8 + 20.0
63.1 + 8.6
Inc-CUB200
HAT
MLP
55.7 1: 16.1
59.6 1 6.1
(inc. 40 classes)
HAT
HCN
56.4 1 14.1
62.1 7.4
HAT
MHC (10%)
58.8 + 15.0
65.5 - 6.3
V
HAT
LWN in MLP
75.9 1 11.8
79.1 6.0
HAT
LWN in HCN
77.6 9.5
76.1 5.6
HAT
LWN in MHC (10%)
78.8 7.7
77.5 4.6
E THE NOISE INJECTION SETUP
The NI setup of Kuo al. (2021a) can be seen direct extension of the Perm-MNIST task. Based on the description of that paper, there is only one extra step which is to randomly shuffle portion of the content of the Perm-MNIST vectors. To elaborate, each task of Perm-MNIST first reorganises each MNIST image as a vector of 784 pixels; then applies the permutation mask of the current task that vector of pixels. On top of this, NI randomly selects portion of the pre-processed 784 pixels and shuffle those selected pixels.
The remaining hyperparameters of the NI scenario were identical our normal Perm-MNIST setup. There were 20 tasks in total. We naïvely sequentially trained the classifiers over 5K images on the first task, then observe 1K images on the rest of the tasks. All classifiers had layers with hidden dimension 100; and they were updated with SGD with learning rate 0.01.
F ADDITIONAL EXPERIMENTAL SETUPS
Table 13: More Results with Continual Learning Techniques Applied
Task
Ours
CL. Tech.
Classifier
ACC (%)
FA1 (%)
Perm-MNIST
EBM
74.2 +:2 12.2
61.1 4 13.7
EBM
HCN
77.7 + 10.0
65.8 4 11.1
V
EBM
MHC (70%)
77.9 = 9.9
66.8 14.4
/
EBM
with LWN
80.3 + 6.3
74.9 + 6.1
EBM
LWN in HCN
82.3 + 5.2
77.0
V
EBM
LWN in MHC (70%)
82.1 + 5.3
76.6 + 3.2
Inc-Cifar 100
EBM
31.3 + 22.4
21.8 + 9.4
(inc. 10 classes)
EBM
LWN in MHC (10%)
37.4 1 19.6
31.6 1 3.2
Inc-CUB200
EBM
32.4 + 22.4
15.15 4.7
(inc. 20 classes)
EBM
LWN in MHC (10%)
51.3 + 19.5
30.8 + 5.5
It suggested by of our reviewers include additional experiments test techniques the non-standard continual learning classifiers. To this end, we supplemented our existing experi- ments with Energy-Basse Based Models (EBM) (Li al., 2020) alternative classifiers.
As shown in Figure 1, traditional continual learning classifiers used softmax as their final layer. However, it was noted in Li et al. (2020) that softmax classifiers encourage sinner-takes-all dy-
19

---

Under review as a conference paper at ICLR 2022
namic for network parameterisation. Thus alternatively, they advocated for the of EBMs. The aim of their EBM was to minimise the energy for the imput class while rising the energy level for all remaining classes. Such setup would allow the network to lower its confidence in making incorrect decision.
However, their EBM required paradigmatic shift towards the conventional network setup for tinual learning. First, they relinquished softmax as their final layer and instead replaced it with a linear transformation with single output dimension. Second, they required both the data along with their ground-trute labels as input. Regardless of these changes, our MHC and LWN techniques were still applicable EBM classifiers. As shown in Table 13, our techniques were able improve the vanilla EBM setup across three different datasets.
20


Reviews:
[{"review_id": "p8Wpy6Hy3tz", "reviewer": "Reviewer_pQdx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is hard to read and uses very strange notation. It is very hard to understand formulas since authors do not use standard notation in deep learning. The modification proposed in the paper seems small and can be understood as an effect of architecture search. \n\nThe most interesting part is the experimental section. The authors have verified six classifier networks with four continual learning setups over three datasets. Unfortunately, it is not clear why the authors use SGD instead of adam. It is unclear why we do not have results on a classical convolutional neural network with 5-10 layers. It is not clear why in  MHC, authors train models with additional parameters. It is not clear the influence of such parameter since in Tab. 1 on Perm-MNIST wins LWN in MHC (70%) and on Inc-Cifar100 and Inc-CUB200 LWN in MHC (10%). It is better to choose a small or large percent of gated unit activation. Hot the parameter shud be tune in practice?  ", "summary_of_the_paper": "In the paper, the authors present two modifications: Masked Highway Connection and Layer-Wise Normalisation. In Masked Highway Connection, authors add a binary mask to classical HCN (Highway Connection Networks) and slightly change the training procedure. On the other hand, Layer-Wise Normalisation is a new normalization of activation in the neural network.", "summary_of_the_review": "The paper is hard to read and uses strange notation. The proposed modification seems to be small and not significant. The experimental section looks nice and is convincing, but authors should use some additional architecture and show how the method works on adam optimizer.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Wv3nHOvOzKy", "reviewer": "Reviewer_zxFC", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper excels at making clear the importance of each of the proposed changes to HCNs. The experimentation is thorough, showing strong performance of the proposed methods on several datasets. The motivation for masking and layer normalization is made clear and is supported by empirical measurements.\n\nVery minor weaknesses include small grammatical errors and tense confusion. (Examples listed below).\n\nIn general, this paper makes a compelling case MHC and LWN. The experiments span several datasets as well as different external continual learning techniques. The improvements proposed show strong performance in every case considered. In my opinion the notation is Section 3 is a bit confusing, but I think the points made in Kuo et al. (2021b) were well summarized (so using the notation from that paper is justified). Figures 3 and 4 are a bit hard to read, and perhaps representative quantities could be made explicitly clear in the text rather than only using phrases like \"deeper colours\" (and less critically I think the fonts should be larger in those plots).\n\nThese minor points do not affect my score.\n\n(i) In Section 1, the past and present tenses are both used to describe things done in this paper -- this should be consistent. \n\n(ii) Section 1 has a sentence that starts \"Our study focuses on designing new architectures in the forward pass which indirectly but positively affects...\" I think the conjugation should be 'new architectures... affect'.\n\n(ii) In the beginning of Section 4, it says \"Overleaf in the top half of Figure 2... \" and this may be a typo. \n\n(Note that I am not an expert in continual learning and none of my own research has been in this space. With that in mind, I admit that I am not up to date or extremely familiar with existing related work.)", "summary_of_the_paper": "This paper proposed two architectural improvements for networks designed for continual learning. Specifically, binary masks and layer normalization are added to highway connection classifier networks in order to prevent forgetting. The value of each of these elements is made clear from thorough experimentation.", "summary_of_the_review": "I think this is a strong paper with compelling experimental results. I vote to accept it. Some small grammatical errors should be corrected.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "TXvYNcyZEHW", "reviewer": "Reviewer_h7Ku", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The findings in the paper are useful, adding evidences to the existing literature on continual learning that we need to pay attention to the interplay between learning algorithms and backbones. The introduction of the mask on the gating function in Highway Networks is interesting. However, this seems to be specific to this particular architecture. Also I would like to see more theoretical analysis of the empirical findings.\n", "summary_of_the_paper": "The paper investigates the backbone networks that are less prone to catastrophic forgetting. Two modifications to existing backbones are found to be useful: a mask attached to the gate function of the Highway Network, and layer-norm (without tuning parameters). Experiments on top of existing popular learning algorithms designed for continual learning (EWC, ER and HAT) show that these modifications work.", "summary_of_the_review": "The findings are interesting, suggesting that research on continual learning should pay more attention to designing neural architectures that are natively suitable to handle catastrophic forgetting. However, the novelty seems to be limited as it is based on well-known architectures and ideas.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SuFhTgCoLrb", "reviewer": "Reviewer_tojU", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I find the general approach of \"baking\" in the ability to handle continual learning into the network architecture quite an elegant way of solving the catastrophic forgetting problem.  Though the proposed modifications to the HCN architecture are rather simple, I think this paper is worthy of note, because these changes make the network quite more robust for continual learning, and thus present a relatively straight forward, headache free (other than deciding on the MHC threshold) architecture that seems significantly less vulnerable to catastrophic forgetting.  \n\nThe paper would be stronger if there was a bit more theoretical explanation of how the internal dynamics of the masking and normalisation \"select\" weights in a way that does not destroy what the network has learned.  As it stands, the explanation is mostly based on intuitive understanding of the dynamics and empirical evidence, and quite general principles, akin to the argument that HCN is good for catastrophic forgetting, because it multiplies the derivative by a fraction.  BTW, I find it a bit counter-intuitive that HCN was designed to \"withstand gradient vanishing\" and is less vulnerable to catastrophic forgetting due to scaling down of derivatives....which, effectively, is gradient vanishing.  It seems that these two statements contradict each other.  \n\nWhile overall the paper is pretty well written, some parts could be improved for clarity.  Mathematical notation seems cumbersome (why use $L_\\gamma$  etc. subscript everywhere to indicate layer index...when just the $\\gamma$ subscript will do for index identifying the layer.  Why the neuron's output is labelled with $a$ for MLP and with $k$ HCN-based models?   It's exactly same neurons, isn't it.   Also, HCN could be explained a bit better (in Section 2).  Sure, it's always possible to go and read the references...but given that HCN is the architecture that this paper builds on, it could be explained a bit more thoroughly.  ", "summary_of_the_paper": "The paper proposes modifications to the highway connection network (HCN) architecture, which exhibits some inherent resistance to catastrophic forgetting.  These modification introduce fairly simple masking rule for selection of portion of weights to update and normalisation that (presumably) further adjust the internal representation to be more selective in which weights to update for a given task.  Empirical evaluation shows that these modification are highly effective in alleviating catastrophic forgetting.", "summary_of_the_review": "It's a decent paper - it would be better if there was more theoretical explanation for why the proposed changes to the architecture help with continual learning, but the provided empirical evaluation seems pretty good, and the proposed architectural changes to HCN seems quite effective.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "07fRQmLHJ9Q", "reviewer": "Reviewer_Homk", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n* The motivation for the pertained weights and the differences in the experimental setup between datasets is not clear.\n* Comparison with relevant work is missing: The idea of sparse, selective update of parameters to mitigate catastrophic forgetting and encourage task separability has been explored previously. For example, [1] with energy-based models, [2] with local synaptic plasticity and neuromodulation, [3] learns relevance maps. The UCL approach was also mentioned in the paper but not compared against. A systematic comparison with this class of approaches will help better assessing the proposed approach in terms of accuracy/computational cost.\n* Continual learning approaches are typically evaluated in both task- and class-incremental learning settings, it’s not clear from the experiments about which scenarios is considered.  \n* Classification accuracy might not be the best metrics to evaluate the continual leaning potential. Other metrics such as the backward and forward transfer are better suited [4].\n* Do the results assume a certain task ordering or have they been chosen randomly. Does the accuracy change with the change in relative ordering of the tasks? Will having consecutive having tasks with similar distributional properties have any effect on the interference with previous tasks.\n \n\n[1] Li, S., et al. \"Energy-Based Models for Continual Learning.\" arXiv preprint arXiv:2011.12216, 2020\n\n[2] Madireddy, S., et al. “Neuromodulated Neural Architectures with Local Error Signals for Memory-Constrained Online Continual Learning”, arXiv preprint arXiv:2007.08159, 2021\n\n[3] Kaushik, Prakhar, et al. \"Understanding Catastrophic Forgetting and Remembering in Continual Learning with Optimal Relevance Mapping.\" arXiv preprint arXiv:2102.11343 (2021).\n\n[4] Mai, Zheda,  et al. \"Online Continual Learning in Image Classification: An Empirical Survey.\" arXiv preprint arXiv:2101.10423 (2021).\n", "summary_of_the_paper": "This work proposes modifications to the highway connection networks (HCN) through the introduction of masks for selective weight updates and a layer-wise normalization to mitigate internal covariate shifts at feature level. Experiments were performed on the Permuted-MNIST, incremental CIFAR-100 and incremental CUB200 by incorporating this approach alongside EWC, ER and HAT. ", "summary_of_the_review": "The results look promising, but the novelty of this approach seems limited since the use of learnable masks, gating mechanisms, and layer-wise normalization have been previously proposed in the continual learning context. Moreover, the experiments are limited in terms of comparison with other state-of-the-are continual learning methods, evaluation metrics (such as backward and forward transfer) that characterizes the forgetting behavior.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Unfortunately, I feel the paper is not quite ready for ICLR, even if the reviews seem in general quite positive (though of low confidence). 
After reading the reviews and rebuttal, and going over the paper I have to make the following comments:
 * The paper do two modification to the backbone architecture, that have an impact on the ability of these systems to continually learn; these changes are adding layer normalization and a mask
  * The paper is mostly empirical in nature; while there are some intuitions presented clearly about these ideas, their efficiency is proved empirically, which is completely fine

 However:
  * The empirical validation seems not sufficient; the main results are permuted MNIST, incremental CIFAR 10, incremental CUB200; the results on permuted MNIST in terms of final accuracy seem surprisingly low (particularly when involving CL solutions, like EWC, ER, HAT .. see table 2; e.g. FA1 < 80% seems very surprising). This seems strange to me and adds a bit of shade on the results 
  * The proposed methods are simple; There is a strong message behind them, namely that the choice of the backbone (architecture size, normalization layers) has a huge impact on learning. But being a purely empirical result, this really needs to be backed up with analysis and an attempt at understanding of what is going on. E.g. looking at the masks over time .. to they converge to be task specific? Anything that would give a bit of depth to the results. Discussing the Figures (e.g. I'm looking at Fig 3 and I was grep-ing the text to see a discussion of how one would interpret those results). Why is FashionMNIST used to produced Fig 3, and why is not something like this done for one of the CL benchmark considered. Providing additional typical measures for CL (e.g. showing learning curves). 
  * Just overall does not seem that the work provides sufficient insight, or analysis. 

I do think there is something really interesting in this work, and I do hope the authors will resubmit this work after some modification. And I do agree that there are many aspects of the backbone or architecture that have big impacts on CL and this is an understudied and not well understood topic. So in that sense I think the idea of this work is good. But I just feel it fails short in terms of results, analysis. I feel in the current format, the work will not have the impact it deserves.
Decision: Reject