Year: 2021

Abstract:
Neural networks usually excel in learning a single task. Their weights are plastic and help them to learn quickly, but these weights are also known to be unstable. Hence, they may experience catastrophic forgetting and lose the ability to solve past tasks when assimilating information to solve a new task. Existing methods have mostly attempted to address this problem through external constraints. Replay shows the backbone network externally stored memories; regularisation imposes additional learning objectives; and dynamic architecture often introduces more parameters to host new knowledge. In contrast, we look for internal means to create less forgetful networks. This paper demonstrates that two simple architectural modifications – Masked Highway Connection and Layer-Wise Normalisation – can drastically reduce the forgetfulness in a backbone network. When naı̈vely employed to sequentially learn over multiple tasks, our modified backbones were as competitive as those unmodified backbones with continual learning techniques applied. Furthermore, our proposed architectural modifications were compatible with most if not all continual learning archetypes and therefore helped those respective techniques in achieving new state of the art1.

heading: 1 INTRODUCTION
text: Continual learning (McCloskey & Cohen, 1989) is a difficult experimental setup for most neural networks. It places a single learning agent in a dynamic environment where it must learn to adapt from a stream of tasks to make new predictions. This setup means that the underlying data are not independent and identically distributed (non-iid). Thus when we use conventional gradient descent methods (Rumelhart et al., 1986) to fine-tune a pre-trained network for a new task, its weights will be modified to satisfy the learning objectives of the new task. As a result, the re-parameterised network will experience catastrophic forgetting (McCloskey & Cohen, 1989; French, 1999) and abruptly lose the ability to solve a previously learnt task.
Recently, continual learning has been of increasing interest. Most of the latest studies can be roughly categorised as the three archetypes of replay, regularisation, and dynamic architecture. Replay techniques such as gradient episodic memory (Lopez-Paz & Ranzato, 2017) and experience replay (Chaudhry et al., 2019) externally store a handful of data per task. Then upon sequentially learning a new task, they re-expose copies of the stored data to the backbone network along those data of the new task. Regularisation methods like learning without forgetting (Li & Hoiem, 2017) and elastic weight consolidation (Kirkpatrick et al., 2017), on the other hand, impose the backbone network with secondary learning objectives to learn robust mappings that are less prone to parametric corruptions. Furthermore, the dynamic architectural approaches of dynamically expandable networks (Yoon et al., 2018) and hard attention to the task (Serra et al., 2018) either introduce more parameters to host new knowledge or explicitly control the backbone synaptic connections to perform inference on a subset of pre-selected weights. Though the three archetypes of continual learning methods may appear to be ideologically different, they all seek external means to retain learnt knowledge. In contrast, this paper will introduce internal modifications to mitigate forgetting.
That is, we aim to create an extremely plastic yet stable network architecture that can withstand catastrophic forgetting even when it is naı̈vely implemented (simply fine-tuned) for sequential learning over multiple tasks. Our study was inspired by the recent work of Kuo et al. (2021b). In that
1The codes will be made publicly available once the paper is published.
paper, the authors compartmentalised their backbone network. They employed a fixed feature extractor and only fine-tuned their classifiers. Unlike the conventional multiple layer perceptron (MLP) classifier of Figure 1(a) used in most studies (such as Mai et al. (2020)), their classifier which they called highway connection networks (HCNs) had the peculiar design shown in Figure 1(b). An HCN included a two-layer perceptron with highway connection (Srivastava et al., 2015) followed by a softmax layer. We used red to denote fully connected layers, yellow for softmax, and blue for the HCN gated unit. They demonstrated that HCNs were more robust against forgetting than MLPs; and even showed that naı̈vely sequential learning with HCNs could often outperform MLP classifiers with continual learning techniques applied. Moreover, they found that highway connections were specifically required and could not be replaced with residual connections (He et al., 2016).
In this paper, we introduce two novel modifications to HCNs. Our new ideas draw inspiration from the stability-plasticity dilemma (Mermillod et al., 2013). Stability is the ability to retain learnt knowledge upon parametric corruption (Sun et al., 2021); whereas plasticity refers to the ease of assimilating new information. Our study focuses on designing new architectures in the forward pass which indirectly but positively affect the updates in the backward pass. This kind of approach was previously investigated in weight normalisation (Salimans & Kingma, 2016). In that paper, the authors found that by reparameterising the weight vectors, they could influence the gradient covariance matrix by making it closer to an identity matrix thereby hastening network configuration.
We propose the Masked Highway Connection (MHC) and Layer-Wise Normalisation (LWN) shown in Figures 1(c) and 1(d) respectively. While similar techniques exist for general machine learning; their applications for continual learning are not well studied. After discussing how HCN mitigates forgetting (Section 3), we will introduce our MHC and show that HCNs can be made even more robust via self-identifying important weights that are worth fine-tuning through a masking mechanism (Section 4). Then, we will alternatively characterise forgetting as shifts that occur in previous distributions (Ebrahimi et al., 2020); and that the LWN can reduce such shifts (Section 5).
Our experiments (Section 6) indicated that both MHC and LWN were robust against forgetting and greatly outperformed their HCN and MLP baselines. In addition, both designs were compatible with the continual learning archetypes of replay, regularisation, and dynamic architecture; and they were able to help those respective techniques to achieve new state of the art. Moreover, MHC and LWN could also be jointly implemented. Together, they often produced even more robust results and this was because they each addressed a different aspect of catastrophic forgetting. In order to suggest the best practice for MHC and LWN for continual learning, we also tested our novel classifiers over several different settings. Importantly, we found that LWN should not be incautiously substituted with batch normalisation (Ioffe & Szegedy, 2015).

heading: 2 THE CONTINUAL LEARNING FRAMEWORK
text: As stated in Lopez-Paz & Ranzato (2017), the continual learning setup exposes a base learner F to a continuum of data with n locally, but not globally, iid tasks ω where
(x11, ω1, y 1 1), . . . , (x i φ, ωi, y i φ), . . . , (x n Φ, ωn, y n Φ). (1)
Each task corresponds to a dataset (xiφ, y i φ) ∈ Dωi with Φ samples φ = 1, . . . ,Φ. Once the learner F observes data from the j-th task, full access to an earlier dataset ωi where i < j is prohibited to consolidate learnt knowledge. The performance is tracked using a matrix A ∈ R(n+1)×n where each column represents a distinct task. When F is initialised, its accuracy over all tasks is recorded in row A1,(1:n). After learning the i-th task, the new performances are updated in row A(i+1),(1:n).

heading: 3 HIGHWAY CONNECTIONS IMPLICITLY RE-SCALE GRADIENTS
text: An intermediate layer Lγ in an MLP can be represented as aLγ = H (aL(γ−1) ,WLγ ) (2) where the layer output aLγ is transformed from the layer input aLγ−1 over the non-linear function H with weight WLγ . In addition, say there are Γ layers then the network output is aLΓ = ŷ. To fine-tune WLγ , we use gradient descent with the derivative of a costC with respect to the parameters
∂C ∂WLγ = ∂C ∂ŷ Γ−1∏ ζ=γ ∂aL(ζ+1) ∂aLζ  ︸ ︷︷ ︸
Rel: Srivastava et al. (2015)
Rel: Kuo et al. (2021b)︷ ︸︸ ︷{ ∂aLγ ∂WLγ } . (3)
Below, we discuss how gradient vanishing and catastrophic forgetting are closely related (Rel:).
Deep MLPs are known to suffer gradient vanishing (Bengio et al., 1994). When the layers increase but the magnitudes of the individual Jacobian terms ∂aL(ζ+1) ∂aLζ
are small (i.e., ∈ [0, 1]), the term∏Γ−1 ζ=γ ∂aL(ζ+1) ∂aLζ diminishes to 0 (and so does ∂C∂WLγ ) and no effective update is made to WLγ .
Highway connection (Srivastava et al., 2015) is well-known for its ability to withstand gradient vanishing. It reformulates an intermediate layer as
kLγ = (1−TLγ ) kLγ−1 + TLγ H (kL(γ−1) ,WLγ ) with (4) gate TLγ = T(kL(γ−1) ,QLγ ) (5)
and the element-wise dot product . The colour-coded gated unit TLγ has its own weight QLγ and its values lie in the range of [0, 1]. This new formulation has the Jacobian term
∂kLγ ∂kL(γ−1) = { I, if TLγ = 0 H ′(kL(γ−1) ,WLγ ), if TLγ = 1 . (6)
By introducing the identity matrix I to the Jacobian (and thus the [.] term of Equation (3)), highway connections prevent the vanishing of gradients and thus facilitate the training of deep networks.
In Kuo et al. (2021b), the authors noted a second property unique to highway connections. While the {.} term of Equation (3) of an MLP was ∂aLγ∂WLγ = ∂H (aL(γ−1) ,WLγ ) ∂WLγ , a highway connection had the alternative form of ∂kLγ ∂WLγ = TLγ ∂H (kL(γ−1) ,WLγ ) ∂WLγ which included the gated unit. Since each entry in TLγ lied in [0, 1], highway connections thus implicitly scaled down the magnitudes of the gradients used in gradient descent. This was desirable for continual learning because less changes were made to a pre-trained weight WLγ ; and HCNs were hence less susceptible to forgetting.

heading: 4 IMPROVING THE HCN ARCHITECTURE
text: Though highway connection is effective against catastrophic forgetting, its best practices and shortcomings remain unclear. In this section, we review the different roles that the gated unit TLγ plays in the forward pass (inference) and in the backward pass (update). Furthermore, we will show that there exists an interesting structural connection between the HCN gated unit TLγ and the derivative ∂H (kL(γ−1) ,WLγ )
∂WLγ which can be exploited to make the network even more robust to forgetting.
Overleaf in the top half of Figure 2, we show that HCN performs a point-to-point gradient reduction. On the left, we superimpose a dotted box on top of HCN. The inference and update that occur within this box are illustrated on the right. Without loss of generality, the scenario in the figure is for 3 hidden dimensions and that each TLγ entry scales down its corresponding feature of the fully connected layer output kLγ during inference. Note that the entries of kLγ are shown in different shades of red; and they correspond to specific rows in the weight WLγ . Likewise for update, we highlight the rows of ∂H (kL(γ−1) ,WLγ )
∂WLγ in different shades of green; and we show how each row of
this derivative is scaled down, in a point-to-point fashion, by the entries of the gated unit TLγ .
But does the entirety of weight WLγ need to be updated at all times? Two prior studies, the lottery ticket hypothesis (LTH) (Frankle & Carbin, 2019) and the uncertainty-regularised continual learning (UCL) (Ahn et al., 2019), suggested not. In network pruning, LTH found that only a small subset of the dense network needed to be updated to achieve competitive performances. Whereas UCL showed that continual learning could be achieved by regularising specific nodes in the weight that deviated from a pre-determined distribution. They both considered the structural relationship among nodes in the base learner. A similar approach can be applied to HCN.
The lower-half of Figure 2 presents the inner workings of our Masked Highway Connecton (MHC) classifier. We add a binary Mask to HCN and change the formulation from Equations (4) and (5) to
kLγ = (1−GLγ ) kLγ−1 + GLγ H (kL(γ−1) ,WLγ ) with (7) a masked gate GLγ = Mask TLγ = Mask T(kL(γ−1) ,QLγ ). (8)
The Mask keeps the largest neural values in TLγ with the top-K function. As shown in Figure 2, when TLγ ∈ R3 and that K = 1, the MHC masked gate GLγ nullifies the two rows with weaker activation values. The mask also nullifies specific updates during optimisation. In our experiments in Section 6, we will demonstrate that this minimalistic update can reduce excessive weight modification thereby improving network stability for the sequential learning setup.

heading: 5 FORGETTING, MEASURED THROUGH FEATURE SHIFTS
text: The gating mechanism in HCN showed that Kuo et al. (2021b) regarded forgetting as the amount of change that occurred in the pre-trained weights. However, both Ebrahimi et al. (2020) and Ahn et al. (2019) quantified forgetting through alternative means. Both studies employed Bayesian neural networks (BNNs) (Mackay, 1992) for continual learning. The former conceptualised forgetting as shifts that occurred in the distributions of weights; and the latter showed that BNNs could become stable by regularising the network with changes in the standard deviations of the weight matrix.
Thus in addition to MHC, we further propose Layer-Wise Normalisation (LWN) to mitigate shifts in the classifier. However, we address internal covariate shifts (ICSs) (Shimodaira, 2000) instead of weight distribution shifts. ICS refers to the change in network activation distributions due to parametric changes during training; and we will show that forgetting can also be mitigated at the feature level. An LWN with hidden dimension H changes Equation 2 to a
normalised layer aLγ = a∗Lγ − µLγ
σLγ . from .a∗Lγ = H (aL(γ−1) ,WLγ ). with (9)
mean: . µLγ = 1
H H∑ η=1 a∗Lγ,η . and . standard deviation: . σLγ = √√√√ 1 H H∑ η=1 (a∗Lγ,η − µLγ )2. (10)
In order to demonstrate the effectiveness of LWNs over MLPs, we prepared a toy example which first trained both classifiers with MNIST (LeCun et al., 1998) and then with FashionMNIST (Xiao et al., 2017). Both classifiers had the hidden dimensions H = 100 with the two layer designs shown in Figures 1(d) and 1(a) respectively. Both datasets contained grey-scale images on 28×28 pixel boxes with 60K and 10K images for training and testing. After pre-training both classifiers on MNIST, we externally stored a batch of MNIST images BMNIST and applied them on the classifiers to collect the output activations of the second layer a(0)L2 . When we fine-tuned the classifiers for FashionMNIST, we re-applied BMNIST and collected the new activation a (ξ) L2
for each ξ-th update. We then plotted the changes in activation in Figures 3(a) and 3(b). See Appendix A for more details in the setup.
The changes in activation were defined as Diffact = abs ( E|BMNIST|b=1 (ba (ξ) L2,η −b a(0)L2,η ) ) (11)
and it measured the magnitudes of the expected differences in the changes of the layer two activation (over samples b in BMNIST). The deeper the colours the larger the differences. Each row represented the changes in a unique feature aL2,η after the classifier received updates to solve FashionMNIST. Each row generally became deeper and this showed that ICS occurred during re-parameterisation and that the learnt mapping for MNIST images BMNIST had diverged. After fine-tuning, we found that the divergence was Diffact = 86.44 in MLP and Diffact = 69.03 in LWN. Forgetting was hence 20%(= [1 − 69.03/86.44] × 100%) less severe in LWN than in MLP. This was because that the differences in LWN features became smaller after redistributing the activation along the feature dimension. LWN also scaled well with larger dimensionality; see Appendix B for details.
Unlike the well-known layer normalisation (Ba et al., 2016), our LWN did not have learnable parameters to reshape the post-normalised activation distribution. This was because that during our experiments, we found that these features were not necessary. Also, we found that LWN was specifically required to mitigate forgetting; and it could not be replaced by the popular batch normalisation (Ioffe & Szegedy, 2015). This will be further discussed in Section 6.3.

heading: 6 EXPERIMENTS AND RESULTS
text: We compared six classifier networks with four continual learning setups over three datasets. The main results were summarised in Tables 1 and 2, with more results in Tables 4 – 12 in the appendices.
The Datasets We tested image permutation (Kirkpatrick et al., 2017) on MNIST (LeCun et al., 1998), and incremental classes (Rebuffi et al., 2017) on Cifar100 (Krizhevsky, 2009) and CUB200 (Wah et al., 2011). MNIST was previously mentioned in Section 5. Cifar100 included 100 classes of 32 × 32 coloured images; with 50K for training and 10K for testing. Whereas CUB200 contained 5, 994 training and 5, 794 test images over 200 coloured bird species on mostly 500× 500 pixel boxes. The Tasks Permuted MNIST (Perm-MNIST) was derived by first reshaping MNIST images as vectors of 784(= 28×28) pixels, then by applying a unique permutation on all vectors of that task. On the other hand, incremental Cifar100 (Inc-Cifar100) and incremental CUB200 (Inc-CUB200) were generated via sequentially introducing a fixed set of classes per task drawn without replacement from the datasets. There were 20 tasks for Perm-MNIST. For Inc-Cifar100, we tested three scenarios with 5, 10, and 20 classes per task. Likewise for Inc-CUB200, we introduced 10, 20, and 40 classes per task. (The Inc scenarios thus had 20, 10, and 5 respective tasks.) We mainly followed Douillard et al. (2020)
to process the datasets2 and setup as domain-incremental learning (Van de Ven & Tolias, 2019).
The Backbone Networks For Perm-MNIST, our backbone networks were simply two-layer MLPs, HCNs, and MHCs. We tested MHCs with a range of different settings – we used top-K to keep between the top 10% to the top 90% of the gated unit activation. In addition, we embedded LWN in MLPs, in HCNs, and in MHCs. All of the normalisation were applied specifically to the input of layer two. Interested readers could find elaborated illustrations in Figure 7 of Appendix C.
For Inc-Cifar100 and Inc-CUB200, our backbone networks were compartmentalised as fixed pretrained feature extractors with sequential learning classifiers. The feature extractors for Inc-Cifar100 were ResNet18s (He et al., 2016) pre-trained on ImageNet (Deng et al., 2009); and those for IncCUB200 were ResNeXt50s (Xie et al., 2017) pre-trained on ImageNet. We replaced the final layer of the feature extractors with our classifiers. Our classifiers here were also two-layer MLPs, HCNs, and MHCs. The MHC again kept top-K with K ∈ [10%, 90%]; however, we embedded the LWNs differently. The normalisation were applied in two different places – first on the input of layer one (extracted by the feature extractors), and then on the input of layer two. Again, interested readers could find elaborated illustrations in Figure 8 of Appendix C.
Additional Training Detials For Perm-MNIST and Inc-Cifar100, our classifiers had hidden dimensions H = 100; and it was H = 400 for Inc-CUB200. The Perm-MNIST backbones observed 5K images on Task 1 then 1K images for all subsequent tasks. Likewise for Inc-Cifar100, it was 5 epochs on Task 1 then 1 epoch for the rest. However, the backbones observed 5 epochs for all Inc-CUB200 tasks. We used batch size B = 10, and updated all classifiers with SGD (Rumelhart et al., 1986) with learning rate 0.01.
The Continual Learning Setups We denoted singular as the scenario when the backbones were naı̈vely and sequentially trained (simply fine-tuned) over all tasks. Then, we tested the regularisation-based elastic weight consolidation (EWC) (Kirkpatrick et al., 2017), the memory-based experience replay (ER) (Chaudhry et al., 2019), and the dynamic architectural-based hard attention to the task (HAT) (Serra et al., 2018)3. Some secondary experiments could be found in Appendix F.
Metrics We used the two metrics below to evaluate the sequential learning performances Average accuracy (ACC) 1n ∑n i=1 An+1,i ; and Final Accuracy of Task 1 (FA1) An+1,1. Both metrics were based on matrix A of Section 2. The classifier plasticity could be indicated through ACC, and FA1 could reflect the classifier stability. All units were in percentage accuracy, the higher the better; and all results were reported as the 95% confidence intervals over 10 seeds.

heading: 6.1 RESULTS ON NAÏVELY SEQUENTIAL LEARNING
text: We summarised a subset of results for the singular setup overleaf in Table 1. It included all results for Perm-MNIST, Inc-Cifar100 with 20 classes per task, and Inc-CUB200 with 40 classes per task. The baseline results for MLPs and HCNs were shown in grey, and we highlighted our results that were better than the baselines in cyan. This table reflected four important remarks.
The first remark was that, our classifier designs could outperform the baselines with a large margin if they were configured properly. However the second remark showed that, the best practice of MHC varied for each task. For Perm-MNIST, the best performce in MHC was recorded when we kept the top K = 70% of gated unit activation; but for Inc-Cifar100 and Inc-CUB200, it was when only the top K = 10% activation was kept. This was potentially due to the contextual complexity of the datasets; such that it was possible to learn with less information from the contextually rich Cifar100 and CUB200. The third remark was that LWN greatly improved both the plasticity (ACC score) and stability (FA1 score) for all classifiers. While it was well-known that normalisation could improve network plasticity (Ba et al., 2016), our work provided empirical evidence that normalisation along the feature dimension was beneficial for mitigating forgetting. In addition, the fourth remark was
2Refer to inclearn/lib/data/datasets.py in Douillard et al. (2020)’s official repository. 3We set the regularisation coefficient λ = 3 for EWC; stored 50 images per task for ER; and applied HAT
only on the first layer of classifier synapses.
that MHC and LWN could be made more powerful when they were jointly implemented. This was not particularly surprising because both techniques addressed forgetting from different perspectives. MHCs considered forgetting as the modifications made to the pre-trained weights (see Section 4); whereas LWNs were effective in minimising the feature shifts in a learnt mapping (see Section 5). We made more results for the singular setup available in Appendix D. We refer to Table 4 for additional results on Inc-Cifar100; and to Table 5 for additional results on Inc-CUB200.

heading: 6.2 RESULTS WITH CONTINUAL LEARNING TECHNIQUES APPLIED
text: After the singular setup, we tested all classifiers with EWC, ER, and HAT applied. Our results were summarised in Table 2 overleaf. Due to space constraints, we selected the best performing LWN in MHC results. Interested readers could find the results for all other backbone network designs in Tables 6 – 12 in Appendix D. We found three additional important remarks from the experiments.
The first remark was that our LWN in MHCs were compatible with all of EWC, ER, and HAT. Since these three techniques were from the most well-known continual learning archetypes, we expected our LWN in MHCs to be compatible with most if not all existing continual learning techniques. Second, our LWN in MHCs achieved better results than the baselines. Hence, they helped the aforementioned techniques in reaching new state-of-the-art status. The third remark was that, the majority of our naı̈vely sequential learning LWN in MHCs from Table 1 (both ACC and FA1) achieved better results than the baseline classifiers with continual learning techniques applied. Hence, we demonstrated the importance of employing the correct classifier design to mitigate forgetting.

heading: 6.3 BATCH NORMALISATION IS NOT GOOD FOR CONTINUAL LEARNING
text: In this section, we tested batch normalisation (BN) (Ioffe & Szegedy, 2015) in MLP. The results for Perm-MNIST in Table 3 showed that BN was detrimental for the backbone networks.
Table 2: A Subset of Results with Continual Learning Techniques Applied
Task Ours CL Tech. Classifier ACC (%) FA1 (%) Perm-MNIST - - EWC MLP 68.9± 18.3 76.5± 7.0
- - EWC HCN 73.7± 14.4 82.3± 5.7 X EWC LWN in MHC (70%) 76.5± 13.3 84.6± 6.2
Perm-MNIST - - ER MLP 72.1± 12.5 80.3± 2.8 - - ER HCN 73.3± 15.6 83.9± 2.1 X ER LWN in MHC (70%) 76.0± 14.3 85.9± 1.9
Perm-MNIST - - HAT MLP 67.2± 20.6 72.3± 18.5 - - HAT HCN 71.6± 15.1 79.2± 6.0 X HAT LWN in MHC (70%) 75.3± 13.5 83.2± 4.0
Inc-Cifar100 - - EWC MLP 43.3± 15.2 36.1± 5.5 (inc. 20 classes) - - EWC HCN 47.0± 10.6 44.0± 4.9 X EWC LWN in MHC (10%) 49.2± 8.6 46.9± 3.2 Inc-Cifar100 - - ER MLP 41.7± 17.0 37.1± 5.5 (inc. 20 classes) - - ER HCN 42.1± 16.2 38.5± 5.2 X ER LWN in MHC (10%) 45.1± 14.4 41.6± 4.8 Inc-Cifar100 - - HAT MLP 43.4± 13.0 38.0± 6.0 (inc. 20 classes) - - HAT HCN 44.2± 13.1 38.6± 8.3 X HAT LWN in MHC (10%) 47.3± 9.1 44.9± 4.8 Inc-CUB200 - - EWC MLP 56.0± 17.8 46.8± 13.5 (inc. 40 classes) - - EWC HCN 58.8± 13.7 51.8± 6.6 X EWC LWN in MHC (10%) 77.4± 14.0 69.7± 7.7 Inc-CUB200 - - ER MLP 51.9± 20.1 47.6± 4.5 (inc. 40 classes) - - ER HCN 52.0± 20.2 48.2± 9.0 X ER LWN in MHC (10%) 71.8± 20.0 63.1± 8.6 Inc-CUB200 - - HAT MLP 55.7± 16.1 59.6± 6.1 (inc. 40 classes) - - HAT HCN 56.4± 14.1 62.1± 7.4
X HAT LWN in MHC (10%) 78.8± 7.7 77.5± 4.6
(a) BN in MLP (b) LWN in MLP
Figure 4: Longitudinal Changes in Activation Values After Task Changing
In order to understand the reason why BN was disadvantageous for continual learning, we repeated the experiments in Section 5 where we first trained BN in MLP for MNIST and then fine-tuned for FashionMNIST. We illustrated the results in Figure 4. This figure was formatted following the style of Figure 3. BN in MLP had much
deeper colours than LWN in MLP. In addition, the divergence was Diffact = 97.66 in BN in MLP and forgetting was hence 41%(= [1− 69.03/97.66]× 100%) more severe than LWN in MLP. The deep colours first appeared after the BN in MLP started to observe data from FashionMNIST. This indicated that the learnt parameters (i.e., the scale and the shift) in BN required readjustments as a response to the change in source data distribution. This was expected since BN normalised neural network values along the batch dimension rather than the feature dimension. That is, while BN could decrease the ICS within a single task, it could not decrease the ICS occurring between tasks. These results showed that LWN was specifically required and that it should not be incautiously replaced with an alternative normalisation.

heading: 6.4 SEQUENTIAL LEARNING IN A NOISY ENVIRONMENT
text: We further concerned the realistic scenario of continual learning under the presence of noise. The noise injection (NI) (Kuo et al., 2021a) scenario randomly shuffled a portion of pixels of Perm-MNIST images. The lowest setting injected 10% noise while the highest setting injected 50% noise. Refer to the details of this setup in Appendix E.
We naı̈vely sequentially trained the classifiers and plotted the results in Figure 5. Both HCNs and LWN in MHCs were able to achieve better results than MLPs on plasticity (ACC). In addition, our novel LWN in MHCs also outperformed HCNs in stability (FA1). We attributed this to LWN in MHC’s minimalistic updates for reducing the impact of input corruption on optimisation.

heading: 7 RELATED WORK
text: Our results highlight the architectural importance for continual learning. The components in the forward pass not only impact plasticity, but also influence the backward pass and thus stability. From this perspective, continual learning shares many important research questions with network pruning and neural architectural search (NAS).
Network pruning was originally proposed to reduce both the computational and memory cost of neural networks. Han et al. (2015) found that large networks could perform with no loss in accuracy with only 10% of its total parameters. Zhou et al. (2019) further demonstrated that neural networks could still perform competitively when the weights were reset but with their original signs kept. This implied that the architecture had a direct influence on the topology of the solution space (Li et al., 2018) and potentially also the capacity of the minima on the loss landscape (Hochreiter & Schmidhuber, 1997). This in turn decided how easy it was to not forget. These insights echoed with some findings in NAS. NAS considered network building as a learning problem (Zoph & Le, 2017). It was recently extended to continual learning in Mundt et al. (2021); and some linear classifiers with random weights were found to perform on par with fully trained deep counterparts.
Besides network pruning and NAS, many work in the lottery ticket hypothesis (LTH) (Frankle & Carbin, 2019) also highlighted the importance of controlled and minimalistic updates. Gohil et al. (2019) found that when configured correctly, a small and sparsified subnetwork within a dense net could be generalised across tasks. Additional studies on the structural relations between subnetworks and feature distributions were also investigated in Ayinde et al. (2019) and Kuo et al. (2021a).

heading: 8 CONCLUSION
text: This paper introduced Masked Highway Connections (MHCs) and Layer-Wise Normalisations (LWNs) as classifiers to mitigate forgetting for backbone networks that sequentially learn over several tasks. Unlike the reply, regularisation, and dynamic architectural continual learning archetypes, our approach did not add external constraints to prevent forgetting. Instead, we focused on internal modifications that were beneficial for the classifier design. To elaborate, our methods were able to stop forgetting without the externally stored data of replay and were thus memory efficient. Similarly, our methods were different to dynamic architectures and did not introduce new modules upon learning a new task. Hence, our methods were also computationally efficient.
We demonstrated that forgetting could be prevented at both the weight level and at the feature level. MHC achieved the former via minimalistic updates to lower the impact of weight corruption. While LWN achieved the latter through feature redistribution to retain previously learnt mappings.
MHCs and LWNs were able to outperform their baselines. In addition, they were compatible with techniques of all continual learning archetypes hence helping those techniques in achieving new state of the art. Moreover, LWN in MHCs were very stable even when they were the naı̈vely implemented; they often achieved better results than baseline classifiers with continual learning techniques applied.
ETHICS STATEMENT
This paper investigated simple yet effective classifier designs to prevent catastrophic forgetting in the continual learning setup. All of the experiments conducted in this paper were still on the nicely crafted datasets of MNIST, Cifar100, and CUB200. Thus, the content of this work does not present any foreseeable societal consequences.
REPRODUCIBILITY STATEMENT
All of the important hyper-parametric settings were described in text. Interested readers should refer to The Backbone Networks and Additional Training Details in Section 6 for more details. In addition, the authors of this manuscript promise to make the codes of this paper publicly available when the paper is published.

heading: A DETAILS ON THE EXPERIMENTAL SETUP OF SECTION 5
text: This section aims to provide more details on the experimental setup for the toy experiment in Section 5. The purpose of the experiment is to compare the changes in the learnt mapping between an MLP classifier and an LWN in MLP classifier when we change from training them on MNIST (LeCun et al., 1998) to FashionMNIST (Xiao et al., 2017).
The design of the baseline MLP classifier looked like that in Figure 1(a), and similarly the LWN in MLP classifier took the shape of that in Figure 1(d). Both classifiers had 2 layers – with input dimension 784, hidden dimension 100, and output dimension 10; and they were trained with SGD (Rumelhart et al., 1986) with learning rate 0.01. As shown in the figure, the normalisation in LWN in MLP was applied at the input of layer two (equivalently, the output of layer one).
The two classifiers were first trained on MNIST. They were exposed to 5K copies of MNIST images in batches of size 10. Prior to input, each image was flattened as a vector of size 784(=28 × 28). After learning from the 5K copies, we externally stored a subset of MNIST images of size 50 (i.e., , |BMNIST|= 50). Then, we applied the saved images BMNIST to the pre-trained classifiers to collect the layer two activation a(0)L2 .
We proceeded to fine-tune the classifiers on 10K images of FashionMNIST. Similar to the previous setup, each image was flattened as vectors of 784 pixels and that the batch size was 10. There were hence 1K updates. After each step of update, we re-applied BMNIST to the fine-tuned classifiers and collected the new activation values at layer two a(ξ)L2 . We then recorded the longitudinal changes in
the learnt mapping for MNIST as abs ( E|BMNIST|b=1 (ba (ξ) L2,η −b a(0)L2,η ) ) and plot the results in Figure 3.
This experiment was later re-complied in Section 6.3. The BN in MLP classifier also had the normalisation applied at the input of layer two.

heading: B LWN SCALES WELL WITH LARGE HIDDEN DIMENSIONS
text: Following the experimental setup as described in Appendix A, we further compared LWNs with 100 and with 50 hidden dimensions. As shown in Figure 6, there were more patches of deeper colours in an LWN with 50 hidden dimensions than there were in an LWN with 100 hidden dimensions. After fine-tuning both classifiers, we found that an LWN with 100 hidden dimensions had a divergence score of Diffact = 69.03 while an LWN with 50 hidden dimensions had Diffact = 37.34. After weighting it equally across the number of dimensions, it was Diffact = 0.69(= 69.03/100) per dimension for the former and Diffact = 0.75(= 37.34/50) per dimension for the latter. Forgetting was hence 8.00%(= [1 − 0.69/0.75] × 100%) less severe in the former setting. This was because that LWN exploited the hidden dimension in neural networks – the higher the dimensionality, the more evenly the features could be redistributed.

heading: C FIGURATIVE CLARIFICATIONS ON THE CLASSIFIERS IN SECTION 6
text: In Figure 7, we illustrated the different classifier designs that we used for the Perm-MNIST experiments. As mentioned previously, the backbone for the Perm-MNIST experiments were solely the classifiers and that no feature extractor was involved. In addition, all LWN were applied at the input of layer two.
Likewise, we presented our different classifier designs for Inc-Cifar100 and Inc-CUB200 overleaf in Figure 8. There were two main differences. First, the backbone networks of the incremental classes were compartmentalised as fixed pre-trained feature extractors and sequential learning classifiers. Second, the normalisation from LWN were applied at two places instead of one. In addition to the usual normalisation for the input of layer two, we added LWN to the input of layer one. That is, normalisation was applied directly after the feature extractors pre-processed the input data.

heading: D MORE RESULTS FROM THE EXPERIMENTS
text: Due to space constraint, we were unable to put all experimental results in the main text. Hence we only showed the most important results, but we stored the rest of the performances in this section.
In Table 4, we provided additional results on Inc-Cifar100 under the naı̈ve sequential learning setup. The results were for the scenarios with class increments of size 5 and size 10. Similarly, Table 5 provided additional results on Inc-CUB200 under the naı̈ve sequential learning setup with class incremental size 10 and size 20.
Then in Tables 6 to 12, we listed more results when the classifiers were trained with continual learning techniques applied. The aditional results in this section provided the extra metric scores for
MHC, LWN in MLP, and LWN in HCN for all scenarios. In addition, we also listed those results for Inc-Cifar100 with incremental class sizes 5 and 10; and similarly Inc-CUB200 with incremental class sizes 10 and 20.

heading: E THE NOISE INJECTION SETUP
text: The NI setup of Kuo et al. (2021a) can be seen as a direct extension of the Perm-MNIST task. Based on the description of that paper, there is only one extra step – which is to randomly shuffle a portion of the content of the Perm-MNIST vectors. To elaborate, each task of Perm-MNIST first reorganises each MNIST image as a vector of 784 pixels; it then applies the permutation mask of the current task to that vector of pixels. On top of this, NI randomly selects a portion of the pre-processed 784 pixels and shuffle those selected pixels.
The remaining hyperparameters of the NI scenario were identical to our normal Perm-MNIST setup. There were 20 tasks in total. We naı̈vely sequentially trained the classifiers over 5K images on the first task, then observe 1K images on the rest of the tasks. All classifiers had two layers with hidden dimension 100; and they were updated with SGD with learning rate 0.01.

heading: F ADDITIONAL EXPERIMENTAL SETUPS
text: It was suggested by one of our reviewers to include additional experiments to test our techniques on the non-standard continual learning classifiers. To this end, we supplemented our existing experiments with Energy-Based Models (EBM) (Li et al., 2020) as alternative classifiers.
As shown in Figure 1, traditional continual learning classifiers used softmax as their final layer. However, it was noted in Li et al. (2020) that softmax classifiers encourage a winner-takes-all dy-
namic for network parameterisation. Thus alternatively, they advocated for the use of EBMs. The aim of their EBM was to minimise the energy for the input class while rising the energy level for all remaining classes. Such a setup would allow the network to lower its confidence in making an incorrect decision.
However, their EBM required a paradigmatic shift towards the conventional network setup for continual learning. First, they relinquished softmax as their final layer and instead replaced it with a linear transformation with a single output dimension. Second, they required both the data along with their ground-truth labels as input. Regardless of these changes, our MHC and LWN techniques were still applicable to EBM classifiers. As shown in Table 13, our techniques were able to improve the vanilla EBM setup across three different datasets.

Review summaries:
The paper proposes modifications to the highway connection network (HCN) architecture, which exhibits some inherent resistance to catastrophic forgetting.  These modification introduce fairly simple masking rule for selection of portion of weights to update and normalisation that (presumably) further adjust the internal representation to be more selective in which weights to update for a given task.  Empirical evaluation shows that these modification are highly effective in alleviating catastrophic forgetting.

In the paper, the authors present two modifications: Masked Highway Connection and Layer-Wise Normalisation. In Masked Highway Connection, authors add a binary mask to classical HCN (Highway Connection Networks) and slightly change the training procedure. On the other hand, Layer-Wise Normalisation is a new normalization of activation in the neural network.

The paper investigates the backbone networks that are less prone to catastrophic forgetting. Two modifications to existing backbones are found to be useful: a mask attached to the gate function of the Highway Network, and layer-norm (without tuning parameters). Experiments on top of existing popular learning algorithms designed for continual learning (EWC, ER and HAT) show that these modifications work.

This paper proposed two architectural improvements for networks designed for continual learning. Specifically, binary masks and layer normalization are added to highway connection classifier networks in order to prevent forgetting. The value of each of these elements is made clear from thorough experimentation.