Year: 2021

Abstract:
We study how different output layers in a deep neural network learn and forget in continual learning settings. The following three factors can affect catastrophic forgetting in the output layer: (1) weights modifications, (2) interference, and (3) projection drift. In this paper, our goal is to provide more insights into how changing the output layers may address (1) and (2). Some potential solutions to those issues are proposed and evaluated here in several continual learning scenarios. We show that the best-performing type of the output layer depends on the data distribution drifts and/or the amount of data available. In particular, in some cases where a standard linear layer would fail, it turns out that changing parameterization is sufficient in order to achieve a significantly better performance, whithout introducing a continual-learning algorithm and instead using the standard SGD to train a model. Our analysis and results shed light on the dynamics of the output layer in continual learning scenarios, and suggest a way of selecting the best type of output layer for a given scenario.

text: We study how different output layers in a deep neural network learn and forget in continual learning settings. The following three factors can affect catastrophic forgetting in the output layer: (1) weights modifications, (2) interference, and (3) projection drift. In this paper, our goal is to provide more insights into how changing the output layers may address (1) and (2). Some potential solutions to those issues are proposed and evaluated here in several continual learning scenarios. We show that the best-performing type of the output layer depends on the data distribution drifts and/or the amount of data available. In particular, in some cases where a standard linear layer would fail, it turns out that changing parameterization is sufficient in order to achieve a significantly better performance, whithout introducing a continual-learning algorithm and instead using the standard SGD to train a model. Our analysis and results shed light on the dynamics of the output layer in continual learning scenarios, and suggest a way of selecting the best type of output layer for a given scenario.

heading: 1 INTRODUCTION
text: Continual deep learning algorithms usually rely on end-to-end training of deep neural networks, making them difficult to analyze given the complexity (non-linearity, non-convexity) of the training dynamics. In this work, we instead propose to evaluate the ability of a network to learn or forget in a simplified learning setting: We decompose our model as a feature extractor part consisting of all but the final layer, and a classifier part which is given by the output layer. Using this decomposition, we can isolate the role of the output layer parameterization in continual learning scenarios.
Recent works on continual learning (CL) point out some drawbacks of parameterizing the output layer as a linear layer, especially in incremental settings (Wu et al., 2019; Zhao et al., 2020; Hou et al., 2019). Among these problems is the unbalance of bias and/or norm of the output layer’s vectors. These works propose some solutions but do not study them independently from the feature extractor. Indeed, in CL scenarios, the function learned by the feature extractor changes, and therefore the embedding space changes accordingly; we call this change the projection drift. This drift characterizes the change through time of the embedding on a given observation.
By isolating the output layer from the feature extractor, we can study it in a controlled environment, i.e., without projection drifts. Decoupling the feature extractor from a sub-network has shown to be helpful, e.g., in reinforcement learning (Lesort et al., 2018; Raffin et al., 2019; Stooke et al., 2020).
We now list our contributions: •We propose an evaluation of a large panel of output layer types in incremental, lifelong, and mixed continual scenarios. We show that depending on the type of scenario, the best output layer may vary.
•We describe the different sources of performance decrease in continual learning for the output layer: forgetting, interference, and projection drifts.
•We propose different solutions to address catastrophic forgetting in the output layer: a simplified weight normalization layer, two masking strategies, and an alternative to Nearest Mean Classifier using median vectors.

heading: 2 RELATED WORKS
text: In most deep continual learning papers, the last layer of the deep neural network is implemented as a linear layer as in typical classifier architectures in i.i.d. settings such as VGG (Simonyan & Zisserman, 2014), ResNet (He et al., 2016), and GoogleNet (Szegedy et al., 2014). Recently, several CL approaches questioned this approach, showing that the norm or the bias can be unbalanced for the last classes observed in class-incremental settings. For example, BIC (Wu et al., 2019) proposed to train two additional parameters using a subset of the training dataset after training the rest of the model to correct for the unbalance of the bias. In the same spirit, (Zhao et al., 2020) proposed to compute a Weight Aligning value to apply to vector to balance the norm of past classes vectors and the new classes vectors of a linear layer. (Zhao et al., 2020) also experiments with a layer, the weight normalization layer (Salimans & Kingma, 2016), that decouples the norm of the weight matrix from its direction in order to normalize the vector of linear output layers. (Hou et al., 2019; Caccia et al., 2021) proposed to apply a cosine normalization of the layer to avoid the norm and bias unbalance.
Linear Discriminant Analysis (LDA) Classifiers in incremental settings have been used for dozens of years (Aliyari Ghassabeh et al., 2015; Chatterjee & Roychowdhury, 1997; Demir & Ozmehmet, 2005). The streaming version SLDA from (Shaoning Pang et al., 2005) has recently been revisited in deep continual learning in (Hayes & Kanan, 2020). SLDA combines the mean and the covariance of the training data features to classify new observations. A more straightforward approach, as implemented in iCaRL (Rebuffi et al., 2017) is to average the features for each class and apply a nearest neighbor classifier also called Nearest Mean Classifier (NMC).
In this work, we study the capability of those various types of top layers with a fixed pre-trained model in incremental and lifelong settings as defined in (Lesort et al., 2021). Continual learning research field studied many strategies, e.g. rehearsal, generative replay, dynamic architectures or regularization. In this study, we do not aim at modifying the training process or proposing a new one. We study how different layer types learn in various continual scenarios without a CL approach.

heading: 3 OUTPUT LAYER TYPES
text: We here review the output layer parameterizations studied in this work. We introduce the notations, and present the main characteristics of linear layers. Then, we explain how standard layer parameterization struggles to learn continually, and we present different parameterizations to overcome the problems. Finally, we present several other types of output layers not trained by gradient descent.

heading: 3.1 NOTATIONS
text: We will study functions fθ(·) parameterized by a vector of parameters θ representing the set of weight matrices and bias vectors of a deep network. In continual learning, the goal is to train fθ(·) on a sequence of task [T0, T1, ..., TT−1], such that ∀(xt, yt) ∼ Tt (t ∈ [0, T − 1]), fθ(x) = y. We can decompose fθ(·) into two parts, (1) a feature extractor φθ−(·) which encodes the observation xt into a feature vector zt, (2) an output layer gθ+(·) which transforms the feature vector into a non-normalized vector ot (the logits). θ− and θ+ are two complementary subsets of θ.

heading: 3.2 LINEAR LAYER
text: A linear layer is parameterized by a weight matrix A and bias vector b, respectively of size N × h and N , where h is the size of the latent vector (the activations of the penultimate layer) and N is the number of classes. For z a latent vector, the output layer computes the operation o = Az + b. We can formulate this operation for a single class i with 〈z,Ai〉+ bi = oi, where 〈·〉 is the euclidean scalar product, Ai is the ith row of the weight matrix viewed as a vector and bi is the corresponding scalar bias. It can be rewritten: ‖z‖‖Ai‖ · cos(∠(z,Ai)) + bi = oi (1) Where ∠(·, ·) is the angle between two vectors and ‖·‖ denotes here the euclidean norm of a vector. Eq. 1 highlights the 3 components that need to be learned to make a correct prediction ŷ = argmaxi(oi): The norm of Ai, its angle with the latent representation, and the bias. In particular,
if the training dynamics makes a particular ‖Ai‖ or bi larger than others, then the network will be considerably biased into predicting the class i. In incremental learning, the vectors Ai and values bi are learned sequentially, hence, the model will grow the ‖Ai‖ and bi for current classes and reduce them for past classes to ease current task ( Figure 1). This phenomenon may lead to forgetting.

heading: 3.3 LINEAR CLASSIFIERS IN CONTINUAL LEARNING AND REPARAMETERIZATIONS
text: In continual learning, assuming that the true function f(x) = y is fixed, there are two main cases of data distribution drift (see e.g. Lesort et al. (2021)): either we get new data of known classes (domain drift −→ lifelong scenario ) or receive new data from new classes (virtual concept drift −→ incremental scenario). For a linear output layer, forgetting is caused by different mechanisms in both scenarios.
In incremental learning, the vectors Ai and bi are learned sequentially (one by one or set by set). When learning a new Ai (and bi), the training algorithm does not have access to past classes, and hence cannot ensure that learning weight vectors for new classes does not interfere with past classes with eq. 1. As an illustration, suppose two classes are alike in the latent space. In that case, their respective latent vectors might be similar, and interference might happen between the two classes (see fig. 2), a fortiori if norm and bias are not balanced.
One solution found in the bibliography to avoid interference is to counteract unbalance between the norms of vectors or bias. We can modify eq. 1 in several ways to mitigate such unbalance:
Removing the bias (Linear_no_bias layer): −→ ‖z‖‖Ai‖ · cos(∠(zt, Ai)) = oi (2)
Normalizing output vectors (WeightNorm layer): −→ ‖z‖ · cos(∠(zt, Ai)) = oi (3) Measuring only the angle (CosLayer): −→ cos(∠(z,Ai)) = oi (4)
WeightNorm (eq. 3) is similar to the original WeightNorm layer (Salimans & Kingma (2016), here denoted by Original WeightNorm) experimented in Zhao et al. (2020) in a continual learning context:
γi‖z‖ · cos(∠(zt, Ai)) + bi = oi (5)
However, in the original WeightNorm, the additional scaling parameter γ and the bias b are learned during training. These parameters are akin to the parameters in BatchNorm layers (Ioffe & Szegedy, 2015), which have been shown to hurt the training of intermediate layers in continual learning (Lomonaco et al., 2020). Hence, our proposed WeightNorm layer (eq. 3) avoids such interference by ensuring a unit norm for all vectors and removing bias and gamma parameters.
Forgetting might also be caused by weight modification: this happens when the optimizer modifies weight vectors from past classes that are not present in the current task. In the literature, regularization strategies (e.g. EWC (Kirkpatrick et al., 2017), KFRA (Ritter et al., 2018)) aim at avoiding such forgetting by penalizing modification of important parameters.
We introduce a more radical strategy to avoid that the update on a specific class affects another past one. The strategy consists in masking some classes during the update step. We propose two types of
masking: in single masking, we only update weights for the output vector of the true target and in group masking, we mask all classes that are not in the mini-batch. With the following strategies, the update step cannot change Ai and bi of past classes and avoid sub-sequential interference. In Figure 1, we illustrate the impact of single masking in a task with only one class from one experiment of the paper. In incremental scenarios, all masking strategies can also be seen as a regularization strategy that strictly forbids the modification of past weights in the last layer.
The masking strategy resembles the update step in a multi-head architecture (van de Ven & Tolias, 2019) for gradient descent since in this case, the gradient is computed only for a subset of selected classes. However, in our setup there is only one head, and the inference is achieved without the help of any external supervision, such as a task label.
In lifelong learning, forgetting happens when new instances of a previously learned class make the output layer forget features that were important for past instances or modify the direction of the output vectors. However, those problems are usually less prone to catastrophic forgetting but may still have a significant impact in complex scenarios. One of the reasons that catastrophic forgetting is less important is that all classes are available simultaneously, which makes it possible for the output vectors to be coordinated and avoid interference.

heading: 3.4 SIMILARITY-BASED CLASSIFIERS (TRAINED WITHOUT GRADIENT)
text: To avoid the interference and weight modification, we can use another type of layers that rely on the similarity between training latent vectors and testing latent vectors.
One of the most popular classifiers that can easily be adapted into a continual or online learning setting is the k-nearest neighbors (KNN) classifier. This classifier searches for the k nearest exemplars of a given observation in the training set and predicts the class based on the class of the k neighbors. This classifier’s hyper-parameters are the number of neighbors k and the distance used (here the usual euclidean norm).
A lighter strategy is the nearest mean classifier (denoted MeanLayer) as proposed in iCarL (Rebuffi et al., 2017), this strategy consists of saving the mean of the features of each class k while learning (called a prototype µk) and predict the class given by the nearest mean for test latent vectors.
In this paper, we propose to evaluate the performance of a median classifier (denoted MedianLayer), where we replace the mean of the features with the median of the features. The median vector is computed by selecting the median value on each dimension. The idea behind this approach is that the prototype might be more central in the feature distribution and maybe avoid interference with similar classes more effectively.
An alternative is the linear discriminant analysis classifier, which uses both the mean of the features and a covariance. The online version of this algorithm is streaming linear discriminant analysis
(SLDA) (Shaoning Pang et al., 2005). We used the approach with online updates of the covariance matrix of (Dasgupta & Hsu, 2007) as in (Hayes & Kanan, 2020) (see appendix B).
Similarity-based classifiers are very effective in the literature. However, they cannot be directly used to train the feature extractor since they are not differentiable. Hence, they need to store samples from past tasks as rehearsal strategies or be applied in a setting without projection drift.
In this section, we introduced the different output layer types that we will use in experiments. We introduce three modifications: (1) a simplified WeightNorm (referred to as WeightNorm), (2) two masking strategies, single masking and group masking, compatible with any reparameterization of the linear layer, and (3) a Nearest Median Classifier.

heading: 4 EXPERIMENTS
text: In this section, we will review how the different output layer types, presented in section 3, can solve various continual scenarios. We isolate the output layer from the remaining of the neural network by using a frozen pre-trained model. This methodology consists of creating a scenario where no projection drift occurs. Indeed, projection drifts may only lead to a deterioration of performance. Hence our study can eliminate approaches that would, in all cases, fail under such conditions. Our findings can also be directly transferred to settings where the projection drift is negligible.
In a preliminary experiment, we benchmark all output layer types in an i.i.d. setting to fix some of their hyper-parameters for further continual training. The conclusion of these experiments are in section 4.1, but the full experiments are in appendix section C and in Table 3.
In a second setup, we train the various output layers in continual learning scenarios. Finally, we experiment with a vanilla continual approach close to a rehearsal approach: we train using only on a subset of the training data, and compare layer parameterizations in this low data regime.

heading: 4.1 DATASETS AND PRE-TRAINED MODELS
text: We conduct our experiments on CIFAR10/CIFAR100 (Krizhevsky & Hinton, 2009) and Core50 (Lomonaco & Maltoni, 2017). Core50 is a dataset composed of 50 objects from 10 categories (e.g. glasses, phone, cup...), filmed in 11 environments (8 for training and 3 for evaluation). We denote Core10, the 10 classes version where we only predict the category, and Core50, the full 50 classes version. We use both to create various types of scenarios.
The preliminary experiments consist of testing the basic output layer (without masking) to select a learning rate (among 0.1, 0.01 and 0.001 ) and architecture for pre-trained models (among ResNet, VGG16 and GoogleNet available on the torchvision library (Paszke et al., 2019)). While choosing pre-trained models, we made sure that pre-training had not been done using the same dataset that we used in our continual learning experiments (e.g. we did not choose a model pre-trained on CIFAR10 for our CIFAR10 experiments), which would otherwise be "cheating" as the pre-training would have been made using all available data, and not been restricted to data sequentially made available as in continual learning.
Following the results (appendix C), we choose a learning rate of 0.01 for the original linear layer and original linear layer without bias (Linear_no_bias) as well as for their masked counterpart and 0.1 for the other layers. Secondly, we found that for Core50 and Core10 experiments, the ResNet model performs best. We also eliminated the CIFAR100 dataset from our incremental experiments because the pre-trained model we tested (pre-trained on CIFAR10) did not permit to achieve a good enough accuracy. We kept CIFAR10, Core10, and Core50 for subsequent experiments. We added CUB200 and a modified version of CIFAR100 (referred to as CIFAR100Lifelong), for continual experiments. Moreover, we see that 5 epochs per task are sufficient on these datasets, and we will keep this number of epochs for all experiments.

heading: 4.2 CONTINUAL EXPERIMENTS SETTINGS
text: In these experiments, we evaluate the capacity of every output layer to learn continually with a fixed feature extractor. We evaluate the CL performance on virtual concept drifts (incremental scenario),
on domain drifts (lifelong scenario) (cf section 3.3). Incremental settings consist of a sequence of tasks where all new tasks bring new unknown classes. Lifelong settings consists of a sequence of tasks where each new task brings new examples of known classes. We also evaluate the best layers in a mixed scenario with both kinds of drifts in the end. Our experiments use the single-head framework (Farquhar & Gal, 2018), as opposed to multi-head where the task id is used for inference to pre-select a subset of classes. We do not use task labels during training nor testing. The change in labels can signal a change of tasks. However, it does not trigger any continual mechanisms in our experiments, and all training are done using stochastic gradient descent with a momentum of 0.9.
Incremental Scenarios: CIFAR10, Core50, CUB200: Incremental scenarios are characterized by a virtual concept drift between two tasks. These settings evaluate the capacity of learning incremental new classes and distinguishing them from the others. The core50 scenario is composed of 10 tasks; each of them is composed of 5 classes, the CIFAR10 scenario is composed of 5 tasks with 2 classes each, and the CUB200 scenario is composed of 10 tasks of 20 classes each.
Lifelong Scenarios: Core10Lifelong, CIFAR100Lifelong: Lifelong settings are characterized by a domain drift between two tasks. The classes stay the same, but the instances change. These settings evaluate the capacity of improving at classifying with new data. The Core10Lifelong scenario is composed of 8 tasks with 10 classes in a given environment, each new task, we visit a new environment. CIFAR100Lifelong is composed of 5 tasks, with 20 labels each that also stay the same. Data are labeled with the coarse labels of CIFAR100. However, data are shared between tasks using the original label to ensure a domain drift between tasks (More detail in appendix in section A).
Mixed Scenario: Core10Mix: In this scenario, a new task is triggered by either a virtual concept drift or a domain drift. In this case, the domain drift is characterized by the transition from one object to another in the same category. This scenario is composed of 50 tasks, where each task corresponds to a new object (with category annotation) in all its training domain. So a new task is always a new object, but it is either data from a new class or from a known class visited through another object.
The results for those experiments are in section 5.1. We report at each epoch the average accuracy over the whole test set of the scenario (with data of all tasks). This measure makes it possible to report the accuracy on a fixed test set and makes the visualization of progress in solving the complete scenario easier. For easy reproducibility, we put in appendix H, how the scenarios are created with the continuum library (Douillard & Lesort, 2021).

heading: 4.3 SUBSET EXPERIMENTS SETTINGS
text: In many continual learning approaches, a subset of the training is saved, either for rehearsal purposes or for validation purposes (Buzzega et al., 2020; Douillard et al., 2020; Prabhu et al., 2020). In this experiment, we evaluate the capacity of each layer to be trained from scratch with such subsets. It could be more efficient to continually train the feature extractor in a continual end-to-end setting and learn a suitable output layer afterward from a subset of data. This procedure is very fast, and it avoids all problems that projection drifts or any other drift can create. The only (not so simple) problem to solve is which examples to store and how many. Nevertheless, the successes in few-shot learning (Lake et al., 2011; Fei-Fei et al., 2006; Wang et al., 2019) or fast adaptation (Caccia et al., 2020) show that when the feature space is good enough, only a small set of data is enough to identify a class.

heading: 5 RESULTS

heading: 5.1 CONTINUAL EXPERIMENTS
text: We split the results of continual experiments into two groups of figures; the first group (Fig. 3) only compares the various parameterizations of a linear layer presented in section 3. The second group ((Fig. 4)) compares the best performing layers of the first group (CosLayer and WeightNorm) to layers that are not trained by gradient descent, namely: KNN, MeanLayer, MedianLayer, and SLDA. The results presented in fig. 3 and fig. 4 lead us to the following conclusions:
Among the reparameterizations of linear layers, Coslayer and WeightNorm are the overall best performing. Indeed, in incremental settings (figs. 3a, 3c, 3e), they are always among the best performing methods. Both of them only rely on the angle between data and output vectors, which
reduces the risk of interference as described in section 3. The only difference between these layers is that in WeightNorm, the predictions are weighted by the norm of the latent vectors (cf Coslayer eq. 4, WeightNorm eq. 3), which scales the gradients thus may change the learning dynamics.
Masking is efficient in incremental settings to avoid weight modifications. In incremental scenarios (figs. 3a, 3c, 3e) the best performing methods are all masked. Except in CUB200 (Fig. 3e) where (only) the WeightNorm layer outperforms the masked layers. It shows that in incremental settings, avoiding weight modification with masking can significantly improve results. However, single masking1 creates instability in lifelong scenarios (figs. 3d, 3b). Masking may prevent a good organization of output vectors together and make the training unstable.
Removing the bias on the linear layer has no effect. We did not observe any improvement by removing the bias from the linear layers in our experiments (Linear and Linear_no_bias almost coincide). This questions Wu et al. (2019)’s observations that correcting the bias can mitigate catastrophic forgetting. Nevertheless, we hypothesize that the effect of the bias in continual learning depends on the task at hand.
Similarity-based layers perform well regardless of the data drift type (fig. 4). These layers are easy to train since they need to see the data only once to achieve their best performance. In fig. 4 these layers look to perform very well in all settings in comparison with linear layers, especially SLDA. Nevertheless, we should not forget that they cannot be used to train an end-to-end neural network. They could, however, be associated with another layer to train the feature extractor as in iCARL (Rebuffi et al., 2017).
In lifelong, we cannot avoid weight modification with masking. In lifelong scenarios, tasks use the same output vectors, i.e., the same weights; therefore, freezing weights forbids further learning. In Core10Lifelong setting (fig. 3b), weight modification does not look to be a problem, and no decrease in performance indicates catastrophic forgetting, while in CIFAR100Lifelong (fig. 3d) all layers seem to forget while learning a new task. This phenomenon is due to the similarity of the different modes
1We removed the layer masked by group in the lifelong experiment because it is useless when all classes are available simultaneously.
of the data distribution, in Core10Lifelong only the background changes from one task to another. At the same time, in CIFAR100Lifelong, the data are very different from one task to another, i.e., for the coarse label “aquatic mammals” the data go from beavers to dolphins to otters to seals to finally whales in separate tasks.
Linear layers do not perform similarly while exposed to different types of drift. A tendency that emerges from these experiments is that all layers are compatible with lifelong learning even if masking may create some instability. In incremental and mixed scenarios, the layers relying only on angles between data and output vectors and masking helps to avoid weight modification.

heading: 5.2 TRAINING WITH A SUBSET
text: In these experiments, we randomly select a sub-sample of the training dataset, and we train the output layers on it. Experiment’s results are gathered in Table 1 (Core50) and Table 4 (other datasets). The goal is to measure how the layers previously experimented can learn with a low amount of data. It gives us also a good continual baseline close to the GDump approach (Prabhu et al., 2020) where we save samples in the memory, and we train at the end of the scenario only on the memory’s samples.
This approach performs particularly well in comparison to many continual approaches. Here are some conclusions of these experiments:
Masking does not work in an iid setting. In table 1, we can note that most of the masked layers underperform in this setting, except the CosLayer, which performs better with a mask.
WeightNorm is still very competitive among linear layers. In most of the experiments, WeightNorm is among the best reparameterization of the linear layer.
Similarity-based layers tend to slightly underperform linear layers. In most of the results, these layers do not perform as well as in the continual scenarios, even if they can still have good accuracy.
MedianLayer does not show much interest in comparison with MeanLayer. MedianLayer works quite well in our experiment but shows no real advantages against MeanLayer; moreover, MeanLayer is easier to compute online. In a dataset with many outliers, MeanLayer could show some advantages against the MeanLayer. In our experiments, only training on CIFAR10 with all data made MedianLayer better than MeanLayer (cf appendix Table 4).
These experiments with iid training on a low data regime show that in such settings, reparameterization of the linear layer are well-performing. In conclusion of all the experiments, we recommend using WeigthNorm (especially in lifelong) or CosLayer in its masked version in continual scenarios.

heading: 6 DISCUSSION
text: In this paper, we studied and proposed various approaches that can help to avoid catastrophic forgetting in the output layer by addressing weights modification and interference, by restricting ourself to a setup where there is no projection drift.
On the opposite, multi-head architectures use the task label at test time to train a head specific to the current task. It avoids interference and weight modification by training separate heads. Interestingly, in multi-heads settings, using regularization (Kirkpatrick et al., 2017; Zenke et al., 2017; Ritter et al., 2018) or dynamic architecture (Rusu et al., 2016; Mallya et al., 2018; Rajasegaran et al., 2019), have been shown to perform well on end-to-end tasks. Then, they are able to deal with projection drifts: the head of task 0 still works when applied to the feature extractor trained up until the last task.
Unfortunately, end-to-end approaches that work in multi-heads can not be directly connected with our findings to create a good end-to-end single head approach. Indeed, single-head models need discriminative inter-task representations that are not necessary for multi-heads settings. Therefore, the constraints on the feature extractor are not the same, and direct transfer is not possible.
Dealing with projection drift in single-head settings stays then one of the biggest challenge in continual learning. Replay might be a simple solution to simulate an iid training and avoid problems with projection drift. However, even if replay might be necessary in continual learning scenarios (Diethe et al., 2018; Lesort et al., 2019), replaying data may have consequences in the memory needs and compute needs. Therefore, it is worth finding training procedures, architectures, or regularizations that do not necessarily replace replay but at least reduce its need.

heading: 7 CONCLUSION
text: In this paper, we conduct an empirical evaluation of the various output layers of deep neural networks in CL scenarios. This evaluation gives us clear insights into how output layers learn continually or on a low data regime. We also showed how data distribution drifts might affect the output layers differently, namely in lifelong and incremental settings. This perspective should incite researchers to study more the connection between data distribution drifts characteristics and catastrophic forgetting.
On one hand, classifiers such as KNN, MeanLayer and SLDA provide very strong baselines that are very effective with a frozen feature extractor. On the other hand, our results show that by a reparameterization of the classical linear layer, we can greatly improve performance, especially in incremental settings. This reparameterization helps to reduce the risks of interference and weight modifications. Some output layer types can learn continually when there is no projection drift with stochastic gradient training. These findings could be integrated in end-to-end training to make continual training more efficient.

heading: APPENDIX

heading: A SCENARIOS DESCRIPTION

heading: A.1 CIFAR10 5 TASKS
text: A classical incremental setting, with 2 classes per task with original CIFAR10 dataset.

heading: A.1.1 CIFAR100LIFELONG 5 TASKS
text: CIFAR100 dataset has two labelization systems: the classes labels and the coarse labels. There are 20 coarse labels gathering 5 classes each. To create CIFAR100Lifelong 5 tasks, we share data into tasks such as having one data for each coarse label in each task. On the other hand, we use class labels to ensure that the data is different for each task.
We have then 5 tasks with data for all coarse labels and from 20 different classes. A class does not appear in 2 tasks, as in an incremental. However, the model should predict the coarse label. We obtain a scenario with 5 tasks with always the same 20 labels. However, the data change, and the model should learn continually to predict the coarse labels for data for all classes correctly. This scenario makes it possible to measure the resistance to mode change (change of class) while continually learning the coarse label classification.

heading: A.2 CORE50 10 TASKS
text: A classical incremental setting, with 5 classes per task with the 50 objects core50.

heading: A.3 CORE10LIFELONG 8 TASKS
text: Core50 is composed of 50 objects, equitably shared into 10 classes filmed in 11 environments (8 environments for train and 3 for tests.) In Core10Lifelong, we have 8 tasks where we explore each time one environment with all objects. Objects are annotated with their class labels. Hence, we have 8 tasks of 10-way classification. The object stays the same from one task to another; only the environment changes. This scenario makes it possible to measure the resistance to background change while learning continually.

heading: A.4 CORE10MIX 50 TASKS
text: In this scenario, each task contains the data corresponding to one object in all of its environments. We have hence a sequence of 50 tasks. However, objects are annotated with their classes labels. Hence all classes are revisited 5 times in the scenario (in a random way). This scenario is a mixture of incremental and lifelong to measure the ability to solve a scenario with several types of data distribution drifts.

heading: B SLDA LAYER
text: The approach we use (proposed by Hayes & Kanan (2020)) stores one mean vector per class µk with a counter ck ∈ R as well as a covariance matrix of the same size as A (common to all classes). The covariance matrix’s update is done with:
Σt+1 = tΣt + ∆t t+ 1
(6)
Where ∆t is:
∆t = t(zt − µk)2
t+ 1 (7)
For prediction, SLDA uses a precision matrix Λ = [(1− )Σ + I]−1, where = 10−4 is called the shrinkage parameter and I ∈ Rd×d is the identity matrix. Then two vectors w and b are computed wk = Λµk, bk = − 12 (µ > k Λµk)
And finally: ok = 〈zt, wk〉+ bk

heading: C PRELIMINARY EXPERIMENTS

heading: C.1 PRELIMINARY EXPERIMENTS SETTING
text: The preliminary experiments are designed to see how the different output layers work without continual learning constraints. This experiment is also conducted to select the learning rate and architecture for further experiments.
We experimented with CIFAR10 and CIFAR100 datasets, and we use resnet20 pre-trained model from here 2 (BSD 3-Clause License). The main idea is to compare those results with our results. We take the frozen pre-trained model and replace the output layer with a new one we train on all data. (1) We experiment with training on CIFAR10 with a neural network pre-trained on CIFAR100 and the opposite. (2) We also trained on CIFAR10 with the model pre-trained on CIFAR10 to see if we would recover the original accuracy, and we did the same for CIFAR100. Doing continual learning on a dataset with a model pre-trained on this dataset is not recommended since it means having access to the whole data from the beginning of training. We did not use (2) setting in the further experiments, and it was just used as a reference.
(3) We also experiment also with Core50 dataset (Lomonaco & Maltoni, 2017). We used models pre-trained on ImageNet available on the torchvision library (Paszke et al., 2019). We used VGG16, GoogleNet and Resnet, which have a latent space of dimension 2048, 1024, and 512. We experimented with the Core50 dataset, Core50 setting with 50 classes corresponding to the object id, and Core10 setting with 10 classes corresponding to the object category.
In these preliminary experiments, we evaluate all layers without their masking variant to verify that the pre-trained models were suited to the continual learning scenarios.

heading: C.2 PRELIMINARY EXPERIMENTS
text: The summary of the preliminary results is reported in Table 2. The full results can be found in Table 3. As described in section C.1, we experiment with the simplest output layer for this preliminary experiment, i.e., linear layer, linear layer without bias (Linear_NB), CosLayer, WeightNorm layer, Original WeightNorm (OWeightNorm), KNN, SLDA, and MeanLayer.
The results of those preliminary experiments are that globally the learning rate of 0.1 is adapted to all settings. Secondly, we found that for Core50 and Core10 experiments, the ResNet model was the best suited. Moreover, we see that 5 epochs per task are sufficient to learn a correct solution. Concerning CIFAR experiments, the reported top-1 accuracy on CIFAR10 and CIFAR100 of the original pre-trained models are respectively 92.60%, 69.83%. The settings CIFAR10 pre-trained on CIFAR10 (same for CIFAR100) show us that the training procedure can recover almost the best performance. In CIFAR100, there is a drop of 6% of accuracy, but it stays reasonable in a 100-way classification task.
Even if not very surprising, one interesting result is that training on CIFAR100 with a model pretrained on CIFAR10 does not work at all. It illustrates that using a pre-trained model is not always a solution in machine learning or continual learning. They should be selected carefully, even if the data might look similar to CIFAR10 and CIFAR100. On the other hand, the experiment on CIFAR10 with CIFAR100 models shows a significant decrease in the performance for all layers.
2https://github.com/chenyaofo/pytorch-cifar-models
These preliminary experiments make us able to fix some hyper-parameters to study all layers in a common setting. It also offers us interesting insight into the drop of performance in simple experiments when using a pre-trained model instead of training end-to-end. The results of this experiment can be used as a baseline for continual experiments and subsets experiments.

heading: D FORGETTING IN LIFELONG SCENARIOS

heading: D.1 CIFAR100LIFELONG
text: In Fig. 5, we can see that the model is forgetting the first tasks and the masking have no effect on it. Moreover, since the single masking makes learning less efficient with many classes, it can make the performance bad from the beginning.

heading: D.2 CORE10LIFELONG
text: Unfortunately, the Core50 test set does not make it possible to evaluate the forgetting in each task of lifelong scenarios. Indeed, the environments of the test set are different from the train environments. Then it is not possible to evaluate with the test set the evolution of accuracy on past tasks. In the lifelong scenario, we can only evaluate the performance on 3 other unknown environments, similarly to an out of distribution evaluation.
E FULL SUBSETS RESULTS
F VISUALIZATIONS
(a) Linear: Angles between the different vectors of the output layer (b) Linear: Mean Angles between the vectors and data class by class. (c) Linear: Interference Risks
(d) WeightNorm: Angles between the different vectors of the output layer (e) WeightNorm: Mean Angles between the vectors and data class by class. (f) WeightNorm: Interference Risks

heading: G TASK ORDERS
text: from continuum.scenarios import create_subscenario import numpy as np
# we suppose scenario, num_tasks and seed already set np.random.seed(seed) task_order = np.arange(num_tasks) scenario = create_subscenario(scenario, task_order)
(a) Code to change task order of the original scenario. We used seeds [0, 1, 2, 3, 4, 5, 6, 7]. Note: for seed 0, we use original order without modification.

heading: H SCENARIO REPRODUCIBILTY

Review summaries:
This paper presents a study on different last output layers, under the continual learning problem. More specifically, the study is conducted in a controlled environment without projection shift. The paper proposed several design choices of the last layer, including a modified weight normalization layer, masking operations, and median vector. Different scenarios of incremental/lifelong and mixed learning have been experimented with. 

This paper studies the output layer in a deep neural network in the continual learning setting. The paper studies multiple types of output layers under different continual learning configurations. Some suggestions are made based on observed empirical trends.

The authors analyzed the effect of changes to the last output layers of the neural network in the continual, incremental learning scenario and effects of it two scenarios 1. weight modification 2.Inference. Authors also claim to propose simplified weight normalization layer, two masking strategies, and an alternative to NearestMean Classifier using median vectors to address catastrophic forgetting in the output layer in both continual, Incremental scenarios.  During the experiment setup authors separated the network into two parts  1. feature extractor part consisting of all but the final layer (kept fixed to avoid projection drift) 2. classifier part which was given by the output layer.[Much of the analysis is concentrated on Part-B]