Continual Learning in Deep Networks: an Analysis of the Last Layer

We study how different output layers in a deep neural network learn and forget in continual learning settings . The following three factors can affect catastrophic forgetting in the output layer : ( 1 ) weights modifications , ( 2 ) interference , and ( 3 ) projection drift . In this paper , our goal is to provide more insights into how changing the output layers may address ( 1 ) and ( 2 ) . Some potential solutions to those issues are proposed and evaluated here in several continual learning scenarios . We show that the best-performing type of the output layer depends on the data distribution drifts and/or the amount of data available . In particular , in some cases where a standard linear layer would fail , it turns out that changing parameterization is sufficient in order to achieve a significantly better performance , whithout introducing a continual-learning algorithm and instead using the standard SGD to train a model . Our analysis and results shed light on the dynamics of the output layer in continual learning scenarios , and suggest a way of selecting the best type of output layer for a given scenario . 1 INTRODUCTION . Continual deep learning algorithms usually rely on end-to-end training of deep neural networks , making them difficult to analyze given the complexity ( non-linearity , non-convexity ) of the training dynamics . In this work , we instead propose to evaluate the ability of a network to learn or forget in a simplified learning setting : We decompose our model as a feature extractor part consisting of all but the final layer , and a classifier part which is given by the output layer . Using this decomposition , we can isolate the role of the output layer parameterization in continual learning scenarios . Recent works on continual learning ( CL ) point out some drawbacks of parameterizing the output layer as a linear layer , especially in incremental settings ( Wu et al. , 2019 ; Zhao et al. , 2020 ; Hou et al. , 2019 ) . Among these problems is the unbalance of bias and/or norm of the output layer ’ s vectors . These works propose some solutions but do not study them independently from the feature extractor . Indeed , in CL scenarios , the function learned by the feature extractor changes , and therefore the embedding space changes accordingly ; we call this change the projection drift . This drift characterizes the change through time of the embedding on a given observation . By isolating the output layer from the feature extractor , we can study it in a controlled environment , i.e. , without projection drifts . Decoupling the feature extractor from a sub-network has shown to be helpful , e.g. , in reinforcement learning ( Lesort et al. , 2018 ; Raffin et al. , 2019 ; Stooke et al. , 2020 ) . We now list our contributions : •We propose an evaluation of a large panel of output layer types in incremental , lifelong , and mixed continual scenarios . We show that depending on the type of scenario , the best output layer may vary . •We describe the different sources of performance decrease in continual learning for the output layer : forgetting , interference , and projection drifts . •We propose different solutions to address catastrophic forgetting in the output layer : a simplified weight normalization layer , two masking strategies , and an alternative to Nearest Mean Classifier using median vectors . 2 RELATED WORKS . In most deep continual learning papers , the last layer of the deep neural network is implemented as a linear layer as in typical classifier architectures in i.i.d . settings such as VGG ( Simonyan & Zisserman , 2014 ) , ResNet ( He et al. , 2016 ) , and GoogleNet ( Szegedy et al. , 2014 ) . Recently , several CL approaches questioned this approach , showing that the norm or the bias can be unbalanced for the last classes observed in class-incremental settings . For example , BIC ( Wu et al. , 2019 ) proposed to train two additional parameters using a subset of the training dataset after training the rest of the model to correct for the unbalance of the bias . In the same spirit , ( Zhao et al. , 2020 ) proposed to compute a Weight Aligning value to apply to vector to balance the norm of past classes vectors and the new classes vectors of a linear layer . ( Zhao et al. , 2020 ) also experiments with a layer , the weight normalization layer ( Salimans & Kingma , 2016 ) , that decouples the norm of the weight matrix from its direction in order to normalize the vector of linear output layers . ( Hou et al. , 2019 ; Caccia et al. , 2021 ) proposed to apply a cosine normalization of the layer to avoid the norm and bias unbalance . Linear Discriminant Analysis ( LDA ) Classifiers in incremental settings have been used for dozens of years ( Aliyari Ghassabeh et al. , 2015 ; Chatterjee & Roychowdhury , 1997 ; Demir & Ozmehmet , 2005 ) . The streaming version SLDA from ( Shaoning Pang et al. , 2005 ) has recently been revisited in deep continual learning in ( Hayes & Kanan , 2020 ) . SLDA combines the mean and the covariance of the training data features to classify new observations . A more straightforward approach , as implemented in iCaRL ( Rebuffi et al. , 2017 ) is to average the features for each class and apply a nearest neighbor classifier also called Nearest Mean Classifier ( NMC ) . In this work , we study the capability of those various types of top layers with a fixed pre-trained model in incremental and lifelong settings as defined in ( Lesort et al. , 2021 ) . Continual learning research field studied many strategies , e.g . rehearsal , generative replay , dynamic architectures or regularization . In this study , we do not aim at modifying the training process or proposing a new one . We study how different layer types learn in various continual scenarios without a CL approach . 3 OUTPUT LAYER TYPES . We here review the output layer parameterizations studied in this work . We introduce the notations , and present the main characteristics of linear layers . Then , we explain how standard layer parameterization struggles to learn continually , and we present different parameterizations to overcome the problems . Finally , we present several other types of output layers not trained by gradient descent . 3.1 NOTATIONS . We will study functions fθ ( · ) parameterized by a vector of parameters θ representing the set of weight matrices and bias vectors of a deep network . In continual learning , the goal is to train fθ ( · ) on a sequence of task [ T0 , T1 , ... , TT−1 ] , such that ∀ ( xt , yt ) ∼ Tt ( t ∈ [ 0 , T − 1 ] ) , fθ ( x ) = y . We can decompose fθ ( · ) into two parts , ( 1 ) a feature extractor φθ− ( · ) which encodes the observation xt into a feature vector zt , ( 2 ) an output layer gθ+ ( · ) which transforms the feature vector into a non-normalized vector ot ( the logits ) . θ− and θ+ are two complementary subsets of θ . 3.2 LINEAR LAYER . A linear layer is parameterized by a weight matrix A and bias vector b , respectively of size N × h and N , where h is the size of the latent vector ( the activations of the penultimate layer ) and N is the number of classes . For z a latent vector , the output layer computes the operation o = Az + b . We can formulate this operation for a single class i with 〈z , Ai〉+ bi = oi , where 〈·〉 is the euclidean scalar product , Ai is the ith row of the weight matrix viewed as a vector and bi is the corresponding scalar bias . It can be rewritten : ‖z‖‖Ai‖ · cos ( ∠ ( z , Ai ) ) + bi = oi ( 1 ) Where ∠ ( · , · ) is the angle between two vectors and ‖·‖ denotes here the euclidean norm of a vector . Eq . 1 highlights the 3 components that need to be learned to make a correct prediction ŷ = argmaxi ( oi ) : The norm of Ai , its angle with the latent representation , and the bias . In particular , if the training dynamics makes a particular ‖Ai‖ or bi larger than others , then the network will be considerably biased into predicting the class i . In incremental learning , the vectors Ai and values bi are learned sequentially , hence , the model will grow the ‖Ai‖ and bi for current classes and reduce them for past classes to ease current task ( Figure 1 ) . This phenomenon may lead to forgetting . 3.3 LINEAR CLASSIFIERS IN CONTINUAL LEARNING AND REPARAMETERIZATIONS . In continual learning , assuming that the true function f ( x ) = y is fixed , there are two main cases of data distribution drift ( see e.g . Lesort et al . ( 2021 ) ) : either we get new data of known classes ( domain drift −→ lifelong scenario ) or receive new data from new classes ( virtual concept drift −→ incremental scenario ) . For a linear output layer , forgetting is caused by different mechanisms in both scenarios . In incremental learning , the vectors Ai and bi are learned sequentially ( one by one or set by set ) . When learning a new Ai ( and bi ) , the training algorithm does not have access to past classes , and hence can not ensure that learning weight vectors for new classes does not interfere with past classes with eq . 1 . As an illustration , suppose two classes are alike in the latent space . In that case , their respective latent vectors might be similar , and interference might happen between the two classes ( see fig . 2 ) , a fortiori if norm and bias are not balanced . One solution found in the bibliography to avoid interference is to counteract unbalance between the norms of vectors or bias . We can modify eq . 1 in several ways to mitigate such unbalance : Removing the bias ( Linear_no_bias layer ) : −→ ‖z‖‖Ai‖ · cos ( ∠ ( zt , Ai ) ) = oi ( 2 ) Normalizing output vectors ( WeightNorm layer ) : −→ ‖z‖ · cos ( ∠ ( zt , Ai ) ) = oi ( 3 ) Measuring only the angle ( CosLayer ) : −→ cos ( ∠ ( z , Ai ) ) = oi ( 4 ) WeightNorm ( eq . 3 ) is similar to the original WeightNorm layer ( Salimans & Kingma ( 2016 ) , here denoted by Original WeightNorm ) experimented in Zhao et al . ( 2020 ) in a continual learning context : γi‖z‖ · cos ( ∠ ( zt , Ai ) ) + bi = oi ( 5 ) However , in the original WeightNorm , the additional scaling parameter γ and the bias b are learned during training . These parameters are akin to the parameters in BatchNorm layers ( Ioffe & Szegedy , 2015 ) , which have been shown to hurt the training of intermediate layers in continual learning ( Lomonaco et al. , 2020 ) . Hence , our proposed WeightNorm layer ( eq . 3 ) avoids such interference by ensuring a unit norm for all vectors and removing bias and gamma parameters . Forgetting might also be caused by weight modification : this happens when the optimizer modifies weight vectors from past classes that are not present in the current task . In the literature , regularization strategies ( e.g . EWC ( Kirkpatrick et al. , 2017 ) , KFRA ( Ritter et al. , 2018 ) ) aim at avoiding such forgetting by penalizing modification of important parameters . We introduce a more radical strategy to avoid that the update on a specific class affects another past one . The strategy consists in masking some classes during the update step . We propose two types of masking : in single masking , we only update weights for the output vector of the true target and in group masking , we mask all classes that are not in the mini-batch . With the following strategies , the update step can not change Ai and bi of past classes and avoid sub-sequential interference . In Figure 1 , we illustrate the impact of single masking in a task with only one class from one experiment of the paper . In incremental scenarios , all masking strategies can also be seen as a regularization strategy that strictly forbids the modification of past weights in the last layer . The masking strategy resembles the update step in a multi-head architecture ( van de Ven & Tolias , 2019 ) for gradient descent since in this case , the gradient is computed only for a subset of selected classes . However , in our setup there is only one head , and the inference is achieved without the help of any external supervision , such as a task label . In lifelong learning , forgetting happens when new instances of a previously learned class make the output layer forget features that were important for past instances or modify the direction of the output vectors . However , those problems are usually less prone to catastrophic forgetting but may still have a significant impact in complex scenarios . One of the reasons that catastrophic forgetting is less important is that all classes are available simultaneously , which makes it possible for the output vectors to be coordinated and avoid interference .

Summary: This paper presents a study on different last output layers, under the continual learning problem. More specifically, the study is conducted in a controlled environment without projection shift. The paper proposed several design choices of the last layer, including a modified weight normalization layer, masking operations, and median vector. Different scenarios of incremental/lifelong and mixed learning have been experimented with.