Discriminative Representation Loss (DRL): A More Efficient Approach than Gradient Re-Projection in Continual Learning

1 INTRODUCTION . In the real world , we are often faced with situations where data distributions are changing over time , and we would like to update our models by new data in time , with bounded growth in system size . These situations fall under the umbrella of “ continual learning ” , which has many practical applications , such as recommender systems , retail supply chain optimization , and robotics ( Lesort et al. , 2019 ; Diethe et al. , 2018 ; Tian et al. , 2018 ) . Comparisons have also been made with the way that humans are able to learn new tasks without forgetting previously learned ones , using common knowledge shared across different skills . The fundamental problem in continual learning is catastrophic forgetting ( McCloskey & Cohen , 1989 ; Kirkpatrick et al. , 2017 ) , i.e . ( neural network ) models have a tendency to forget previously learned tasks while learning new ones . There are three main categories of methods for alleviating forgetting in continual learning : i ) regularization-based methods which aim in preserving knowledge of models of previous tasks ( Kirkpatrick et al. , 2017 ; Zenke et al. , 2017 ; Nguyen et al. , 2018 ) ii ) architecture-based methods for incrementally evolving the model by learning task-shared and task-specific components ( Schwarz et al. , 2018 ; Hung et al. , 2019 ) ; iii ) replay-based methods which focus in preserving knowledge of data distributions of previous tasks , including methods of experience replay by episodic memories or generative models ( Shin et al. , 2017 ; Rolnick et al. , 2019 ) , methods for generating compact episodic memories ( Chen et al. , 2018 ; Aljundi et al. , 2019 ) , and methods for more efficiently using episodic memories ( Lopez-Paz & Ranzato , 2017 ; Chaudhry et al. , 2019a ; Riemer et al. , 2019 ; Farajtabar et al. , 2020 ) . Gradient-based approaches using episodic memories , in particular , have been receiving increasing attention . The essential idea is to use gradients produced by samples from episodic memories to constrain the gradients produced by new samples , e.g . by ensuring the inner product of the pair of gradients is non-negative ( Lopez-Paz & Ranzato , 2017 ) as follows : 〈gt , gk〉 = 〈 ∂L ( xt , θ ) ∂θ , ∂L ( xk , θ ) ∂θ 〉 ≥ 0 , ∀k < t ( 1 ) where t and k are time indices , xt denotes a new sample from the current task , and xk denotes a sample from the episodic memory . Thus , the updates of parameters are forced to preserve the performance on previous tasks as much as possible . In Gradient Episodic Memory ( GEM ) ( Lopez-Paz & Ranzato , 2017 ) , gt is projected to a direction that is closest to it in L2-norm whilst also satisfying Eq . ( 1 ) : ming̃ 12 ||gt − g̃|| 2 2 , s.t.〈g̃ , gk〉 ≥ 0 , ∀k < t. Optimization of this objective requires a high-dimensional quadratic program and thus is computationally expensive . Averaged-GEM ( A-GEM ) ( Chaudhry et al. , 2019a ) alleviates the computational burden of GEM by using the averaged gradient over a batch of samples instead of individual gradients of samples in the episodic memory . This not only simplifies the computation , but also obtains comparable performance with GEM . Orthogonal Gradient Descent ( OGD ) ( Farajtabar et al. , 2020 ) projects gt to the direction that is perpendicular to the surface formed by { gk|k < t } . Moreover , Aljundi et al . ( 2019 ) propose Gradient-based Sample Selection ( GSS ) , which selects samples that produce most diverse gradients with other samples into episodic memories . Here diversity is measured by the cosine similarity between gradients . Since the cosine similarity is computed using the inner product of two normalized gradients , GSS embodies the same principle as other gradient-based approaches with episodic memories . Although GSS suggests the samples with most diverse gradients are important for generalization across tasks , Chaudhry et al . ( 2019b ) show that the average gradient over a small set of random samples may be able to obtain good generalization as well . In this paper , we answer the following questions : i ) Which samples tend to produce diverse gradients that strongly conflict with other samples and why are such samples able to help with generalization ? ii ) Why does a small set of randomly chosen samples also help with generalization ? iii ) Can we reduce the diversity of gradients in a more efficient way ? Our answers reveal the relation between diversity of gradients and discriminativeness of representations , and further show connections between Deep Metric Learning ( DML ) ( Kaya & Bilge , 2019 ; Roth et al. , 2020 ) and continual learning . Drawing on these findings we propose a new approach , Discriminative Representation Loss ( DRL ) , for classification tasks in continual learning . Our methods show improved performance with relatively low computational cost in terms of time and RAM cost when compared to several state-of-theart ( SOTA ) methods across multiple benchmark tasks in the setting of online continual learning . 2 A NEW PERSPECTIVE OF REDUCING DIVERSITY OF GRADIENTS . According to Eq . ( 1 ) , negative cosine similarities between gradients produced by current and previous tasks result in worse performance in continual learning . This can be interpreted from the perspective of constrained optimization as discussed by Aljundi et al . ( 2019 ) . Moreover , the diversity of gradients relates to the Gradient Signal to Noise Ratio ( GSNR ) ( Liu et al. , 2020 ) , which plays a crucial role in the model ’ s generalization ability . Intuitively , when more of the gradients point in diverse directions , the variance will be larger , leading to a smaller GSNR , which indicates that reducing the diversity of gradients can improve generalization . This finding leads to the conclusion that samples with the most diverse gradients contain the most critical information for generalization which is consistent with in Aljundi et al . ( 2019 ) . 2.1 THE SOURCE OF GRADIENT DIVERSITY . We first conducted a simple experiment on classification tasks of 2-D Gaussian distributions , and tried to identify samples with most diverse gradients in the 2-D feature space . We trained a linear model on the first task to discriminate between two classes ( blue and orange dots in Fig . 1a ) . We then applied the algorithm Gradient-based Sample Selection with Interger Quadratic Programming ( GSS-IQP ) ( Aljundi et al. , 2019 ) to select 10 % of the samples of training data that produce gradients with the lowest similarity ( black dots in Fig . 1a ) , and denote this set of samples as M̂ = minM ∑ i , j∈M 〈gi , gj〉 ||gi||·||gj || . It is clear from Fig . 1a that the samples in M̂ are mostly around the decision boundary between the two classes . Increasing the size of M̂ results in the inclusion of samples that trace the outer edges of the data distributions from each class . Clearly the gradients can be strongly opposed when samples from different classes are very similar . Samples close to decision boundaries are most likely to exhibit this characteristic . Intuitively , storing the decision boundaries of previously learned classes should be an effective way to preserve classification performance on those classes . However , if the episodic memory only includes samples representing the learned boundaries , it may miss important information when the model is required to incrementally learn new classes . We show this by introducing a second task - training the model above on a third class ( green dots ) . We display the decision boundaries ( which split the feature space in a one vs. all manner ) learned by the model after 4 2 0 2 4 6 x 4 2 0 2 4 6 8 y class 0 class 1 M ( a ) Samples with most diverse gradients ( M̂ ) after learning task 1 , the green line is the decision boundary . 4 2 0 2 4 6 x 4 2 0 2 4 6 8 y class 0 class 1 class 2 memory ( b ) Learned decision boundaries ( purple lines ) after task 2 . Here the episodic memory includes samples in M̂ . 4 2 0 2 4 6 x 4 2 0 2 4 6 8 y class 0 class 1 class 2 memory ( c ) Learned decision boundaries ( purple lines ) after task 2 . Here the episodic memory consists of random samples . ( a ) Splitting samples into several subsets in a 3-class classification task . Dots in different colors are from different classes . ( b ) Estimated distributions of β when drawing negative pairs from different subsets of samples . ( c ) Estimated distributions of α− δ when drawing negative pairs from different subsets of samples . Figure 2 : Illustration of how Pr ( 2β > α − δ ) in Theorem 1 behaves in various cases by drawing negative pairs from different subsets of a 3-class feature space which are defined in Fig . 2a . The classifier is a linear model . y-axis in the right side of ( b ) & ( c ) is for the case of x ∈ S1 ∪ S2 . We see that α− δ behaves in a similar way with β but in a smaller range which makes β the key in studying Pr ( 2β > α − δ ) . In the case of x ∈ S3 the distribution of β has more mass on larger values than other cases because the predicted probabilities are mostly on the two classes in a pair , and it causes all 〈gn , gm〉 having the opposite sign of 〈xn , xm〉 as shown in Tab . 1. task 2 with M̂ ( Fig . 1b ) and a random set of samples ( Fig . 1c ) from task 1 as the episodic memory . The random episodic memory shows better performance than the one selected by GSS-IQP , since the new decision boundaries rely on samples not included in M̂ . It explains why randomly selected memories may generalize better in continual learning . Ideally , with M̂ large enough , the model can remember all edges of each class , and hence learn much more accurate decision boundaries sequentially . However , memory size is often limited in practice , especially for high-dimensional data . A more efficient way could be learning more informative representations . The experimental results indicate that : 1 ) more similar representations in different classes result in more diverse gradients . 2 ) more diverse representations in a same class help with learning new boundaries incrementally . Now we formalise the connection between the diversity of gradients and the discriminativeness of representations for the linear model ( proofs are in Appx . A ) . Notations : Negative pair represents two samples from different classes . Positive pair represents two samples from a same class . Let L represent the softmax cross entropy loss , W ∈ RD×K is the weight matrix of the linear model , and xn ∈ RD denotes the input data , yn ∈ RK is a one-hot vector that denotes the label of xn , D is the dimension of representations , K is the number of classes . Let pn = softmax ( on ) , where on = W Txn , the gradient gn = ∇WL ( xn , yn ; W ) . xn , xm are two different samples when n 6= m. Lemma 1 . Let n = pn − yn , we have : 〈gn , gm〉 = 〈xn , xm〉〈 n , m〉 , Theorem 1 . Suppose yn 6= ym , and let cn denote the class index of xn ( i.e . yn , cn = 1 , yn , i = 0 , ∀i 6= cn ) . Let α , ||pn||2 + ||pm||2 , β , pn , cm + pm , cn and δ , ||pn − pm||22 , then : Pr ( sign ( 〈gn , gm〉 ) = sign ( −〈xn , xm〉 ) ) = Pr ( 2β > α− δ ) , Theorem 2 . Suppose yn = ym , when 〈gn , gm〉 6= 0 , we have : sign ( 〈gn , gm〉 ) = sign ( 〈xn , xm〉 ) For a better understanding of the theorems , we conduct empirical study by partitioning the feature space of three classes into several subsets as shown in Fig . 2a and examine four cases of pairwise samples by these subsets : 1 ) . x ∈ S0 , both samples in a pair are near the intersection of the three classes ; 2 ) . x ∈ S0∪S1 , one sample is close to decision boundaries and the other is far away from the boundaries ; 3 ) . x ∈ S3 , both samples close to the decision boundary between their true classes but away from the third class ; 4 ) . x ∈ S1 ∪ S2 , both samples are far away from the decision boundaries . Theorem 1 says that for samples from different classes , 〈gn , gm〉 gets an opposite sign of 〈xn , xm〉 with a probability that depends on the predictions pn and pm . This probability of flipping the sign especially depends on β which reflects how likely to misclassify both samples to its opposite class . We show the empirical distributions of β and ( α− δ ) obtained by a linear model in Figs . 2b and 2c , respectively . In general , ( α− δ ) shows similar behaviors with β in the four cases but in a smaller range , which makes 2β > ( α − δ ) tends to be true except when β is around zero . Basically , a subset including more samples close to decision boundaries leads to more probability mass on large values of β , and the case of x ∈ S3 results in largest mass on large values of β because the predicted probabilities mostly concentrate on the two classes in a pair . As shown in Tab . 1 , more mass on large values of β leads to larger probabilities of flipping the sign . These results demonstrate that samples with most diverse gradients ( which gradients have largely negative similarities with other samples ) are close to decision boundaries because they tend to have large β and 〈xn , xm〉 tend to be positive . In the case of x ∈ S1 ∪ S2 the probability of flipping the sign is zero because β concentrates around zero . According to Lemma 1 〈gn , gm〉 are very close to zero in this case because the predictions are close to true labels , hence , such samples are not considered as with most diverse gradients . Theorem 2 says 〈gn , gm〉 has the same sign as 〈xn , xm〉 when the two samples from a same class . We can see the results of positive pairs in Tab . 1 matches Theorem 2 . In the case of S0 ∪ S1 the two probabilities do not add up to exactly 1 because the implementation of cross-entropy loss in tensorflow smooths the function by a small value for preventing numerical issues which slightly changes the gradients . As 〈xn , xm〉 is mostly positive for positive pairs , 〈gn , gm〉 hence is also mostly positive , which explains why samples with most diverse gradients are not sufficient to preserve information within classes in experiments of Fig . 1 . On the other hand , if 〈xn , xm〉 is negative then 〈gn , gm〉 will be negative , which indicates representations within a class should not be too diverse . Extending this theoretical analysis based on a linear model , we also provide empirical study of non-linear models ( Multi-layer Perceptrons ( MLPs ) ) . As demonstrated in Tab . 1 , the probability of flipping the sign in MLPs are very similar with the linear model since it only depends on the predictions and all models have learned reasonable decision boundaries . The probability of getting negative 〈gn , gm〉 is also similar with the linear model except in the case of S1 ∪ S2 for negative pairs , in which the MLP with ReLU gets much less negative 〈gn , gm〉 . As MLP with tanh activations is still consistent with the linear model in this case , we consider the difference is caused by the representations always being positive due to ReLU activations . These results demonstrate that non-linear models exhibit similar behaviors with linear models that mostly align with the theorems . Since only negative 〈gn , gm〉 may cause conflicts , reducing the diversity of gradients hence relies on reducing negative 〈gn , gm〉 . We consider to reduce negative 〈gn , gm〉 by two ways : 1 ) .minimize the representation inner product of negative pairs , which pushes the inner product to be negative or zero ( for positive representations ) ; 2 ) .optimize the predictions to decrease the probability of flipping the sign . In this sense , decreasing the representation similarity of negative pairs might help with both ways . In addition , according to Fig . 2b x ∼ S3 gets larger prediction similarity than x ∼ S0 due to the predictions put most probability mass on both classes of a pair , which indicates decreasing the similarity of predictions may decrease the probability of flipping the sign . Hence , we include logits in the representations . We verify this idea by training two binary classifiers for two groups of MNIST classes ( { 0 , 1 } and { 7 , 9 } ) . The classifiers have two hidden layers each with 100 hidden units and ReLU activations . We randomly chose 100 test samples from each group to compute the pairwise cosine similarities . Representations are obtained by concatenating the output of all layers ( including logits ) of the neural network , gradients are computed by all parameters of the model . We display the similarities in Figs . 3a and 3b . The correlation coefficients between the gradient and representation similarities of negative pairs are -0.86 and -0.85 , which of positive pairs are 0.71 and 0.79 . In all cases , the similarities of representations show strong correlations with the similarities of gradients . The classifier for class 0 and 1 gets smaller representation similarities and much less negative gradient similarities for negative pairs ( blue dots ) and it also gains a higher accuracy than the other classifier ( 99.95 % vs. 96.25 % ) , which illustrates the potential of reducing the gradient diversity by decreasing the representation similarity of negative pairs .

Summary: This paper presents a novel way of making full use of compact episodic memory to alleviate catastrophic forgetting in continual learning. This is done by adding the proposed discriminative representation loss to regularize the gradients produced by new samples. Authors gave insightful analysis on the influence of gradient diversity to the performance of continual learning, and proposed a regularization that connects metric learning and continual learning. However, there are still some issues to be addressed as below.