Title: BETTER KNOWLEDGE RETENTION THROUGH METRIC LEARNING

Abstract: In a continual learning setting, new categories may be introduced over time, and an ideal learning system should perform well on both the original categories and the new categories. While deep neural nets have achieved resounding success in the classical setting, they are known to forget about knowledge acquired in prior episodes of learning if the examples encountered in the current episode of learning are drastically different from those encountered in prior episodes. This makes deep neural nets ill-suited to continual learning. In this paper, we propose a new model that can both leverage the expressive power of deep neural nets and is resilient to forgetting when new categories are introduced. We demonstrate an improvement in terms of accuracy on original classes compared to a vanilla deep neural net.

1. INTRODUCTION:

Despite the multitude of successes of deep neural networks in recent years, one of the major unsolved issues is their difficulty in adapting to new tasks while retaining knowledge acquired from old ones. This problem is what continual learning aims to tackle. To successfully develop a continual learning method, one major limitation that must be overcome is the problem of catastrophic forgetting (McCloskey & Cohen, 1989) , which describes the phenomenon that a machine learning model finetuned for a new task performs poorly on the old task it was originally trained on.

Various continual learning methods make different assumptions about whether the task identities and boundaries are known at training time and test time (van de Ven & Tolias, 2019; Hsu et al., 2018; Zeno et al., 2018) . In the most restrictive setting, task identities are known at both training and test time and the method is only asked to classify among the possible classes within the given task, a setting known as task learning. In domain learning, the assumption that the task identity is known at test time is removed (with all others in place) and each task is assumed to have the same number of classes with similar semantics; in this case, the method is asked to classify among the possible classes within any given task. Class learning differs from domain learning in that each task may contain a different number of classes that may have non-overlapping semantics with the classes in other tasks, and the method is asked to classify among the possible classes across all tasks. Discrete task-agnostic learning further generalizes this by removing the assumption of known task identity at training time and replacing it with an assumption of known boundaries between different tasks at training time. In the most general setting, task boundaries between different tasks are also unknown, even at training time. This setting is known as continuous task-agnostic learning, which is the setting we focus on in this paper.

Note that the same method can yield very different performance under different settings; in particular, some methods may work very well on more restrictive settings, but undergo a significant performance degradation under less restrictive settings. This is because different underlying strategies employed by various methods may be especially dependent on certain assumptions. As a result, care must be taken when interpreting results across different papers to ensure that evaluation is conducted under the same setting. Refer to (van de Ven & Tolias, 2019; Hsu et al., 2018) for an extensive study and discussion on the performance of various methods under different settings.

Existing continual learning methods can be divided into two broad categories: those that bias the parameters towards parameter values learned on old tasks, and those that expands the model size to accommodate new tasks. In this paper, we propose a new approach that is orthogonal to these categories. Our approach neither penalizes parameter changes nor expands model size. Our key intuition is that neural nets forget because parameters at all layers need to change to adapt to new tasks. Therefore, one way to improve retention is to keep parameter changes localized. Because upper-layer parameters depend on lower-layer parameters, upper-layer parameters must change when lower-layer parameters change; hence, lower-layer parameters must be kept relatively static to prevent parameter changes throughout the network. One way to achieve this would be to explicitly regularize parameter changes in the lower layers; this is less than ideal, however, because it would reduce the network's expressive power and therefore its capability to learn on new tasks. How do we achieve this without compromising on expressive power?

To arrive at a solution to this problem, we need to consider the underlying cause of parameter changes at all layers when fine-tuning on new tasks. In a neural net, each layer on its own has quite limited expressive power and is simply a linear model. As a result, if the features computed by the penultimate layer on training examples for the new task are not linearly separable, then the parameters in the layers up to the penultimate layer must change to successfully learn the new task. To avoid this, we can make the last layer more expressive and replace it with a nonlinear classifier. To this end, we propose replacing the softmax activation layer with a k-nearest neighbour classifier. We will show that this simple modification is surprisingly effective at reducing catastrophic forgetting.

2. RELATED WORK:

Our work offers a novel solution to the continual learning problem by leveraging ideas from metric learning.

Continual learning is the idea that machine learning models should be able to emulate humans' ability to learn new information while retaining previously learned skills or knowledge. This capability is generally relevant -for example, in robotics it is desirable for robots to adapt to a given environment while retaining a general functionality across environments; another example from computer vision is the desired ability to update a classifier online with information on new classes while retaining the ability to detect old classes. While it is always possible to accomplish this goal of continual learning through retraining a model on the combined old and new datasets (e.g., as in (Caruana, 1997) ), this approach is computationally expensive or infeasible due to the inability to store old data. One commonly used shortcut is to instantiate a model trained on the old dataset and then finetune that model on the new data (e.g., as in (Hinton & Salakhutdinov, 2006; Girshick et al., 2014) ); however, this method has the side effect of (significantly) decreased performance on the old data.

Recent interest in continual learning has resulted in many proposed methods for continual learning with deep neural networks. They can be grouped into two broad categories: those that dynamically expands the model architecture, and those that constrains the parameter values to be close to those obtained after training on previous tasks.

An example of a method in the former category is (Rusu et al., 2016) , which creates a new network for each new task, with connections between the networks (from old task networks to new ones), thus ensuring the network weights for the old network are preserved while enabling adaptation to the new task. Another method (Masse et al., 2018) randomly zeros a subset of hidden units for each task, which can be viewed as a way to implicitly expand the model architecture, since the hidden units that are zeroed are effectively removed from the architecture.

In the latter category are two subtypes of methods: those based on regularization and those based on data replay. Regularization-based methods are all based on the idea of protecting important parameters and introduce different regularizers to penalize changing the values of these parameters from what they were after training on previous tasks. Different methods differ in the way they determine importance of each parameter. Examples of methods of this flavour include elastic weight consolidation (EWC) (Kirkpatrick et al., 2017) , online EWC (Schwarz et al., 2018) and synaptic intelligence (SI) (Zenke et al., 2017) .

Whereas regularization-based methods can be viewed as constraining the parameter values explicitly, data replay-based methods do so implicitly. Generally speaking, data replay works by generating simulated training data from the model trained on previous tasks and using it to augment the training data used to train on the new task. For example, Learning without Forgetting (LwF) (Li & Hoiem, 2018) performs distillation (Hinton et al., 2015) on the training data for the new task, or in other words, takes simulated labels to be the predictions of the model trained on previous tasks on the input training data for the new task. Deep Generative Replay (DGR) (Shin et al., 2017) uses a deep generative model to generate simulated input data and labels them with the model trained on previous tasks. Other related methods (Rebuffi et al., 2017; Wu et al., 2018; Venkatesan et al., 2017; van der Ven & Tolias, 2018) use training data from previous tasks, either directly or to train a deep generative model, the samples from which are used to augment the current training set. For a more comprehensive review of this area, we refer readers to (Parisi et al., 2019) .

Note that all these methods at least require task boundaries to be known. For example, methods that dynamic expand the model architecture need to know when to expand the architecture, regularizationbased methods need to know the point in time at which the parameter values are used to constrain future parameter updates, and data replay-based methods need to know when to generate simulated training data. Some methods, like (Masse et al., 2018) , require stronger assumptions, namely that task identities be known as well. In contrast, we consider the setting where neither task identities nor task boundaries are known.

Metric learning involves learning a (pseudo-) distance metric. This can be combined with k-nearest neighbours (k-NN) classifier to yield classification predictions. Standard k-NN works by finding the closest k training data points to a test data point and classifying the test data point as the most common label among the training data. The distance metric used for determining the "closest" k points is commonly the Euclidean distance. As proposed by Weinberger et al. (2006) , it is possible and desirable to learn a Mahalanobis distance metric that improves the accuracy of k-NN. The loss function attempts to enforce clustering of labels with large margins between clusters.

More recent work like (Schroff et al., 2015) has used similar ideas in the context of deep neural networks where the distance between two data points is defined as the Euclidean distance between their embeddings computed by the network. Most methods work by pulling examples whose labels are the same as that of the current training example closer to the training example and pushing examples whose labels are different away. The challenge becomes designing a loss function that is conducive to efficient training of the network. Various loss functions like those used in (Schroff et al., 2015; Tamuz et al., 2011; Van Der Maaten & Weinberger, 2012) have been proposed, to varying degrees of success on varying datasets.

3. METHOD:



3.1. MODEL:

The key observation we make is that in order for a neural net to learn on non-linearly separable data, the parameters in the layers below the last layer must change. So, if the training data for the new task is not linearly separable from the training data for the previous tasks, the parameters below the last layer will be changed when training on the new task. Because the parameters in the layer below the last layer are shared across all classes, the classes seen during previous tasks must depend on the parameters below the last layer. And so changes in those parameters would result in a performance drop on these classes, leading to catastrophic forgetting.

To avoid catastrophic forgetting, we must make sure the parameters below the last layer will not change significantly even as the model is trained on new tasks. To this end, we must make the last layer more expressive than a linear classifier. Therefore, we replace it with a nonlinear classifier, namely a k-nearest neighbour classifier using a learned Mahalanobis distance metric, which is equivalent to replacing the softmax activation layer in a neural net with a k-nearest neighbour classifier using a Euclidean distance metric.

More formally, we can characterize the test-time prediction of a vanilla neural net classifier and the proposed classifier as follows:

EQUATION

where Eq. 1 is a vanilla feedforward neural network classifier and Eq. 2 is the proposed classifier.

Here In practice, we choose one anchor for each class, which means that we only need to store one example for each class.

x i is the image input, f (x i ) ∈ R d denotes

The loss function we use is a modified form of the triplet loss (Schroff et al., 2015) :

EQUATION

)

where Φ denotes the model, i.e.: a function that maps the input to embedding, x a i denotes an anchor, x p i denotes a positive example and x n i denotes a negative example, and M is a hyperparameter representing the desired minimum margin of separation between the positive and negative examples.

Ideally, we would like the training examples of the same class to cluster together. In practice, we found that training using L triplet results in overlap between clusters for different classes, as shown in Figure 2 . To discourage this, we add another term to the loss function to encourage tight clusters:

EQUATION

)

where α is a hyperparameter for balancing different terms in the loss. 

3.2.2. TRAINING PROCEDURE:

Algorithm 1 Training Procedure Sample set X a of k a anchors for l = 1, . . . do Sample subset X e of k e training examples for x ai ∈ X a do X p ← set of positive examples of x ai in X e X n ← set of negative examples of x ai in X e D ai,pj ← || Φ(x ai ) -Φ(x pj )|| 2 2 ∀x ai ∈ X a , x pj ∈ X p D ai,nk ← || Φ(x ai ) -Φ(x nk )|| 2 2 ∀x ai ∈ X a , x nk ∈ X n end for for x ai ∈ X a do T xai ← {(x ai , x pj , x nk ) : x pj ∈ X p , x nk ∈ X n , D ai,pj < D ai,nk } Txai := {(t a , t p , t n ) ∈ T xai : t n is among the d closest negative examples to t a in T xai }

Train model Φ on Txai for m steps end for end for As shown in Figure 1 , to train the model, we take a random training example from each class and combine them into a set of anchors. In each iteration, we select a batch of training examples and compute the distances between each anchor and the training examples in the batch. Then we construct the minibatch of triplets used to train the model. We only include triplets where the positive example is closer to the anchor than the negative example and the said negative example is within the d closest to the anchor among negative examples that are farther from the anchor than the positive example. The variable d is a hyperparameter and is known as the dynamic margin, because the distance to the farthest negative example to the anchor that is included varies depending on the scale of the embedding. The reason we do not include triplets where the positive example is farther from the anchor than the negative example is because the model would tend to map all examples to zeros otherwise, which is in line with the observations of (Schroff et al., 2015) .

While training, we need to retrieve the closest examples to the anchor. To this end, we use an efficient k-nearest neighbour search algorithm that is scalable to large datasets and high dimensions known as Prioritized DCI (Li & Malik, 2017) .

3.3. EMBEDDING NORMALIZATION:

We normalize the output embedding of our model, so that the embedding vector for each example has an 2 norm of 1. This can be interpreted as making sure k-nearest neighbors returns the k closest vectors by angle, which in this case is the same as return the vectors that are closest by Euclidean distance. So, our classifier is essentially basing its decision on the maximum cosine similarity between the image embeddings.

4. EXPERIMENTS:

We consider the setting where training data from a subset of categories are presented to the learner in stages. In the first stage, a subset of the categories are presented (which will henceforth be referred to as Set A), and the learner can iterate over training data from these categories as many times as it likes.

In the second stage, the remaining categories are presented (which will henceforth be known as Set B); while the learner can look at training data from these categories as often as it desires, it no longer has access to the training data from the categories presented in the first stage.

4.1. TRAINING AND TESTING PROCEDURE:

We consider the following training procedure:

1. Train on Set A.

2. Starting from the parameters obtained from the first step, train on Set B.

We evaluate performance on Set A after each step in order to assess the extent each method retains its performance on Set A after training on Set B.

4.2. METHODS:

For the baseline, we consider a vanilla neural net classifier with softmax activation in the final layer trained with cross-entropy loss. To train on Set B, because the weights of the last layer are specific to categories in Set A, we do not use them and instead initialize the last layer randomly. After training on Set B, for the purposes of evaluation on Set A, we discard the weights of the last layer and train a new set of the weights for the last layer (while keeping the weights of all other layers fixed) on Set A.

For the proposed method, which replaces the softmax activation with a k-nearest neighbour classifier, we use the same architecture as the baseline when training on the smaller set (which is our case is always Set B). Thus, the number of parameters is identical to that used by the baseline. To train on Set A, to reduce the amount of training time on large datasets, we first pretrain using cross-entropy loss and then switch to the proposed method later on. To train on Set B, unlike for the baseline, we do not need to do anything special for the last layer of weights, because the output dimension of the last layer does not need to change when the number of classes changes.

On MNIST, which is the dataset most existing continual learning methods are tuned for, we also compare to seven continual learning methods that operate under more restrictive assumptions (where knowledge of task boundaries is required), including Elastic Weight Consolidation (EWC), online EWC, Synaptic Intelligence (SI), Deep Generative Replay (DGR), DGR with Distillation, Replaythrough-Feedback (RtF) and Learning without Forgetting (LwF).

4.3. DATASETS:

We consider four different datasets/splits:

MNIST We first consider MNIST, which is the simplest dataset. We divide it into 2 tasks: Set A consists of the odd digits and Set B includes the even digits.

CIFAR-10:

We then consider CIFAR-10 and take Set A to be the six animal categories (bird, cat, deer, dog, frog and horse), and set B to be the four remaining categories corresponding to man-made objects (airplane, automobile, ship, truck). This split is designed to be more challenging than just a random split, since we don't expect many intermediate-level features to be shared across Sets A and B, making it easier to forget about Set A when training on Set B.

CIFAR-10 (5 Tasks) We also consider a variant of CIFAR-10 with 5 tasks, where each task corresponds to a pair of consecutive categories. This is more challenging than the two-task setting above, because many more examples would be seen after training on the earlier tasks, and so preventing forgetting on the earlier tasks is harder.

ImageNet (Random Split) Next we consider ImageNet, which is larger and more complex than CIFAR-10. We randomly select 100 classes to serve as Set A, and 100 classes to serve as Set B.

ImageNet (Dog Split) Finally, we consider ImageNet and use a more challenging split. For Set A, we take it to be all 880 categories that are not dogs; for Set B, we take it to be all 120 categories that correspond to particular breeds of dogs. Similar to the split we used for CIFAR-10, this split is designed to be more challenging by minimizing semantic overlap between Sets A and B. Table 4 : Performance comparison between our method and the vanilla baseline on ImageNet. Performance evaluated as top-1 percent correct on test data for the original task. Baseline represents the network trained with cross-entropy loss. "Original" denotes the network after being trained on the original task (classes in Set A) and "After Seeing Set B" denotes the network after it is fine-tuned on the new task (classes in Set B). See corresponding sections for more details on the dataset and network used for each row.

4.4. NETWORK ARCHITECTURE:

We used standard network architectures that have proven to work well on the datasets we consider.

For MNIST, following (van de Ven & Tolias, 2019), we use a network with two hidden layers of 400 units each. We use this same architecture for our method and all baselines. For CIFAR-10, we use the architecture from the Tensorflow CIFAR-10 tutorial1 , which consists of two convolutional layers followed by three fully-connected layers. For ImageNet, we use the AlexNet architecture (Krizhevsky et al., 2012) , which consists of five convolutional layers followed by three fully-connected layers. All networks are trained using Adam (Kingma & Ba, 2014).

4.5. KEY FINDINGS:

The test accuracy of each method is given in Tables 1 and 4 in the sub-column labelled "Original". After fine-tuning, we compare the accuracy of the network relative to the test-accuracy from set Athese numbers are given in Table 4 in the sub-column labeled "After Seeing Set B". For the MNIST comparison, we also include the test-accuracy from set B. proposed method can both adapt to the new task and retain knowledge about the previous task, which is especially remarkable considering its simplicity and ability to operate without knowledge of task boundaries.

Table 4 shows the results of the proposed method and the baseline on more complex datasets, such as CIFAR-10 and ImageNet with random split and dog split. As shown, even though our method achieves a lower test accuracy compared to the baseline after only training on Set A, it is better at remembering the knowledge it learned once it is trained on new data. This is true in each of the datasets we tested it on, and the gap is substantial (it achieve nearly double the test accuracy of the baseline) for ImageNet (random split) and CIFAR-10. The gap is smaller on ImageNet (dog split), which can be attributed to the diminished similarity between set A and set B. Since set B only consists of dog images, fine-tuning on this data may wipe out much of the knowledge the model has learned about images in general. In contrast, since ImageNet (random split) does contain a random sampling of images in each set, it is likely that fine-tuning on set B preserves more general concepts about images that are still useful in classifying the images in set A.

4.6. ANALYSIS:

We further examine the performance of the proposed method on the ImageNet dataset with random split.

4.6.1. SENSITIVITY ANALYSIS:

We varied the hyperparameter settings and took a look at the impact of changes in hyperparameters. We tried the following settings: reduce number of negative samples, increase number of positive sample and different embedding dimensions. As shown in Table 5 , increasing the number of positive examples showed a slight improvement, whereas decreasing the number of negative examples resulted a slight performance drop. A similar trend can be observed with the embedding dimension, where proposed method does not benefit from the increase in embedding dimension, neither does it deteriorate significantly when the embedding dimension is decreased. Overall, we found the method to be reasonably robust to hyperparameter changes.

4.6.2. ABLATION STUDY:

We tested the proposed method by disabling some key components to determine their contributions.

The following components are analyzed: embedding output normalization and dynamic margin. As shown in Table 6 , by disabling embedding output normalization, the performance on both tasks drops significantly. When the dynamic margin is removed and replaced with a static margin, the method fails to capture new information from set B. In summary, we found that embedding normalization and dynamic margin contribute enormously to the performance of the proposed method. Disabling dynamic margin also greatly decreases the performance on Set B. Overall, we found that embedding normalization and dynamic margin are crucial for the method to be able to adapt well to new tasks.

5. CONCLUSION AND FUTURE WORK:

As discussed in Section 4, we achieve consistent results across four different datasets/splits. These results suggest that the method is successful at addressing the catastrophic forgetting problem, and hint at a possible general strategy for reducing catastrophic forgetting, namely the idea of replacing the last layer of a neural net with a non-linear classifier.

In the future, we would like to explore combining the proposed approach with other orthogonal approaches for tackling catastrophic forgetting. We also plan to explore applications that can benefit from continual learning approaches, such as reinforcement learning.

:

https://www.tensorflow.org/tutorials/images/deep_cnn

Figures and Tables:

Figure 1: Graphical depiction of the loss function in Eq. 4.

Figure 2: Comparison of the two loss function variants in Eq. 3 and 4. Here, a 5-layer CNN is trained on the CIFAR-10 dataset to output a 2-dimensional embedding. The labeled loss function is used to train the network for each figure. The displayed figures generated by sampling training examples and plotting their embeddings. Plots of test examples appear similar, albeit slightly more spread out for both loss functions.

Performance of the proposed method compared to eight other continual learning methods on MNIST. SGD with dropout, EWC, online EWC and SI suffer from catastrophic forgetting on Set A after seeing Set B, but were able to adapt well to the new task (Set B). DGR, DGR + distillation and RtF forget more on Set A compared to our method. LwF retains knowledge learned on Set A; however, it fails to adapt to Set B. We also note that all baseline methods assume knowledge of task boundaries, whereas the proposed method does not.

Table content: <table><tr><td>Method</td><td>Set A: Original</td><td>Set A: After Seeing Set B</td><td>Set B</td></tr><tr><td/><td/><td>(Difference from Original)</td><td/></tr><tr><td>Our Method</td><td>97.5%</td><td>95.1% (-2.4%)</td><td>95.6%</td></tr><tr><td>Very Forgetful &amp; Adaptive:</td><td/><td/><td/></tr><tr><td>SGD with dropout (ratio: 0.5)</td><td>98.62%</td><td>0.0% (-98.62%)</td><td>98.62%</td></tr><tr><td>Elastic weight consolidation (EWC)</td><td>98.62%</td><td>0.0% (-98.62%)</td><td>98.46%</td></tr><tr><td>Online EWC</td><td>98.62%</td><td>0.0% (-98.62%)</td><td>98.46%</td></tr><tr><td>Synaptic intelligence (SI)</td><td>98.62%</td><td>0.0% (-98.62%)</td><td>98.13%</td></tr><tr><td>Forgetful &amp; Adaptive:</td><td/><td/><td/></tr><tr><td>Deep Generative Replay (DGR)</td><td>98.78%</td><td>89.32% (-9.46%)</td><td>98.40%</td></tr><tr><td>DGR with distillation</td><td>98.78%</td><td>91.70% (-7.08%)</td><td>98.05%</td></tr><tr><td>Replay-trough-Feedback (RtF)</td><td>98.96%</td><td>93.22% (-5.74%)</td><td>98.62%</td></tr><tr><td>Not Forgetful &amp; Not Adaptive:</td><td/><td/><td/></tr><tr><td>Learning without Forgetting (LwF)</td><td>98.62%</td><td>98.07% (-0.55%)</td><td>6.84%</td></tr></table>

Performance of the proposed method compared to eight other continual learning methods on CIFAR-10. SGD with dropout, EWC, online EWC and SI suffer from catastrophic forgetting on Set A after seeing Set B, but were able to adapt well to the new task (Set B). DGR, DGR + distillation and RtF also forget on Set A, this could relate to the fact that CIFAR-10 data is more complex and it is difficult for these methods to train a generative model for replaying. LwF retains some knowledge learned on Set A; however, it fails to adapt to Set B compared to our method.

Table content: <table/>

Performance of the proposed method compared to eight other continual learning methods on 5-task CIFAR. SGD with dropout, EWC, online EWC and SI suffer from catastrophic forgetting on previous tasks after seeing newer tasks, but were able to adapt well to the last task. DGR, DGR + distillation and RtF forget more on the previous tasks compared to our method. LwF retains knowledge learned on first set; however, it fails to adapt to newer tasks.

Table content: <table><tr><td/><td/><td>Our Method</td><td/><td>Baseline</td></tr><tr><td/><td colspan="4">Original After Seeing Set B Original After Seeing Set B</td></tr><tr><td>ImageNet (Random split)</td><td>69.5%</td><td>59.2%</td><td>72.7%</td><td>30.2%</td></tr><tr><td>ImageNet (Dog split)</td><td>67.0%</td><td>52.9%</td><td>70.0%</td><td>30.4%</td></tr></table>

shows the results of the proposed method and existing continual learning methods on MNIST. As shown, regularization-based methods including EWC, online EWC and SI are very forgetful of the

Table content: <table><tr><td/><td colspan="2">Set A (Original) Set A (After Seeing set B)</td><td>Set B</td></tr><tr><td>Original</td><td>69.5%</td><td>59.2%</td><td>69.2%</td></tr><tr><td>Number of Positive Examples (5 → 30)</td><td>+0.4%</td><td>+0.1%</td><td>+0.3%</td></tr><tr><td>Number of Negative Examples (40 → 10)</td><td>-0.9%</td><td>-1.8%</td><td>-0.6%</td></tr><tr><td>Embedding Dimension (100 → 200)</td><td>-0.1%</td><td>+0.1%</td><td>-0.5%</td></tr><tr><td>Embedding Dimension (100 → 50)</td><td>-1.3%</td><td>-0.3%</td><td>-0.1%</td></tr></table>

Test accuracy under the original hyperparameter setting and the change in test accuracy under different hyperparameter settings relative to the original setting. Increasing the number of positive examples (from 5 to 30) slightly improves performance. Decreasing the number of negative examples (from 40 to 10) slightly reduces test accuracy. Increasing the embedding dimension (from 100 to 200) results in similar performance as the original setting, whereas decreasing the embedding dimension (from 100 to 50) moderately lowers the performance. Overall, the method is fairly robust to hyperparameter changes.

Table content: <table/>

Test accuracy with all components, and changes in test accuracy relative to the original proposed method after various components are removed. Not normalizing the embedding output dramatically lowers test accuracy, and especially limits the ability to adapt to the new task (Set B).

Table content: <table/>

Peer review (iclr2020):
This paper considers the use of a metric learning approach in a continual/lifelong classification settings. Experiments show in the case of two tasks forgetting can be minimized by using the approach. Methods The proposed method appears to be a standard triplet loss. The authors add a second term to the triplet loss that is essentially making the loss a combination of the triplet and siamese loss. It’s not really explained anywhere why they do this and whether its essential to the performance. Is there anything specific to continual learning done or is the paper essentially pointing out this existing method (metric learning + nearest neighbor) is surprisingly effective for forgetting. If this is the case the authors should present it in this way I think. Although triplet loss can often yield reasonably performance on classification problems it tends to not perform as well as cross entropy loss, this is observed in other works as well as this one. A major question of mine: it is not clear from the method nor experiments what samples are stored after task A for the kNN classifier. Is it all of the data samples from the previous task? Experiments The experimental results consider a custom continual learning setup where there is two sets of categories. Overall the experiments seem lacking at the moment in rigorous comparisons. MNIST experimental comparisons are currently suspect. It is very surprising that LwF does so poorly, do the authors have some explanation for this. LwF is typically a reasonable baseline for these 2 task settings (e.g. https://arxiv.org/pdf/1704.01920.pdf). Similarly the well known EWC is shown to simply not work at all for the very task it was designed for on the MNIST dataset. LwF and EWC simply not working to any degree seem to me like rather dramatic claims to make without any explanation. Cryptically the fine-tuning baseline described in 4.2 is not shown here for MNIST? This seems a major oversight CIFAR10/Imagenet Experiments It is not clear if the baseline finetuning is done on only the top weights or the entire network. Both of these baselines should be considered. Another good baseline to consider is finetuning with cosine distance and only the top weights as in https://arxiv.org/pdf/1804.09458.pdf and other recent works should also be considered Why do the authors not include any of the baselines from MNIST experiments here, for example LwF. Ablations study the need for normalization and dynamic margin, it seems these are helpful for accuracy and forward transfer (and not as critical for minimizing forgetting). The author state their method is agnostic to the task boundaries, its a bit unclear what this means in this context. The procedure is not online and the labels of the samples are being used? If the authors are referring to the need to add additional outputs to the “vanilla” model this seems like it can be trivially addressed by simply saying outputs are added the first time a new class is seen thereby making it agnostic to the boundary in the same sense as this method. Clarity Can be problematic at times. Although all the elements of the approach are outlined the motivations are overly wordy and repetitive making them actually hard to follow. -(minor) first/2nd paragraph of 3.1 seems a bit redundant making it hard to follow Overall I think the idea to consider metric learning and local adaptation for continual learning is interesting, however the current work is currently lacking in both experimental evidence (appropriate comparisons) and clear motivation/difference to existing work for its particular instantiation of this idea. ++++Post Rebuttal++++ Thank you for your detailed responses. The clarification about “task-agnostic” for the experiments does make them look more relevant than I had previously assessed. I do want to note that the language used for this is inconsistent with the ones used in other papers, which typically calls this a “shared-head” setting (https://arxiv.org/pdf/1801.10112.pdf, https://arxiv.org/pdf/1805.09733.pdf, https://arxiv.org/pdf/1903.08671.pdf ). It is also somewhat inconsistent with the authors own definition of “task agnostic learning” given in the introduction of this paper which implies it is something related to task boundaries at training time, in fact this is something related to availability of the task id at test time. I suggest the authors to make this more clear. Furthermore, the authors should highlight all this in the experiment text, e.g. noting EWC does poorly but this is because we use a different protocol than this and this paper etc. Regarding the experiments under this light they do look more reasonable. Indeed it has been observed that EWC works poorly in the shared-head setting https://arxiv.org/pdf/1801.10112.pdf Regarding the new 5 task CIFAR-10 the results are interesting, however I will point the authors to the work above (Rwalk) which also reports results in this setting better than theirs (but not by too much). I do however still have issues regarding the memory usage of the method, specifically which data needs to be stored from previous tasks. It is still not completely clear and I find obfuscated since just one sentence not even fully answering the concern about this was added to the manuscript despite myself and another reviewer asking about it. My understanding based on the (somewhat conflicted responses) of the authors is they store a substantial amount of prior task data, but most of this is only used at test time. For example for imagenet as much as 1000 images/class are stored for testing time. This begs the question why not use this data for training as well if it is allowed to be used by the model at testing time (and therefore preserved from the first task), why is the storage cost of this data not considered and how do the authors justify this still being a lifelong learning setup. As an alternative, why can*t one use a much bigger fully parametric model that uses the same amount of storage as the authors model + stored images. It seems it is not fair to compare these to methods that cant utilize this large storage amount. Finally its not clear if this data is stored as raw images or somehow stored as embeddings. If it is stored as embeddings this would require some discussion on how the authors avoid representation drift when the next task is training. If the authors store raw images, it means at evaluation time the entire raw dataset needs to be re-encoded, therefore the model can’t perform easily anytime inference. Unfortunately the discussion period ended but I would have liked more clarification on this, on the other hand these pieces of information should really have been in the manuscript in the first place. Overall, my impression of the paper is improved. But I do think it could use some further writing revisions to emphasize/clarify key points: a) the method is not new (it says e.g. in abstract “new model” which is misleading) but its application in CL is under-explored b) the experiments show poor performance on existing methods because most of those are not designed nor work well for the shared head “task agnostic” setting, while metric learning handles it gracefully. c) be explicit about what is the memory being stored when moving onto the next task (this should be somewhere visible and explicit) and how this is justified