Title: FEATURE TRANSFORMERS: A UNIFIED REPRESENTA-TION LEARNING FRAMEWORK FOR LIFELONG LEARN-ING

Abstract: Despite the recent advances in representation learning, lifelong learning continues to be one of the most challenging and unconquered problems. Catastrophic forgetting and data privacy constitute two of the important challenges for a successful lifelong learner. Further, existing techniques are designed to handle only specific manifestations of lifelong learning, whereas a practical lifelong learner is expected to switch and adapt seamlessly to different scenarios. In this paper, we present a single, unified mathematical framework for handling the myriad variants of lifelong learning, while alleviating these two challenges. We utilize an external memory to store only the features representing past data and learn richer and newer representations incrementally through transformation neural networks -feature transformers. We define, simulate and demonstrate exemplary performance on a realistic lifelong experimental setting using the MNIST rotations dataset, paving the way for practical lifelong learners. To illustrate the applicability of our method in data sensitive domains like healthcare, we study the pneumothorax classification problem from X-ray images, achieving near gold standard performance. We also benchmark our approach with a number of state-of-the art methods on MNIST rotations and iCIFAR100 datasets demonstrating superior performance.

1. INTRODUCTION:

Deep learning algorithms have achieved tremendous success on various challenging tasks like object detection, language translation, medical image segmentation, etc. Lifelong learning -the ability to adapt, benefit and sustain performance post deployment with more data and feedback, is an important goal for artificial intelligence (AI) and extremely crucial for sustained utility of these algorithms in domains like healthcare (learning from rare cases), self driving (learning new object detectors), etc. While there is no single standard definition of lifelong learning, most of the research in this field can be classified into one of the following sub-categories:

1. Incremental learning -encountering new data, with no change in distribution, for same task 2. Domain adaptation -data from modified target distributions but for the same task 3. New-task learning -data from tasks that were not presented before Majority of the successful techniques study these variants in isolation. However, a realistic lifelong learning scenario would involve a mixture of these manifestations over time. Another major impediment to a successful lifelong learner is data privacy. In domains like healthcare, it is near impossible to have data access beyond scope (both time and geography) which leads to catastophic forgetting (McCloskey & Cohen, 1989 )-the inability of machine learning models to retain past knowledge while learning from new data. In this paper, we provide a unified framework -feature transformers -for practical lifelong learning to handle data privacy and catastrophic forgetting.

In summary, the major contributions of our work are as follows:

• Define a realistic and a challenging lifelong learning scenario and formulate a unique and generic mathematical framework -feature transformer, to work in this setting successfully. • Ensure data privacy by storing only features from previous episodes while successfully combating catastrophic forgetting.

• Principled way of utilizing additional neural-compute to tackle complex incremental tasks.

• Demonstrate state-of-the-art results on common continuous learning benchmarks for new task learning like MNIST rotations, iCIFAR dataset and a challenging healthcare problem. • Demonstrate exemplary performance even under severe constraints on memory and compute, thus operating under the assumptions of practical lifelong learning.

Following review of related work in section 2, we describe the mathematical formulation of feature transformers and their practical realization in section 3. Section 4 contains experiments and results, followed by discussion in section 5.

2. RELATED WORK:

A conventional deep learning classifier can be viewed as an automatic feature extractor followed by a classifier, trained jointly. For lifelong learning, the methods in literature can be broadly classified into: 1) learning with fixed feature representations and 2) incrementally evolving representations.

Fixed Represenations: In this class of approaches, a representation for the data is learnt from the initial task and remains frozen for ensuing tasks, while only the classification layers are modified.

The simplest baseline approaches include fine-tuning (Oquab et al., 2014) and Mensink et al. (2012) where the idea of using fixed representations was extended further by using the nearest class mean classifier. In spite of their simplicity, these algorithms do not perform well in practice due to the limiting constraint of a fixed representation throughout the incremental learning phase.

Incrementally evolving representations::

The last few years have witnessed renewed interest in developing methods that allow changing data representation with the addition of new tasks. Naive methods of retraining with only new data suffer from catastrophic forgetting (McCloskey & Cohen, 1989) . To overcome this problem, most methods have attempted different manifestations of rehearsal (Robins, 1995) -replaying data from the previous tasks. However, access to all the previous data is usually not feasible due to size, compute and privacy concerns, researchers have attempted to reproduce past through proxy information, known as pseudorehearsal. Pseudorehearsal techniques can be further classified depending on how the previous information is stored /used:

• Knowledge distillation based approaches: An example method for lifelong learning using knowledge distillation (Hinton et al., 2015) is learning without forgetting (LwF) (Li & Hoiem, 2017 ) -where distillation loss was added to match output of the updated network to that of the old network on the old task output variables. LwF and its extensions does not scale to a large number of new tasks, suffers from catastrophic forgetting and importantly do not address incremental learning or domain adapatation settings. • Network regularization strategies: Methods belonging to this class aim to identify and selectively modify parts of the neural network which are critical to remember past knowledge or by explicitly penalizing loss of performance on old tasks. Kirkpatrick et al. (2017) ; Liu et al. (2018) use the Fischer information matrix to identify the most crucial weights responsible for prediction of a given task and lower the learning rate for these tasks. Lee et al. (2017) propose a dynamically expanding network to increase the capacity for new tasks if the previous architecture is insufficient to represent the data.

Our framework lies at the intersection of pseudorehearsal methods and progressive neural networks, with scope for judiciously utilizing extra capacity, while resisting catastrophic forgetting.

3. LIFE-LONG LEARNING VIA FEATURE TRANSFORMATIONS:

Before we present the feature transformer method, we introduce the terminologies and notations. We consider training a deep neural network which classifies an input to one of the classes c ∈ [C] {1, 2, • • • , C}. We refer to the operation of classifying an input to a particular class as a task. To this end, the classifier is trained with training dataset (X, Y ), drawn from a joint distribution (X , Y).

We view the deep neural network, defined by the parameters (θ, κ), as the composition of a feature extractor Φ θ : X → F , and a classifier

Ψ κ Ψ κ • Φ θ : X → [C],

(1) where X is the space of input data, and F is a space of low-dimensional feature vectors.

We concisely denote the training of the neural network by TRAIN(θ, κ; D), which minimizes a loss function on training data D = (X, Y ) and produces the network parameters (θ, κ). Let us also define the set of all computed features on input F = Φ θ (X).1 

When the loss function only penalizes misclassification, the network is expected to learn only the class separation boundaries in the feature space. However, as we demonstrate experimentally, good separation of class specific features enables stable learning of representations, which directly has a bearing on the performance of life-long learning. Therefore, in all our training procedures, we also use a feature loss which promotes feature separation model loss

EQUATION

In the lifelong learning context, we denote the time varying aspect of the network, training data and the classes, by using the time symbol t ∈ N {0, 1, 2, • • • } in superscript on these objects. Realistically, at any time t > 0, the classifier encounters any of the following canonical situations:

1. The number of classes remain the same for the classifier as at t -1. The model encounters new training data for a subset of classes, without change in their distribution: ) ] and ∀τ ∈ T

∀t > 0, C (t) = C (t-1) , T (t) 1 ⊆ [C (t

(t) 1 , (X (t) τ , Y (t) τ ) ∼ (X (t) τ , Y (t) τ ) = (X (t-1) τ , Y (t-1) τ ).

2. The number of classes remain the same for the classifier as at t -1. The model encounters new training data from modified input distribution(s), for a subset of classes: ) ] and ∀τ ∈ T

∀t > 0, C (t) = C (t-1) , T (t) 2 ⊆ [C (t

(t) 2 , (X (t) τ , Y (t) τ ) ∼ (X (t) τ , Y (t) τ ) = (X (t-1) τ , Y (t-1) τ

). -1) ] and ∀τ ∈ T

3.. The model encounters new class(es) and corresponding new data:

: ∀t > 0, C (t) > C (t-1) , T (t) 3 = [C (t) ]\[C (t

EQUATION

τ ).

In the most generic scenario, a combination of all the three situations can occur at any time index t, with training data available for the classes in the set

T (t) T (t) 1 ∪ T (t) 2 ∪ T (t)

3 . However, it is important to note that at every index t, the classifier is trained to classify all the C (t) classes.

3.1. FEATURE TRANSFORMATION BY AUGMENTING NETWORK CAPACITY:

At any time t -1, the classifier is optimized to classify all the classes [C (t-1) ] and also the set of features F (t-1) are well separated according to classes. At t, when new training data

D (t) = ∪ τ ∈T (t) (X (t) τ , Y (t)

τ ) is encountered, the features extracted using the previous feature extractor

EQUATION

are not guaranteed to be optimized for classifying the new data and new classes. In order to achieve good performance on new data and classes, we propose to change the feature representation at time t, just before the classification stage. We achieve this by defining a feature transformer

EQUATION

parameterized by ∆θ (t) , which maps any feature extracted by Φ θ (t-1) to a new representation. The new feature extractor is now given by Φ θ

(t) Φ ∆θ (t) • Φ θ (t-1)

, where θ (t) θ (t-1) ∪ ∆θ (t) . Practically, this is realized by augmenting the capacity of the feature extractor at each time t by using one or more fully connected layers2 . It is however possible that Φ ∆θ (t) could be simply an identity transform and feature transformers learnt in previous episodes could be adapted for new data. This helps in controlling the growth of network capacity over time and this aspect of our work is described in section 4.5.

The feature transformer is trained, along with a new classifier layer, using the composite loss function of the form in equation 2, by invoking TRAIN(∆θ (t) , κ (t) ; D (t) ), with D (t) = (∂F (t) , Y (t) )3 . This ensures that the classifier performs well on the new data. However, strikingly, training a feature transformer at t does not involve changing the feature extractor Φ θ (t-1) at all, and this helps us in alleviating catastrophic forgetting by efficiently making use of already computed features F (t-1) through a memory module.

3.2. REMEMBERING HISTORY VIA MEMORY:

The set of all extracted features F (t-1) serves as a good abstraction of the model, for all the tasks and data encountered till t-1. Therefore, if F (t-1) is available to the model when it encounters new tasks and data, then the feature transformer at t can take advantage of this knowledge to retain the classification performance on previous tasks and data as well. To this end, we assume the availability of an un-ending memory module M, equipped with READ(), WRITE() and ERASE() procedures, that can store F (t-1) and can retrieve the same at t. In situations where memory is scarce, only a relevant subset of F (t-1) can be stored and retrieved.

We train the feature transformer at any t > 0 by invoking TRAIN(∆θ (t) , κ (t) ; D (t) ), with the combined set of features

D (t) = (∂F (t) ∪ F (t-1) , ∪ t ∈[1,2,••• ,t] Y (t ) ), ∀t > 0.

(5)

We then obtain the new set of features

EQUATION

and replace F (t-1) in memory by (a subset of) F (t) .

With the assumption of infinite memory and capacity augmentation at every episode, our feature transformers framework is presented in algorithm 1.

3.3. FEATURE TRANSFORMERS IN ACTION:

Before we present experimental results to demonstrate the ability of feature transformers for lifelong learning of new tasks, we first show how feature transformers operate when simply new episodes of data are presented. As described in section 3, we use a composite loss comprising of classification loss and feature loss to train the classifier. To promote feature clustering/separation, we propose to use center-loss as described in Wen et al. (2016) .

Dropping the time index t for the brevity, with the convention that the ground truth class for each x is encoded using one-hot vector

ȳ = [y 1 , y 2 , • • • , y C ] T , and (Ψ κ • Φ θ (x)) c

is the c th component of the classifier output, we use the following loss function for all the classifier training procedures:

EQUATION

)

where D = ∪ τ ∈T (X τ , Y τ ) is the given training data set, and µ c is the centroid of all features corresponding to input data labelled as c.

Figure 1 provides snapshot of feature transformation algorithm when a new episode of data is encountered. We consider X-Ray lung images (from Wang et al. (2017a) ) consisting of two classes: (i) normal and (ii) pneumothorax. At a time index (t -1), the classifier model is trained on 6000

Input Task set T (t) , and training data

∪ τ ∈T (t) (X (t) τ , Y (t) τ ), ∀t ≥ 0 Output (θ (t) , κ (t) ), ∀t t ← 0, ERASE(M) /* Set initial time, erase memory */ D (0) ← ∪ τ ∈T (0) (X (0)

τ , Y (0) τ ) /* Obtain initial tasks and training data */ TRAIN(θ (0) , κ (0) ;

D (0) ) /* Train initial network */ F (0) ← ∪ τ ∈T (0) (Φ θ (0) (X (0) τ )) /* Compute features */ WRITE(M, (F (0) , Y (0) ))

/* Write features to memory */ while T RU E do

t ← t + 1, obtain T (t) , ∪ τ ∈T (t) (X (t) τ , Y (t) τ )

/* Obtain current tasks and training data */ Compute ∂F (t) using equation 3/* Compute old model features on new data */ (F (t-1) , Y (t-1) ) ← READ(M) /* Read previously computed features from memory */ Form D (t) using equation 5/* Form composite training data */ TRAIN(∆θ (t) , κ (t) ;

D (t) ) /* Train feature transformer */ Φ θ (t) ← Φ ∆θ (t) • Φ θ (t-1)

/* Obtain new feature extractor */ Compute F (t) using equation 6/* Compute new features */ ERASE(M), WRITE(M, (

F (t) , ∪ t ∈[1,2,••• ,t] Y (t ) ))

/* Erase & write new features*/ end Algorithm 1: The life-long learning framework

 (t)

randomly permute the rotation angles (to test domain adaptation). For every rotation angle, we randomly permute the class labels (0-10) and divide them into two subsets of 5 classes each (learning new tasks). We divide the number of training samples into two different subsets (incremental learning). Table 1 details the episodes of a lifelong learning experiment from the described procedure. We used a basic CNN architecture composed of 3 conv-layers of 32 filters and kernel-size -(3x3) followed by 2 dense layers "fc1" and "fc2" of feature length 256 and a softmax. Our feature transformer networks at every episode aims to transform "fc1" features using 2 additional dense layers of feature length 256. Feature transformers from previous episode serve as initialization for current episode and these models were optimized for the cumulative loss (equation 7), with λ = 0.2. All the feature transformers were trained for only 3 epochs, with batch size of 32.

We compare our results to the two obvious life-long learners -naive and cumulative training, which serve as the lower and upper bounds of performance respectively in the absence of other competitive algorithms for this setting. While the naive learner finetunes the entire network on the latest episode data, cumulative learner accumulates data from all the episodes seen so far and retrains the model. Fig. 2a highlights the remarkable performance of the proposed approach. We show the performance evolution over the first 25 episodes in Fig. 2a while, performance comparisons over 80 episodes across multiple experiment runs are averaged and shown in Table . 2b. We demonstrate that just by storing features in memory and learning feature transformers at every episode, we achieve almost similar performance to the gold-standard result. Further, it also underlines the applicability of our approach as single framework to combat different requirements of lifelong learning.

Finally, we also compare our results (figure 3 ) in the conventional MT setting described in Lopez-Paz et al. (2017) , whose open source implementation we used for our experiments. For details on the terminology and compared methods, the readers are encouraged to refer to Lopez-Paz et al. (2017) . Fig. 3a demonstrates clear superiority of our approach over multiple methods compared. Additionally, backward transfer (BWT) -quantitative metric that models the deterioration of performance on older tasks while learning new tasks, is also negligible for our method, which shows resistance to catastrophic forgetting. Fig. 3b highlights this further as shown by accuracy on first task after learning subsequent tasks.

4.2. PNEUMOTHORAX IDENTIFICATION FROM X-RAY IMAGES:

We simulate a practical manifestation of lifelong learning where a model trained to detect pneumothorax is deployed in a hospital with data arriving incrementally. We utilize a subset of ChestXRay (Wang et al., 2017b) dataset, which consists of chest X-rays labeled with corresponding diagnoses. We simulated incremental learning by providing the 8k training images in incremental batches of 2k and measured the performance on held-out validation set of 2k images. Fig. 4a establishes the baselines for the experiment. As in previous experiment, naive and cumulative training define the performance bounds. To clearly highlight the value of our feature transfom, we also add another strong baseline -naive Learner with center loss, which learns on the recent batch but with an augmented loss function (equation 7). In spite of a gain of 5% due to center loss, there is still a loss of 4% performance in the incremental learning set-up in the fourth batch of 2k.

Experimental Details and Results: We used a pre-trained VGG-network (Simonyan & Zisserman, 2014) as the base network and explored the use of features from different layers of the VGG-network . 4b also presents the performance of feature transformer depending upon the base features used. It can be noted that performance is lowest for the layer that is closer to the classification layer -fc 2. This is intuitively satisfying because, the further layers in a deep neural network will be more finely tuned towards the specific task and deprives the feature transform of any general features. We present the 100 classes from CIFAR100 dataset in a sequence of 20 tasks comprising of 5 classes each. Similar to Lomonaco & Maltoni (2017) , we use the definitions for MT and single-incremental task (SIT). In an MT setting, evaluation is performed only on the new tasks exposed to the learner in the current episode. We start with a VGG type architecture, pretrained on iCIFAR10 dataset as suggested by Lopez-Paz et al. (2017) and Lomonaco & Maltoni (2017) . Base features are extracted from flatten layer (before the fully connected layers) and our feature transformers included two dense layers with feature length = 256. Similar to our earlier experiments, the feature transformer module from the previous episode initializes the current episode transformer and is optimized for the cumulative loss, with NoEpochs = 30 and batch size = 32. Unsurprisingly, naive learner performs very poorly, while feature transformer displays exemplary performance numbers of 40% validation acccuracy after encountering all 20 episodes, while compared to cumulative learner at 50%. Our method significantly outperforms AR1 (Lomonaco & Maltoni, 2017 ), while iCaRL (Rebuffi et al., 2017) achieves best-in-class performance close to gold-standard cumulative learner. This is not surprising because iCaRL is an explicit rehearsal technique, where exemplar images from previous episodes are stored and replayed while learning new tasks, while we only store low dimensional features.

4.4. EFFECT OF LIMITED MEMORY:

One of the drawbacks of the proposed approach is the assumption of infinite memory and need to store features computed on all samples observed so far. To understand the extent of this limitation, we performed ablation experiments limiting the amount of history replayed as well as computing the storage requirements involved.

Storing all features is not necessary We studied the effect of size of memory by limiting the number of samples stored to a smaller percentage. We observed that performance dropped from 97% to 94% when we reduced the memory size to only 25% the original size on MNIST rotations ( In the description of the feature transformer framework in section 3.1 and section 3.2, we provided a generic treatment of the method, where, at the end of the each episode, the features are transformed up to date and then stored in memory for the next episode. With this scheme, it becomes imperative to always augment the capacity of the network in order to learn new representations, resulting in ever-growing network capacity. However, this problem can be easily alleviated by partitioning the entire network into a base feature extractor and feature transformer layers. The base feature extractor remains always fixed, and it is only the output of base feature extractor that is always stored in memory. With this scheme, the feature transformer layers need not grow in capacity and only a few already existing layers can be adapted for the new data. When the capacity of the feature transformer layers is not sufficient, then it can be augmented by adding one or more extra layers. In either case, the stored base features are sufficient to train.

Effect of varying additional capacity:

We varied the size of feature transformers and observed the difference in performance. Table 4 shows that by halving the additional capacity does not change the performance on MNIST rotations dataset at all. In addition, we froze the capacity of feature transformers after 5th episode and adapted them till end of 80 episodes. It is striking that performance is still high. Similarly, for Pneumothorax classification, Table 5 shows the performance comparisons with varying capacity of 2 , 1 and zero fc layers post third episode. These experiments (along with Sec 4.4) clearly demonstrate that power of the proposed approach comes from learning separable representations continually and not necessarily from storing all features or additional capacity. 

5. DISCUSSION:

In the final section, we discuss various points concerning our proposed approach.

Bayesian interpretation of Feature transformers:

Our feature transformers framework -learning a new representation with every new episode of data, can be interpreted as a maximum-a-posteriori representation learning, with the previous representation acting as prior. The implementation described in this paper using the combination loss (equation 7) is one instantiation of a general incremental representation learning possible in our framework. We have cast the MAP estimate problem to a tractable optimization problem constrained by a center-loss. In future, we plan to explore other manifestations of our approach with different loss functions that can ensure better separability.

Information Loss, Incremental Capacity, Data Privacy

As shown in Table . 4b, feature transformer becomes less effective if the base features do not contain enough relevant information. This also means that additional capacity that every feature transformer adds may not help or in-fact be counter-productive. If the base features are extracted from layers close to the input image, there will be problem of traceability which violates the data privacy requirement we want to accomplish. We feel this a potential trade-off between performance and data privacy which we will investigate in future.

Model compaction for cascade of feature transformers:

Another approach to control the growth of network capacity is model compaction, which will be our future work. At any point in time, the entire set of feature transformer layers can be replaced by a smaller and simpler network (possibly using distillation techniques) and then again allowed to grow subsequently. This cycle of grow-and-purge can be used to effectively manage the overall capacity of the network.

:

Though Φ θ is a mapping which acts on individual vectors, we abuse the notation here by using it with sets.

There is no specific restriction on the kind of layers to be used, but in our present work we use only fully connected layers.

Y (t) = ∪ τ ∈T (t) Y (t) τ

Figures and Tables:

Figure 2: (a) Performance evolution over first 25 episodes and (b) Average performance across multiple runs at the end of lifelong experiments

Figure 3: Comparison of proposed approach in conventional multi-task setting

Figure 4: (a) Performance Comparison on validation dataset and (b) Comparison of feature transformers from different base layers

Figure 5: Comparison with state-of-the-art methods -iCIFAR100 dataset

Fig 5ademonstrates the superiority of our approach by a significant margin of >10%. Further, we have negligible Backward Transfer.4.3.2 SINGLE INCREMENTAL TASK SETTINGFig 5b captures the performance over 20 batches of data for our method along with cumulative and naive learner. Unsurprisingly, naive learner performs very poorly, while feature transformer displays exemplary performance numbers of 40% validation acccuracy after encountering all 20 episodes, while compared to cumulative learner at 50%. Our method significantly outperforms AR1(Lomonaco & Maltoni, 2017), while iCaRL (Rebuffi et al., 2017)  achieves best-in-class performance close to gold-standard cumulative learner. This is not surprising because iCaRL is an explicit rehearsal technique, where exemplar images from previous episodes are stored and replayed while learning new tasks, while we only store low dimensional features.

Table. 2). We performed similar experiments on Pneumothorax classification problem and achieved similar trends as shown Table. 3, clearly demonstrating the resistance of the proposed method.Storing all features is not prohibitive Additionally, calculations of size-on-disk suggests that storing features of the entire history is not prohibitive. A typical natural/medical image is 256*256*3 integers or more, whereas our representation is only 4096 floats (16kb). Even the largest available medical image repository of 100k X-ray images takes 1.6GB which is not huge. These are con-

Table content: <table><tr><td>% of</td><td>Feature</td><td>Cumulative</td><td>% of</td><td>Feature</td></tr><tr><td colspan="2">Samples stored Transformer</td><td>Learner</td><td colspan="2">Samples stored Transformer</td></tr><tr><td>from history</td><td>Val Acc</td><td>Val Acc</td><td>from history</td><td>Val Acc</td></tr><tr><td>25</td><td>94%</td><td>95%</td><td>25</td><td>80.95%</td></tr><tr><td>50</td><td>94.8%</td><td>98%</td><td>50</td><td>82.95%</td></tr><tr><td>75</td><td>96%</td><td>98.5%</td><td>75</td><td>85.85 %</td></tr><tr><td>100</td><td>97 %</td><td>99 %</td><td>100</td><td>86.94 %</td></tr><tr><td colspan="3">Table 2: MNIST rotations</td><td colspan="2">Table 3: Pneumothorax dataset</td></tr><tr><td/><td colspan="4">Table: Performance comparisons with limited memory budget</td></tr><tr><td colspan="5">servative estimates. A standard medical image can be of much larger size (1024*1024) and in 3-D</td></tr><tr><td colspan="5">(minimum 10 slices). Any exemplar-based method (iCARL) will have severe storage limitations</td></tr><tr><td colspan="5">than our method. Additionally, storing 50 low-dimensional features occupies same memory as stor-</td></tr><tr><td colspan="5">ing one exemplar image. This directly leads to storing more history compactly while addressing</td></tr><tr><td colspan="3">catastrophic forgetting and privacy.</td><td/><td/></tr><tr><td colspan="4">4.5 CONTROLLING THE GROWTH OF NETWORK CAPACITY</td><td/></tr></table>

Pneumothorax dataset Table: Performance comparisons with limited incremental compute

Table content: <table><tr><td colspan="2">Incremental capacity Feature Transformer</td><td colspan="2">Incremental capacity Feature Transformer</td></tr><tr><td>added per episode</td><td>Val Acc</td><td>added per episode</td><td>Val Acc</td></tr><tr><td>2 dense layers</td><td>96.4%</td><td>2 dense layers</td><td>86.94%</td></tr><tr><td>1 dense layers</td><td>96.5%</td><td>1 dense layers</td><td>86.43%</td></tr><tr><td>No additonal capacity</td><td/><td>No additonal capacity</td><td/></tr><tr><td>(after 5 th episode)</td><td>96.2%</td><td>(after 3 rd episode)</td><td>86.41%</td></tr><tr><td colspan="2">Table 4: MNIST rotations</td><td/><td/></tr></table>

Peer review (iclr2019):
 Summary: a method is presented for on-going adaptation to changes in data, task or domain distribution. The method is based on adding, at each timed step, an additional network module transforming the features from the previous to the new representation. Training for the new task/data at time t relies on re-training with all previous data, stored as intermediate features. The method is shown to provide better accuracy than naïve fine tuning, and slightly inferior to plain re-training with all the data. While the method is presented as a solution for life long learning, I think it severely violates at least two demands from a feasible solution: using finite memory and using finite computational capacity (i.e. a life-long learning cannot let memory or computation demands to rise linearly with time). Contrary to this, the method presented induces networks which grow linearly in time (in number of layers, hence computation requirements and inference time), and which use a training set growing indefinitely, keeping (representations of) all the examples ever seen so far. If no restrictions on memory and computation time are given, full retraining can be employed, and indeed it provides better results that the suggested method. In the bottom line, I hence do not see the justification for using this method, either as a life-long learner or in another setting. Pros: + the method shows that for continuous adaptation certain representations can be kept instead of the original examples Cons: - The method claims to present a life long learning strategy, yet it is not scalable to long time horizon (memory and inference costs rise linearly with time) - Some experiments are not presented well enough to be understood. More detailed comments: Page 3: - Eq. 2 is not clear. It contains a term ‘classification loss’ and ‘feature_loss’ which are not defined or explained. While the former is fairly standard, leaving the latter without definition makes this equation incomprehensible. o I later see that eq. 7 includes the details. Therefore eq.2 is redundant. Page 4: - Eq. 5 seems to be flawed, though I think I can understand what it wants to say. Specifically, it states two sets: one of examples (represented by the previous feature extractor) and one of labels (of all the examples seen so far). The two sets are stated without correspondence between examples and labels – which is useless for learning (which requires example-label correspondence). I think the intention was for a set of (example, label) pairs, where the examples are represented using feature extractor of time t-1. - Algorithm 1 seems to be a brute force approach in which the features of all examples from all problems encountered so far are kept (with their corresponding labels). This means keeping an ever growing set of examples, and training repeatedly at each iteration on this set. These are not realistic assumptions for a life-long learner with finite capacity of memory and computation. o For example, for the experiment reported at page 6, including 25 episodes on MNist, each feature transformer is adding 2 additional FC layers to the network. This leads to a network with >50 FC layers at time step 25 – not a reasonable and scalable network for life ling learning Page 6: - The results show that the feature transformer method achieve accuracy close to cumulative re-training, but this is not too surprising, since feature transformer indeed does cumulative re-training: at each time step, it re-trains the classifier (a 2 stage MLP) using all the data at all times steps (i.e. cumulative retraining). The difference from pure cumulative re-training, if I understand correctly, is that the cumulative re-training is done not with the original image representations, but with the intermediate features of time t-1. What do we earn and what do we loose from this? If I understand correctly, we earn that the re-training is faster since only a 2-layer MLP is re-trained instead of the full network. We loose in the respect that the model gorws larger with time, and hence inference becomes prohibitively costly (as the network grows deeper by two layers each time step). Again, I do not think this is a practical or conceptual solution for life long learning. - The experiment reported in figure 3 is not understandable without reading Lopez-Paz et al., 2017 (which I didn’t). the experiment setting, the task, the performance measurements – all these are not explained, leaving this result meaningless for a stand-alone read of this paper. - Page 8: it is stated that “we only store low dimensional features”. However, it is not reported in all experiment exactly what is the dimension of the features stored and if they are of considerably lower dimension than the original images. Specifically for the MNIst experiments it seems that feature stored are of dimension 256, while the original image is of dimension 784 – this is lower, but no by an order of magnitude (X10). - The paper is longer than 8 pages.