Title: HC-NET: MEMORY-BASED INCREMENTAL DUAL-NETWORK SYSTEM FOR CONTINUAL LEARNING

Abstract: Training a neural network for a classification task typically assumes that the data to train are given from the beginning. However, in the real world, additional data accumulate gradually and the model requires additional training without accessing the old training data. This usually leads to the catastrophic forgetting problem which is inevitable for the traditional training methodology of neural networks. In this paper, we propose a memory-based continual learning method that is able to learn additional tasks while retaining the performance of previously learned tasks. Composed of two complementary networks, the Hippocampus-Net (H-Net) and the Cortex-Net (C-Net), our model estimates the index of the corresponding task for an input sample and utilizes a particular portion of itself with the estimated index. The C-Net guarantees no degradation in the performance of the previously learned tasks and the H-Net shows high confidence in finding the origin of an input sample.

1. INTRODUCTION:

The main difference between the human brain and the machine learning methodology is the ability to evolve. Using neurophysiological processing principles, human brains can achieve and organize knowledges throughout their lifespan. Having the neuroplasticity, human brains can transfer an activating region of a given function to a different location or control the creation and destruction of synapses according to its experiences. Usually, artificial neural network (ANN) models consist of finite numbers of filters. Also, parameters and operations in the ANN do not possess the ability corresponding to the memory system of human brains. This structural limit leads to the problem called catastrophic forgetting, i.e., the newly coming information diverts the model from previously learned knowledge. The field of researches trying to solve this problem is referred to as continual or lifelong learning. Kirkpatrick et al. (2017) suggested a method applying a penalty on the quadratic distance between the old parameters and the new ones. Called elastic weight consolidation (EWC), this method can be applied to either supervised or reinforcement learning scenarios. Jung et al. (2018) proposed a method using feature regularization referred as less-forgetful learning (LF). When training a new dataset, the feature just before the fully connected layer is regularized to become similar to that of the previous model. A considerably small learning rate is used in order to prevent a drastic change of the model. Despite of such efforts, as more and more tasks are added, the model eventually collapses with poor performances in old datasets. Li & Hoiem (2017) proposed a method using a similar regularization method, referred as learning without forgetting (LwF). Continual learning is conducted to a single convolutional neural network (CNN), adding an additional classifier whenever a dataset is newly trained. Classifiers in the newly trained network are regularized to reproduce the outputs of the old network while the new classifier is trained in a classical way. LwF shows better performance than LF, but still suffers from the catastrophic forgetting with multiple tasks. These approaches using regularization basically focus on alleviating the deformation of the original network. Therefore, the more the new knowledge flows in the model, the more the performance on the original task degrades. To make a single network deal with a large number of datasets, instead of using all parameters of a single network for each task, PackNet suggested a model which allocates the datasets to specific weights of filters (Mallya & Lazebnik, 2018) . As only the specialized weights for a particular task are involved in its classification process, PackNet shows a remarkable result in multiple tasks. However, the information about from which task the input originates and which Figure 1 : The overall architecture of our method. Blue arrows depicts the training process for the new task (Task C) of our method, where Task A and Task B have been already trained. H-Net is trained each time a dataset for a new task is provided and then the C-Net is trained in a standard way such as using the cross-entropy loss. We allow only the trainable filters (Blue parameters) to be updated. Red arrows shows the inference procedure of our method. H-Net finds out the task index J (Task C) from which the input came and let the C-Net to use the filters specialized for task J (Blue + Orange + Grey). Class labels are also switched to those of task J . group of weights should be used must be given in advance. Typically, images do not contain such prior knowledges and this makes the PackNet hard to apply in real world situation.

In this paper, inspired by the human brain system, we propose a network which efficiently increases its complexity without degrading the performance of previous tasks. Instead of saving all the memory in the cortex and trying to access it directly, the hippocampus finds the index of the memory scattered over the cortex where knowledges are well organized inside. Under the inspiration from the quote in (Buzsaki, 2006) , "Think of the cortex as a huge library and the hippocampus as its librarian", we have built our network using two complementary sub-networks as shown in Figure 1 . The Cortex-Net (C-Net) keeps the knowledge from several independent tasks. After training a network from the previous task, additional filters in the convolutional modules are attached and trained with the next task. To infer the newly learned data, the model utilizes the newly trained filters along with the previously learned filters, whereas the data from the old task is inferred only using the previously learned filters. In order to determine which combination of filters to use, we adopted a memory network which can distinguish the origin of a given input sample. Referred as the Hippocampus-Net (H-Net), it can recall the information not only which group of filters to use but also which group of class labels to use. This endows our method label-expandability, which enables our method to be practically applied in real world situation. We suggest three methods, each of which respectively using generative adversarial networks (GAN) (Goodfellow et al., 2014) , variational auto-encoder (VAE) (Kingma & Welling, 2013) , and principal component analysis (PCA) (Jolliffe, 2011) for H-Net and report the properties and performance of each method. Our H-Net can be combined with any other works such as LwF or PackNet. The combined H-Net and C-Net referred as HC-Net can prevent the catastrophic forgetting using a densely organized architecture and a highly reliable memory system. The contributions of this paper can be summarized as follows:

• Filters in the convolutional layers are allocated to each task and parameters from the previously learned tasks are shared among post tasks and used for the initialization of post tasks parameters, which makes the overall model compact and efficient.

• Attaching filters whenever a new dataset is trained allows the model to be expandable as far as the physical constraints, such as memory and processing time constraints, allow.

• To identify which portion of convolutional filters should be used to classify a data, a memory network (H-Net) is proposed with several methodologies.

• The overall method is applicable to multi-task continual learning whose number of class labels increases over tasks. 

2. METHOD:

We propose a memory-based incremental dual network system for continual learning that satisfies three properties as stated below.

Property 1. The Hippocampus-Net (H-Net) offers the index about which part of the Cortex-Net (C-Net) should be used.

Property 2. The C-Net learns the knowledge on the new task under the presence of the previously learned knowledge.

Property 3. H-Net and C-Net are flexible to the expansion of the network and as many new tasks can be learned in one network as physical constraints allow.

In this respect, our proposed H-Net works as a hippocampus of a human brain in that it gives a summarized information to the C-Net. Also, the C-Net corresponds to the human cortex in terms of its ability to learn semantic knowledge.

2.1. CORTEX NETWORK (C-NET):

We propose an efficient way to allocate the capacity to a network according to the given tasks. C-Net is an expandable network where additional filters can be attached whenever a new task is given. It can be applied to many typical networks such as VGG and ResNet (Simonyan & Zisserman, 2014; He et al., 2016) . The only difference with the original versions of those networks is that it uses different parts of filters for different tasks. As shown in (Li et al., 2016) , typical CNN are capable of maintaining their performance even when the majority of their parameters are pruned. This implies that filters in a CNN contains a lot of unnecessary or redundant information. Also, sharing the filters among different tasks inevitably leads to the catastrophic forgetting of the network since the parameters trained by an old task should be replaced with those trained by a new task. For these reasons, we utilize a network divided into several parts and refer it as C-Net where each part takes charge of a particular task.

While training the C-Net with several tasks, each task uses a different part of the C-Net. For the convenience of referring which segments of the C-Net are to be activated, we introduce a filter index I J for the J -th task. The filter index I J defines the range of filters to use for the specific task J in every layers. For example, if the filter index I 1 equals to ten, the first task uses filters from the first to the tenth filter.

Depicted in Figure 2, I J k denotes the filter index in the k-th layer corresponding to the task J . This equals to the number of output channels in the sequent feature map. I P k denotes the filter index in the k-th layer for the previous task. As the same principle is applied to the precedent (k -1)-th layer, the number of input channels for the filters in the k-th layer will change likewise. That is, total of I P k filters have the channel length of I P k-1 when the previous task is under training. As the new task is given, the length of the filters become I J k-1 since the filters in the prior layer will deliver a feature

Algorithm 1 C-Net Training:

Input: W J , I P , I J Output: W J * 1: Freeze the W P ⊂ W J using the information I P 2: Initialize the W T with W P and a random noise 3: W J * ← arg min(L c ) {Update W T using backpropagation} map whose length is I J k-1 . Therefore, both the number of input and output channels increases as the number of tasks increases until the physical constraints allow.

Weight parameters of J -th task in filters of the k-th layer are referred as

W J k ∈ R x×x×I J k-1 ×I J

k , where x means the size of the kernel while

W P k ∈ R x×x×I P k-1 ×I P

k are the weight parameters of the previous task. When training the new task, W J k is used for inference but only the parameters

w ∈ W T k (W T k = W J k \ W P k

) are updated while leaving W P k fixed. Note that biases in the filters can also be trained likewise as explained above. The fully connected layer located just before the linear classifier can be divided as well into the sections of I J and I P .

To enhance the performance, slices of trained parameters W P from the old task are duplicated to the newly available task parameters W T and Gaussian noise is added to them for the sake of good initialization. Then, the network is trained in a standard way, e.g., using a cross-entropy loss L c . Independent linear classifiers are used for every tasks. This scheme of C-Net can also be applied to structures using shortcut connections such as ResNet. The overall training process is shown in Algorithm 1.

2.2. HIPPOCAMPUS NETWORK (H-NET):

Even if the C-Net is well trained separately with several tasks, the task index J must be given at the time of inference. Especially when the class labels expand, this prior information J is necessary to estimate which group of class labels to use. In the testing step, existing methods needs the exact origination of the input data. However in the real world, a given data for classification normally does not offer any information about its origination. Therefore, this prior knowledge should be estimated by an independent network.

To solve this problem, we introduce the Hippocampus network (H-Net) using several different methods (GAN, VAE and PCA), which is able to estimate the task J of the given input data and inform this to the C-Net to specify which filters to use. In this paper, we explain the GAN method as a representative and other methods are explained in Appendix A.

2.2.1. GENERATIVE ADVERSARIAL NETWORKS (GAN):

This method is inspired by the on-line replay method used for continual learning (Shin et al., 2017) . We train a task-specific generative model G J using a GAN to generate pseudo-samples of task J . The generator is trained using the adversarial loss as follows:

min G J max D J V (D J , G J ) = E x∼P J data [log(D J (x)] + E z∼pz(z) [log(1 -D J (G J (z))].

(1)

Here, D J is a discriminator for the task J and x is a sample from the task J .

After generating samples of all tasks, a task-wise binary classifier B J is trained to classify whether the given input is from the task J or not. Note that only the generated samples are involved for the training of each binary classifier B J . This means that positive samples are from the generator G J for the task J and negative samples are from all the other task generators G K (∀K = J ).

In the testing phase, each classifier produces the probability of how likely the given unknown input data is from the task J . The classifier B i with the highest probability assumes that the input data is from the task i. H-Net can figure out the task J of input data with maximum probability across the set of classifiers. H-Net gives this estimated task index J to the C-Net. The H-Net using GAN can be summarized as below:

EQUATION

3 EXPERIMENT

We evaluate our method on several image classification datasets. First, we verify the effectiveness of our method with MNIST (LeCun et al., 1998) , SVHN (Netzer et al., 2011) and CIFAR-10 ( Krizhevsky & Hinton, 2009) which are widely used to evaluate image classification performance.

Then, we evaluate our method on two subsets of ImageNet (Russakovsky et al., 2015) which is a realistic image dataset. More details are summarized in Table 1 .

We compare our method to various methods such as Elastic Weight Consolidation (EWC), Learning without forgetting (LwF), less-forgetful learning (LF) and PackNet as well as networks trained for a single target task which shows the performance without conducting continual learning. We use the GAN method for H-Net in all the experiments as a representative. To solely examine the influence of H-Net on our method (HC-Net), we have conducted experiments with the same experimental scenarios on the HC-Net experiments. In Table 2 , each row shows the result of one among GAN, VAE and PCA. In Table 3 and Table 4 , the result of 'HC-Net (without H-Net)' shows the performance of the C-Net only. That is, it assumes that the H-Net never fails and the task index J is always given correctly. Likewise, LwF and PackNet need prior knowledge of a given input to know which classifier to use. As there is no structure like H-Net in the original paper, The actual comparison must be done with 'C-Net'.

3.1. CONTINUAL LEARNING USING BASIC IMAGE CLASSIFICATION DATASETS:

We have built three experimental scenarios to evaluate our method using basic image classification datasets. We compare the performance of learning a consecutive two task pair among MNIST, SVHN and CIFAR-10 dataset. After that, we conduct multiple-task continual learning with these datasets. In these experiments, we use the same learning hyperparameters and network architectures suggested in (Jung et al., 2018) used for 'Tiny image classification'. Training details are reported in Appendix B.

3.1.1. MNIST → SVHN:

After selecting a subset of SVHN to equalize the number of training data between MNIST and SVHN, images are resized to 28 × 28 as in (Jung et al., 2018) . The results of Modified LwF, EWC and LF are referred from (Jung et al., 2018) . Modified LwF is sligthly different from the original LwF in that the old output Ŷo and new outputs Ŷn is united into a single output. The 'single network' in Table 3 is a network of full capacity trained using a single dataset. The column 'Old' represents the test accuracy on the previously learnt task and 'New' is that of the newly trained task . The value γ for EWC is determined as in (Kirkpatrick et al., 2017) . A knowledge distillation loss with hyper-parameters of T = 2 and T = 1 is used for LwF, referred as temperature in (Hinton et al., 2015) .

As MNIST and SVHN highly resemble each other, the accuracy of all methods are relatively high. LF demonstrate better average classification rates than Modified LwF and EWC. LwF can control the performance trade-off between the old and new task by changing the temperature parameter T . The bigger T is, the lower the accuracy drops but it still outperforms those of the methods mentioned above. PackNet almost maintains the performance of the single network for the first task with less than 2% drop in accuracy. HC-Net outperforms all other methods in the baseline methods. Nonetheless, our full model, HC-Net also outperforms PackNet, especially on the second task by around 0.7%.

3.1.2. SVHN → CIFAR-10:

In this experiment, we want to see the performance of the proposed method when the statistics of the two tasks are quite different. The datasets SVHN and CIFAR-10 have a clear distinction in image styles and colors. For LF whose classifier must be shared between the old and new tasks, this inevitably leads to a significant decrease in performance. LwF and PackNet show similar trends as that of 'MNIST to SVHN'. Higher temperature (T ) induces better performance in the new task but a bit of degradation in average performance still occurs. PackNet experiences almost no performance degradation. The pruning method of PackNet acts like a generalization method and helps the model to maintain its capacity. The average performance of C-Net is slightly lower than that of PackNet by 0.35%. However, note that PackNet requires additional finetuning after the pruning process and takes additional time for training. On the other hand, C-Net requires no additional pruning or finetuning precedure and yet retains the performance.

3.1.3. MNIST → SVHN → CIFAR-10:

Training three or more tasks is unmanageable for the models using regularization methods. LF scored an average accuracy of no more than 69.03%, which is much more lower than any other methods. No matter how we change the temperature (T ), LwF suffers from a drop in average performance, especially in the third task. Also, even though the proportion allocated to each task is equivalent between the PackNet and our methods (HC-Net and C-Net), our method highly outperforms the PackNet in the third task. This implies that HC-Net is more suitable for multi-task sequential learning situation.

3.2. CONTINUAL LEARNING USING REALISTIC DATASETS:

ImageNet contains images more realistic than other datasets used in this paper. Higher resolution and complex backgrounds make the classification even harder. Table 5 : Performances of the old (MNIST) and new (SVHN) tasks with different initialization methods: 1) initialization with only random Gaussian noise and 2) the pretrained parameter added by random Gaussian noise are referred to as ImageNet-A and ImageNet-B respectively in this paper. To show the adaptability of our method on structures having shortcut connections, ResNet-50 is used for the experiment. We use the same experimental setting as in (He et al., 2016) with a batch size of 128. We compare our C-Net with LwF and PackNet.

Table 4 shows the results on the ImageNet dataset. The 'Old' accuracy of all methods slightly dropped from those of single networks. Like in the 'SVHN to CIFAR-10' experiment, PackNet shows a better result than the LwF regardless of the value of T . The result of C-Net is almost identical to PackNet with a slight increase in the average accuracy.

PackNet actually utilizes more parameters for each task than C-Net does. Since the masks force weights with no influence to be zero and utilize the well trained remaining weights, PackNet can make good use of the entire network. However in C-Net, just adding filters gradually without any other post-processing performs well enough compared to PackNet. When the model of an initially designed size is fully occupied by several tasks, can just add more filters and train them along with the trained filters while this is not the case of PackNet. Furthermore, PackNet requires additional memories to store the masks for all filters in it. C-Net needs only one integer per layer for each task. For these reasons, C-Net is far more efficient than the PackNet with no loss in performance.

3.3. ABLATION STUDY ON THE EFFICIENCY OF C-NET:

As mentioned in Section 2.1, our C-Net appends filters whenever a new task is to be trained. In this process, parameters learned from previous tasks are utilized altogether and the parameters which are to be trained newly are initialized using the existing ones. We conduct experiments to show the effect of these methods. Experiments are carried out in 'MNIST to SVHN' case as a representative.

3.3.1. INITIALIZATION WITH PRETRAINED PARAMETERS:

To analyse the effectiveness of initialization using pretrained old task parameters with random Gaussian noise, we compare the accuracy between a model initializing the new parameters with just a random Gaussian distribution and a model initializing the new task parameters using the pretrained old task parameters with additional random Gaussian noise. In Table 5 , the model initialized by our method obviously shows higher accuracy on the SVHN task. This result implies that a good initialization prevents the model from going through a local minima.

3.3.2. PARAMETER SHARING:

We verify the effect of the parameter sharing between the old task and the new task. The baseline method is a model where the old parameters are filled with random Gaussian initialization and no further training is done. The new parameters in the model have to learn knowledges from the new task without the aid of old parameters since they are fixed from the beginning. On the other hand, the old parameters in our method is trained by the old task and the new parameters can make use of these learnt knowledges to learn the new task. To solely observe the effect of the parameter sharing, both methods do not use our parameter initialization method mentioned in the previous section. The task indices I 1 and I 2 are {16,16,32} and {18,18,34} respectively. The model using parameter sharing converges far more faster than the model with no parameter sharing which is shown in Figure 3 . Also the final error rate of our method is lower than the other. Increasing the number of indices enhances our method by 6.38% and the other model by 7.91% shown in Table 6 . This implies that the parameter sharing improves the model and allow it to be compact with fewer numbers of filters.

3.4. H-NET:

The results of H-Net are reported in Table 2 . In 'MNIST to SVHN', all methods approximates the results to perfection. Despite of the similar appearance of these two datasets, VAE has been able to generate distinct outputs. If a sample in the MNIST dataset is given, the result from the VAE trained for SVHN resembles those of the SVHN dataset. In other words, auto-encoders of each task tends to memorize only what they used for training. PCA is completely able to discriminate the origin of the images from MNIST and SVHN. Particularly in this experiment, even when using a single eigenvector with the least eigen-value showed a result of 100% and 99.7%. According to the dataset, PCA can highly reduce the computational cost as well as the memory resource.

In 'SVHN to CIFAR-10', GAN showed the best discriminating performance. VAE shows a poor result on SVHN while the performance on CIFAR-10 remains more than 97%. This means that the auto-encoder trained for CIFAR-10 reconstructs SVHN images well, leading to low reconstruction losses in both of the auto-encoders. With PCA, both datasets are able to separate from each other with chances more than 92%. The experiments using all three datasets showed almost the same results as the experiments above. GAN definitely shows the best results among all methods introduced.

3.4.1. LIMITATION:

Generative models usually have difficulties in generating realistic images with high resolution and there are few researches experimented on ImageNet data. This incompetence makes the H-Net hard to be applied to the ImageNet data. Also, as the subgroups share the same statistics, PCA hardly catch the characteristic of each set. Experiments have shown poor results on ImageNet and we leave this for a future work.

4. CONCLUSION:

In this paper, we have proposed a novel framework which is able to divide its capacity into several parts and utilize them according to the given input. Composed of two networks, H-Net is responsible for memorizing "where the data came from" and complicated knowledges such as "what the data is" are engraved to the C-Net. To the best of our knowledge, our work, H-Net, is the first attempt to estimate the origin of data which used to be assumed as given in the previous works. As well as overcoming the catastrophic forgetting, HC-Net allows extra class labels when training a new dataset and also has expandability to the network architecture. In the data mining field, multi-task learning can be seen as learning from data streams. Therefore, our method can be considered to be one of researches on learning methods from data streams (Gama & Gaber, 2007; Gama, 2010) .

:



Figures and Tables:

Figure 2: Trainable parameters W T k (Orange) and frozen parameters W P k (Grey) in the C-Net training process. In the training process, the parameters (Orange), which are trainable parameters except frozen parameters (Grey), are updated. x denotes the spatial dimension of filters.

Figure 3: Error rate comparison between sharing the parameters which learned previous task and random parameters.

Figure 5: SVHN dataset

Figure 6: CIFAR-10 dataset

Details of datasets.

Table content: <table><tr><td/><td/><td/><td/><td/><td/><td>Datasets</td><td/><td>Methods</td><td/><td colspan="2">Old(%) New(%) Avg. (%)</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td colspan="3">single network (MNIST) 99.49</td><td>-</td><td>-</td></tr><tr><td/><td/><td/><td/><td/><td/><td>MNIST</td><td/><td colspan="2">single network (SVHN)</td><td>-</td><td>92.82</td><td>-</td></tr><tr><td>Datasets MNIST SVHN</td><td/><td colspan="4">#of Train data #of Test data #of Class 60,000 10,000 10 73,275 26,032 10</td><td>↓ SVHN</td><td/><td colspan="3">Modified LwF EWC (γ = 2.32 × 10 4 ) 94.15 94.78 LF (λ e = 1.6 × 10 -3 ) 97.37</td><td>83.77 79.31 83.79</td><td>89.28 86.73 90.58</td></tr><tr><td>CIFAR-10</td><td/><td>50,000</td><td/><td>10,000</td><td>10</td><td/><td/><td colspan="2">LwF (T = 1)</td><td>98.27</td><td>86.40</td><td>92.34</td></tr><tr><td>ImageNet-A ImageNet-B</td><td/><td>64,750 64,497</td><td/><td>2,500 2,500</td><td>50 50</td><td/><td/><td colspan="2">LwF (T = 2) PackNet</td><td>97.33 99.45</td><td>86.97 91.49</td><td>92.15 95.47</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td>HC-Net</td><td/><td>99.43</td><td>92.20</td><td>95.82</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td colspan="3">HC-Net (without H-Net) 99.43</td><td>92.37</td><td>95.90</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td colspan="3">single network (SVHN) 92.94</td><td>-</td><td>-</td></tr><tr><td colspan="6">Datasets Methods Old(%) New(%) MNIST GAN 100 99.94 ↓ VAE 100 99.85</td><td colspan="2">SVHN ↓ CIFAR-10</td><td colspan="2">single network (CIFAR) LF (λ e = 2.0 × 10 -3 ) LwF (T = 1) LwF (T = 2)</td><td>-83.18 91.76 90.19</td><td>79.69 48.99 70.57 71.94</td><td>-66.09 81.17 81.07</td></tr><tr><td>SVHN</td><td/><td>PCA</td><td/><td>100</td><td>100</td><td/><td/><td>PackNet</td><td/><td>92.84</td><td>76.78</td><td>84.81</td></tr><tr><td>SVHN</td><td/><td colspan="2">GAN</td><td>97.00</td><td>99.90</td><td/><td/><td>HC-Net</td><td/><td>90.05</td><td>76.62</td><td>83.34</td></tr><tr><td>↓</td><td/><td>VAE</td><td/><td>71.07</td><td>97.06</td><td/><td/><td colspan="3">HC-Net (without H-Net) 92.21</td><td>76.70</td><td>84.46</td></tr><tr><td colspan="2">CIFAR-10</td><td>PCA</td><td/><td>93.28</td><td>92.5</td><td>Datasets</td><td colspan="2">Methods single network (MNIST)</td><td colspan="3">MNIST(%) SVHN(%) CIFAR(%) Avg(%) 99.44 ---</td></tr><tr><td>Datasets</td><td colspan="5">Methods MNIST(%) SVHN(%) CIFAR(%)</td><td>MNIST</td><td/><td>single network (SVHN)</td><td>-</td><td>92.94</td><td>-</td><td>-</td></tr><tr><td>MNIST → SVHN → CIFAR-10</td><td/><td>GAN VAE PCA</td><td>100 100 100</td><td>96.65 71.22 93.28</td><td>99.83 97.07 92.5</td><td>↓ SVHN ↓</td><td/><td>single network (CIFAR) LF (λe = 1.0 × 10 -3 ) LwF (T = 1)</td><td>-89.57 95.06</td><td>-72.1 86.80</td><td>79.69 45.43 68.98</td><td>-69.03 83.61</td></tr><tr><td/><td/><td/><td/><td/><td/><td>CIFAR-10</td><td/><td>LwF (T = 2)</td><td>86.09</td><td>85.07</td><td>69.63</td><td>80.26</td></tr><tr><td colspan="6">Table 2: Experimental results of H-Net on the basic datasets</td><td/><td colspan="2">PackNet HC-Net HC-Net (without H-Net)</td><td>99.38 99.41 99.41</td><td>91.93 89.36 91.84</td><td>66.34 74.93 75.02</td><td>85.88 87.90 88.76</td></tr></table>

Mean classification results on the basic datasets (5 runs).

Table content: <table/>

To save the training time, two subsets of the ImageNet dataset each having 50 randomly chosen classes have been used for evaluation. They Mean classification results for the realistic dataset (2 runs)

Table content: <table><tr><td>datasets</td><td>Methods</td><td/><td colspan="3">Old(%) New(%) Avg. (%)</td></tr><tr><td/><td colspan="2">single network (ImageNet-A)</td><td>83.28</td><td>-</td><td>-</td></tr><tr><td>ImageNet-A</td><td colspan="2">single network (ImageNet-B)</td><td>-</td><td>85.28</td><td>-</td></tr><tr><td>↓</td><td colspan="2">LwF (T = 1)</td><td>82.2</td><td>86.72</td><td>84.46</td></tr><tr><td>ImageNet-B</td><td colspan="2">LwF (T = 2)</td><td>80.92</td><td>86.96</td><td>83.94</td></tr><tr><td/><td>PackNet</td><td/><td>82.16</td><td>88.72</td><td>85.44</td></tr><tr><td/><td colspan="2">HC-Net (without H-Net)</td><td>83.30</td><td>88.66</td><td>85.98</td></tr><tr><td/><td>Initialization</td><td colspan="3">Old(%) New(%) Avg.(%)</td></tr><tr><td/><td>Random</td><td>99.42</td><td>91.67</td><td>95.55</td></tr><tr><td colspan="2">Pretrained parameters</td><td>99.43</td><td>92.37</td><td>95.90</td></tr></table>

The performances on the new task while increasing the filter index of the model using parameter sharing and the model with fixed random parameters.

Table content: <table/>

Peer review (iclr2019):
 Summary This paper proposes an extension of Progressive Networks [Rusu et al. NIPS 2016] (unfortunately, not cited) where the task id is not given at test time. This is inferred by a battery of classifiers trained on data produced by generative models trained on task specific data. The authors argue strong connections to similar mechanisms in the human brain and demonstrate this method on a stream of 2 or 3 vision tasks. However, the interpretation of these results is dubious. Novelty: given prior work on Progressive Nets and other methods using generative models for continual learning, novelty is limited. Relevance: the motivation and aim of this work is certainly relevant for ICLR. Clarity: the paper is overall clear, although it needs a bit of rewriting to improve fluency (see for instance sec. 3.4.1). References: the authors should definitely cite Progressive Networks and their extension "Progress and Compress" (Schwarz ICML 2018), as their approach is an extension of the former with the only difference that the task id is inferred at test time by using a battery of binary classifiers. Empirical validation: The empirical validation is limited because of: a) lack of comparison to Progressive Nets, b) lack of simple baselines (e.g., how about replacing H-Net with an inference process like task_id = argmin_i=1..T loss(C-Net, task = i) ), c) unclear interpretation of the provided results (how can the accuracy on MNIST be 100%? are the authors reporting training accuracy?) d) very limited number of tasks considered (up to 3) General comments Major drawbacks of the proposed approach are: 1) training on new tasks can never improve performance on past tasks (unlike other methods like GEM (Lopez-Paz et al. NIPS 2017), 2) the number of parameters grow linearly with the number of tasks (an issue addressed by the Progress and Compress paper above), and 3) the overall approach is not efficient as it requires lots of data from each task in order to train the generative models. Finally, I think all the connections and inspiration from how the human brain works should be toned down. Statements like "the C-Net corresponds to the human cortex..." should be at the very least rephrased appropriately.