Title: ENSEMBLENET: A NOVEL ARCHITECTURE FOR IN-CREMENTAL LEARNING

Abstract: Deep neural networks are used in many state-of-the-art systems for machine perception. Once a network is trained to do a specific task, it cannot be easily trained to do new tasks as it leads to catastrophic forgetting of the previously learned tasks.We propose here a novel architecture called EnsembleNet that accommodates for newer classes of data without having to retrain previously trained submodels. The novelty of our model lies in the fact that only a small portion of the network has to be retrained which makes it computational efficient and also results in higher performance compared to other architectures in the literature. We demonstrated our model on MNIST Handwritten digits, MNIST Fashion and CI-FAR10 datasets. The proposed architecture was benchmarked against other models in the literature on Ω new , Ω base , Ω all metrics for MNIST-Handwritten dataset. The experimental results show that Ensemble Net on overall outperformed every other model in the literature.

1. INTRODUCTION:

Many times in the real world setting, it becomes essential for an existing neural network to learn a new class. For example: A Deep Neural Network (DNN) that is trained to recognize cats might have to be trained to recognize a new species of cat. However, current DNNs are not capable of meeting the need. Whenever a DNN is trained on a new subset of data, the weights that are responsible for retaining the old information are disturbed. This leads to forgetting of the old data that it learnt and this is called as Catastrophic Forgetting. Human brains are very good at this and make it look easy, but emulating this behaviour in machine learning models has proven to be a challenging task. To solve this problem, many techniques have been proposed in the past . Most of them include freezing the weights that are important for the previous data (Fernando et al. (2017) ) (Kirkpatrick et al. (2017) ) or retraining the entire neural network on the previous data. Retraining the entire model on entire previous data is a computationally intensive task and might require weeks of retraining in few cases. It would be computationally very efficient if we can retrain only a small portion of the model instead of the entire neural network.

In this paper we proposed a model that will address this issue. The proposed architecture performs rehearsal of the previous data on only a small portion of the entire model. The proposed architecture contains two layers of neural networks and only the neural network that is present in the final layer is exposed to rehearsal. We have shown that this partial rehearsal technique is sufficient to achieve state of the art performance on Ω base , Ω new , Ω old metrics Kemker et al. (2018) where Ω base is the ability of a model to retain the base knowledge, Ω new is the ability of a model to retain the new knowledge and Ω all is the overall knowledge retention capacity of a model.

As a consequence of the proposed models architecture, it can be trained on a distributed computing network in an asynchronous manner on heterogeneous systems. Many models and architectures have been proposed earlier which train a neural network using a distributed computing system Dean et al. (2012) . However, all of the proposed methods involve data transfer between the systems in the distributed network. To ensure smooth data transfer between systems, it is highly essential that all the systems in the network are synchronised and are online. This requirement reduces the robustness of those methods. However, our proposed architecture does not require any data transfer between systems. This allows all the systems in the network to be asynchronous. The training will not be affected even if one system goes offline.

2. RELATED WORK:

Many architectures have been proposed in the past which are capable of incremental learning. Some of the previously proposed architectures utilize Ensembling methods while other architectures propose using specialized architectures. For example in Polikar et al. (2001) and He et al. (2011) ensemble methods were used to achieve incremental learning. They are capable of learning new data that is flowing into the system as well as learning new classes incrementally. Connolly et al. (2008) , Polikar et al. (2001) , Gabrys & Bargiela (2000) , Fritzke (1995) , Bouchachia (2006) were proposed which do not use any ensembling techniques. These algorithms have smaller memory footprint compared to ensemble methods but are incapable of incrementally learning new classes.

Our proposed architecture is similar to a stacking method which can learn new incoming data as well as incrementally learn new classes.Apart from incremental learning capabilities the proposed architecture can be trained in a distributed manner in an asynchronous fashion on heterogeneous computers in parallel. Some work has already been done in trying to achieve distributed training of machine learning models like the two algorithms proposed in Dean et al. (2012) where the neural network is split to smaller parts and then trained in parallel. Some of the existing distributed systems infrastructure like Map Reduce Dean & Ghemawat (2008 ), Graph Lab Low et al. (2012) has been utilized by the machine learning community to implement their algorithms. But none of the proposed distributed training methods are capable of incremental learning. The novelty of our architecture lies in the fact that our model is capable of incremental learning new classes with minimal rehearsal and also being capable of trained in a distributed manner.

3. PROPOSED ARCHITECTURE:

The architecture consists of two levels of classifiers. In the first level, there are k classifiers where k is the number of classes in the data that the model has been trained on till now. Each classifier in the first layer is trained to recognize datapoints that belong to only one class. If they see a datapoint that they are not trained to recognize then they output 0. In the second level there is only one classifier and it is called as the final classifier. The final classifier takes input from all the classifiers in level 1 and predicts the class of the given input sample point. The proposed architecture is similar to StackingC architecture Seewald (2002) . However, StackingC architecture doesnot append new classifier to the first layer of classifiers in-order to learn new classes incrementally. Another key difference between our architecture and StackingC is that unlike StackingC, in our architecture the final classifier takes in the confidence of the classifier and not the predicted output of the level 1 classifier.

Figure 1: Architecture of EnsembleNet:

The architecture proposed is built using an iterative algorithm where each new class is added in a one-vs-rest fashion. For adding a class 'K', the training data is split into positive and negative data, where positive points correspond to class K points. The negative data corresponds to points of all other classes than K. The negative data is only a subset of points from each of the other classes.

The algorithm is presented in [reference to algorithm] table. The Algorithm 1 describes the training procedure of the proposed architecture. The training procedure takes a data point x i where i is the class to which the datapoint belongs. If there is already a classifier l i in the first level of the EnsembleNet that is trained to recognize datapoints belonging to class i, then the new datapoint is fed to l i as a training point. If the label is not covered, then a new model is built and added to the net. The training data for this new classifier consists of X i , where X i is the set of all the new points belonging to class i and X i-1,i-2,i-3.. is the subset of points belonging to all the previous classes. Here, X i is considered as positive class and X i-1,i-2,i-3 

if i ∈ I then 12:

A New classifier l is created 13: l is trained on with i class as positive points and subset of rest classes as negative points 14:

L = L ∪ l i 15: else 16: l i is trained on x i 17: end if 18: end for 19: for x in X do 20: Θ = {} 21: for l in L do 22: Θ = Θ ∪ l(x) 23: end for 24: F C is trained on Θ 25: end for Algorithm 2 EnsembleNet -Prediction algorithm 1: X T // Test data 2: for x i in X T do 3: Θ = {} 4: for l in L do 5: Θ = Θ ∪ l(x) 6: end for 7: prediction = F C (Θ) 8:

print(prediction) 9: end for In the prediction phase, a given input point x is passed through each of the existing classifiers in layer 1.[Algorithm 2]. A classifier in the first level outputs it's confidence only when it recognizes the point. This is called as opinion of the classifier. For every point, opinion of all the classifiers in the first level is gathered and is passed to the final classifier for classification.

3.1. VECTOR SPACE TRANSFORMATION OF DATAPOINTS:

Whenever datapoints are projected in a vector space, they are usually represented in the ndimensional feature vector space (Figure 2 .a), where x 1 , x 2 , x 3 ... are features. When these datapoints are passed to the level 1 of the EnsembleNet a feature transformation takes place and the points are now projected on a different vector space where the vectors are the confidence of a point belonging to a particular class. We can call this new vector space as class vector space. There are two possible cases that could occur. In the first case, only one classifier from layer 1 recognises the point and the point will lie along one axis in the class vector (Figure 2.b) . If all the points in the dataset fall under case 1, then the final classifier can be a single perceptron. But this is not the case always. Many times, for a given datapoint, more than one classifier in layer 1 is triggered and the points donot lie along any axis as shown in Figure 3 . Hence, a multi-layer perceptron was used to perform the final classification task.

3.2. TRAINING TIME OF ENSEMBLENET:

The architecture of EnsembleNet allows it to be trained on an asynchronous distributed system with ease. All the classifiers in the first level are independent of each other and can be trained independently. Hence, parallelizing the training of the first level of classifiers by training each classifier on one system in a distributed network will greatly reduce the training time. If one classifier is trainined on one sub-system of a distributed system, then the training time (T ) will be

T = max(T l1 , T l2 , T l3 ..T l k ) + T F C (1)

where T l k is the time consumed by a system to train the l k classifier on one sub-system in the distributed system and T F C is the time taken to train the final classifier. If the distributed system contains only n nodes, then equation 1 can be re-written as

T = max(T 1 , T 2 , T 3 ..T n ) + T F C (2)

where T 1 is the time spent by the first node in the system on training the first classifier and T 2 is the time spent by the second node in the system on training the second classifier and so on. n denotes the total number of nodes in the distributed system. The proposed architecture's training time was benchmarked against a standard CNN and standard CNN with rehearsal on Multi task learning experiment. The results have been tabulated in the Experimental results section of the paper in Table 4 . The proposed architecture was evaluated against the following architectures: 1) Standard Multi-layer Perceptron 2) MLP with Elastic Weight Consolidation 3)GeppNet 4)Fixed Expansion layer architecture and 5) GeppNet+STM on MNIST -Handwritten dataset. The benchmarking performance of each model on MNIST Handwritten dataset was taken from Kemker et al. (2018) . The experiment that was proposed there was replicated using the same dataset without any changes. The evaluation of the Ω base , Ω new and Ω all were based on the α ideal that were described in the paper.

4.2.1. STANDARD MULTI-LAYER PERCEPTRON:

A standard Multi Layer Perceptron was used as a baseline by Kemker et al. (2018) . A hyperparameter search for the number of units per hidden layer (32-4,096) and number of hidden layers (2-3) was done to obtain the optimal model for the given task. This model was tested for incremental class learning by declaring the number of classes initially and then adding new sessions of data.

4.2.2. ELASTIC WEIGHT CONSOLIDATION:

In EWC, the weights that are important for remembering the previous data are disturbed comparatively less when new data is being added to the model. This is achieved by using a modified loss function

L(θ) = L t (θ) + i λ 2 F i (θ i -θ A,i ) 2

F is the Fisher information matrix which is used to constrain the weights that are important for retaining previously learned data. A detailed description of the loss function can be found at Kirkpatrick et al. ( 2017 Only some of the units in the first layer are updated because only a subset of the FEL layer units are allowed to have non-zero output to the final classification layer. Moreover, only a subset of units in the FEL layer are connected to the units in the first layer.

4.3. EXPERIMENTAL RESULTS:



4.3.1. INCREMENTAL CLASS LEARNING:

The experiment that was performed was Incremental class learning, where new classes are added to the model iteratively and the ability of the model to retain the base knowledge, new knowledge and the overall knowledge is measured. For all the three datasets, classes from 0 to 4 were considered as base knowledge and classes 5 to 9 were added iteratively to the model.

For MNIST-Handwritten digits dataset, although MLP and FEL showed better capability to retain new data, they both performed poorly when their ability to retain Base data and Overall data were measured. Elastic weight consolidation technique demonstrated higher capability of retaining base knowledge, but the weight consolidation prevented the model from learning new data effectively.

GeppNet and GeppNet+STM performed better compared to other models, but our model outperformed both the models in all the metrics. EnsembleNet showed significantly higher ability to retain new knowledge comapared to the GeppNet variants. The Geometric mean of Ω Base , Ω All and Ω new has been calculated and is represented as Ω G . EnsembleNet outperformed all other models in Ω G as well.

The model was also tested on MNIST-Fashion dataset and CIFAR10. 0-4 classes were considered as the base knowledge and 5-9 classes were incrementally added in successive sessions. 

4.3.2. MULTI MODAL INCREMENTAL LEARNING:

The experiment evaluates the ability of a model to learn multiple tasks sequentially. In this experiment, the model is first trained on one dataset and then it is expected to learn a new dataset. The first data on which the model has been trained is considered as the base dataset and the second dataset on which the model is trained is considered as the new dataset. MNIST Handwritten digits and MNIST Fashion datasets were used for the experiment. In one iteration, Handwritten digits dataset was trained as the base dataset and Fashion dataset was trained as the new dataset. In the second iteration, Fashion dataset was considered as the base dataset and Handwritten digits dataset was considerd as the new dataset. The model's behaviour has been benchmarked against standard Convolutional Neural Network Krizhevsky et al. (2012) and a standard Convolutional Neural Network that was allowed to rehearse the base dataset after being trained on the new dataset. The experimental results show that the proposed architecture has better capacity to retain the base knowledge and as well as learn new knowledge. Four variations of experiments were conducted on CNN and the CNN was allowed to rehearse the base knowledge in two experiments. Even after rehearsal, the standard CNN failed to show results that were comparable to our proposed architecture. The architecture showed an overall of higher performance on Ω all and Ω new metrics.

Model:



5. DISCUSSION:

The working philosophy of EnsembleNet can be understood by understanding the Stability-Plasticity Dilemma Abraham & Robins (2005) . The stability-plasticity dilemma basically states that, if a neural network's weights are disturbed easily when it is trained on new data, then the weights that are responsible for retention of old memories are distorted and the ability of the network to retain and recall the old data is lost. However, if the neural network's weights remain unchanged even when it is trained on new data, then it will fail to learn the new data. In the former case, the neural network can be considered as being too plastic and in the later case, the neural network can be considered as being too stable. The standard Multi-layer perceptron and Convolutional Neural networks are good examples of neural architectures that are too plastic in nature. The experimental results on Multi-modal experiments clearly show that CNNs are capable of learning new data very well, however their capacity for retaining base knowledge is low. Neural Networks that incorporate partial or total weight freezing perform poorly on learning new data as their architecture is too stable and crucial weight updations donot happen. Examples of architectures that utilize partial or total weight freezing are Fixed expansion layer network and Elastic weight consolidation networks.

The high performance of EnsembleNet can be explained by EnsembleNet's ability to keep a portion of it's network stable while leaving other portion of it's network to be plastic. The classifiers in the first level remain un-affected when the model is trained on new data. This protects the crucial weights in the network that are responsible for retaining old data. However, the weights in the newly added classifier and the final classifier are subjected to change which helps in the learning of new data. The lack of updation of weights in the level 1 makes EnsembleNet stable while rapid updation of weights in the final classifier and the new classifier makes it plastic. This inherent nature of remaining stable while having the flexibility to be plastic explains the high performance that has been observed in the experiments.

6. CONCLUSIONS:

In this paper, we proposed a novel architecture which is capable of incrementally learning new object classes. We demonstrated that EnsembleNet is capable of retaining both recently learned data and newly learned data by only retraining a small portion of the entire network. In addition we showed that EnsembleNet can be trained much faster than traditional neural networks on a distributed system in an asynchronous manner. Future work will include exploration of ways to avoid storing of entire training data. Pseudorehearsal will be incorporated into the EnsembleNet architecture to make it more memory efficient. We are also planning to test the proposed architecture on datasets with higher number of classes like CIFAR100 and CUB200.

7. SUPPLEMENTARY MATERIAL:



7.1. TRAINING TIME RELATIONSHIP BETWEEN AND ORDINARY NETWORK:

In order to deduce the training time relationship between and Ordinary network, we need to make certain assumptions that are general and sensible in practice.

Assumptions or Control settings:

• Training time of each of the sub-modules is proportional to the size of its corresponding input training data T i ∝ |S i |

• Each of the sub-models have similar complexities and take same training time, i.e.

T i T j = |S i | |S j | → T i = α * |S i |

• Comparisons are drawn against a single neural network of multi class prediction type

• The training time of the final layer (say, T f ) of the is proportional to the number of input classes

• In a given scenario of comparison between Ordinary network and , we can assume T f = c, some constant value.

In case of the parallel training scenario, the effective time required for training the non-final layer is,

T b = max(T 1 , . . . , T n ) .

Let S max ⊆ S be the partition corresponding to the sub-module that took the maximum time. Let |S max = β * |S|.

The total time required to train the is therefore shown in (Equation 3).

EQUATION

The total time required to train the ordinary network is (Equation 4).

EQUATION

Now, the relationship between T ord and T en is shown in (Lemma 7.1).

Let us consider the ratio of training times of and Ordinary network, We require that,

T en T ord . T en T ord = T max + c T ord = α * |S max | + c β * |S| = α * β * |S max | + c γ * |S|

T en T ord < 1

, which is implied by (Equation 5).

-→ |S| * α 2 + c α * |S| < 1

EQUATION

If sub-module and ordinary are of similar architectures, then T en ≤ T ord Upon enforcing (Equation refeqn:enet-ord-axiom1) and due to (Lemmas 7.1.1, 7.1.2, 7.1.3) , we can conclude that T en ≤ T ord .

7.1.1. REAL VALUED α:

We required that, α is a real value, which means the quantity under square root to be positive (Equation 6).

When the number of training points in the data set are at least 4 times larger than the number of classes, then each sub-module is non-trivially trainable.

Refer to (Equation 5) and consider the quantity inside the square root. Refer to (Equation 5) and consider the minimum value of α.

α 2 -α + c |S| < 0 -→ α ∈ [ 1 -1 -4 * c |S| 2 , 1 ± 1 + 4 * c |S| 2 ] α = 1 -1 -4 * c |S| 2 ≥ 0 Consider 1- √ 1-4 * c |S| 2 ≥ 0, 1 -1 -4 * c |S| 2 ≥ 0 -→ 1 -4 * c |S| ≤ 1 -→ 1 -4 * c |S| ≤ 1 -→ -4 * c |S| ≤ 0 .

Therefore, (∀c, |S|) : α ≥ 0.

7.1.3. HANDLING THE UPPER BOUND, α ≤ 1:

The condition α ≤ 1 is satisfied all valid values of number of classes and sizes of training set of points.

Refer to (Equation 5) and consider the maximum value of α.

We required that, α ≤ 1 .

EQUATION

This is again true, (∀c, |S|) : α ≤ 1. 

:



Figures and Tables:

.. considered as negative class. Finally, the entire training data is rehearsed to the final classifier by freezing the classifiers present in the first level and only retraining the final classifier. Algorithm 1 EnsembleNet -Training algorithm 1: X N // New data set 2: X // Entire data set 3: x i // Datapoint belonging to class i 4: I // Set of all classes that EnsembleNet already knows 5: l i // Classifier trained to recognize i class points 6: l(x) // Opinion of classifier l on datapoint x. 7: L // Set of all classifiers in Level 1 8: Θ // Opinion vector 9: F C // Final classifier 10: for x i in X N do 11:

Figure 2: Comparision of vector space representations

Figure 4: Parallel training of level 1 classifiers on a distributed system

When sub-module and the ordinary network are of similar architectures, γ ≈ β . When all the sub-modules are similar, β ≈ α .T en T ord = α * β * |S max | + c γ * |S| = α * α * |S| + c α * |S|

condition α ≥ 0 is satisfied all valid values of number of classes and sizes of training set of points.

Figure 5: Relationship between sub-module error ξ and overall error δ for n = 10

Figure 6: Relationship between sub-module error ξ and overall error δ for n = 1, 2

Incremental class learning results on MNIST Handwritten digits dataset. The benchmarking results were taken fromKemker et al. (2018) .

Table content: <table><tr><td>The proposed</td></tr></table>

Incremental class learning results on MNIST-Fashion and CIFAR10 datasets model showed similar performance on Fashion and CIFAR10 dataset. The α ideal was set as 92% and 67.3% respectively. It has to be observed that EnsembleNet showed values greater than 0.92 for all Ω values on all the tested datasets.

Table content: <table/>

Multi modal incremental learning results on MNIST-Fashion and MNIST-Handwritten digits datasets

Table content: <table><tr><td/><td colspan="3">Base Dataset New Dataset Ω new</td><td>Ω base</td><td>Ω all</td><td>T B</td><td>T N</td><td>T F</td></tr><tr><td>CNN</td><td>MNIST</td><td>Fashion</td><td colspan="4">1.000 0.0013 0.5114 60s 60s 120s</td></tr><tr><td>CNN+rehearsal</td><td>MNIST</td><td>Fashion</td><td colspan="4">0.758 0.9945 0.9314 120s 60s 180s</td></tr><tr><td>CNN</td><td>Fashion</td><td>MNIST</td><td colspan="4">0.9937 0.2535 0.6787 60s 60s 120s</td></tr><tr><td>CNN+rehearsal</td><td>Fashion</td><td>MNIST</td><td colspan="4">0.013 0.9262 0.521 120s 60s 180s</td></tr><tr><td>Ours</td><td>MNIST</td><td>Fashion</td><td colspan="3">0.907 0.9728 0.985</td><td>6s</td><td>6s 112s</td></tr><tr><td>Ours</td><td>Fashion</td><td>MNIST</td><td colspan="3">0.9648 0.9272 0.9842</td><td>6s</td><td>6s 112s</td></tr></table>

Peer review (iclr2020):
The authors propose an ensemble model with deep networks. The authors suggest its relevance for incremental learning. The proposed model is evaluated appropriately. There are many issues with the work. The presentation and description needs a lot of improvement. What exactly do the authors mean by rehearsal? It is confusing term. *... architecture contains two layers of neural networks and only the neural network in final layer is exposed to rehearsal ...*. Networks themselves have layers, so what does layers of networks mean here? The language needs a lot of improvement. Assuming that rehearsal means re-training or tuning or some sorts, the authors propose to split the network into two parts and retrain the second set of layers. How is this fundamentally different from retraining last few layers which is what most state of the art methods do? Further, architecture wise how is the proposal different from learning a layer of length C with softmax activation output for each class-specific index? What are we gaining in terms of learnability here? As per description only a fraction of points from the rest of the classes are utilized as negative examples for learning a given class? This fraction seems very important/sensitive? What is its influence in evaluations? When a new class comes in in the form of incremental learning, one needs to retrain all the class specific models! Unlike claimed in the introduction, how is this small rehearsal or how is this different from retraining like in the conventional case? In section 3.1, we still have a vector space, and nothing really special. Not need to tag them this separately! For tables 2 and 3 we need confidence intervals for better interpreting the performance.