# Transfer Incremental Learning Using Data Augmentation
## 2 Related Work  
There has been an interest in incremental learning for a long time [11; 12; 13]. For example, methods have been proposed [14; 15; 16] to address this problem with the aim at bounding memory footprint (see criterion 3). These approaches perform learning one subset at a time using Support Vector Machines (SVMs). More precisely, a new SVM is trained for each batch of new data, exploiting previous support vectors. Since the latter are not conveying the full extent of previous data, the newly trained SVM suffers from _catastrophic forgetting_[2; 3], and thus violate criterion 2 defined in the introduction. Another incremental learning algorithm, called \"Learn++\" was introduced [17; 18]. This algorithm adds weak one-vs-all classifiers to accommodate new classes. Therefore, it may result in an excessive computational complexity and memory usage, disobeying criterion 3. It also needs training data for all classes to occur repeatedly, which contradicts criterion 1. Research showed also the possibility for the sequential learning of data [19]. However, this requires to choose a correct ordering of the whole dataset, which does not fulfil criterion 1. In [20], the authors proposed use of a pre-trained and unchanged DNN as a feature extractor followed by the Nearest Class Mean classifier (NCM). NCM summarises each class using the average feature vector of all examples observed for the class so far. Classification processes by assigning the class of the most similar average vector using a metric that can be learned from data. Compared to other parametric classifiers [20; 21; 22], NCM showed better performances in incremental learning scenarios. However, NCM gives a lower accuracy than state-of-art methods even when it uses all the dataset, hence does not fulfil criterion 2. In [23], a quite different incremental method called Budget Restricted Incremental Learning (BRIL) was proposed. BRIL combines \"transfer learning\" [7; 8] with binary associative memories. A pre-trained DNN is used as feature extractor, as mentioned in [20], while binary associative memories act as a classifier. A product random sampling is performed as an intermediate between the pre-trained DNN and the classifier. Despite being compliant with criteria 1 and 3, the accuracy remains significantly lower than existing counterparts, which violates criterion 2. [PERSON] et al. [24] showed that new classes can be added to a multi-class classifier with limited impact on accuracy when the classifiers can be retrained from at least a small amount of data belonging to all classes. Using this, in [10] the authors proposed an incremental learning method called \"Incremental Classifier and Representation Learning\" (iCaRL), based on a trainable DNN feature extractor, followed by a single classification layer. The classification process is inspired by NCM: it computes the mean of feature vectors for each class, and assigns the label of the nearest prototype. However, memory usage can easily increase, especially when the dataset is made of high resolution images such as ImageNet, which may violate criterion 3. Moreover, the iCaRL method, when trained on data streams containing only a few classes at a time, provides low accuracy as shown in [10], hence iCaRL does not respect criterion 2. To reach good performances and a comparable accuracy to state-of-art methods, iCaRL thus needs to be trained over batches of data containing a large part of the dataset, which does not correspond to an incremental learning scenario and infringes 1. In this paper, we introduce TILDA that builds upon previously proposed work, attempting to cover all three criteria for efficient incremental learning. As in iCaRL and BRIL, TILDA uses a pre-trained DNN as feature extractor. TILDA also uses an NCM-inspired classifier over the feature vectors obtained from by the pre-trained DNN. Data augmentation is performed on both training and classification datasets, aiming to improve accuracy.