ON THE USE OF CONVOLUTIONAL AUTO-ENCODER FOR INCREMENTAL CLASSIFIER LEARNING IN CONTEXT AWARE ADVERTISEMENT

1 INTRODUCTION . Recently , deep neural networks ( DNNs ) have shown remarkable performance in several classification tasks in many domains such as speech , image , natural language processing , etc. , . Among deep networks , Deep Convolutional Neural Networks ( DCNNs ) are the most successful models for supervised image classification and set the state-of-the-art in many benchmarks competitions Lecun et al . ( 1998 ) , Krizhevsky ( 2009 ) where the numbers of image categories have already been defined . It is common that machine learning systems with neural networks such as DCNN are trained in batch mode in which training samples from all the classes are available Wu et al .. Hence , DCNN based learning framework fits well with the requirements of benchmark evaluations . However , in real-world applications , more and more image categories will be available as time goes on and network will need to learn new tasks . The drawback of DCNN is inability to learn new information without redoing the whole learning process using all old and new information . If previously learned information is not available at the time of adding new information , DCNN tends to forget old memories . This is referred to as catastrophic forgetting problem . Human brain has ways to learning new information without forgetting old memories . Here , our focus is on incremental learning process which is similar to the process in human biological brains . Human memory is composed of three storage systems : 1 ) encoding , 2 ) long-term memory and 3 ) recall networks . Encoding system convert new information to a particular form for storage by paying attention Dorta . And , long-term memory has two main parts which are episodic and semantic memory and it acts as long-term warehouse and stores memories related to important personal events , feelings , silly facts , etc. , . Finally , the third network is recall system which is able to find information stored in our brain and ” pull it out ” efficiently . Humans learn something new everyday and every moment which makes human brain to perform incremental learning process most of the time . To store information for long time in the brain , humans need to pay sufficient focus , time or attention on them at the time of learning . Furthermore , in order not to forget the previously learned knowledge , humans must have sufficient practise , usage which involves revisiting to the stored memory Kirby ( 2013 ) . If we compare DCNN with human brain , DCNN only has batch learning capability . It has no modules for information encoding , storing encoded information and recalling it for the purpose of practising or revisiting it later which are important for incremental learning process . In this paper , we integrate these missing modules to DCNN for incremental learning . One concern of incremental learning system is the need to store training samples of categories trained previously . Previous methods such as state-of-the-art iCaRL Rebuffi et al . ( 2017b ) and Roy et al . ( 2018 ) keep the old images for revisiting purpose . However , keeping original images long time in our CAA framework is not practical due to privacy and legal concerns . For example , it is not good to store negative images such as violence , accident scenes . In this paper , we address two issues : 1 ) fulfilling the privacy and legal requirement 2 ) dealing with catastrophic forgetting problem with simulation of human brain system . The experimental results show that proposed solution is effective in addressing privacy and legal concerns as well as deling with catastrophic forgetting problem to maintain system performance especially for CAA framework . 2 RELATED WORK . Recently , there has been increased interest in incremental learning or lifelong learning and several approaches have been proposed to solve the problem . The approaches can be categorized into 3 main groups . The approaches in first group store the small subsets of training data from the previously learned tasks . The stored exemplars are used in training the network when adding the new tasks to alleviate the catastrophic forgetting problem . Example of such method is iCaRL Rebuffi et al . ( 2017b ) which keeps a few exemplars to rehearse the previous tasks . Another recent method in this group is based on Gradient Episodic Memory ( GEM ) model Lopez-Paz & Ranzato ( 2017 ) . However the usage of stored exemplars in GEM based approach is different from iCaRL Rebuffi et al . ( 2017b ) which keeps the predictions of exemplars from past tasks invariant by means of distillation . GEM based approach define inequality constraints on loss to allow positive backward transfer for increasing performance on some preceding tasks . In second group , the approaches avoid storing any original training data from previously learned tasks . However , these approaches retain statistics of training samples such as means and standard deviations Kemker & Kanan ( 2018 ) of individual class distributions for tackling the catastrophic forgetting problem . Finally , the third group of algorithms totally avoid keeping any samples of previously learned tasks . Some of the approaches in this group tackle the catastrophic forgetting problem by dynamically expending the network for each new task Rusu et al . ( 2016 ) , Rebuffi et al . ( 2017a ) , Yoon et al . ( 2018 ) , Terekhov et al . ( 2015 ) . Although these approaches are simple , they have scalability issues as the size of network grows with the increasing numbers of tasks . To avoid the scalability issue , some recent studies explore regularization methods over the network parameters Zenke et al . ( 2017 ) , Kirkpatrick et al . or the study in Wu et al . investigates on generating synthesized images . Example of regularization method is Elastic Weight Consolidation ( EWC ) Zenke et al . ( 2017 ) , Kirkpatrick et al. , Lee et al . ( 2017 ) , Liu et al . ( 2018 ) , Chaudhry et al . ( 2018 ) based approach . EWC based approaches use a regularization term so that network parameters remain close to the parameters of the network trained for the previous classes . Similarly , Learning Without Forgetting ( LWF ) Li & Hoiem ( 2016 ) performs regularization on predictions instead of network weights . Our proposed approach falls under the second group and the prior work in this category is FearNet Kemker & Kanan ( 2018 ) which is brain-inspired algorithm for incremental class learning . The FearNet system involves three brain-inspired sub-modules : 1 ) short-term memory storage for recently learned knowledge which is inspired by the hippocampal complex ( HC ) of human brain 2 ) long-term memory storage inspired by the medical prefrontal cortex ( mPFC ) and 3 ) a sub-module to choose the memory system between HC and mPFC to use for a particular example . Catastrophic forgetting problem is addressed by using pseudorehearsal which allows the network to revisit the previous memories during incremental training . Although FearNet uses HC model as short-term storage and the model is erased after the information is transferred to long-term storage ( mPFC ) network . mPFC stores class statistics for pseudorehearsal in the form of mean and covariance matrix for each class to generate new exemplars ( pseudo-examples ) . In human ’ s brain , encoding is crucial step to creating new memory Mastin ( 2018 ) . Encoding process convert newly perceived knowledge into a construct for storage to recall it later . During encoding process , human pay more attention especially for memorable events . This causes neurons to fire more frequently , making the event to be encoded as a memory . For example , emotional events tend to increase attention and leading it to unforgettable memories . Hence , the more intense the attention , the more effective the encoding and this results the storing acquired knowledge for long time . In this paper , we propose the brain inspired incremental learning system and investigate the effect of quality of encoded information on incremental learning . We integrate Convolutional AutoEncoder ( CAE ) in incremental learning process for encoding knowledge . We show that our brain inspired model is efficient and CAE is useful for encoding for revisiting the stored knowledge later . The state-of-the-art system such as iCaRL Rebuffi et al . ( 2017b ) requires to storing original training examples for all the previously learned classes , making it challenging for privacy and legal requirements especially for CAA framework which has limitations in storing original images ( or exemplars ) owned by others . To address privacy issues , the study in Wu et al . uses images generated by Generative Adversarial Networks ( GAN ) to replace the exemplar set of original training samples . However GANs have limitations on image quality and mode dropping . GAN based method works relatively well on simple images , however , it is not easy to get realistic scene images such as plane crash ( for example ) which we are interested . In our approach , we take care of privacy and legal by integrating CAE in our system . 3 DATASETS . Experiments are conducted on the following datasets . CIFAR-100 contains 100 image classes . Each class has 500 training images and 100 testing images . Following the class-incremental benchmark protocol in iCaRL [ 22 ] on this dataset , 100 classes are arranged in a fixed random order and come in as P parts . Each part is with C = 100/P classes . A multi-class classifier is built with the first part that contains C classes . Then this classifier is adapted to recognize all 100 classes . IMDB-CAA We collect an IMage Database for Context Aware Advertisement ( IMDB-CAA ) . Most of these images are negative scenes with violence , accidents , gambling etc . The database has about 12,000 color or gray-scale images with all images are manually labeled and irrelevant images are removed . The categories of the images in IMDB-CAA are listed in Table 1 . Each category has 500 training images and 100 testing images . 4 PROPOSED BRAIN INSPIRED INCREMENTAL LEARNING SYSTEM . In this paper , we investigate a strategy to simulate the encoding and storing information as in human brain learning process . By encoding the original images using Convolutional Auto-Encoder ( CAE ) , the negative images become not easily viewable during storage . By using the incremental learning mechanism , only encoded information is kept for future learning once new data is available . In the following section we discuss about Convolutional Auto-Encoder ( CAE ) and presents its use to address privacy issues as well as to simulate human brain encoding mechanism . 4.1 CONVOLUTIONAL AUTO-ENCODER ( CAE ) . CAEs are very popular in deep learning research for unsupervised learning methods . CAEs can be used for extracting useful features from unlabelled data as well as for detecting and removing input redundancies to preserve only essential aspects of information for robust and discriminative representations Masci et al . ( 2011 ) , Turchenko & Luczak ( 2015 ) . Unsupervised training process of CAEs tends to avoid local minima and improves the network ’ s performance and stability Erhan et al . ( 2010 ) . As an auto-encoder , CAEs are based on encoder-decoder paradigm in which input data or image is mapped to a lower-dimensional space ( referred to as encoder part ) and then , encoded information is expended to regenerate the input data or image ( referred to as decoder part ) . As discussed above , CAEs have encoder and decoder parts which we need for simulating the human brain . Encoder part is useful for information encoding , storing encoded information . And , decoder part is for decoding stored information for practising or revisiting it in future . For DCNN to achieve the capabilities as those in human brain , we propose to integrate the auto-encoder , CAE , to DCNN . By doing so , we expect DCNN to achieve the capability of information encoding , storing , and practising or revisiting the stored information later in order to overcome the catastrophic forgetting problem of DCNN . To overcome the catastrophic forgetting problem and address privacy requirements , we propose to use CAEs to encode the original images and keep encoded pseudo-examples instead of keeping original images . There are several advantages of using CAEs over GANs . These include 1 ) possibility of recalling or regenerating high quality images from stored encoded information for future revisiting . 2 ) There is freedom to control on the size of the pseoduexemplars by designing autoencoder as required . If accuracy is more important we can design the network for larger pseoduexemplars . However , if the storage is very limited , we can build the network for smaller pseoduexemplars with the certain expense of system performance . One may argue that instead of using CAEs , we may use some encryption method . The focus in encryption and cryptography are security , encryption/decryption time , etc. , Mahajan & Sachdeva ( 2013 ) . It is common that security of the algorithm depends on the length of the key . The longer key length will always support to good security feature Oad et al . ( 2014 ) . And , longer key length also means larger file size . Hence , most of the encrypted file is larger than before it is encrypted . However , our focus is on privacy , storage , performance and simplicity for easy scalability . Storage and performance should be able to be adjusted according to requirement of the application . The CAE network involves 6 convolution layers and the first 3 layers are for encoder part and the last 3 layers are for decoder part . The numbers of filters in convolutional layers are set as { 16 , 8 , 8 , 8 , 8 , 16 } respectively . The decoder part is a mirror of the encoder part . We use binary cross entropy as loss function and adaptive learning rate method for optimization function . We use subset of ImageNet database Deng et al . ( 2009 ) consisting of 150,000 images to train the CAE network . We set the heights and widths of the images as 224 to use as input for training the network . We use encoder part for encoding images and obtain pseudo-examples . Then , we employ decoder part to regenerate images from stored pseudo-examples . Figure 1 shows the comparisons of CAE regenerated images from different pseudo-exemplar sizes with original images and images generated by GAN . In Figure 1 , we gradually reduce the size of pseudo-examples . The reconstructed image becomes blur when we reduce the size . At the 16.6 % reduced size we notice that edges of the object becomes sharper with increased noise . This phenomena can be explained by the findings in image enhancement using deep autoencoders . In image enhancement task , there is a trade-off between denoising capability and perceived sharpness of enhanced image Lore et al . ( 2016 ) . The model with higher denoising capability generates smoother edges and generated images becomes less sharp which eventually lose structural information . It can be seen in 3rd row from bottom in Figure 1 for 16.6 % pseudo-exemplar size that object edges appears sharper at the cost of having more noise if we compare these images with reconstructed images of other pseudo-exemplar sizes . In fact , the characteristics such as sharper edges and textures are desirable properties especially for image classification with Convolutional Neural Network ( CNN ) .

Summary: The paper adresses the problem of incremental learning when data from new classes are available as a stream and one wants to be able to update to learn new observed classes without forgetting the older ones. There is a budget issue here and one does not want to just keep the whole training set of all known previously observed classes but rather one wants to consider a maximum memory budget allowed to store what is necessary for an optimal incremental learning (typical examples, statistics etc). There is also a privacy issue preventing from storing original training samples.