ConVAEr: Convolutional Variational AutoEncodeRs for incremental similarity learning

1 INTRODUCTION . In machine learning , incremental learning is the process of updating a model as new data becomes available or extended to support further tasks . An incrementally trained model should ideally retain previously attained knowledge while incorporating any new knowledge made available as it trains Syed et al . ( 1999 ) ; Polikar et al . ( 2001 ) . Many machine learning algorithms can not retain prior knowledge or do so in an unsatisfactory manner . Models that do not incrementally learn new tasks whilst retaining prior knowledge suffer from catastrophic forgetting . Catastrophic forgetting typically occurs during training on new data that contains no or highly imbalanced examples drawn from prior learned distributions McCloskey & Cohen ( 1989 ) ; Ratcliff ( 1990 ) . Catastrophic forgetting in deep neural networks and virtually all of the tasks supported by them remains an open research problem Goodfellow et al . ( 2013 ) ; Fernando et al . ( 2017 ) ; Robins ( 1995 ) ; Draelos et al . ( 2017 ) . Historically , analyses have been focused almost entirely on incremental supervised classification in multi-layer perceptrons ( MLP ) neural networks such as typically encountered in computer vision tasks . However , there persists a lack of evidence detailing the degree to which similarity learning tasks and the underpinning pair mining and loss functions are affected by catastrophic forgetting . Further , most techniques aimed at overcoming catastrophic forgetting in deep neural networks have been engineered with classification tasks in mind Rannen et al . ( 2017 ) ; Rebuffi et al . ( 2017 ) ; Choi et al . ( 2019 ) . Previous work by Huo & van Zyl ( 2021 ) compared four state-of-the-art algorithms for reducing catastrophic forgetting during incremental learning . These algorithms included Fully Connected VAEs ( FullVAE ) , Elastic Weight Consolidation Kirkpatrick et al . ( 2017 ) , Encoder-Based lifelong learning Rannen et al . ( 2017 ) , and incremental Classifier and Representation Learning iCaRL Rebuffi et al . ( 2017 ) . Their work analyzed several loss functions on MNIST , EMNIST , FashionMNIST , and CIFAR-10 . All the considered techniques were effective but to an unsatisfactory extent with FullVAE and iCaRL shown to be the most robust across numerous loss functions . However , given that the datasets used are somewhat trivial , these results only partially indicate what is to be expected on real-world data . This paper builds on this prior body of research and presents conclusions on the influence of catastrophic forgetting in incremental metric learning . We approximated the catastrophic forgetting test procedure of Kemker et al . ( 2018 ) which is described for classification tasks . Our work examines three-loss functions , contrastive , centre and triplet loss , using CUB200 Welinder et al . ( 2010 ) and CAR196 Krause et al . ( 2013 ) in metric learning . We contrast the current state of the art for catastrophic forgetting during incremental metric learning , FullVAE and iCaRL , to our solution ConVAEr . ConVAEr uses convolutional Variational Autoencoders ( VAE ) to generate representations fed into the convolutional layers to supplement previously seen knowledge without regenerating entire images or the need to keep a collection of previous data . We present results that allow the reader to explore which technique retains the most base knowledge , new knowledge , and overall knowledge during incremental metric learning . 1 . We show that our method yields better average knowledge retention across all experiments . 2 . We support the importance of keeping prior knowledge or data during incremental similarity learning . 3 . We demonstrate that injected VAE generated representations work as well as images exemplars . 4 . We show that intercepting exemplars from the convolutional layers retained the highest ratio of base knowledge . 5 . We note that using embeddings from the linear layers leads to better performance on new knowledge than convolutional embeddings . 2 RELATED WORK . 2.1 CATASTROPHIC FORGETTING . Goodfellow et al . ( 2013 ) investigated catastrophic forgetting in gradient-based neural networks used for classification . The results showed that various combinations of activation function and learning were affected differently by catastrophic forgetting . The work by Rannen et al . ( 2017 ) demonstrated the problem of catastrophic forgetting in deep convolutional neural networks ( DNN ) AlexNet . They highlighted the classification performance dropped in a previously learned task when a DNN is fined-tuned for newer classification tasks . The authors proposed using lightweight autoencoders to “ store ” the embedding learned by the base network . A new autoencoder is trained and kept after the network learns each new task . The method significantly reduced catastrophic forgetting during incremental classification . Choi et al . ( 2019 ) proposed the use of an autoencoder-based incremental metric learning method for classification without the use of a softmax classification layer . The work is premised on the notion of a metric-based classification method , nearest-class-mean ( NCM ) Mensink et al . ( 2013 ) . A pre-trained fixed network was used as a feature extractor and the autoencoder is trained on the feature embeddings . To overcome catastrophic forgetting while fine-tuning the autoencoder to new classes , the authors use regularization techniques : Synaptic Intelligence ( SI ) Zenke et al . ( 2017 ) , and Memory Aware Synapses ( MAS ) Aljundi et al . ( 2018 ) . The methods demonstrated good memory retention without the need to train on older data . Our work used VAE for knowledge preservation to supplement the update of the network during incremental similarity learning whereas the authors used a fixed network to supply the autoencoder for incremental classification and not similarity learning . 2.2 INCREMENTAL CLASSIFIER AND REPRESENTATION LEARNING ( ICARL ) . Incremental Classifier and Representation Learning ( iCaRL ) is a method proposed by Rebuffi et al . ( 2017 ) for reducing catastrophic forgetting . It is reported to learn classes incrementally over a longer period than other methods . iCaRL relies primarily on storing exemplars of previously seen classes . Each class ’ s exemplar set is constructed , using the herding algorithm , from the k closest images to the class ’ s mean representation . The stored exemplars are used to supplement the incremental learning phase of new classes using knowledge distillation . Since iCaRL does not directly translate to similarity learning tasks we implemented the modified version as described in the paper by Huo & van Zyl ( 2021 ) . 2.3 ENCODER-BASED LIFELONG LEARNING . Rannen et al . ( 2017 ) proposed Encoder-Based Lifelong Learning ( EBLL ) for incremental learning in classification tasks . The method modifies how the convolutional layers of a network are updated . After each incremental learning task , a single autoencoder is trained to reconstruct the “ images ” at the output of the convolutional layers . The reconstructed images are passed through the network ’ s remaining fully connected layers to calculate their resulting classification loss . The reconstruction loss , together with the classification loss , is used to update the autoencoder . The previous task ’ s classification layer ( output layer ) is detached for each new incremental learning task , and a new classification layer is attached . A frozen copy of the previous optimal network is made before training the next incremental task . The new images are passed through both the new and frozen network during training for the new task to calculate the distillation loss and added to the new classification loss and the encode loss to update the new network Rannen et al . ( 2017 ) . However , only the new network is updated . For updating the network ’ s weights , the images ’ convolutional layers outputs of the new and frozen network are passed into the autoencoder up to the bottleneck layer , where the mean square error is calculated and added to the classification loss and propagated through the autoencoders ’ network ’ s weights . This process constrains the weight update of the convolutional network layers to compromise between new and old tasks . 2.4 FULLY CONNECTED VAE . Huo & van Zyl ( 2021 ) proposed using a VAE to represent images in a representation that can be passed through intermediate layers in the network . The authors focused on preserving knowledge from the fully connected layers ( flattened output from the last CNN layer ) . Their method was shown to outperform iCaRL and other methods using similar test metrics to Kemker et al . ( 2018 ) during incremental similarity learning . The concern with their work was that they experimented using a simple network and a variety of simple datasets that are not representative of real-world performance . 2.5 OUR APPROACH ( CONVAER ) . Rannen et al . ( 2017 ) constrain the weights of the feature extraction layers ( convolutions ) that were optimal for previous tasks with an autoencoder . The solution is robust when reusing the feature extraction layers ( convolutions ) on new tasks . Each task is tested independently from the others with its classification layer—the approach yields encouraging results by finding a middle-ground across the base and new knowledge . Huo & van Zyl ( 2021 ) , however , have previously demonstrated that this approach is not practical for incremental metric learning . The iCaRL method by Rebuffi et al . ( 2017 ) depends on the storage of a large number of exemplars . As reported , the performance of iCaRL decreases with time as the number of exemplars per class is reduced to accommodate new classes . Eventually , the stored exemplars are not sufficient to rep- resent all classes . Further , retaining previous exemplars is hardly akin to overcoming catastrophic forgetting . Instead , this represents retaining previous data rather than previously learned knowledge . However , iCARL does present as a state-of-the-art method and , as such , provides a reasonable baseline for comparison . Huo & van Zyl ( 2021 ) combine ideas from Rannen et al . ( 2017 ) and Rebuffi et al . ( 2017 ) . They train a new variational autoencoder ( VAE ) for each class . The VAEs learn representations at the end of the convolutional layers . The use of VAEs allows them to generate examples from previously seen classes . The method requires that the convolutional layers be frozen after initial training or pretrained frozen convolutional layers from a base model are used . The VAEs generate representations from previously seen classes combined with the new classes during incremental metric training to perform incremental metric learning . The work of Huo & van Zyl ( 2021 ) more closely aligned with the intent of overcoming catastrophic forgetting . That is , using previously learned representations and knowledge for future metric learning tasks . We extend on the work by Huo & van Zyl ( 2021 ) by using the authors method and moving the VAE to intercept input to the convolution layer instead of at the input to the fully connected layers show in figure 2 . The autoencoder ’ s reconstruction loss function is required to vary depending on the network ’ s interception layer ’ s activation function . For example , in our case , the convolutional interception layers use ReLU activation , and consequently , we used the Binary Cross-Entropy objective function to determine the reconstruction errors summed with the Kullback-Leibler divergence . The loss function to update the VAEs is given as : LV AE = − 1 N N∑ i=1 yi · log ( p ( yi ) ) + ( 1 − yi ) · log ( 1 − p ( yi ) ) + 1 2 ( exp ( σ2 ) + µ2 − 1 − σ2 ) , ( 1 ) where σ2 is the variance of the entire dataset and µ is the mean . The first term is the Binary CrossEntropy reconstruction loss , and the second term is the Kullback–Leibler divergence . We also make use of the angle-wise distillation loss seen the paper by Huo & van Zyl ( 2021 ) on the examples generated from the VAEs . This usage with the network update during incremental learning , is similar to the approach taken by iCaRL Rebuffi et al . ( 2017 ) . The incremental learning step makes use of the following loss function defined as : Lincremental = lmetric learning + λdistil ∗ LA ( 2 ) , where lmetric learning are the three possible metric learning functions , we state below . LA is the angle-wise distillation for metric learning . λdistil is the importance that we place on the angle-wise distillation loss . The student output is the convolution image output we get from the network incrementally training at each new train step . The teacher output is the convolution output image we get from the frozen network before performing the new incremental train step . For center loss , we used a different distillation loss as defined in Hinton et al . ( 2015 ) defined as : LD ( to , so ) = − l∑ i=1 tiolog ( s i o ) ( 3 ) where l is the number of labels , tio and soi are the modified versions of the teacher model outputs and the current student model outputs . So the modified loss function during the incremental step for center loss is defined as : Lincremental = lmetric learning + λdistil ∗ LD , ( 4 ) where LD is defined above , and the rest remains the same .

Summary: This paper presents the method for incremental similarity learning using feature replay with VAEs. The experimental section investigates the impact of different loss functions on the final performance of the model. Yet, the discussion of the results lacks any insights which could be beneficial for the community. The idea to replay features instead of images is not novel and the proposed method is just a minor modification of the method introduced in the paper [1] (which was published on arXiv 4.10.2021 -- two days before the deadline of ICLR submissions). Due to the lack of novelty in this paper, it should be regarded as a form of ablation study for the paper [1] rather than a separate research paper.  [1] Incremental Class Learning using Variational Autoencoders with Similarity Learning, J. Huo et al. [2] GDumb: A Simple Approach that Questions Our Progress in Continual Learning, A. Prabhu et al. (edited)