ConVAEr: Convolutional Variational AutoEncodeRs for incremental similarity learning

Abstract: Due to catastrophic forgetting, incremental similarity learning in neural networks remains an open challenge.   Previous  work  has  shown  that  keeping image  exemplars during  incremental  similarity  learning  is  effective  for  preserving  base knowledge (past learnt features and embeddings).   It is also generally accepted that the output layers learn more task-specific feature embeddings during the later training stages compared to the input layers’ general features earlier on. Building on these insights, we start by freezing the input layers of a neural network.  We then investigate the viability of generating “embedding” exemplars from a VAE that can protect base knowledge in the intermediate to output layers of the neural networks.  These generated exemplars replace the necessity for retaining images from previously learned classes. We experimented with three metric learning loss functions on the CUB-200 and CARS-196 in an incremental similarity learning setup.  We train different VAEs to generate exemplars from the intermediate convolution layers and linear output layers. We use these generated exemplars to rep-resent base knowledge. We compared our work to a previous technique that stores image exemplars.  The comparison is done for base knowledge, new knowledge and average knowledge preservation as metrics. The results show that generating exemplars from the linear and convolutional layers retained the highest ratio of base knowledge.  We note that using embeddings from the linear layers leads to better performance on new knowledge than convolutional embeddings.  Overall our methods yield better average knowledge performance across all experiments. These results support the view that for incremental similarity learning to overcome catastrophic forgetting, emphasis can be placed on learning embedding exemplars for intermediate to output layers. Further, we note that most incremental similarity learning for new classes depends on the linear layers rather than the convolutions. Further investigation into the relationship between transfer learning and similarity learning and the protection of intermediate layer embedding space for catastrophic forgetting is required.

Reviews and author responses:

This paper investigates the viability of generating “embedding” exemplars from a VAE that can protect base knowledge in the intermediate to output layers of the neural networks. Experiments on the CUB-200 and CARS-196 datasets with an incremental similarity learning setup are conducted. Generally, I think the novelty of this paper is somehow limited. Furthermore, It would strengthen the paper if there is some theory showing that why using the proposed CON-VAE can boost the performance. The motivation of using VAE seems not very clear.

The authors should claim the motivation of the tested datasets (CUB-200 and CARS-196). To my knowledge, these datasets are used for test fine-grained classification tasks. Also, why only metric learning tests are evaluated in the experiments? How about its performance on other tasks?

Will be good if the author can articulate other impactful contributions of this approach beyond the CON-VAEr selling point. My sense is that there may be other good properties of this approach. More analyzes should be conducted.

Page 4, Paragraph 2: “combine ideas from Rannen et al. (2017) and Rebuffi et al. (2017)”. This might be a typo. Overall, although this research topic seems an interesting problem, the novelty of the paper is somehow limited.

The authors propose some changes to an existing approach called FullVAE for incremental metric learning for the problem of catastrophic forgetting. Both ConVAER and FullVAE use VAEs for generating feature representation prototypes that could be passed through the intermediate layers of a network. The proposed changes are the position of feeding the generated prototypes from the VAEs to the network and a modified VGG network instead of a simple ConvNet. They evaluate their method on 2 real-world datasets and show improvement compared to the baselines. Weaknesses: 
1. The evaluation of this work is weak. The reviewed literature seems not to be very recent. There are only a few papers from the 2020 or 2021 referenced. The authors are comparing their method to iCarl from 2017, and FullVAE (which again does not compare to works after iCarl). Recent SOTA works in incremental learning such as PODNet (Douillard et al., 2020) and CCIL (Mittal et al., 2021) are not referenced or compared to. The method is not evaluated on standard incremental learning datasets such as Cifar100, ImageNet-100, ImagNet.

2. The paper is not well organized and structured. Parts of the method are in related work and instead, the experimental setup is written in the methodology section which should be in the results section.

3. The contributions of the work are very limited, compared to FullVAE the only difference is moving the VAEs to the Conv layers instead of FC layers and a new architecture.

4. The text, equations and figures are heavily borrowed and have a lot of overlap with FullVAE. Even if the authors are the same, this could count as self-plagiarism since the authors consider ConVAER an extension of FullVAE.

5. There is no ablation study to show whether the change in the architecture or the change in the position of VAEs is affecting the improvement.

Strength: 
1. The proposed changes are shown to improve FullVAE. 
2. The method is evaluated on 2 datasets and using 3 losses.
 Even though the suggested improvements by the authors seem to be effective, there are a lot of weaknesses in the work. I suggest rejection of this work since there are currently many issues with the current version.

This paper presents the method for incremental similarity learning using feature replay with VAEs. The experimental section investigates the impact of different loss functions on the final performance of the model. Yet, the discussion of the results lacks any insights which could be beneficial for the community.
The idea to replay features instead of images is not novel and the proposed method is just a minor modification of the method introduced in the paper [1] (which was published on arXiv 4.10.2021 -- two days before the deadline of ICLR submissions).
Due to the lack of novelty in this paper, it should be regarded as a form of ablation study for the paper [1] rather than a separate research paper.

[1] Incremental Class Learning using Variational Autoencoders with Similarity Learning, J. Huo et al.
[2] GDumb: A Simple Approach that Questions Our Progress in Continual Learning, A. Prabhu et al. (edited)  Strengths:
- The idea of exchanging data with its features to improve past experience replay is interesting and inline with the current trends in continual learning.

Weaknesses:
- The paper is poorly written and unstructured (e.g. the method is explained in the related work section, the related work section is not coherent) which makes this paper hard to read.
- The literature review is insufficient. Citing three loosely related papers is far below the standards of the ICLR conference. Suggesting that results from 2017 (iCARL, EEBL, EWC) are currently the state of the art is simply untrue. For reference see [2]
- This paper simply stacks a few previously introduced mechanisms/methods (VAE, feature replay, knowledge distillation, metric learning). However, there is no explanation why is it worth doing so and how these results could be useful for others. 
- The experimental details are confusing and should not be presented as incremental learning evaluations. The authors use two datasets with 196 and 200 classes. Yet, their experiments are performed with 160 classes. How these 160 classes are drawn and why the rest is abandoned? Additionally, the authors pretrain their models with 120 classes (75% of the whole dataset) in a standard (supervised learning) way and present the results for the remaining 40 classes learned incrementally. Such experiments are not challenging and bring no information about the method.  See the comments above.

Meta-review: The paper studies the problem of metric learning for handling catastrophic forgetting. All the reviewers recommended clear reject because of writing issues and lack of experimental investigation to support the ideas. The authors did not provide a rebuttal. Hence, the reviewers' opinion still remains the same. AC agrees with the reviewers and believes that the paper is not yet ready.

Decision: Reject