Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: In continual learning settings, one of the important technique for avoiding catastrophe forgetting is to replay data points from the past. For memory efficiency purposes, representative samples can be generated from a generative model, such as GANs, rather than replaying the original samples which can be large in number. It is argued that GANs generate new samples which may not belong exactly to one of the classes, so a new generative model is proposed. Experimental results are appealing.

I have some basic concerns. 
(1) First of all, the idea of generating new samples for replay is not motivated well enough; in the experiments, even the model replaying on original images takes memory in few hundred MBs. 
(2) Second, why is there a need for generating new samples? Why not select a representative subset of the original set of samples. There are tons of methods to select samples informatively, within the paradigm of active learning and beyond.
(3) Why not use conditional-GANs for generating data points specific to a class?

Since autoencoders have a problem of generating blurry images, the proposed autoencoding model borrows ideas from neural style transfer algorithm. Specifically, besides the reconstruction loss, content loss is introduced utilizing the idea of content transfer from the neural style transfer algorithm. It seem to make sense in reference to Fig. 2, though it needs better explanation.   

For memory efficiency of the autoencoder itself, it is stores centroids and covariances of the episodes (representations of images), from which pseudo-encoded episodes are generated. This doesn't seem very principled, and may or may not work in different empirical settings, depending upon the intrinsic dimensionality of images, and the difficulty of the task. 

review_title: Problem is not well motivated - some basic concerns - model is not principled enough 

final_score: rating: 6: Marginally above acceptance threshold
review_content: ##########################################################################

Summary:

 
The paper proposes an approach which trains autoencoders with Neural Style Transfer to encode and store images. The method is applied to the problem of continual learning to overcome the catastrophic forgetting and memory limitation on the storage data. The authors report that the presented approach increases the classification accuracy by 13-17% over SOTA methods 

##########################################################################

Reasons for score: 

 
Overall, I vote for accepting. The paper presents nice ideas, outperforms SOTA methods and is clearly written. The contribution could have been stronger with a more detailed evaluation and better presentation.
 
##########################################################################

Pros: 

 
1. I really like the idea of using NST to train the autoencoder. 

2. Clarity of the paper.
 
 
##########################################################################

Cons: 

1. The system uses a pretraied Style Transfer Network with Imagenet. Does it offer an unfair advantage over other approaches?
2. It would be great to compare the NST Autoencoder with Variational or Denoising Autoencoder. 
3. The validation has been performed using low-resolution images (32x32)
4. The improvement on Imagnet-50 A5 against iCaRL-S is 6.28. So, the difference should be 6.28 and not 17.4. 
5. It would be great to see the results for A30 and A50 with imagenet as it was done by (Ostapenko et.al. 2019)
6. Figure 4 shows the accuracy on ImageNet-50 with different budgets. It seems that the accuracy is still increasing, did you tried with larger values?

Minor points: 
- Table 1 - it would be great to include the reference paper of each method


 
##########################################################################

Questions during rebuttal period: 

 
Please address and clarify the cons above 

 
#########################################################################

Some typos: 

(1) Page 5: corariance -> covariance
(2) Page 7: the the accuracy --> the accuracy


Update after rebuttal:
My initial concerns were clarified by the authors and the paper substantially improved. 
review_title: Good Paper

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary: 

The paper tackles catastrophic forgetting for continual learning. It proposes to train autoencoders with Neural Style Transfer to generate previous images. The encoded episodics can be converted into centroids and covariances matrices in order to save memory usage. It shows significant improvements in the experimental parts.

Strengths:

- The results on tiny datasets (MNIST, SVHN and CIFAR-10) are promising. On Tiny ImageNet-50, the performance compared to other methods looks good, but it is relatively bad compared to some other datasets.
- It is well written and easy to follow.


Concerns:

- The overall idea is very close to what FearNet proposed by using autoencoders. The main difference is to use autoencoders on images instead of features.  Neural Style Transfer is helpful to generate more realistic images, therefore it makes sense to help during incremental learning to replay previous knowledge. However, it seems obvious improving image quality will result in better performance for methods based on replay.

- From the experimental results compared to other methods, it is still limited to relative small resolution images and a few tasks. It would be interesting to see the performance on more challenging datasets with large resolution and more tasks.

review_title: The idea of paper is quite similar to existing method FearNet using autoencoders. The experiments are limited to small images and  a few tasks.

Meta-review: This paper uses an autoencoder with neural style transfer to generate images from previously seen classes to avoid catastrophic forgetting in continual learning.

While reviewers had some concerns about the paper (experiments on high-resolution images, comparison with FearNet), authors have addressed all the concerns. R1's concern about the motivation for generation instead of replaying actual images is not necessary since this is not the first work to use generative replay.
Decision: Accept (Poster)