Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: [Summary]
This paper proposes a new method for overcoming catastrophic forgetting in continual learning, based on distribution-based regularization using the sliced Cramer distance, i.e. Sliced Cramer Preservation (SCP). Unlike previous work on catastrophic forgetting, this paper tackles unsupervised learning scenarios as well as supervised learning. They evaluate the proposed SCP on permutated MNIST, sequential learning in autoencoder task, and sequential learning for segmentation. 

[Pros]
- This paper tackles unsupervised learning scenarios beyond the classification on benchmark datasets.
- This paper employes sliced Cramer distance with theoretical justification.
- The analysis on EwC and MAS in terms of geometric view
- Experimental results look promising.

[Cons]  
- Even if MAS[1] describes the details on synaptic concept and Hebbian rule, many readers might be not familiar with synaptic or neuro-science terms. So, in prelimiary session, more explanation can help readers to understand.
- In addition to permute-MNIST, it is required to be evaluated on more conventional tasks such as MNIST->SVHN or CIFAR-10, 100 datasets. Also, more recent work should be compared such as IMM [2] and PGMA [3] for supervised learning scenarios.
- All graph result figures can be improved for enhancing legibility. In Figure 6, In particular, the subtitle of Summer case confuses me.
 
[1] Aljundi et al. Memory Aware Synapses: Learning what (not) to forget, ECCV 2018.
[2] Lee et al. Overcoming Catastrophic Forgetting by Incremental Moment Matching, NIPS 2017.
[3] Hu et al. Overcoming Catastrophic Forgetting for Continual Learning via Model Adaptation, ICLR 2019.
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: I think the paper is written quite well, and the approach makes a lot of sense. I think the idea of replacing generalizing the KL to sliced Cramer distance is quite interested. The authors put in some effort in explaining why they propose this alternative distance between distributions. 

I think overall this is a great paper, very informative. If I need to nitpick, I think the experimental section relies heavily on MNIST (e.g. permuted MNIST, auto-encoding MNIST), which I think is not a hard enough task. It is though widely used in the continual learning community, though maybe it should not anymore. But I think given the reliance of the field on these datasets it makes sense, plus I think the strength of the paper is not in the empirical evaluation but rather in the derivation of the method. 

I think while the authors put quite a bit of effort in explaining the difference between KL and Cramer distance, I would have appreciated a even more detailed exposition. I think the difference between these metrics is not well understood by the majority in the community. 
review_title: Official Blind Review #1

Meta-review: The paper addresses an important problem (preventing catastrophic forgetting in continual learning) through a novel approach based on the sliced Kramer distance. The paper provides a novel and interesting conceptual contribution and is well written. Experiments could have been more extensive but this is very nice work and deserves publication.
Decision: Accept (Spotlight)