Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: I quite liked the revival of the dual memory system ideas and the cognitive (neuro) science inspiration. The paper is overall well written and tackles serious modern datasets, which was impressive, even though it relies on a pre-trained, fixed ResNet (see point below).

My only complaint is that I felt I couldn’t understand why the model worked so well. A better motivation for some of the modelling decisions would be helpful. For instance, how much the existence (and training) of a BLA network really help — which is a central new part of the paper, and wasn’t in my view well motivated. It would be nice to compare with a simpler baseline, such as a HC classifier network with reject option. I also don’t really understand why the proposed pseudorehearsal works so well. Some formal reasoning, even if approximate, would be appreciated.

Some additional comments below:

- Although the paper is in general well written, it falls on the lengthy side and I found it difficult at first to understand the flow of the algorithm. I think it would be helpful to have a high-level pseudocode presentation of the main steps.

- It was somewhat buried in the details that the model actually starts with a fixed, advanced feature pre-processing stage (the ResNet, trained on a distinct dataset, as it should). I’m fine with that, but this should be discussed. Note that there is evidence that the neuronal responses in areas as early as V1 change as monkeys learn to solve discrimination tasks. It should be stressed that the model does not yet model end-to-end learning in the incremental setting.

- p. 4, Eq. 4, is it really necessary to add a loss for the intermediate layers, and not only for the input layer? I think it would be clearer to define the \mathcal{L} explictily somewhere. Also, shouldn’t the sum start at j=0?
review_title: Interesting cognitive science take, tested on modern datasets

final_score: rating: 6: Marginally above acceptance threshold
review_content: 
This paper addresses the problem of incremental class learning with brain inspired memory system. This relies on 1/ hippocampus like system relying on a temporary memory storage and probabilistic neural network classifier, 2/ a prefrontal cortex-like ladder network architecture, performing joint autoencoding and classification, 3/ an amygdala-like classifier that combines the decision of both structures. The experiments suggests that the approach performs better than state-of-the-art incremental learning approaches, and approaches offline learning.
The paper is well written. The main issue I have with the approach is the role of the number of examples stored in hippocampus and its implication for the comparison to state-of-the art approaches.
Comments:
It seems surprising to me that the network manages to outperform other approaches using such a simplistic network for hippocampus (essentially a Euclidian distance based classifier). I assume that the great performance is due to the fact that a lot of examples per classes are stored in hippocampus. I could not find an investigation of the effect of this number on the performance. I assume this number corresponds to the mini-batch size (450). I would like that the authors elaborate on how fair is the comparison to methods such as iCaRL, which store very little examples per classes according to Fig. 2. I assume the comparison must take into account the fact that FearNet stores permanently relatively large covariance matrices for each classes.
Overall, the hippocampus structure is the weakness of the approach, as it is so simple that I would assume it cannot adapt well to increasingly complex tasks. Also, making an analogy with hippocampus for such architecture seems a bit exaggerated.

review_title: Suprizingly good results with an rather simple architecture. Is the comparison to SotA fair??

final_score: rating: 7: Good paper, accept
review_content: Quality: The paper presents a novel solution to an incremental classification problem based on a dual memory system. The proposed solution is inspired by the memory storage mechanism in brain.

Clarity: The problem has been clearly described and the proposed solution is described in detail. The results of numerical experiments and the real data analysis are satisfactory and clearly shows the superior performance of the method compared to the existing ones.

Originality: The solution proposed is a novel one based on a dual memory system inspired by the memory storage mechanism in brain. The memory consolidation is inspired by the mechanisms that occur during sleep. The numerical experiments showing the FearNet performance with sleep frequency also validate the comparison with the brain memory system.

Significance: The work discusses a significant problem of incremental classification. Many of the shelf deep neural net methods require storage of previous training samples too and that slows up the application to larger dataset. Further the traditional deep neural net also suffers from the catastrophic forgetting. Hence, the proposed work provides a novel and scalable solution to the existing problem.

pros: (a) a scalable solution to the incremental classification problem using a brain inspired dual memory system
          (b) mitigates the catastrophic forgetting problem using a memory consolidation by pseudorehearsal.
          (c) introduction of a subsystem that allows which memory system to use for the classification

cons: (a)  How FearNet would perform if imbalanced classes are seen in more than one study sessions?
          (b) Storage of class statistics during pseudo rehearsal could be computationally expensive. How to cope with that?
          (c) How FearNet would handle if there are multiple data sources?
review_title: The paper presents an interesting problem of incremental classification inspired by the dual memory-system of brain. I feel the paper  explicitly describes the problem and explains the proposed methodology in great detail.

Meta-review: A novel dual memory system inspired by brain for the important incremental learning and very good results.
Decision: Accept (Poster)