Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper addresses lifelong learning setting under resource constraints, i.e. how to efficiently manage the storage and how to generalise well with a relatively small diversity of prior experiences. The authors investigate how to avoid storing a lot of original training data points while avoiding catastrophic forgetting at the same time.
The authors propose a complex neural network architecture that has several components. One of the components is a variational autoencoder with discrete latent variables, where the recently proposed Gumbel-softmax distribution is used to efficiently draw samples from a categorical distribution (Jang et al ICLR 2017). Discrete variables are categorical latent variables using 1-hot encoding of the class variables. In fact, in the manuscript, the authors describe one-hot encoding of c classes as l-dimensional representation. Why is it not c-dimentional? Also the class probabilities p_i are not defined in (7). 
This design choice is reasonable, as autoencoder with categorical latent variables can achieve more storage compression of input observations in comparison with autoencoders with continuos variables. 
Another component of the proposed model is a recollection buffer/generator, a generative module (alongside the main model) which produces pseudo-experiences. These self generated pseudo experiences are sampled from the buffer and are combined with available real samples during training to avoid catastrophic forgetting of prior experiences. This module is inspired by episodic training proposed by Lopez-Paz and Ranzato in ICLR2017 for continual learning. In fact, a recollection buffer for MNIST benchmark has 50K codes to store. How fast would it grow with more tasks/training data? Is it suitable for lifelong learning? 

My main concern with this paper is that it is not easy to grasp the gist of it. The paper is 11 pages long and often has sections with weakly related motivations described in details (essentially it would be good to cut the first 6 pages into half and concentrate on the relevant aspects only). It is easy to get lost in unimportant details, where as important details on model components are not very clear and not structured. Second concern is limited novelty (from what I understood). 



review_title: This paper presents important and timely problem of lifelong learning under resource constraints; the manuscript lacks clarity and structure; limited novelty.

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents an approach to lifelong learning with episodic experience storage under resource constraints. The key idea of the approach is to store the latent code obtained from a categorical Variational Autoencoder as opposed to the input example itself. When a new task is learnt, catastrophic forgetting is avoided by randomly sampling stored codes corresponding to past experience and adding the corresponding reconstruction to a batch of data from a new problem. The authors show that explicitly storing data provides better results than random sampling from the generative model. Furthermore, the method is compared to other techniques relying on episodic memory and as expected, achieves better results given a fixed effective buffer size due to being able to store more experience.

While the core idea of this paper is reasonable, it provides little insight into how episodic experience storage compares to related methods as an approach to lifelong learning. While the authors compare their method to other techniques based on experience replay, I feel that a comparison to other techniques is important. A natural choice would be a model which introduces task-specific parameters for each problem (e.g.  (Li & Hoiem, 2016) or (Rusu et al., 2016)).

A major concern is the fact that the VAE with categorical latents itself suffers from catastrophic forgetting. While the authors propose to "freeze decoder parameters right before each incoming experience and train multiple gradient descent iterations over randomly selected recollection batches before moving on to the next experience", this makes the approach both less straight-forward to apply and more computationally expensive. 

Moreover, the authors only evaluate the approach on simple image recognition tasks (MNIST, CIFAR-100, Omniglot). I feel that an experiment in Reinforcement Learning  (e.g. as proposed in (Rusu et al., 2016)) would provide more insight into how the approach behaves in more challenging settings. In particular, it is not clear whether experience replay may lead to negative transfer when subsequent tasks are more diverse.

Finally, the manuscript lacks clarity. As another reviewer noted, detailed sections of weakly related motivations fail to strengthen the reader's understanding. As a minor point, the manuscript contains several grammar and spelling mistakes.
review_title: Deep Lifelong learning with recollections under resource constraints.

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes an architecture for efficient deep lifelong learning. The key idea is to use recollection generator (autoencoder) to remember the previously processed data in a compact representation. Then when training a reasoning model, recollections generated from the recollection generator are used with real-world examples as input data. Using the recollection, it can avoid forgetting previous data. In the experiments, it has been shown that the proposed approach is efficient for transfer knowledge with small data compared to random sampling approach.

It is an interesting idea to remember previous examples using the compact representation from autoencoder and use it for transfer learning. However, I think the paper would be improved if the following points are clarified.

1. It seems that reconstructed data from autoencoder does not contain target values. It is not clear to me how the reasoning model can use the reconstructed data (recollections) for supervised learning tasks. 

2. It seems that the proposed framework can be better presented as a method for data compression for deep learning. Ideally, for lifelong learning, the reasoning model should not forget previously learned kwnoledge embeded in their weights. 
However, under the current architecture, it seems that the reasoning model does not have such mechanisms.

3. For lifelong learning, it would be interesting to test if the same reasoning model can deal with increasing number of tasks from different datasets using the recollection mechanisms.

 




review_title: Recollections for efficient deep lifelong learning

Meta-review: The reviewers were uniformly unimpressed with the contributions of this paper. The method is somewhat derivative and the paper is quite long and lacks clarity. Moreover, the tactic of storing autoencoder variables rather than full samples is clearly an improvement, but it still does not allow the method to scale to a truly lifelong learning setting. 
Decision: Reject