Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: - Second paragraph in Section 1: Nice motivation. I am not sure though whether the performed experiments are the most expressive for such motivation. For instance, is the experiment in Section 5.1 a common task in that sequential lifelong learning setting?

- Section 4, which is the main technical section of the paper, is quite full of lengthy descriptions that are a bit equivocal. I reckon each claim really needs to be supported by a corresponding unequivocal mathermatical formulation.

- An example of the last point can be found in Section 4.2: "The synthetic samples need to be representative of all the previously observed distributions ...": It will be much clearer how such samples are representative if a formulation follows, and that did not happen in Section 4.2.

- "1) Sampling the prior can select a point in the latent space that is in between two separate distributions ...": I am not sure I got this drawback of using the standard form of VAEs. Could you please further elaborate on this?

- "we restrict the posterior representation of the student model to **be close to that of the teacher** for the previous distributions** accumulated by the teacher. This allows the model parameters to **vary as necessary** in order to best fit the data": What if the previous distributions are not that close to the new one?

- Distribution intervals: Will it be the case in reality that these intervals will be given? Otherwise, what are the solutions to that? Can they be estimated somehow (as a future work)?


Minor:
- "we observe a sample X of K": sample X of size K, I guess?
- "... form nor an efficient estimator Kingma (2017)": citation style.
- "we illustrates ..."
review_title: Good initiative for using VAEs in a new framework. The work needs to be a bit more principled though.

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: We have seen numerous variants of variational autoencoders, most of them introducing delta changes to the original architecture to address the same sort of modeling problems. This paper attacks a different kind of problem, namely lifelong learning. This key aspect of the paper, besides the fact that it constitutes a very important problem, does also addes a strong element of freshness to the paper.

The construction of the generative model is correct, and commensurate with standard practice in the field of deep generative models. The derivations are correct, while the experimental evaluation is diverse and convincing. 
review_title: At last, a deep generative model addressing a fresh problem

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper proposed a teacher-student framework and a modified objective function to adapt VAE training to streaming data setting. The qualitative experimental result shows that the learned model can generate reasonable-looking samples. I'm not sure about what conclusion to make from the numerical result, as the test negative ELBO actually increased after decreasing initially. Why did it increase?

The modified objective function is a little ad-hoc, and it's unclear how to relate the overall objective function to Bayesian posterior inference (what exactly is the posterior that the encoder tries to approximate?). There is a term in the objective function that is synthetic data specific. Does that imply that the objective function is different depending on if the data is synthetic or real? What is the motivation/justification of choosing KL(Q_student||Q_teacher) as regularisation instead of the other way around? Would that make a difference in the goodness of the learned model? If not, wouldn't KL(Q_teacher||Q_student) result reduction in the variance of gradients and therefore a better choice?

Details on the minimum number of real samples per interval for the model to be able to learn is also missing. Also, how many synthetic samples per real samples are needed? How is the update with respect to synthetic sample scheduled? Given infinite amount of streaming data with a fixed number of classes/underlying distributions and interval length, and sample the class of each interval (uniformly) randomly, will the model/algorithm converge? Is there a minimum number of real examples that the student learner needs to see before it can be turned into a teacher?

Other question: How is the number of latent category J of the latent discrete distribution chosen?

Quality: The numerical experiment doesn't really compare to any other streaming benchmark and is a little unsatisfying. Without a streaming benchmark or a realistic motivating example in which the proposed scheme makes a significant difference, it's difficult to judge the contribution of this work.
Clarity: The manuscript is reasonably well-written. (minor: Paragraph 2, section 5, 'in principle' instead of 'in principal')
Originality: Average. The student-teacher framework by itself isn't novel. The modifications to the objective function appears to be novel as far as I am aware, but it doesn't require much special insights.
Significance: Below average. I think it will be very helpful if the authors can include a realistic motivating example where lifelong unsupervised learning is critical, and demonstrate that the proposed scheme makes a difference in the example.



review_title: Adapted VAE training to streaming data setting. Experiment shows the learned model can generate reasonable-looking samples. Unsure about quantitative results.

Meta-review: Thank you for submitting you paper to ICLR. The paper studies an interesting problem and the solution, which fuses student-teacher approaches to continual learning and variational auto-encoders, is interesting. The revision of the paper has improved readability. However, although the framework is flexible, it is complex and appears rather ad hoc as currently presented. Exploration of the effect of the many hyper-parameters or some more supporting theoretical work / justification would help. The experimental comparisons were varied, but adding more baselines e.g. comparing to a parameter regularisation approach like EWC or synaptic intelligence applied to a standard VAE would have been enlightening.

Summary: There is the basis of a good paper here, but a comprehensive experimental evaluation of design choices or supporting theory would be useful for assessing what is a complex approach.
Decision: Reject