Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The author extends existing PAC-Bayes bounds to multi-task learning, to allow the prior to be adapted across different tasks. Inspired by the variational bayes literature, a probabilistic neural network is used to minimize the bound. Results are evaluated on a toy dataset and a synthetically modified version of MNIST. 

While this paper is well written and addresses an important topic, there are a few points to be discussed:

* Experimental results are really week. The toy experiment only compares the mean of two gaussians. Also, on the synthetic MNIST experiments, no comparison is done with any external algorithms. Neural Statistician, Model-Agnostic Meta-Learning and matching networks all provide decent results on such setup. While it is tolerated to have minimal experiments in a theoretical papers, the theory only extends Pentina & Lampert (2014). Also, similar algorithms can be obtain through variational-bayes evidence lower bound. 

* The bound appears to be sub-optimal. A bound where the KL term vanishes by 1/n would probably be tighter. I went in appendix to try to see how the proof could be adapted but it’s definitively not as well written as the rest of the paper. I’m not against putting proofs in appendix but only when it helps clarity. In this case it did not.

* The paper is really about multi-task learning. Lifelong learning implies some continual learning and addressing the catastrophic forgetting issues. I would recommend against overuse of the lifelong learning term.

Minors:
* Define KLD
* Section 5.1 : “to toy”
* Section 5.1.1: “the the”

review_title: Well written paper, with very week experiments.

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper considers multi-task setting of machine learning. The first contribution of the paper is a novel PAC-Bayesian risk bound. This risk bound serves as an objective function for multi-task machine learning. A second contribution is an algorithm, called LAP, for minimizing a simplified version of this objective function. LAP algorithm uses several training tasks to learn a prior distribution P over hypothesis space. This prior distribution P is then used to find a posterior distribution Q that minimizes the same objective function over the test task. The third contribution is an empirical evaluation of LAP over toy dataset of two clusters and over MNIST.

While the paper has the title of "life-long learning", the authors admit that all experiments are in multi-task setting, where
the training is done over all tasks simultaneously. The novel risk bound and LAP algorithm can definitely be applied to life-long setting, where training tasks are available sequentially. But since there is no empirical evaluation in this setting, I suggest to adjust the title of the paper. 
 
The novel risk bound of the paper is an extension of the bound from [Pentina & Lampert, ICML 2014]. The extension seems to be quite significant. Unlike the bound of [Pentina & Lampert, ICML 2014], the new bound allows to re-use many different PAC-Bayesian complexity terms that were published previously. 

I liked risk bound and optimization sections of the paper. But I was less convinced by the empirical experiments. Since 
the paper improves the risk bound of [Pentina & Lampert, ICML 2014], I expected to see an empirical comparison of LAP and optimization  algorithm from the latter paper. To make such comparison fair, both optimization algorithms should use the same base algorithm, e.g. ridge regression, as in [Pentina & Lampert, ICML 2014]. Also I suggest to use the datasets from the latter paper.  

The experiment with multi-task learning over MNIST dataset looks interesting, but it is still a toy experiment. This experiment  will be more convincing with more sophisticated datasets (CIFAR-10, ImageNet) and architectures (e.g. Inception-V4, ResNet). 

Minor remarks:
Section 6, line 4: "Combing" -> "Combining"
Page 14, first equation: There should be "=" before the second expectation.
review_title: Interesting risk bound but empirical evaluation is not convincing

final_score: rating: 6: Marginally above acceptance threshold
review_content: I personally warmly welcome any theoretically grounded methods to perform deep learning. I read the paper with interest, but I have two concerns about the main theoretical result (Theorem 1, lifelong learning PAC-Bayes bound).
* Firstly, the bound is valid for a [0,1]-valued loss, which does not comply with the losses used in the experiments (Euclidean distance and cross-entropy). This is not a big issue, as I accept that the authors are mainly interested in the learning strategy promoted by the bound. However, this should clearly appear in the theorem statement.
* Secondly, and more importantly, I doubt that the uaw of the meta-posterior as a distribution over priors for each task is valid. In Proposition 1 (the classical single-task PAC-Bayes bound), the bound is valid with probability 1-delta for one specific choice of prior P, and this choice must be independent of the learning sample S. However, it appears that the bound should be valid uniformly for all P in order to be used in Theorem 1 proof (see Equation 18). From a learning point of view, it seems counterintuitive that the prior used in the KL term to learn from a task relies on the training samples (i.e., the same training samples are used to learn the meta-posterior over priors, and the task specific posterior).  

A note about the experiments:
I am slightly disappointed that the authors compared their algorithm solely with methods learning from fewer tasks. I would like to see the results obtained by another method using five tasks. A simple idea would be to learn a network independently for each of the five tasks, and consider as a meta-prior an isotropic Gaussian distribution centered on the mean of the five learned weight vectors.

Typos and minor comments:
- Equation 1: \ell is never explicitly defined.
- Equation 4: Please explicitly define m in this context (size of the learning sample drawn from tau).
- Page 4, before Equation 5: A dot is missing between Q and "This".
- Page 7, line 3: Missing parentheses around equation number 12.
- Section 5.1.1, line 5: "The hypothesis class is a the set of..."
- Equation 17: Q_1, ... Q_n are irrelevant.

=== UPDATE ===
I increased my score after author's rebuttal. See my other post.
review_title: Interesting algorithm based on a theoretical study, but the main theorem might contain a flaw

Meta-review: The author's revisions addressed clarity issues and some experimental issues (e.g., including MAML results in the comparison). The work takes an original path to an important problem (transfer learning, essentially). There is a question of significance, and this is due to the fact that the empirical comparisons are still very limited. The task is an artificial one derived from MNIST. I would call this "toy" as well. On this toy task, the approach isn't that much different from MAML, which is not in of itself a problem, but it would be interested to have a less superficial discussion of the differences.

The authors mention that they didn't have time for a larger empirical study. I think one is necessary in this case because the work is purposing a new learning algorithm/framework, and the question of its potential impact/significance is an empirical one.
Decision: Reject