Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors verify the importance of train-validation split in meta-learning theoretically, which is commonly used in the meta-learning paradigms. By analyzing the linear centroid meta-learning problem, the authors show that the splitting method converges to the optimal prior as expected, whereas the non-splitting method does not in general, without structural assumptions on the data. The authors validate the theories empirically through both simulations and real meta-learning tasks.

The paper provides new insights on the usage of the train-validation splits in meta-learning theoretically. Some popular methods such as iMAML are also used as comparisons in the experiments. 

Here are some suggestions:
1. Will the batch-size, the meta-training way influence the results in Table 1. More ablations like Table 2 should be investigated.

2. The authors could discuss whether the theory can be applied to other kinds of meta-learning methods and how the theoretical results can guide the design of new meta-learning methods.
review_title: A theory to show the importance of train-val split in meta-learning

final_score: rating: 6: Marginally above acceptance threshold
review_content: In meta-learning, a common practice is to do a train/validation split of the data within each that, so that optimization of meta-parameters is performed on validation, not training, losses. In this paper, the author argue that this split is important for correcting model misspecification, but if the model is correctly specified, not doing a split might actually lead to better learning rates. They provide theoretical justification under a simple linear model, and some experiments on synthetic and real data.

Disclaimer: I have good expertise in random matrix theory but I am not as knowledgeable about the meta-learning literature. Thus, my review will mostly focus on the results themselves, with the hope that other reviewers can better evaluate how this work compares to other work in the area.

Globally, I think the results of this paper are interesting. The proof of Theorem 7 (the trickiest result) is clever. I think the high-level structure of the paper could be improved and results better presented, but the "core" of the paper holds its ground. Per the disclaimer I cannot judge the novelty of the work. Overall I would lean towards acceptance.

[pros]
1. The central result of the paper is, in my mind, Corollary 8. Its precursor, Theorem 7, has a clever proof that was different from what I expected. Namely, I thought it would be based on linear spectral statistics results as found in Bai and Silverstein (2004), but instead they noticed their statistic of interest can be computed from the nice trick of directly differentiating (a generalization of) the Stieltjes transform. It's very elegant.

2. I think Propositions 2-3 are the other interesting results, namely by showing that consistency of the methods are intimately linked to model specification.

3. To obtain a workable theoretical analysis, the authors analyzed a simple linear model, (6). However, this model also leads to counterintuitive results, such as Corollary 6. What happens when the model is very flexible, like a neural net? Issues of model misspecification become less relevant, so one might think we are more in the setting of Section 4. However, it's possible that the gap in learning rates between train-val and train-train is an artifact of how simple the linear model is. I think the "real data" experiments in Section 5 are interesting in that respect and suggest that it remains true with deep models, so this is an important contribution as well.

[cons]
1. I think the structure of the paper could be much improved. Since Corollary 8 is the most important result, it should be given the status of a Theorem. Theorem 4, 5, and 7 should disappear. I think Corollary 6 and its ensuing discussion should disappear as well. Theorem 4 is ok, but maybe should be given a lesser status. I think if Proposition 2-3 could be merged into a single Theorem it would be nice too. I would try to eliminate or shorten as much as possible Section 2.1, I found this a bit distracting. The rest of Section 2 is fine for me. Otherwise, I would really emphasize the two scenarios: under model misspecification (Section 3) and under correct model specification with a working example, the linear model (Section 4), perhaps by changing the titles of the current sections. Finally, as discussed in my "pro" point 3, the "real data" experiments in Section 5 are really important, and it's a pity a lot of details are relegated to the appendix. Since I suggest cutting a lot of theoretical results, I would maybe bring back a lot of details currently dumped in Section A.2, since I think this deep, real data example is really crucial for making the case that the paper is relevant to normal practice.

2. I would like to see more details as to why interchange of the derivative and both the limit and the integral is justified in the proof of Theorem 7 (uniform convergence).

Other comments:
- Calling \hat{w}_{0, T}^{tr-val} in Proposition 2 an "algorithm" doesn't make much sense, I think "estimator" is more accurate.
- There seems to be a major typo in the statement of Theorem 7: you talk of the train-val method but the math has tr-tr written.

References

[1] Bai, Z. D., and Jack W. Silverstein. "CLT for Linear Spectral Statistics of Large-Dimensional Sample Covariance Matrices." Annals of Probability (2004): 553-605.
review_title: Interesting, non-trivial theoretical result

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper compares two approaches of data splitting in meta-learning: train-validation split and train-train split. This paper shows that the best of the two approaches depends on the specific problem.

Strength: It is easy to follow the paper and it is very convincing why the train-train approach could be better than the train-validation approach.

Weakness: I am not an expert in meta-learning and I could be wrong with my comments. But my concerns are:

1) Both inner loop and outer loop are solving the linear regression problems. In fact, because the estimate of the inner loop $\mathcal{A}(w_0; X, y)$ is linear in $w_0$, we can combine the inner loop and outer loop together, and our estimate for $w_0$ is a linear regression estimate where the new features are constructed by the original features and response, and the loss is not only averaged by the sample size $n$ of each task but also the number of tasks $T$. It is unclear to me how ''this problem captures
the essence of meta-learning with non-linear models (such as neural networks) in practice'' as claimed on page 2. Furthermore, as $T$ goes to infinity, it becomes the classic linear regression regime where dimension fixed and sample size goes to infinity. Because of this, it is intuitive and looks straightforward to check that the train-validation split gives a consistent estimate and the train-train split gives a consistent estimate in the realizable setting but not in general. I would say to improve the significance of the paper, the authors could like at the regime where d, n, T both grows to infinity at certain rates e.g. $d=\Theta(Tn)$.   

2) One major reason that the train-validation split is inferior to the train-train split is that a significant amount of data is missed in the training. My question is why we do not apply K-fold cross-validation instead of a simple train-validation split. In the case of ridge regression, it is possible to do a leave-one-out cross-validation efficiently, i.e., we can obtain the leave-one-out prediction risk without fitting the regression $n$ times but using the estimate trained by the entire data. For general meta-learning, K-fold cross-validation should be a better approach than a train-validation split. In the linear regression setting, leave-one-out cross-validation should be a more proper approach. Then it will be kind of fair to compare K-fold cross-validation with the train-train split and it will be interesting to see train-train split can be still better.

3) Because of the linear regression settings with $T\rightarrow \infty$, I think it is closely related to a Bayesian hierarchical model with the Gaussian prior on the coefficients (due to the ridge penalty in the inner loop). Is the Bayesian estimator be consistent as well when $T\rightarrow infty$? Some discussion about this connection would be good.

Because of these concerns, I currently recommend a reject. But it is possible that I will change my mind if the authors can address my concerns well.
review_title: More discussions are required for the model settings. 

final_score: rating: 5: Marginally below acceptance threshold
review_content: In this paper, the authors study the theoretical properties of meta-learning. In particular, the train-validation split to tackle the linear centroid meta-learning problem is investigated using statistical asymptotic theory. First, the authors proved that the train-validation method has statistical consistency, while the train-train method has a statistical bias to the centroid. Under the noise-free setting, however, both methods have statistical consistency. Furthermore, the train-train method is superior to the train-validation method in the sense of the asymptotic MSE. Based on the asymptotic analysis the optimal ratio of the data splitting for the train-validation method was also derived. The theoretical findings are confirmed by some numerical experiments. 

In this paper, the meta-learning is formulated as a multiple ridge regression problem and the standard statistical asymptotic theory is used to derive some theoretical results. The impact of the results will be rather limited. 

In the paper, the difference between the train-val method and the train-train method was investigated. The theoretical properties are well-known in the standard context of statistics and machine learning. That is, the overfitting to the training data yields a serious bias to the generalization error. Information criteria such as AIC, BIC, etc provide the bias correction to fill the gap. I guess that the authors are aware of the close relationship between the present formulation of the meta-learning and the classical statistical theory. The authors could mention its relationship and could go to further intriguing problems such as the analysis of meta-learning using lasso-type regularization. 

In my option, the theoretical contribution of this paper is quite limited. 

Some more comments below:
- In each task, the same number of data is assumed to be observed. Was that just for the sake of simplicity? How is the theoretical results modified when the sample size varies from task to task?
- In Proposition 1, the meaning of the Cov(\nabla\ell_t) is not very understandable.
- The authors analyzed the statistical properties of the meta-learning using realizable linear models, or in other words, the noise-free models. The noise-free model is extremal situation, and a more general situation could be studied.
- In Corollary 6, can the author provide an intuitive reason why splitting with zero training and all validation is optimal.
- In section 4.2, the proportional limit was studied. Is it possible to derive a theoretical bound in the case that T, n, and d take comparable numbers?
- The authors did not provide any interpretation of the conclusion of Corollary 8. Showing a persuasive reason is of great help to the readers. 

review_title: In this paper, the meta-learning is formulated as a multiple ridge regression problems and the standard statistical asymptotic theory is used to derive some theoretical results. The impact of the results will be rather limited. 

Meta-review: The authors investigate theoretical properties of meta-learning. In particular, the train-validation split to tackle the linear centroid meta-learning problem is investigated in asymptotically regimes (a non-asymptotic analysis of the train-train estimator has been added later on). It is shown that the train-validation method has statistical consistency, while the train-train method has a statistical bias to the centroid. Yet, in the noise-free setting both methods have statistical consistency. Furthermore, the train-train method is superior to the train-validation method in the sense of the asymptotic MSE. Based on the asymptotic analysis the optimal ratio of the data splitting for the train-validation method is derived. The theoretical findings are corroborated by some numerical experiments.
The main theoretical result suggests that the train-train method + optimally tuned regularization is a strong alternative to the train-validation split, hence the authors' recommendation that for meta-learning train-train method should be preferred.

The reviewers and the area chair appreciated the theoretical findings and the subsequent effort to improve the paper the authors put in place during the discussion period. Unfortunately, the tight competition among papers in this year's edition of the conference makes this specific submission not compelling enough for publication. 
I would like to encourage the authors to sumbit to another ML venue in the near future, while considering improvements in their experimental validation (as also suggested by some reviewers).

Decision: Reject