Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper analyzes the train-validation split in meta-learning for classification problems with a two-layer CNN model. It proves that the train-validation split is necessary to learn a good prior model compared with a train-train model. Experiments justify the theoretical claims.
strength_and_weaknesses: Strength:
- There is few research work studying train-validation split of meta-learning with neural networks, so it is great to have such work targeting on the theoretical foundations.
- The theoretical analyses seem to be solid but I have not checked the proof details.

Weakness:
- What does it mean by "FOMAML" (first appears at Page 2)? It seems no where the term is introduced.
- There is a little gap between the theories and the experiments, namely it would be great to have some synthetic data and experiments to meet the theoretical assumptions and to see how it matches to the theoretical claims. For example, we can generate synthetic data by Definition 3.2, then we can use a two-layer CNN described in the theoretical analysis as the backbones. To verify Thm 4.4 and 4.5, we can set different values of $d$ in synthetic data and to confirm the training loss is at the order of O(1/poly(d)) as claimed in the theorems. Similarly for the test loss, we want to see a figure plotting the test loss evolution, where train-train test loss converges to some constant while train-validation test loss continues to decrease. I think the theories and experiments can be more closely connected in this way.
clarity,_quality,_novelty_and_reproducibility: Overall, the quality and clarity are good, and the originality is great since there is few work studying train-validation split of meta-learning with CNN.
summary_of_the_review: Based on the novelty and strengths discussed above, I am close to an acceptation of the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors conduct an investigation into the importance of having train-validation sets for episodic few-shot learning, learned by meta-learning. They do as such by both theoretical proofs and empirical evaluation. 

While the intuitions behind having a train - validation set for meta-learning are quite clear and reasonable, this is a very good complementary study that showcases that the intuitions do in fact measure up to reality. 
strength_and_weaknesses: Strengths:

- Clear empirical evaluation, with thorough experiments. 
- Inclusion of both theoretical justification and empirical results.

Weaknesses:

- Writing quality is clumsy, and at worst significantly affecting clarity and precision of what is being communicated. 
clarity,_quality,_novelty_and_reproducibility: The clarity and quality of the work are its main problem, with the writing clarity affecting both the overall clarity of the work as well as its precision. 

The work seems reproducible from what is in the paper.
summary_of_the_review: I recommended a weak reject due to the writing quality being problematic enough to cause precision and clarity issues in the work. If that is improved, I'd be happy to accept this paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this work, authors provide a theoretical analysis regarding the usage of training data only during episodic training in a meta-learning setup against train each episode using train and compute the loss on a validation split to propagate back through the network. The second strategy is commonly used by practitioners, however, some existing relevant work argue that it might not be required. In this work, authors show that using a train-val split strategy can achieve lower test loss in a meta-learning setup (when used with a two-layer convolutional network and first-order MAML as the meta-learning algorithm).
strength_and_weaknesses: **Strengths**

* The idea of training using a train split and back-propagating using the loss on the validation split has been a longstanding practice in the meta-learning literature - however, there has not been a lot of interest in analyzing that, either empirically of theoretically. Only existing work on this is from [Bai et al.](https://arxiv.org/pdf/2010.05843.pdf) which the authors discuss here. Theoretically analyzing whether train-val split is needed for meta-learning is an important topic.
* Most of the assumptions made in terms of proving the bounds make sense to me. Authors use a Huberized-ReLU instead of a normal ReLU to simplify the proof and show that empirically it does not make much difference. I have not looked at the derivation in details provided in the appendix - the lemmas stated in the main manuscript makes sense to me at a high-level. 
* Authors perform experiments using synthetic datasets and also using a two-layer network on standard datasets like Mini-ImageNet which corroborates their theoretical findings.

**Weakness**

For me, the biggest weakness or doubt about this paper that I have in mind is how well these results will extrapolate to other type of networks or networks with more layers. There is a discrepancy in terms of findings between this work and Bai et al. where Bai et al. claimed train-train split is enough with a linearized network and this work claims that their hypothesis is not valid when using a two-layer convolutional network. However, the empirical results of Bai et al. speaks a different story - if we see their Table 1 (page 11), they experiment with the same convolutional backbone as this paper on Mini-ImageNet and Tiered-ImageNet and show that train-train works equal or better compared to train-val, which contradicts the claims made in this paper. 

Along the same line, my other question is - is there any guarantee that claims made in this paper will hold true for a different kind of architecture e.g. vision transformers or if we simply increase the number of layers in the convolutional network/add a normalization layer/add a residual connection? If these networks are too complicated to be analyzed theoretically, I think some empirical evaluation using train/train vs train/val strategy will help to justify the claims made in this paper. 

clarity,_quality,_novelty_and_reproducibility: **Clarity**

The paper is written in a succinct way - it unambiguously states the problem it wants to analyze, clearly defines the notations and describes the lemmas. Empirical evaluation setup is also described well.

**Quality and Novelty**

The proposed work is one of the very few which explores the usefulness of train-train vs train-val strategy used in episodic meta-learning setup. The theoretical framework borrows ideas from Bai et al., however, the results they found are different so there is novelty in that aspect. The quality of the paper is still in doubt - especially with respect to the contradictory empirical vs theoretical results when compared to Bai et al. I would like authors to comment on it before I finalize my thoughts on the quality/novelty aspect.

**Reproducibility**
This is mostly a theoretical paper and implementation ideas analyzed in this paper are well-known.
summary_of_the_review: The paper is well-written, analyzes an important implementation details from the meta-learning literature from a theoretical viewpoint and provides bounds which show that train-val is a better strategy for the model to generalize as opposed to train-train which is the common practice in the literature. However, the empirical and theoretical results are in conflict with the previous work from Bai et al. which casts doubts about the contributions of this paper.

I'll be willing to modify my score if authors can clarify my confusion about the results from Bai et al.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a theoretical results of FOMAML that shows train-train and train-validation methods can achieve a small training loss. It is necessary to perform a train-validation split in the task data to get good generalization results. Compared to previous work that only studies linear models . This paper advances the results to two-layer convolutional networks.

strength_and_weaknesses: **Strength**:

* This paper presents theoretical results of FOMAML and advances previous results to convolutional neural networks. 

* The paper is well-written

**Weakness**:

* Does the theoretical results or analysis also applicable for second-order MAML, implicit MAML[1], closed-form solver[2], etc?


* If the number of samples is large, how do the theoretical results change with different numbers of samples? In this case, does train-validation still better train-train meta learning?


* How does the results compare to Reptile[3], which does not need a train-validation split.

References;

[1] Meta-Learning with Implicit Gradients. NeurIPS 2019

[2] Meta-learning with differentiable closed-form solvers. ICLR 2019

[3] On First-Order Meta-Learning Algorithms


clarity,_quality,_novelty_and_reproducibility: The paper is clearly written.

The novelty is moderate.

Results seem to be reproducible. 
summary_of_the_review: This paper proposes a theoretical results of FOMAML that shows train-train and train-validation methods can achieve a small training loss. The results advance previous results and provide a deeper understanding of meta-learning.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper focuses on the benefits of the train-validation split for FOMAML and shows both theoretically and empirically that it is better to use a train-valid split than using train-train setups. 

The main criticism against this paper is that the results in this paper contradict the results of Bai et al. 2021. However, the authors have made a very convincing rebuttal explaining the problem with the setup of Bai et al. 2021 and why there is a contradiction. Further, the authors have added more results which strengthen their claims.

Given that I am convinced by their rebuttal, I recommend an acceptance. However, I strongly recommend the authors add all the new results in the paper and also improve the clarity of the paper. Also, make sure to not overclaim the results (which I see that you have already done in the revision).


justification_for_why_not_higher_score: It is a good paper that is worth accepting. The paper claims the importance of train-valid splits for a very specific meta-learning algorithm - FOMAML. Hence the recommendation for poster and not spotlight.
justification_for_why_not_lower_score: This is a very useful discussion to have in the meta-learning community and there are no major issues with the paper that requires rejection.
Decision: Accept: poster