Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: ##########################################################################
Summary:
 
The paper reviews common assumptions made by recent theoretical analysis of meta-learning and applies them to meta-learning methods as regularization. Results show that these regularization terms improve over vanilla meta-learning.

##########################################################################
Reasons for score: 

Overall, I vote for reject. The main idea of applying theory to practice is reasonable, but the regularization methods proposed are mainly known. Regularizing the singular value is similar to the spectral normalization proposed in [1]. The Frobenius norm regularization is similar to the commonly used weight decay.

##########################################################################
1.	Assumption 1 in Du et al. states that the ground truth weight should cover all directions evenly. It cannot be ensured when the tasks are fixed. The proposed regularization penalizes the condition number of the weight matrix during training, which is more similar to the spectral normalization proposed in [1]. As to regularizing the Frobenius norm, there exist a line of literature showing weight decay works for general settings apart from meta-learning. Thus, I think the regularization proposed in this paper is known.
2.	The experimental results indeed improve over vanilla meta-learning. However, as shown in [2], even by with some simple tricks, meta-learning can be more stable and achieves better results. This casts doubt on the value of the proposed method.

[1] Spectral Normalization for Generative Adversarial Networks, ICLR 2018
[2] HOW TO TRAIN YOUR MAML, ICLR 2019

review_title: The idea of bridging theory and practice is good, but the proposed regularization is not novel.

final_score: rating: 4: Ok but not good enough - rejection
review_content: The main motivation of this paper is based on the theoretical results of meta-learning. To ensure the assumptions of the theories, the authors propose a novel regularizer, which improves the generalization ability of the model. Some results on few-shot learning benchmarks show the proposed method improves w.r.t. those baselines.

Here are the main concerns of this paper:
1. The proposed method in this paper is based on the meta-learning theory as stated in Section 2. However, the theoretical setting here is not fully consistent with the few-shot learning setting. For example, there is no validation set in Eq. 1. The authors should make more discussions here to show will these differences influence the final results.
2. One main theoretical assumption in meta-learning theory is the task distribution. Could the authors make this notion clear? Should we do empirical results on those tasks with different kinds of task distributions?
3. The meta-learning loss in Eq. 4 is a bit different from the popular meta-learning objective. For example, in MAML, we do not optimize the classifier W till convergence while only a limited number of gradient steps are used. 
4. The authors should list those baseline values in Table 1, which are still important for reference.
review_title: A theory inspired method for meta-learning

final_score: rating: 5: Marginally below acceptance threshold
review_content: To improve the practical performance of meta-learning algorithms, this paper proposes two regularization terms that are motivated by two common assumptions in some recent theoretical work on meta-learning, namely (1) the optimal (linear) predictors cover the embedding space evenly, and (2) the norms of the optimal predictors remain bounded as the number of tasks grow. Numerical experiments show that the proposed regularization terms help achieve better performance of meta-learning in some tasks.

This work serves as a nice attempt to instruct the practice of meta-learning with theoretical insights. Below are some of my concerns.

- In some experimental results, the improvement due to the proposed regularization seems to be at the same level of the standard deviation, as well as the difference between the reproduced results of existing meta-learning algorithms and those reported in earlier papers. This casts doubt on the true efficacy of the proposed methods.

- For the loss function in Eq. (4), it is more reasonable and natural to introduce two weighting parameters (as tunable hyperparameters) for the proposed regularization terms.

- The authors often talk about "enforcing/ensuring the assumptions". However, from my understanding, whether the assumptions (on the optimal linear predictors, or "ground-truth" predictors) hold or not depends on the learning problem itself, NOT on the algorithms. Therefore, there is no way we can enforce/ensure these assumptions. I would prefer using the phrase "respecting the assumptions" (used by the authors on Page 8); this seems more accurate and reasonable. 

- Following the previous point, I'm curious about one question: if the learning problem actually doesn't satisfy the two assumptions, then is it still helpful to add the proposed regularization terms to the loss function? (I'm not sure, but my guess is no; indeed, it might even hurt.) To solve puzzles like this, I would encourage the authors to conduct some synthetic experiments, where they can design the data generating process (e.g. they can control whether the true linear predictors satisfy the assumptions or not). Since this work is a connection between theory and practice, I believe that experiments with synthetic data can help explain things more clearly and make the claims more convincing.
review_title: Improving practical performance of meta-learning, with inspiration from theoretical results

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:
In this paper, the authors aim at bridging the gap between the practice and theory in meta-learning approaches. Specifically, they propose two regularization terms to 1) capture the diversity of the tasks and 2) control the norm of the prediction layer, thereby satisfying the assumptions in meta-learning theory.

Strength:
+ The motivation of this paper is interesting, before proposing the methodology. These theoretical assumptions have not been paid enough attention before.
+ The paper is well-organized and clearly written. 
+ The experimental setting is designed in a good manner and the results are promising.

Weakness:
- I am skeptical of the novelty of the second regularize in Eq.(4). According to Section 3.2, it is equivalent to ||w||_{2}=O(1). So what is its difference to a simple l2 weight decay?
- According to Section 2, the outer-level parameters are restricted as a linear layer. Is this means the proposed regularizes would become trivial while applied on top of a more complicated model, e.g., LEO[1]?
- Too few competitors. It would be better to add some comparisons with recent methods.
- The details to calculate the subgradients of the singular values, which is quite complicated, are missing. Especially seeing that there is no guarantee that an auto-differentiation tool will do that correct.

Ref:
[1] Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, Raia Hadsell: Meta-Learning with Latent Embedding Optimization. ICLR 2019

Above all, since the contribution and the technical details to calculate the subgradients are not clear to me, I have to currently recommend a weak reject. 



review_title: Putting Theory to Work: From Learning Bounds to Meta-Learning Algorithms

Meta-review: This paper is a systematic study of how assumptions that are present recent theoretical meta-learning bounds are satisfied in practical methods, and whether promoting these assumptions (by adding appropriate regularization terms) can improve performance of existing methods. The authors review common themes in theoretical frameworks for a meta learning setting that involves a feature learning step, based on which linear predictors for a variety of tasks are trained. Statistical guarantees for such a framework (that is, statistical guarantees for the performance of trained on an additional target task) are based on the assumption that the set of weight vectors of the linear predictors span the space (ie exhibit variety) and that the training tasks all enjoy a similar margin separability (that is, that the representation is not significantly better suited for some of the tasks than others).

The current submission, cleanly reviews the existing literature, distills out these two properties and then proposes a regularization framework (that could be added to various meta-learning algorithms) to promote these properties in the learned feature representation. 

Finally, the authors experimentally evaluate to what degree the properties are already observed by some meta learning methods, and whether the proposed additions will improve performance. It is established that adding the regularization terms improves performance on most tasks. The authors thus argue that incorporating insights obtained form recent theoretical frameworks of analysis, can lead to improved performance in practice. Naturally, the purpose of the presented results is not to establish a new state of the art on a set of benchmark tasks, but to systematically study and compare the effect of adding regularization terms that will promote the properties that are desirable for a  feature representation based on statistical bounds.

I would argue that the research community should support this type of studies. The work is well presented and conducted. Most importantly, the study has a clear and general message, that will be valuable for researchers and practitioners working in on meta-learning.  

However, the reviewers did not recommend publishing this type of study for ICLR. The authors are encouraged to resubmit their work to a different venue.
Decision: Reject