Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presents the idea of enforcing better condition numbers for inner-loops in meta-learning frameworks. They reformulated meta-learning loss as the least-square formulation, which enables them to easily approximate condition numbers. Then, it was used as an additional loss term to minimize it, which results in better performance in few-shot classification tasks. They demonstrated that their approach converged rapidly, especially for the first few iterations.
strengths_and_weaknesses: I generally like the idea of achieving rapid convergence without introducing more parameters. The paper is well written and easy to follow. The idea of using least-square reformulation to easily approximate eigenvalues is quite interesting. However, I have some concerns about experimental results below in the question section. I would revise my score depending on the authors' response.
questions: 1. All 5 step training was used for all experiments for both MAML and the proposed method. Since MAML might overfit the initial parameters for 5 step performance, so they learned not to update too much in the first few iterations. I am curious how MAML performs, using 1 step training, and 1 step testing accuracy, which is a fairer comparison I think.

2. Missing MAML results on CUB for step 5 - step 100 in Table 3.

3. Missing comparisons to prior arts. Only MAML baseline is not good enough as time of now. For examples, miniImageNet results, the accuracies are way lower than current state-of-the-arts methods. I am not complaining about not achieving SOTA, but it’s better to be upfront with the readers. Especially, you should have shown the performance of methods related to ‘preconditioning’, e,g, [1-3], you can still argue that your method does not require additional parameters.

4. checklist 3-(c), the answer was yes, but I can’t find any error bars w/ different random seeds. I was expecting the ‘shaded area’ in training curves, e.g., standard deviation or confidence interval, at least for the main curves. The standard deviations you reported in the tables are not w/ different random seeds. It’s average over test tasks from my understanding. So, your answers for checklist 3-(c) should be ‘no’.

5. How did you compute eigenvalues? is it differentiable so that you can easily use them as a part of loss function? no explanation about this in the current texts. Also, are there other related works that are using eigenvalues as a part of loss function? 

6. Figure 2-(c), you plotted L_k, which is considered as condition numbers. But, was it approximated (by eq 8) and also surrogate condition numbers (by eq (9))? Since you are using L_k as an additional loss function, it is obvious that this number decreases. I am curious if the real condition number is decreasing.

7. In figure 1, (a) and (b), were they a toy example? or from MAML? If MAML, how did you pick only two dimensions in the parameters? not clear given the current texts.

[1] Meta-Curvature, Park et al., NeurIPS 2019
[2] On Modulating the Gradient for Meta-Learning, Simon et al., ECCV 2020
[3] Meta-learning with warped gradient descent, Flennerhag et al., ICLR 2020
limitations: The authors addressed some limitations in the main text. and I do not see any negative societal impact.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: - The authors propose a regularization method for improving the conditioning of MAML.
- Experiments show the method consistently reduces number of steps for adaption, and improves accuracy
strengths_and_weaknesses: Strengths:
- method seems very effective at reducing number of required MAML steps
- consistently outperforms MAML
- simple additional loss during training, no cost at inference time
- authors study effect of using parameter subset

Weaknesses:
- precondition loss only applied in top layers (as mentioned by authors), but this may not be a big problem, as the top layers are the ones that are adapted the most during meta-testing
questions: Is it the real condition number or the variance of logs in Figure 2?
limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a regularisation term for the outer-loop of MAML to encourage a well conditioned parameter space that improves inner loop adaptation. The experiments suggest that the condition number is correlated to the performance within a few gradient updates. Training with the proposed regularisation term is shown to empirically improve upon MAML.  


strengths_and_weaknesses: 
Strengths:
 - Does not require additional parameters for adaptation unlike existing preconditioning methods. 
 - Interesting analysis regarding the connection between condition number and few-step performance on an unseen task. 
 - Promising experiment results show an empirical benefit of preconditioning. 

Weaknesses:
 - Requires higher-order gradients (computationally expensive) like MAML
 - Computing the conditioning term is expensive. As such, instead of applying the conditioning constraint to the entire model, only a small subset of parameters can have the conditioning constraint applied.
 - Missing comparison with existing preconditioning methods (For example, see Section 2.1 of WarpGrad) 

Overall, I would rate the paper:

Novelty: Medium

Clarity: High

Significance: Medium


questions: 
It is not clear to me why MAML's training collapses in Figure 2. 


Although there has been previous work in preconditioning, this is the first method that I'm aware of that does so without additional parameters. On Page 9 Line 299-300, it is mentioned that existing preconditioning methods "[incur] an often extensive number of additional parameters". This seems to suggest that some methods may not require a lot more parameters. If so, how does the number of parameters required compare to the existing preconditioning methods and how do the performance differ? This would allow a potential user to make a decision between which method to try.

I find it interesting that the proposed method can scale to many more gradient steps (Table 3). How does MAML fare when scaling to those same number of gradient steps at test time? Does MAML continue to perform similarly or better than with 5 gradient steps? I am not convinced of the benefit of choosing the number of adaptation steps at inference steps. For example, at test time, it makes sense to evaluate the model on the same number of inner loop updates as it was trained on. Although, it is beneficial if improvements can be further gained with more training (as we see in Table 3). 


On Page 9 Line 316, it's mentioned that the proposed method "significantly [outperforms] unconstrained methods during initial adaptation". However, the only method compared is MAML, so it would be more precise to say "significantly [outperforms] its unconstrained counterpart during initial adaptation"

If the parameter space is well-conditioned, then I would expect that a single large gradient step from the meta-initialization could already train a good model. For example, instead of evaluating the model by fine-tuning with 5 inner loop updates with an inner loop learning rate of 0.1, I'm curious as to the performance of fine-tuning with 1 inner loop update with an inner loop learning rate of say 0.5.
limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Improving the performance of a deep neural network on a new, unseen task with a limited number of new datapoints and adaptation epochs is one of the central problems in modern deep learning. The authors propose to improve the performance of MAML, a benchmark algorithm for few-shot adaptation problems, by pre-conditioning the parameter space using the condition number of the network. Instead of directly using the condition number to condition the bi-level optimization problem of MAML, the authors propose to consider the distribution of all eigenvalues using the approximated logarithmic eigenvalues for increased expressiveness. This simple modification to MAML is shown to be highly effective at improving the few-shot adaptation process across diverse datasets.
strengths_and_weaknesses: Strength
- The proposed method is theoretically sound, interesting, and novel. Although it is a simple modification to an existing algorithm (MAML), there is enough novelty to be recognized in the idea to better-condition a parameter space using the condition number of the Hessian matrix. In addition to the strong theoretical motivation behind the proposed approach, the authors experimentally demonstrate the relationship between the condition number of a network and its adaptation capabilities. 
- The proposed method is shown to be highly effective at allowing more rapid few-shot adaptation. According to the experimental results, the proposed method improves adaptation performance of MAML across all adaptation steps, but the degree of improvement is particularly noteworthy under a limited number of adaptation steps (1 or 2). 
- The writing is concise and clear. It was easy to follow how the modified objective function is derived by introducing the conditioning constraint to the MAML bi-level optimization problem. 

Weakness
- Maybe more baseline approaches from meta-learning and/or few-shot learning, other than MAML, could be added for comparison. Can MAML + the conditioned parameter space (proposed method) outperform more recent meta-learning algorithms? Also, correct me if I'm wrong, but it appears that the parameter space conditioning can be integrated with other meta-learning algorithms as well. Does it lead to performance improvement regardless of the choice of base meta-learning algorithm?
questions: - Can the proposed method also improve MAML's performance in reinforcement learning setting? Experimentally demonstrating the applicability of the proposed method across different problem set-ups or tasks would strengthen the paper's contribution.
- The authors mention that "computing the condition number by using the max and min eigenvalues... would unnecessarily weaken the training signal" (section 3.2). Does using the condition number as opposed to Hessian's logarithmic eigenvalues actually lead to a meaningful difference in adaptation performance?
- Some of the adaptation works have found that only adapting bath norm parameters, instead of the whole network parameters, to reduce the computational cost can be sufficient to improve the adaptation performance. In that regard, it is interesting that adapting cls+emb+eBN parameters performs worse than adapting cls+emb parameters (Table 2), any insight on why additionally adapting bn parameters leads to worse performance?
limitations: The authors adequately acknowledge the computational complexity of their method in section 5.
Please refer to weakness and questions for additional concerns and questions.
review_title: null

Meta-review: metareview: This paper was quite well received by reviewers, with scores of 5, 6, 6, 8.
Reviewers felt the paper was well written, clear and expressed an interesting core idea.
Experimental results compare against MAML and show clear improvements.

The key idea here was inspired by preconditioning, and the method here aims to increase adaptation speed for gradient-based meta-learning methods without incurring extra parameters. The paper recasts the optimisation to a non-linear least-squares formulation and propose a way to enforce a well-conditioned parameter space for meta-learning through the condition number and local curvature perspectives. Experiments show that the approach outperforms unconstrained optimization significantly and does particularly well during initial adaptation phase of optimization.

The AC recommends acceptance.
confidence: Certain
Decision: Accept