Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a new meta-learning algorithm. Following the meta-regularization method, this paper proposes a new proximal regularizer instead of the traditional regularizer to reduce the computational complexity significantly. The algorithm is reasonable and interesting which also shows high performance in experiments.
main_review:  Originality:

In meta-learning community, how to extract meta-knowledge for fast adaptation to a future task is the key problem. Metric-based algorithms learn meta-knowledge using a shared feature extractor and gradient-based meta-algorithms learn meta-knowledge by learning a common initialization. The above two branches draw a lot of attention these years, while meta-regularization, learning a common mean, is natural and reasonable but less popular. This paper discovers the potential of the meta-regularization algorithms.

This paper proposes a new meta-regularizer which is interesting and novel.

Quality:
pos: The idea is interesting and novel. The theoretical analysis and the experiments are solid.

cons: The motivation and implications of the new proposed regularizer are not very clear. Why we need to propose a new regularizer instead of the commonly used mean square loss?  Apart from the computational cost, is there any key difference between the proposed one and the traditional one?
limitations_and_societal_impact: It will be better to clarify the key motivation and implications to apply the proposed regularizer.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors proposed a kernelized proximal regularization method (MetaProx) for meta learning. MetaProx combines deep kernel and proximal meta regularization, allowing learnable regularization by reformulating the proximal problem in dual space. Thorough theoretical analysis are provided as well as empirical studies on regression and classification task.
main_review: Overall the paper is well written and easy to follow. The proofs are clear. The derived propositions supported the motivation of introducing learnable regularization. 
My major concerns about this paper are as follow:
1. The author mentioned that the proposed method can introduce nonlinearity to the model other than CommonMean method. By using a linear kernel as base kernel, the only difference between MetaProx and CommonMean is that MetaProx utilizes nonlinear neural networks as embedding network while CommonMean doesn’t. And that’s also the reason it outperforms CommonMean. It makes the work looks a bit incremental from this perspective. Though I think the author made a good point about why it outperforms MetaOpt-SVM work. 

2. In few-shot classification task, the original MetaOpt-SVM work mentioned that their method works better in large backbone network setting (e.g. ResNet12). However, in smaller backbone network setting, they do not have much advantages and hard to outperform ProtoNet. It might be fairer and more convincing to compare under different backbone setting to show consistent results.
limitations_and_societal_impact: There is no negative societal impact of this work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper introduces a new functional regularisation for meta learning. Often times meta learning algorithms can overfit to the training datasets and hence an introduction of a regularisation in terms of the outer loop parameters is a sensible choice. In particular, they build upon the Common mean algorithm and consider the dual form in which they can derive closed form solutions to the optimisation problem. Their main contribution is that the method remains computationally cheap, competitive in terms of performance and provide some theoretical insights into their proposed method MetaProx.
main_review: Reason to accept the paper:
- The paper introduces a novel regularisation for meta learning in the function space and obtains closed-form solutions to the ir optimisation problem. 
-  The proposed method is computationally efficient and achieves competitive results compared to previous methods.
-  In particular they show that setting f_{\theta} = 0 reverts back to the popular MetaOPTNet algorithm and demonstrate that their additional regularisation improves upon the original method.
-  They show empirically that their proposed methods performs better than most methods in particular MetaOPTNet
-  (I have not gone through the proofs of their theoretical claims and will base my opinion on this aspect from other more confident reviewers in this area)


Reason to reject/neutral the paper and areas that need clarification:

- Given that you have dropped the hyperparameter \lambda in your objective expression (You say you are learning the kernel so this is not necessary anymore), it seems to me that initialization of your networks seems crucial and could potentially make optimization very hard. Can you comment on this?

- The top of page 5 is very confusing and not very clear. I was wondering if you could give a more intuitive explanation of what is happening and why for example the implicit function theorem has to be used etc. I understand that this is to reduce complexity but a clearer explanation of this would be helpful.

- What is K in your paper? Is this the kernel that is learning through the feature maps? You define a K on the Z's. Have you tried to use a gaussian kernel on top of the deep kernel features instead of the just a linear kernel? You use the word RKHS and seems quite misleading (As they are usually associated with Infinite dim feature) if in reality all you are doing is deep kernel learning. This is not a problem perse just wanted this clarified as K is never properly defined (line 138-139)

-  Your Fig 1/2 is very misleading as you only show MAML papers such as [1] or [2] should be added. Else i do not see the point of this figure. Also the width is crucial in MAML and hasnt been properly studied 

- Table 1/2/3 is missing [1, 2] which has very well documented code and beats MetaOptNet. Hence this comparison should not be omitted.

- Table 3: 5-shot, your method is not statistically significantly better than Proto-net and hence bolding your number is misleading



[1] : https://arxiv.org/pdf/1912.02738.pdf CODE: https://github.com/jinxu06/metafun-tensorflow
[2]: https://arxiv.org/pdf/2003.11539.pdf CODE: http://github.com/WangYueFt/rfs/
limitations_and_societal_impact: Overall the paper proposes a new simple method for regularising the meta learning training. The write up of the paper could be improved in particular the dual problem setup as well as their proposed computational tricks as mentioned above. Also in terms of the experiments i believe that methods mentioned above should be included for completeness or at least justified why they weren't added. I understand that the proposed method improves upon metaoptnet but that doesn't mean that it is necessarily the best method and hence should be compared to others as well.

I will be awaiting the rebuttal and am happy to raise my score if the above have been addressed.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors proposed an effective meta-regularization algorithm (MetaProx) for meta-learning. A learnable proximal regularizer with deep kernel is introduced to learn the base learner. The computation complexity of meta-gradient is linear in the number of meta-parameters. Experiments on synthetic dataset and mini-ImageNet show that the proposed method outperforms the popular few-shot learning methods.
main_review: 1.	This paper is well-written. The research problem and proposed method are clearly introduced. 
2.	It is good to have theoretical analysis on the smoothness of the meta-loss and the convergence. 
3.	The proposed method is verified to be effective in few-shot regression and classification tasks.
4.	The paper states that the new method is much faster than other meta-learning methods e.g. MAML. However, the paper lacks the comparison of training and converging time. 
5.	In few-shot classification experiments, the simple Conv4 network is used. However, the state-of-the-art few-shot learning methods use ResNet and Wide Residual Networks as the backbone. Does the proposed method still achieve better performance than the competitors with large models?

limitations_and_societal_impact: The limitation is not provided. No negative societal impact.
review_title: null

Meta-review: The paper initially received mixed ratings. Most concerns from the reviewers have been well addressed by the rebuttal and after the discussion period, all reviewers are now on the accept side. In particular, the reviewers have been convinced by the experimental results provided in the rebuttal. The area chair agrees with their assessment and follows their recommendation. 

The final version of the paper should include these results and take into account the discussions that took place during the rebuttal period.
Decision: Accept (Poster)