Title: Meta-learning with differentiable closed-form solvers
Authors: Luca Bertinetto
Joao F. Henriques
Philip Torr
Andrea Vedaldi
Venue: ICLR 2019
Keywords: few-shot learning
one-shot learning
meta-learning
deep learning
ridge regression
classification

Abstract: Adapting deep networks to new concepts from a few examples is challenging, due to the high computational requirements of standard fine-tuning procedures.
Most work on few-shot learning has thus focused on simple learning techniques for adaptation, such as nearest neighbours or gradient descent.
Nonetheless, the machine learning literature contains a wealth of methods that learn non-deep models very efficiently.
In this paper, we propose to use these fast convergent methods as the main adaptation mechanism for few-shot learning.
The main idea is to teach a deep network to use standard machine learning tools, such as ridge regression, as part of its own internal model, enabling it to quickly adapt to novel data.
This requires back-propagating errors through the solver steps.
While normally the cost of the matrix operations involved in such a process would be significant, by using the Woodbury identity we can make the small number of examples work to our advantage.
We propose both closed-form and iterative solvers, based on ridge regression and logistic regression components.
Our methods constitute a simple and novel approach to the problem of few-shot learning and achieve performance competitive with or superior to the state of the art on three benchmarks.

TL;DR: We propose a meta-learning approach for few-shot classification that achieves strong performance at high-speed by back-propagating through the solution of fast solvers, such as ridge regression or logistic regression.

Review discussion:
This paper proposes a meta-learning approach for the problem of few-shot classification. Their method, based on parametrizing the learner for each task by a closed-form solver, strikes an interesting compromise between not performing any adaptation for each new task (as is the case in pure metric learning methods) and performing an expensive iterative procedure, such as MAML or Meta-Learner LSTM where there is no guarantee that after taking the few steps prescribed by the respective algorithms the learner has converged.
For this reason, I find that leveraging existing solvers that admit closed-form solutions is an attractive and natural choice.
Specifically, they propose ridge regression as their closed-form solver (R2-D2 variant). This is easily incorporated into the meta-learning loop with any hyperparameters of this solver being meta-learned, along with the embedding weights as is usually done. The use of the Woodbury equation allows to rewrite the closed form solution in a way that scales with the number of examples instead of the dimensionality of the features;
therefore taking advantage of the fact that we are operating in a few-shot setting.
While regression may seem to be a strange choice for eventually solving a classification task, it is used as far as I understand due to the availability of this widely-known closed-form solution. They treat the one-hot encoded labels of the support set as the regression targets, and additionally calibrate the output of the network (via a transformation by a scale and bias) in order to make it appropriate for classification. Based on the loss of ridge regression on the support set of a task, a parameter matrix is learned for that task that maps from the embedding dimensionality to the number of classes. This matrix can then be used directly to multiply the embedded (via the fixed for the purposes of the episode embedding function) query points, and for each query point, the entry with the maximum value in the corresponding row of the resulting matrix will constitute the predicted class label.
They also experimented with a logistic regression variant (LR-D2) that does not admit a closed-form solution but can be solved efficiently via Newton’s Method under the form of Iteratively Reweighted Least Squares.
When using this variant they restrict to tackling the case of binary-classification.
A question that comes to mind about the LR-D2 variant: while I understand that a single logistic regression classifier is only capable of binary classification, there seems to be a straightforward extension to the case of multiple classes, where one classifier per class is learned, leading to a total of N one-vs-all classifiers (where N is the way of the episode). I’m curious how this would compare in terms of performance against the ridge regression variant which is naturally multi-class. This would allow to directly apply this variant in the common setting and would enable for example still oversampling classes at meta-training time as is done usually.
I would also be curious to see an ablation where for the LR-D2 variant SGD was used as the optimizer instead of Newton’s method. That variant may require more steps (similar to MAML), but I’m curious in practice how this performs.
A few other minor comments:
- In the related work section, the authors write: “On the other side of the spectrum, methods that optimize standard iterative learning algorithms, [...] are accurate but slow.” Note however that neither MAML nor MetaLearner LSTM have been showed to be as effective as Prototypical Networks for example.
So I wouldn’t really present this as a trade-off between accuracy and speed.
- I find the term multinomial classification strange. Why not use multi-class classification?
- In page 8, there is a sentence that is not entirely grammatically correct: ‘Interestingly, increasing the capacity of the other method it is not particularly helpful’.
Overall, I think this is good work. The idea is natural and attractive. The writing is clear and comprehensive. I enjoyed how the explanation of meta learning and the usual episodic framework was presented. I found the related work section thorough and accurate too. The experiments are thorough as well, with appropriate ablations to account for different numbers of parameters used between different methods being compared. This approach is evidently effective for few-shot learning, as demonstrated on the common two benchmarks as well as on a newly-introduced variant of cifar that is tailored to few-shot classification.
Notably, the ridge regression variant can reach results competitive with SNAIL that uses significantly more weights and is shown to suffer when its capacity is reduced. Interestingly, other models such as MAML actually suffer when given additional capacity, potentially due to overfitting.
We thank the reviewer for the insightful comments and analysis.
> “One-vs-all classifiers” for LR-D2
This is a great suggestion, and we are not quite sure how we missed it. We will update the results for 5-way classification incorporating this method.
> “ablation where for the LR-D2 variant SGD was used ...
instead of Newton’s method”
We previously did exactly this experiment, although for the R2-D2 (ridge regression) variant. We did not include it due to space constraints. It is equivalent to MAML, which also uses SGD, but adapting only the classification layer for new tasks (instead of adapting all parameters).
We tested this variant on miniImageNet with 5 classes, with the lowest-capacity CNN (which is the most favorable model for MAML/SGD). It yields 45.4±1.6% accuracy for 1-shot classification and 61.7±1.0% for 5-shot classification. Comparing it to Table 1, there’s a drop in performance compared to our closed form solver (3.5% and 4.4% less accuracy, respectively), and also compared to the original MAML (3.3% and 1.4% respectively).
Although we expect the conclusions for logistic regression (LR-D2) to be similar, we will extend the experiment to this case and report the results.
> “Neither MAML nor MetaLearner LSTM have been showed to be as effective as Prototypical Networks for example”
We agree, and will amend the text. Their interest may lie more in their technical novelty.
> Suggestions on multinomial term and sentence grammar
These do improve the readability of the text and will be corrected.
Summary: The paper proposes an algorithm for meta-learning which amounts to fixing the features (ie all hidden layers of a deep NN), and treating each task  as having its own final layer which could be a ridge regression or a logistic regression. The paper also proposes to separate the data for each task into a training set used to optimize the last, task specific layer, and a validation set used to optimize all previous layers and hyper parameters.
Novelty: This reviewer is unsure what the paper claims as a novel contribution.
In particular training multi-task neural nets with shared feature representation and task specific final layer is probably 20-30 years old by now and entirely common. It is also common freeze the feature representation learned from the first set of tasks, and to simply use it for new tasks by modifying the last (few) layer(s) which would according to this paper qualify as meta-learning since the new task can be learned with very few new examples.
We thank the reviewer for the comment.
However, we believe that the low score originates from a misunderstanding of our proposal.
Below, we try to bring some clarity by disambiguating between what the reviewer refers to and our method.
If our interpretation of what the reviewer refers to as “entirely common” is incorrect, it would be great to be provided with at least one reference, so that we can continue the conversation on the same ground.
> “novel contribution?” , “training multi-task neural nets with shared feature representation and task specific final layer is probably 20-30 years old by now and entirely common.”
“It is also common freeze the feature representation learned from the first set of tasks, and to simply use it for new tasks by modifying the last layer”
We understand that the reviewer is hinting at the common multi-task scenario with a shared network and task-specific layers (e.g. Caruana 1993). He/she also refers to basic transfer learning approaches in which a CNN is first pre-trained on one dataset/task and then adapted to a different dataset/task by simply adapting the final layer(s) (e.g. Yosinski et al. “How transferable are features in deep neural Networks?” - NIPS 2014; Chu et al. “Best Practices for Fine-tuning Visual Classifiers to New Domains” - ECCVw 2016).
If so, then this is significantly different to our work, and in general to all of the previous literature on meta-learning applied to few-shot classification (e.g. Finn et al. 2017, Ravi & Larochelle 2017, Vinyals et al. 2016, etc).
Notably, these methods and ours take into account adaptation *already during the training process*, which requires back-propagating errors through the very fine-tuning process.
Within this setup, our main contribution is to propose an adaptation procedure based on closed-form regressors, which have the important characteristic of allowing different models for different episodes while still being fast because of 1) their convergence in one (R2-D2) or few (LR-D2) steps, 2) the use of the Woodbury identity, which is particularly convenient in the few-shot data regime, and 3) back-propagation through the closed-form regressor can be made efficient.
To better illustrate our point, we conducted a baseline experiment.
First, we pre-trained the same 4-layers CNN architecture, but for a standard classification problem, using the same training samples as our method. We simply added a final fully-connected layer (with 64 outputs, like the number of classes in the training splits) and used the cross-entropy loss.
Then, we used the convolutional part of this trained network as a feature extractor and fed its activation to our ridge-regression layer to produce a per-episode set of weights.
On miniImagenet, the drop in performance w.r.t. our proposed R2-D2 is very significant: 13.8% and 11.6% accuracy for the 1 and 5 shot problems respectively.
Results are consistent on CIFAR, though less drastic: 11.5% and 5.9%.
This confirms that simply using a “shared feature representation and task specific final layer” as commented by the reviewer is not what we are doing and it is not a good strategy to obtain results competitive with the state-of-the-art in few-shot classification.
Instead, it is necessary to enforce the generality of the underlying features during training explicitly, which we do by back-propagating through the fine-tuning procedure (the closed-form regressors).
We would like to conclude remarking that, probably, the source of confusion arises from the overlap that exists in general between the few-shot learning and the transfer/multi-task learning sub-communities.
We realize that the two have developed fairly separately while trying to solve very related problems, and unfortunately the similarities/differences are not acknowledged enough in few-shot classification papers, including our own. We intend to alleviate this problem in our related work section, and invite the reviewer to suggest more relevant works from this area.
I disagree with Reviewer #2 and the authors about the novelty. The delta from just simple multi-task learning approach of eg Caruana 93 is extremely small -- the same algorithms are trivially extended to deal with meta-learning. The mere fact of using closed form ridge regression in this setting does not feel like sufficient contribution to warrant an ICLR paper to this reviewer. Merely because some other paper also had small novelty and got accepted in the past I can not see why this paper should also get accepted with minimal novel contributions.
The reviewer has not refuted any of the points we made above.
Namely:
- That meta-learning approaches (like ours) back-propagate errors through the fine-tuning process, a major departure from standard multi-task/transfer learning.
- That *not doing so* incurs a large performance penalty, as demonstrated by our experiments.
We invite the reviewer to address these points, rather than just reiterate a subjective judgment over the value of meta-learning.
While we respect this opinion, our paper cannot be rejected based solely on the reviewer’s opinion that meta-learning papers are not novel in general (compared to multi-task learning).
The 3 meta-learning papers developed new techniques and/or models for meta-learning (which have never been proposed or used in multi-task learning), while this paper applies existing multi-task learning technique in the meta-learning setting. The contributions in these two cases are very different. I think it is misleading to indicating that the contribution of this paper is as novel as the 3 meta-learning papers.
It is fine to apply multi-task learning technique to the meta-learning problem.
To some extent, meta-learning can be explained as a generalization of multi-task learning in the way that meta-learning applies to any set of tasks sampled from certain *task distribution*, while the set of tasks in multi-task learning are fixed. They both need knowledge transfer between different tasks.
However, using a multi-task technique in meta-learning setting cannot be treated as a novel or original contribution.
1) We wrote: ““[multi-task learning] is different to our work, and in general to all of the previous literature on meta-learning applied to few-shot classification (e.g. Finn et al. 2017, Ravi & Larochelle 2017, Vinyals et al. 2016, etc).
Notably, these methods and ours take into account adaptation *already during the training process*, which requires back-propagating errors through the very fine-tuning process.””
2) R1 answered with: ““Merely because some other paper also had small novelty and got accepted in the past I can not see why this paper should also get accepted””
3) We then observed that *R1 did not refute any of our point of rebuttal* (long answer in this thread) and seems to be dismissive of the above papers, which are widely accepted by the community.
> ““ However, using a multi-task technique in meta-learning setting cannot be treated as a novel or original contribution.””
Again, it is not what we do - we amply addressed this point both on OpenReview (last two answers to the reviewer) and in the paper.
We would like to repeat that if this were true, the baseline experiment we described (applying ridge regression in the manner that the reviewer refers to as standard) would not have been possible, since our method and the baseline would then be the same (which they are not -- both in methodology and results).
This paper proposes a new meta-learning method based on closed-form solutions for task specific classifiers such as ridge regression and logistic regression (iterative). The idea of the paper is quite interesting, comparing to the existing metric learning based methods and optimization based methods.
I have two concerns on this paper.
First, the motivation and the rationale of the proposed approach is not clear.
In particular, why one can simply treat \hat{Y} as a scaled and shifted version of X’W?
Second, the empirical performance of the proposed approach is not very promising and it does not outperform the comparison methods, e.g., SNAIL.  It is not clear what is the advantage.
We thank the reviewer for the comments and questions.
> “Why one can simply treat \hat{Y} as a scaled and shifted version of X’W?”
In the case of logistic regression, the scaling and shifting is not needed, and we have \hat{Y}=X’W. This is because logistic regression is a classification algorithm, and directly outputs class scores. These scores are fed to the (cross-entropy) loss L.
However, ridge regression is a regression algorithm, and its regression targets are one-hot encoded labels, which is only an approximation of the discrete problem (classification). This means that an extra calibration step is needed (eq. 6), to allow the network to tune the regressed outputs into classification scores for the cross-entropy loss L.
> “The empirical performance of the proposed approach is not very promising and it does not outperform the comparison methods, e.g., SNAIL”
Our method actually outperforms SNAIL on an apples-to-apples comparison, with the same number of layers. We would like to draw the reviewer’s attention to the last paragraph of the “Multi-class classification” subsection (page 8).
The result mentioned by the reviewer uses a ResNet, while we use a 4-layer CNN to remain comparable to prior work. SNAIL with a 4-layer CNN ([11] Appendix B) performs much worse than our method (7.4% to 10.0% accuracy improvement).
Even disregarding the great difference in architecture capacity, our proposal's performance coincides with SNAIL on miniImageNet 5way-5shot and it is comparable on 3 out of 4 Omniglot setups. We would have liked to establish a comparison also on CIFAR, but unfortunately the official code for SNAIL hasn’t been released.
Borrowing the words of AnonReviewer2: “Notably, the ridge regression variant can reach results competitive with SNAIL that uses significantly more weights and is shown to suffer when its capacity is reduced.”
We hope that this addresses the two concerns raised by the reviewer. We will be happy to answer any other question about the paper.
Using closed-form base learner is an interesting idea.
However, the results are underwhelming.
As shown in https://github.com/gidariss/FewShotWithoutForgetting , Prototypical Networks can be quite powerful with some modifications. The modifications include:
1. add data augmentation
2. use SGD with momentum optimizer
3. scale the output of the euclidean distance to a suitable range
Using a 4-Conv backbone with 64 channels, Protypical Networks are able achieve remarkable results in MiniImagenet:  1-shot: 53.30% +/- 0.79 5-shot: 70.33% +/- 0.65
Even without data augmentation, in my experiments, Protypical Networks can still get 5-shot accuracy around 68.8%.
Considering this, the proposed method has not demonstrated superior empirical results than Protypical Network yet.
We thank the anonymous commenter for pointing out a GitHub repo with improvements. We note that neither data augmentation nor the optimizer schedule are mentioned at all in the associated published paper.
Additionally, the mentioned improvements are not specific to prototypical networks (or to any method for that matter), and can also be applied to ours.
As such, we fail to see how this says anything about the merits of our proposal.
In our experiments, we compare against prototypical networks using the same setup of the original paper (Adam optimizer, halving LR every 20 epochs; no data augmentation).
In this fair comparison, we outperform it.
We would gain no knowledge by showing that “proto-nets with data augmentation and optimizer improvements” (as suggested) beats “R2D2 with no data augmentation”, or that “MAML with a ResNet base” beats “R2D2 with 4 layers”. These are apples-to-oranges comparisons which make any scientific conclusion very hard to draw.
Instead, a proper comparison is to take the innovation of each paper -- the prototype layer in proto-nets, and the ridge regression layer in R2D2 -- and compare them, with everything else fixed. This includes data augmentation, as well as network model and initialization.
Carefully controlled comparisons are a core part of the scientific method, and ignoring them will lead to unsubstantiated conclusions.
I understand your points.
Overall, I like the idea of using closed-form base learner which also demonstrate good performance when the backbone network is shallow.
However, as a practioner, I may not adopt the proposed method for now.
In my opinion, meta-learning is about learning a data-driven inductive bias for few-shot learning. Closed-form regression itself introduces a strong inductive bias which is not learned.
Therefore, it is interesting to investigate whether the inductive bias of closed-form regression is needed when the backbone network gets deeper.
As shown in the Figure 3 of https://openreview.net/pdf?id=HkxLXnAcFQ , the performance gap between different meta-learning methods diminishes as the backbone gets deeper. One intersting point in the figure is that ProtoNet typically outperforms other methods when the network is deeper.
Thank you for pointing us to this interesting paper! We agree that methods with limited inductive bias such as protonets are attractive, and there is indeed a good case for their performance scaling better with computation and data.
We are looking forward to try out the proposed testbed. One possible advantage of using our  R2-D2 with the deeper architectures of their setup is that we can concatenate activations from multiple layers together without increasing the computational burden of the base-learner thanks to the Woodbury identity.
We would like to thank both reviewers and anonymous commenters for their feedback and participation.
In light of the discussion, the Appendix of the paper has been updated:
* Section B offers a runtime analysis which reveals that R2-D2 is several times faster than MAML and almost as fast as a simple (fixed) metric learning method such as prototypical networks, while still allowing per-episode adaptation.
* Section A reports the accuracy of the 1-vs-all variant of LR-D2 (as suggested by AnonReviewer2), which is comparable with the one of R2-D2.
* Finally, Section C extends the discussion sparked here on OpenReview about a) the nature of our contribution b) the disambiguation with the multi-task learning paradigm .
The ICLR'19 camera-ready version of the paper has been uploaded.
Code available at https://github.com/bertinetto/r2d2

Meta-review:
role: AC
data: metareview: The reviewers disagree strongly on this paper. Reviewer 2 was the most positive, believing it to be an interesting contribution with strong results. Reviewer 3 however, was underwhelmed by the results. Reviewer 1 does not believe that the contribution is sufficiently novel, seeing it as too close to existing multi-task learning approaches.

After considering all of the discussion so far, I have to agree with reviewer 2 on their assessment. Much of the meta learning literature involves changing the base learner *for a fixed architecture* and seeing how it affects performance. There is a temptation to chase performance by changing the architecture, adding new regularizers, etc., and while this is important for practical reasons, it does not help to shed light on the underlying fundamentals. This is best done by considering carefully controlled and well understood experimental settings. Even still, the performance is quite good relative to popular base learners.

Regarding novelty, I agree it is a simple change to the base learner, using a technique that has been tried before in other settings (linear regression as opposed to classification), however its use in a meta learning setup is novel in my opinion, and the new experimental comparison regression on top of pre-trained CNN features helps to demonstrate the utility of its use in the meta-learning settings.

While the novelty can certainly be debated, I want to highlight two reasons why I am opting to accept this paper: 1) simple and effective ideas are often some of the most impactful. 2) sometimes taking ideas from one area (e.g., multi-task learning) and demonstrating that they can be effective in other settings (e.g., meta-learning) can itself be a valuable contribution. I believe that the meta-learning community would benefit from reading this paper.


Decision: Accept (Poster)