Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: * This paper proposes an approach to few-shot classification developing on related works such as MAML and Body Only Inner Loop (BOIL).
* In few-shot learning settings which involve domain shift between the training and evaluation settings, 
models necessarily require some adaptation in their parameters in order to be able to solve the testing time tasks. This is because the representations learned by the earlier layers need to change in order to be effective on the new task.
* The BOIL method approaches this problem by forcing the inner layers of the network to adapt in the inner loop of meta-learning.
* However, the BOIL method is constrained in that one must pre-determine the number of output classes at training (and therefore at testing time, which must be equal to the number at train time) because this architectural property of the model cannot change once it is trained. This is a limiting factor when seeking to apply few shot models in practice, when testing scenarios may necessarily be different from training scenarios.
* This work proposes a method to combine a non-parametric classification head with an adaptation procedure for a network body. This allows both the lower level representations of the model adapt and also the number of classes at test time to vary.
* In evaluation, the paper standardises a setting where a 4-conv architecture (used in the original MAML paper) is trained on MiniImageNet, and then considers various evaluation paradigms. These include: testing on MiniImageNet (in-domain), testing on FC100 (out of domain), varying the number of ways/shots at evaluation time vs training time.
* The paper also conducts representational similarity analysis on the methods.
* The key empirical finding is that the proposed strategy can improve performance in settings over BOIL, MAML, and ANIL (uses no inner loop during training for the network body) where domain adaptation and changes in ways/shots between train and test time occur. 
* In settings where one or both of these do not change, the method mostly performs competitively with the other strategies.


strength_and_weaknesses: ## Major comments

The paper's key idea to try to make an algorithm like BOIL more flexible by allowing a non-parametric head makes sense and is generally well-motivated. The experimental results are also overall encouraging, though I am not sure why the proposed method performs poorly compared to baselines in Table 3c, in-domain. 

However, the paper was hard to read and I feel that the main methodological idea was not that well explained or motivated. Here are a concrete set of improvement areas:

* There are a number of typos/lack of careful definitions in the presentation of the mathematical part of the paper. It is possible to decipher some of these, but it makes understanding the method a lot harder. For example: in eqn 9, what is $d$? What is $l$? I think $l$ is a typo for $k$, but this makes following it harder. In Equation 10 and 11, why are we subtracting $c_i$ and $c_j$? Are these meant to be commas (like in equation 9)? What are $d_{\text{support}}$ and $d_{\text{query}}$? What is $g_{\text{body}}(S;\theta)$?I don't think these are clearly defined enough, and it makes the contribution very hard to follow.
* The intuition behind the objective in Eqns 10 and 11 was not explained clearly enough. The figure in the appendix helps, but given this is a very important part of the paper, I think it needs more explanation. Why is this particular form of the objective better than something else?
* Relatedly, the paper says that the BOIL formulation cannot work well with the NIL head. I am not fully sure why this is the case, and the explanation in the paper was not clear enough -- this seems like the simplest baseline to try. Also, even if it doesn't work, it should be evaluated in the experiments as the most obvious baseline method.
* Given a major part of the study is domain shift, studying only one setting of domain shift from MiniImageNet to FC100 is in my opinion, not sufficient. I think the paper should at least present analysis on Cars, CUB, or something like that from the BOIL paper. In addition, it would be good to evaluate methods at least overall on Meta-Dataset, given it's a more realistic testbed for FSL methods.
* Unless I missed it, Table 3 doesn't get much attention in the text, and I would be curious to know more about why the method does worse in the in-domain setting here. The performance deltas in Table 3c, MiniImageNet are quite substantial.


## Minor
Some other typos/suggestions:
* BOIL could use a citation in the intro.
* In eq 8, the transpose operator and leading dimension are both using $T$. It's better to use a different letter for the dimension, or change the way the transpose is typeset.
* Details on proto-MAML are not very clear -- what is the subscript $k$? 
* End of section 2.2.2: typo when defining the last convolutional block?
* Table 1 is missing a heading -- I think the FC100 heading is in the wrong place, and MiniImageNet heading is missing. 


clarity,_quality,_novelty_and_reproducibility: The key idea in the work is to the best of my understanding original, but the explanations in the paper and attention to detail in presentation are not sufficient, making it very hard to follow and appreciate. 
summary_of_the_review: Overall, I think this idea has promise. However, there are too many errors in the paper's presentation and a general lack of motivation about the method used for it to be accepted at this stage. I think the paper needs substantial revisions and cleaning up in the methodological presentation, as well as some more comprehensive experiments in the domain-shift setting (with other datasets), before I can recommend it for acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new variant of the MAML algorithm, called "NP-MAML" for non-parametric MAML. The method uses a non-parametric head (final layer) using distances to a prototype to assign labels to new data, while still updating the body of the network in the meta-learning inner loop. The paper compares this method to related methods (BOIL/ANIL/MAML) on few-shot image classification tasks.
strength_and_weaknesses: Strengths:
- Simple idea, well motivated
- Clear experiments that support the main conclusions of the paper

Weaknesses:
- Would be nice to see experiments on other domains (besides images)
- The representation similarity experiments are intriguing, but could be further developed

Minor comments:
- In the introduction, the BOIL acronym is introduced in the fourth paragraph without explaining what it is.
- Missing header in Table 1
clarity,_quality,_novelty_and_reproducibility: The work is to the best of my knowledge novel. The background, methods, experiments and discussion are all clearly written.
summary_of_the_review: Overall, I thought this was a nice and interesting modification to MAML. I think the paper could be improved by demonstrating the performance of NP-MAML on additional tasks, as well as providing more intuition for what types of changes are happening in the representation (extending the CKA analysis—although perhaps this latter point is better left for future work).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors investigate MAML's inner loop adaptation dynamics in a variety of different task distributions and find that in some cases, earlier layer inner loop adaptation is key for improvements, i.e. heterogeneous tasks. They also propose a method that builds on MAML that is able to improve performance in heterogeneous tasks, while remaining competitive in the remainder of the tasks.
strength_and_weaknesses: Strengths:

- Excellent and thorough empirical evaluations in terms of task formulations
- Interesting empirical insights about MAML
- Novel proposed method that is both well-motivated and effective

Weaknesses:

- The paper seems to be missing key related work, such as [1, 2, 3] and many others, but those three in particular studied MAML and came up with lots of useful insights that could have been integrated into this work.
- Possibly a side-effect of the above, the empirical experiments don't seem to incorporate some of the more powerful MAML-related models or ablation with such SOTA models along with the proposed method on top. As a result, it's hard to judge whether the proposed model is unique in its improvement in heterogeneous cases or whether its benefits disappear when used in conjunction with other powerful and existing SOTA models.

*Request: You state that there are 'no straightforward' ways of making MAML heterogeneous, however, what I've seen done, is learning a single 'class' weight matrix, that is cloned N times for an N-class problem, and then fine-tuned in the inner layer -- therefore learning a generic weight matrix that is quickly adaptable for any new N-way task coming in, no matter what N is. I'd love to see that baseline.


1. Rusu AA, Rao D, Sygnowski J, Vinyals O, Pascanu R, Osindero S, Hadsell R. Meta-learning with latent embedding optimization. arXiv preprint arXiv:1807.05960. 2018 Jul 16.
2. Antoniou A, Edwards H, Storkey A. How to train your MAML. arXiv preprint arXiv:1810.09502. 2018 Oct 22.
3. Antoniou A, Storkey AJ. Learning to learn by self-critique. Advances in Neural Information Processing Systems. 2019;32.

For additional few-shot papers that could be of interest, to ensure completeness, might be worth looking at

4. Hospedales T, Antoniou A, Micaelli P, Storkey A. Meta-learning in neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence. 2021 May 11;44(9):5149-69.
clarity,_quality,_novelty_and_reproducibility: The work is clearly communicated and is of high quality. The novelty of the work is about 6/10, because it tackles a diverse set of under-explored task distributions, and discovers cool insights about MAML.
summary_of_the_review: I gave this paper a weak accept due to its interesting empirical insights and novel direction of exploring heterogeneous task distributions, and would be willing to increase my rating if:

- More SOTA models are included in the ablation studies.
- The proposed method is incorporated on top of existing and suitable SOTA methods to check if there are any benefits once a model is sufficiently strong.
- Some experiments are ran with the baseline stated in the strengths and weaknesses section, as it would dispel any doubts about how a more 'default' MAML model would perform in heterogeneous settings when it is enabled to do so by learning a quickly adaptable 'single class' weight matrix, that is cloned N times for a given N-way task and fine-tuned on the inner loop.
- More care is taken to include relevant related work, as it feels like the currently included work includes some historical work, and some more recent papers but misses much of the in-between work that is directly related with what the authors are doing in this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Traditional MAML is well used in few-shot learning domains. However, recent works show that MAML does not perform well in a fast adaptation. This paper gives a detail about feature reuse phenomenon and found the relationship between lower layer and the distribution differences in training and testing.
strength_and_weaknesses: Pros

In this work, the problem of MAML is divided into two main challenges including algorithmic challenge and architectural challenge . algorithmic challenge represents the performance of cross domain under the domain shift and architectural challenge is related to the model structure such as pixel input and the dimension of the output. These two challenges are reasonable and the results seem good in the experiment section.

Cons

NP-MAML updates the body layer with a distance using prototypes. In this respect, for the fair comparison, the architectural challenge needs to be compared with embedding methods and it is similar to embedding few-shot learning methods such as a prototypical network. 
clarity,_quality,_novelty_and_reproducibility: In my opinion, this work is the combination of a prototypical network and MAML framework. There is no code for reproducibility.

summary_of_the_review: I think this method needs to be compared to SOTA-embedding few-shot learning methods for verifying the effectiveness of MAML + prototypical network not the BOIL and ANIL which are using a fully-connected layer.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The idea of the paper is to build a few-shot learning model which can be seen as a combination of prototypical networks and MAML.
1) The method uses neg-distances to prototypes (similarly to PN) as logits. This allows to have a different number of classes at test time.
2) The method adjust the encoder using several steps of gradient descent (GD) similarly to MAML. This allows the model to improve its performance in cross-domain adaptation.

The central element of this idea is the loss which is minimized with GD to adapt the encoder. The authors propose to *maximize* the distance to the prototypes, which is a somewhat unintuitive choice.
The authors compare the methods to existing few-shot learning algorithms on a few benchmarks and report good results obtained with the proposed algorithm.

During the discussion, the reviewers raised the following concerns:

- The lack of clarity of the presentation. Although the clarity has been improved a lot during the discussion, the current version of the paper still contains errors and typos. The central part of the proposed method (equations 11 and 12) is not explained well and it required further corrections and clarifications on the openreview discussion page. The algorithm box that the reviewers posted on openreview was also very helpful and should be included in the paper.

- The motivation for the update rule (12) (which is the central contribution of the paper) remains unclear. The authors do not provide a satisfying explanation/intuition to why maximising the distance is a good idea. The explanation provided on openreview "we found it apt to choose a contrastive-type loss, disentangling intra-class from extra-class features, which is fairly common in self-supervised learning [2]." is not very convincing. It is unclear why it is a contrastive loss and what kind of disentanglement it encourages. The readers would benefit from seeing a discussion of other possibilities for the loss function and why the proposed loss is a good choice. For example, why not to use the standard PN loss for adapting the encoder?

- Too few benchmarks and baselined used in the experiments. The authors have addressed this concern by including more benchmarks (CIFAR-FS and CUB) and more baselines (Unicorn MAML and PN).

In conclusion, the reviewers think that this is a promising piece of work and the paper is very close to meet the acceptance criteria. However, given a relatively minor novelty, they believe that more polishing is required to fully meet the acceptance criteria. The authors are encouraged to incorporate all the clarifications requested by the reviewers and provide more intuition/motivation for the loss used in the adaptation phase.
justification_for_why_not_higher_score: unsatisfying explanation of the central part (eq. 12)
justification_for_why_not_lower_score: The experimental results are good.
Decision: Reject