Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a simple large-margin-nearest-neighbors-based penalty for few-shot learning. The proposed method (POODLE) learns the inductive bias of deep neural networks for few-shot learning by leveraging out-of distribution data. Specifically, POODLE penalizes the large distance between the prototypes and positive samples and the small distance between the prototypes and negative samples. Experiments demonstrate the effectiveness of the proposed method.
main_review: *Strengths
1. The paper is well written and easy to follow.
2. The idea of viewing classification problems as conditional reasoning is interesting. 
3. The authors present adequate experiments on several protocols, including standard inductive, transductive, cross-domain FSL, and imbalanced query set.

*Weaknesses
1. The contribution of this paper is limited. The main contribution is the newly proposed loss function. But the idea of pushing negative samples and pulling positive samples is not rare in metric learning.
2. Although the illustration in Figure 1 is logically correct, it is still hard for readers to accept the motivation from the following experiments. Is it possible to present more analysis to the effectiveness of the proposed method using more than just classification accuracy?
3. The authors should show some analysis on selecting different number of negative samples. It is also not fair to use 400 samples for each novel task, which has only 5 (for 5-way-1-shot) or 25 (for 5-way-5-shot) samples.
4. The comparison in Figure 2 is disadvantageous. In Figure 2(a), the accuracy gain of the proposed method only outperforms SSL in Conv-4-512 backbone. In Figure 2(b), the backbone information is not provided.
5. The result of using simple baseline is not well enough to outperform the state-of-the-art methods. For example, the result of Simple + POODLE on mini-ImageNet is 63.79 and 81.41 in inductive setting, which is not presented in Table 3.

==============

I appreciate the authors' efforts in their response, and most of my concerns have been solved.

The newly added experimental results are also convincing for me. In this sense, I would like to increase my rating to 6 and recommend the authors to include some new experimental results in their revision and make a much more clear clarification.

limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors improve the performance of metric-based few-shot learners by adding a new term based on out-of-distribution samples' to the loss for ProtoNets. 
main_review: The paper is written well. They describe the contributions clearly. However, the idea of adding out-of-distribution samples requires more ablation study from my point of view.

Minor comments:
Line 26: If X has P X is Q. I do not see how this relates to this work in particular? I understand that you are talking about using features to classify inputs, but this is just a general intuition about how all models in deep learning work. My point is that this sentence does not have to do anything with network architecture or choices in this paper. So it is better to remove this claim from the introduction or clarify more how this sentence is related to your work.

Line 40: What do you exactly mean by little overhead in inference? Are you talking about finding the distance from each class prototype? What if we have around 10000 classes? Is that overhead still negligible?
Related work is also well written and covers important related work. In line 74, I liked how the authors clarify transductive and inductive settings in the paper. 

With regards to Formula 2, d(w_k, x_k). Should not these both be the output of the feature extractor? since in line 112 it is mentioned that d denotes the distance function. This literally means you apply a distance function on raw inputs. Please fix this.

Questions/Suggestions:
Equation 1 is the same as in the ProtoNets[1] paper. Also, Line 115 inline formula is exactly how ProtoNets perform this (Formula 1 in ProtoNets). So it should be clarified that this is where it is coming from. Right now it seems like that this is a new idea proposed by this paper. There are two differences with the formula in ProtoNets though: 
1- Using \gamma
2- Normalizing the prototype vectors

What is the value of \gamma in the experiments? Why is it useful? How does it impact the results? What is the impact of normalizing the prototype vectors? Furthermore, I believe that increasing the number of shots should eventually resolve the ambiguity. I suggest that authors perform an experiment to find what is the number of shots after which their method does not improve the performance. Right now, it seems that both in 1-shot and 5-shot settings, their method outperforms other baselines.
I increase my score if I see ablation studies for these since this could be really useful for someone who wants to use metric-based methods.

My main concern with regards to inductive bias and Figure 1: How can we be sure that out-of-distribution examples are really out-of-distribution? What if it was an image of a child with a suit? Are you making sure you do not pick that one? In line 151 it is mentioned that there is prior knowledge whether the data point is in- or out-of-distribution. Is that possible to do this in the real world? Since we do not have labeled data. Furthermore, what if the task was to classify based on clothes instead of person? It does not really solve the ambiguity. In fact, it can even make the classifier perform worse. Why this does not happen?

I think the claim in Line 173 that this is the first loss function for both transductive and inductive inference is wrong. MAML[2] objective is also effective for both. ProtoNets objective is also effective for both cases. You can just decide to use batch norm information from the whole batch or not. Please clarify this or remove this claim. 

Line 159 - 161: The objective does not take into account class assignments for positive instances. But in Formula 3, it seems that you take into account the class assignment for both positive and negative instances. Can you clarify this, please?

[1] Snell J, Swersky K, Zemel R. Prototypical networks for few-shot learning. In Proc. of the 31st Int'l Conf. on Neural Information Processing Systems 2017.

[2] Finn C, Abbeel P, Levine S. Model-agnostic meta-learning for fast adaptation of deep networks. In Int’l Conf. on Machine Learning 2017.

===============Post Rebuttal ==================

I thank the authors for their quick and accurate responses. I really appreciate their hard work in answering the reviewers' concerns. Unfortunately, I am not going to increase my score and I think this work needs to focus on some substantial improvements that I will explain. I just want to clarify that from my point of view, there is nothing wrong with using well-known techniques such as pull and push terms that are commonly used in deep learning. In fact, it is a good idea to use them to solve new problems and the novelty does not depend on using them, but on how they are assembled to solve a new problem. Thus there is much value in this work that studies adding a pull term for out-of-distribution samples. However, I think to make this work stronger the authors need to study the choice of out-of-distribution samples in much more detail. There are also some claims which I think are not correct. For example, the claim in line 173. The authors argue that *For transductive, the pull loss considers samples of both support and query set (minimizing the weighted distance to support and query features).* This seems not correct to me at all especially for metric-based few-shot learning that is based on minimizing the distance between support and query. It is still not a good practice but acceptable by the few-shot learning community if something like batch normalization data leaks from the query set, but using query set and support set to minimize the weighted distance of features is really bad practice. Regardless of this, I think using 400 out-of-distribution samples actually contradicts few-shot learning. My biggest concern is that in practice, how can we choose out-of-distribution samples? For example, if I want to train a classifier on clothes or different cars. If the authors study this part in their paper thoroughly, their work's impact will be much larger and I strongly suggest they focus on this for future versions of this work.

===============Post Rebuttal After Clarification by Authors==================

Thanks to the authors, my concern about transductive settings were resolved. After discussion with other reviewers and carefully reading the authors' responses I decided to increase my score to 7. The authors did a great job in their rebuttal and I strongly suggest including all of these studies and comparisons in the final draft and supplemental material. I think after these discussions this paper is a valuable resource for anyone who wants to explore using out-of-distribution samples for metric-based few-shot learning. The studies can provide insights for researchers and practitioners in the future. 

limitations_and_societal_impact: The authors explained the limitation and broader impact of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Authors examine the use of out-of-distribution samples for conditioning few-shot classifiers as they adapt to novel classes. Authors fine-tune a classifier layer at test time, encouraging in-distribution samples to lie close to class representations while out-of-distribution samples lie far away. This simple approach achieves consistent accuracy gains in multiple evaluation settings. 
main_review: STRENGTHS:
The approach and motivation are interesting. The loss itself is simple, intuitive, and fairly versatile. Experiments are extensive and cover a wide range of settings. Authors do not hesitate to re-implement their and other models over many settings (see also supplementary). Ablation in supplementary is detailed, yet straightforward. Paper is well-organized and clearly presented. 

WEAKNESSES:
Unfortunately, I am not convinced that POODLE works as described, and cannot judge its significance or impact. Three reasons:

- Tables 1 and 2 are interesting, but not convincing. Without confidence intervals over the evaluation trials I cannot tell whether the observed improvements are significant. And more fundamentally, while using the simple CELoss model as the baseline works in demonstrating improvement, it does not indicate improvement that is meaningful. The CE baseline is an inductive model for vanilla FSL, but your evaluation settings in Tables 1 and 2 are semi-supervised SSL and transductive SSL respectively. Thus it is impossible to tell if improvement is coming from your novel loss function, or from the fundamental shift from inductive evaluation to a more friendly evaluation setting. Appropriate baselines would be other semi-supervised learning methods and losses (few-shot or otherwise) such as [1] or [2] for table 1, or other transductive methods for table 2 (as in table 4). As is, the improvement shown is deceptively large. 

- No analysis or convincing explanation for why this works. All results appear as accuracy scores; it is impossible to tell if POODLE works for the reasons described (suppressing distractor features). There are even some indicators that it doesn’t: the fact that uniform-POODLE is competitive and sometimes superior to base-class-POODLE indicates that the loss function may not be suppressing distractor features at all, but rather acting as a simple regularizer on labels akin to label smoothing.

- Lack of simple ablations. Some design decisions for the POODLE loss go unexplored, with important baseline ablations absent. 1) What is the performance of a model that simply assigns all negative samples to a distractor class? 2) The pushing loss incentivizes negative samples to be equidistant from all class vectors. What is the performance of a model that does this explicitly, by maximizing entropy of class predictions for negative samples? And as per above: how would this differ from a similar level of label smoothing? 3) As mentioned in sec.5.4, weighting the pull loss by the class predictions can lead to problematic behavior where large classes dominate behavior. Why not use the ground-truth class assignment instead of the predicted one?

In sum, it is not clear to me why and to what degree POODLE actually works. The second point is the most fundamental issue. Additionally:
 
- Not actually complementary to many FSL methods. Lines 38-39, 84-85 imply that POODLE can be applied to FSL methods broadly. This is not the case: it only applies to methods that fine-tune a classifier layer at test time, which precludes all metric-based approaches to FSL. Thus POODLE is not nearly as versatile as claimed. 

- I hesitate to call POODLE SOTA for inductive settings. In a fair comparison without bells or whistles, [3] outperforms simple-POODLE in all settings, and going strictly by the highest published numbers, [3] outperforms the best POODLE on tiered-ImageNet and CUB. Also, DeepEMD is cited as related work but not included in Table 3, though in this case that doesn’t change the results. 

These last two issues are less important and can probably be fixed with small language changes. 

SMALL COMMENTS (feel free to ignore in discussion):
- Missing citation: [4] is a more recent and SOTA example of employing a self-supervised auxiliary loss for FSL
- Where does the name POODLE come from? I assume it’s “Penalizing Out Of Distribution sampLEs” but you never actually explain it. 
- In sec4 and later, you use “prototype” to refer to the class representation vector in the learned classification layer. I would consider making explicit the fact that you are referring to this learned vector and not the “class prototype” in the sense of prototypical networks (even if yours are at first initialized to the same thing). 
- Line 220 is misleading, if not outright factually wrong: your improvement in table 1 is lower than 1pp in many settings (eg. Rot+KD improves by only .09pp on 5-shot CUB). 
- I suspect you have mislabeled Table 2: L_push should be L_pull and vice versa. As-is, your notation conflicts with eq.4 and text lines 224-227 (and makes results difficult to interpret). 

[1] Ren et al. ICLR 2018: Meta-Learning for Semi-Supervised Few-Shot Classification

[2] Saito et al. ICCV 2019: Semi-Supervised Domain Adaptation via Minimax Entropy

[3] Wertheimer, Tang and Hariharan CVPR 2021: Few-Shot Classification with Feature Map Reconstruction Networks 

[4] Doersch, Gupta and Zisserman NEURIPS 2020: CrossTransformers: spatially-aware few-shot transfer

POST-REBUTTAL:

I'd like to echo the first half of reviewer e7JL's post-rebuttal comment: I really appreciate the clear effort authors put in to address our concerns, there's good value in this work, and in my eyes too novelty is sufficient. Regarding my own review: authors mostly addressed my concerns, with the sole exception that we still have no empirical analysis of POODLE behavior beyond accuracy scores, meaning that we still cannot be entirely sure what POODLE is actually doing in practice. However, after additional clarification the method does make intuitive sense so this is not as troubling to me as it was initially.
limitations_and_societal_impact: No negative impacts are discussed. That said, any risks or dangers inherent to this method are shared with few-shot learning more broadly, so a brief mention of this would likely suffice. Also, if POODLE does in fact work by suppressing distractor features, the ability to target certain distractors (i.e. age, race, gender) could be a very large impact, though of course that is outside the scope of this particular paper to examine or demonstrate.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The paper proposes to learn from out-of-domain examples in order to help few-shot learning (FSL). The idea is to maximize the distance from prototypes to out-of-distribution samples while minimizing that of in-distribution samples. Experiments are done on standard FSL benchmarks.

main_review: Pros:
- Easy to read

Cons:
- It is hard to sense the technical novelty of this method. Using this kind of margin distance to learn in-domain and out-of-domain examples is a very known technique. I did not see the novelty in Section 4.1.
- The experiments are relatively weak and distracting.
    - The method only compares with very basic methods on standard FSL benchmarks. Not sure how it compares with sota.
    -  The experiments also show smaller computation cost and better performance on imbalanced setup. I am confused why these 
        experiments are useful to the main contribution of the paper.
    -  The setup of the key experiments (cross-domain FSL) is not convincing. 
- The paper even did not follow the submission guideline. The manuscript still has a broader impact section which is not needed 
   this year.

limitations_and_societal_impact: Yes. Not sensitive area.
review_title: null

Meta-review: Out of the 3 reviewers who spend the most time on this paper, they all thought that the submission was above the acceptance threshold. Reviewers and I especially appreciated the authors' active engagement and helpful responses. 

The idea of using unlabeled out of domain data to improve few-shot learning should be generally useful, and the authors show good empirical performance. As the reviewers pointed out, the presentation is clear. For these reasons, I recommend the acceptance of this paper. 

Decision: Accept (Poster)