Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors present a discussion on two main-stream meta-leaners with attentional classifiers and threshold meta-learners under a unique view of polythetic classification. And to address the limitations of both, attention-based feature selection is introduced. 
Improved performance is demonstrated by both synthetic and real-world few-shot learning tasks.

main_review: **Strengths**

This paper discusses meta-learner from a very unique perspective, with the shortcomings of both ProtoNet-liked threshold meta-learners and MatchingNet-liked attentional meta-learner discussed solidly with intuitive examples and convincing derivation of misclassification rates. 

The proposed attentional feature selection is simple yet effective, and can potentially guide and stimulate many follow-up improvements to meta-learning. 

**Weaknesses**

My main concern is that this paper tends to isolate itself from the entire research of meta-learning, and focus on a corner instead. 
ProtoNets (representing threshold meta-learners) and MatchingNets (representing attentive meta-learners) are insufficient to cover the entire research of meta learning. There are many other directions of meta-learning that are completely overlooked in the discussion. 
The most representative case is the inner-loop optimization-based meta-learner like MAML [1], where the inner-loop adaptation to the feature extractor can potentially solve the challenge of 'not all features are relevant in all tasks and that the support is unlikely to span the input domain' pointed out in this paper. 

And many other directions and methods, e.g., methods based on Hebbian rules [2] might share a similar spirit with the proposed method, and the presentation can be further improved by incorporating more comprehensive discussion to the latest progress of meta-learning and the connections of the proposed method to others. 

**Minor**

The experimental settings included in this paper seem a bit weak. More common benchmarks on real-world meta-learning (classification) tasks, like miniImageNet, and the challenging cross-domain settings can better support the discussions.

The overall writing is good, but some further improvements are expected. For example, is the very first sentence of this paper grammatically incorrect? I believe it's better to say 'that need to be neither universal nor ...'

[1] Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks, ICML2017

[2] Differentiable plasticity: training plastic neural networks with backpropagation, ICML 2018
summary_of_the_review: I give an initial recommendation of score 6 mainly for appreciating this novel view to mainstream meta-learners, and solid discussions supporting the points. However, to meet and standard of ICLR and demonstrate a clear contribution to the research of meta-learning, I believe further efforts on comprehensive discussions are highly expected.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Authors propose the general problem of few-shot polythetic classification, where class membership is determined by the combinations of present and absent features and the salient features and combination patterns change at test time. Authors demonstrate that prototypical approaches with linear decision boundaries respond poorly to these highly non-linear problems, while attention-based soft nearest neighbors paradigms work well but overfit to specious cues. An attention-based feature refinement technique is proposed and evaluated, beating both baselines on specifically polythetic few-shot tasks. 
main_review: STRENGTHS:

The few-shot polythetic classification problem is interesting and novel, and the shortcomings of baseline approaches are readily apparent. Work is well-grounded in older prior literature and the XOR_alpha task is extensively analyzed both theoretically and empirically. Writing is clear and concise (though not always easy to understand, see below). The proposed method performs well in its intended setting. 

WEAKNESSES/ISSUES (in no particular order):

- The XOR_alpha meta-learning task is never formally described (it is only described as a function of bits on page 3), and so subsequent sections become difficult to follow. Please state clearly in section 2 that XOR_alpha also refers to the collection of (presumably?) all alpha-variable XOR “tasks” in a given binary n-space with n>=alpha, with a (presumably?) random partition of train and test tasks. Perhaps also consider using different notation when referring to the meta-learning problem (i.e. caption for Fig.3 right) vs the task (i.e. second to last paragraph of pg.4). Either way, the problem setup for this and other experimental tasks more broadly (i.e. the non-standard tiered-ImageNet task) should be elaborated up front and more clearly. As is I would have a difficult time re-implementing some of these benchmarks without the provided code. 
- Contrary to claims in the abstract and conclusion, the paper does not show that the embedding space of a threshold classifier must grow exponentially with the number of features. Pg.4 discusses embedding growth but has it at O(n^alpha), which for given alpha is only polynomial in the number of features. The embedding space does grow exponentially with task complexity alpha, which is itself O(n), but they are not the same and these claims should be clarified (assuming that this is in fact authors’ intended argument). 
- On a much broader level, I have a hard time understanding the motivation for this work. What is the envisioned use case scenario for this kind of approach? The synthetic examples, while interesting, are very contrived (which is fine since they’re synthetic) but the paper does not provide a strong motivation for the “real world” examples (omniglot and tiered-ImageNet) either. These are also somewhat contrived: while the fine-to-coarse generalization task under discussion is interesting, it’s the opposite coarse-to-fine generalization that has a clear use case. What is the envisioned use-case for a polythetic meta-learner? The introduction suggests certain kinds of fine-grained classification, but if this is the case then it should be investigated directly, such as with CUB or tiered meta-iNat benchmarks. 
- The first paragraph of page 2 is difficult to parse, both because sentences are somewhat long and mostly because it is not immediately clear how conclusions are following from premises (i.e. the described 45degree rotation/”change of basis” of the OR function was initially confusing to me because this geometric operation does not translate intuitively into pure Boolean algebra; the rates of growth (2^2^n and 2^n^2) are not immediately obvious). This section could use some elaboration and clarification. 
- I do not find Appendix A convincing. Appendix A claims to be a demonstration that protonets do not generalize to unseen variable combinations. While it is true that train and test combinations do not overlap, by my understanding the chance performance here is clearly due to the fact that training time noise features are test time signal features, and vice versa, and the network has simply (and correctly!) learned to suppress the “relevant” noise features. As such, no learning-based classifier should be able to solve this problem as presented; it is less a demonstration of the limits of protonets and more a demonstration of the no free lunch theorem. A much more useful and interesting set of results would be to partition the combinations of active variables randomly over train and test, so that combinations do not overlap but features are equally active in expectation at both train and test time. By my understanding this is exactly the experiment in Fig.3 right, though, so perhaps this ought to just be removed entirely. 
- Authors use a pre-trained ResNet-18 for the tiered-ImageNet experiment, but the provided source for the model is PyTorch itself. How was this model pretrained? If the model was pretrained on ImageNet, then it has been trained on the test images and these results are not valid. 
- Typo in Algorithm 1 Input: the an arbitrarily ordered matrix -> an arbitrarily ordered matrix

summary_of_the_review: The paper proposes an interesting problem, and based on theoretical and empirical analysis provides a neat solution. Issues stem from the perhaps overly concise writing; many aspects of the paper are in need of elaboration. These include the motivation, problem setups for the various benchmarks, and the reasoning behind certain conclusions. Currently I am scoring the paper as if my understanding is correct and authors’ intended arguments are simply wrong, but if these issues can be clarified I’d be happy to raise my score. 

POST-DISCUSSION:

Authors mostly address the above issues, in discussion and in the revised manuscript. I have some lingering concerns, shared with reviewer q98z, regarding the lack of _demonstrated_ practical benefit, and the fact that accuracy gains in the more “real-world” benchmarks shrink substantially relative to the motivating XOR problem. However, the revised paper is much improved, conceptual novelty remains high, and on the whole the paper is of sufficient quality for acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper discusses monothetic and polythetic classifications in the context of few-shot learning: distinguishing similar classes often require reasoning with combinations of certain features, which can be especially challenging when only a few training examples are available. Two common types of few-shot methods, ProtoNets and Matching Networks, are shown to be threshold classifiers and attentional classifiers, respectively, each with their advantages and disadvantages. The authors introduce a simple method based on self-attention as a solution to these challenges, with experiments on several toy tasks and the more real-world TieredImageNet.
main_review: Pros:
1.	Strong motivation and introduction of monothetic and polythetic classifications, drawing from prior work from other fields
2.	Interesting analysis and excellent visualizations
3.	Uses simple Boolean tasks to explain background conceptually
4.	Well-designed toy datasets and experiments to verify hypotheses of the nature of ProtoNets and Matching Networks in the context of the paper
5.	Well-written

Cons:
1.	Practical value isn’t clearly demonstrated; improvements over matching networks for the only non-toy dataset (TieredImageNet) is marginal at best.
2.	Limited baselines in experiments

More detailed comments:

This paper examines few-shot learning in the context of monothetic and polythetic classifications. This perspective sheds some light on how ProtoNets and Matching Networks perform classification of features, and there is some interesting analysis to back these hypotheses up. Concepts from the paper are well-explained with several simple examples, and toy settings are designed to illustrate them empirically. Furthermore, the analysis leads to a simple self-attention-based solution to improve features for classification. Overall, this paper reads well, and the analysis leads to some interesting insights into few-shot learning.


Empirically, the results are a little more disappointing. While several toy experiments (e.g. XOR binary strings, polythetic MNIST) are well-designed to illustrate the advantages of considering polythetic patterns, it’s not as clear how much the real-world exhibits these characteristics. The proposed method strongly outperforms the baselines in the toy settings, but improvements over the Matching Networks baseline on TieredImageNet is marginal at best. As such, as nice as these insights are, it’s not clear if it’s practically useful. Additionally, while I understand the focus was comparing with ProtoNets and Matching Networks as representatives of threshold-based and attentional classifiers, it would have been nice to have included more baselines in the experiments.


Questions:
1.	While conceptually easy-to-understand, I was previously unaware of categorizing classifiers as either attentional or threshold-based. Are there any other kinds, or must a non-attentional classifier be threshold based (and vice versa)?
2.	How do other few-shot methods fit into this taxonomy? For example, what are kinds of features (monothetic vs polythetic) are optimization-based methods (e.g. MAML) learning? What about simply training a classifier, or SVM?



=====Post-Discussion=====

I thank the authors for all their effort during the discussion phase to clarify various questions about their paper and for running additional baselines. After reading the other reviews and seeing the authors' responses, my recommendation remains mostly the same. The new perspective on meta-learning provided by the authors is an interesting one, but the practical benefits of this approach can still be more concretely demonstrated. Additional experiments in such use cases (e.g. the DNA example mentioned in one of the discussions, if such a dataset exists) would significantly strengthen this paper.
summary_of_the_review: As stated in the main reviews, the empirical results aren’t particularly impressive, but overall the paper does provide some interesting analysis that may encourage new ways of thinking about few-shot learning problems. As such, I think this paper may be of interest to the ICLR community.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper first considers the limitations of threshold and attentional classifiers. They proposed an attention-based method for feature selection to address the problems of threshold classifiers and attentional classifiers.  The experiments on  several synthetic and real-world few-shot learning tasks seem good. 
main_review: Strengths: 

(1) The explanation about the challenges of threshold (Prototypical networks) and attention classifiers (Matching network) is interesting.

(2) The motivation on the polythetic classification is well-motivated. 

(3) Well-written and easy to follow.


Weaknesses:

(1) Lack of technological innovation. The proposed feature-selection mechanism is too simple and the only technological innovation.  
They only use self-attention to obtain a better representation rather than directly using average pooling (Prototypical networks) or all the support data (matching network).

(2) How to choose repetitions R?  It is a hyper-parameter, is the bigger the R the better? Does it have no upper bound?

(3) Lack the experiments for comparison with threshold classifiers and attentional classifiers in the main paper. It is crucial to show the problems of threshold classifiers and attentional classifiers.

(4)  Self-attention also has some parameters for the transformation, why is the proposed method non-parametric?

(5) The related work is too little. Maybe consider adding some self-attention work?



summary_of_the_review: The implementation details and related work of the proposed paper should be further added.
review_title: null

Meta-review: This paper analyzes problems of existing threshold meta-learners and attentional meta-learners for few-shot learning in polythetic classifications. The threshold meta-learners such as prototypical networks require exponential number of embedding dimensionality, and the attentional meta-learners are susceptible to misclassification. The authors proposed a simple yet effective method to address these problems, and demonstrated its effectiveness in their experiments. This paper discusses meta-learning from a very unique perspective as commented by a reviewer, and clearly explained problems of widely-used meta-learning methods. However, this paper focus on prototypical networks and matching networks even though there have been proposed many meta-learning methods. Some existing methods seem not to have the problems of prototypical networks and/or matching networks. In addition, the practical benefits of the proposed approach are not well demonstrated. Although the additional experiments in the author response addressed some concerns of the reviewers, they are not enough to demonstrate the effectiveness of the proposed method.
Decision: Reject