Attentional meta-learners for few-shot polythetic classification
Authors: Ben Day
Ramon Viñas Torné
Nikola Simidjievski
Pietro Lio
Venue: ICLR 2022 Submitted (2022)
Keywords: Meta-learning
self-attention
feature-selection

Abstract: Polythetic classifications, based on shared patterns of features that need neither be universal nor constant among members of a class, are common in the natural world and greatly outnumber monothetic classifications over a set of features. We show that threshold meta-learners, such as Prototypical Networks, require an embedding dimension that is exponential in the number of features to emulate these functions. In contrast, attentional classifiers, such as Matching Networks, are polythetic by default and able to solve these problems with a linear embedding dimension. However, we find that in the presence of task-irrelevant features, inherent to meta-learning problems, attentional models are susceptible to misclassification. To address this challenge, we propose a self-attention feature-selection mechanism that adaptively dilutes non-discriminative features. We demonstrate the effectiveness of our approach in meta-learning Boolean functions, and synthetic and real-world few-shot learning tasks.

Under review conference paper at ICLR 2022
ATTENTIONAL META- FOR FEW-SHOT POLYTHETIC CLASSIFICATION
Anonymous authors Paper under double-bling review
ABSTRACT
Polythetic classifications, based on shared patterns of features that need neither be universal nor constant among members of a class, are common in the natural world and greatly outnumber monothetic classifications over a set of features. We show that threshold meta-leamers, such as Prototypical Networks, require an embedding dimension that is exponential in the number of task-relevant features to emulate these functions. In contrast, attentional classifiers, such Matching Networks, to solve these problems with a linear embedding dimension. However, we find that in the presence of fask-irrelevant features, inherent to meta-learning problems, attentional models are susceptible to misclassification. To address this challenge, we propose a self- attention feature-selection mechanism that adaptively dilutes non-discriminative features. We demonstrate the effectiveness of our approach in meta-learning Boolean functions, and synthetic and real-worke few-shot learning tasks.
polythetic by default and able
INTRODUCTION
Classification meta-learning is typically approached from the perspective of few-shot learning: Can we train a model that generalises to unseen 'natural' classes at test time? For example, in the Omniglot task (Lake al. 201T) we have access to a labelled set of handwritten characters during training and we are tasked with distinguishing new characters, from unseen writing systems, at test time. From this perspective each example is associated with a consistent class and members of that class share a common set of properties (e.g. all handwritten characters have the shape of the underlying character class). Alternatively, we may consider meta-learning over unseen ways of categorising: Can we train a model on character recognition that generalises to alphabet recognition? or to distinguishing upper from lower case letters? In this setting, features need to be understood in relation to a given classification: For instance, when tasked with distinguishing equids (horses, zebras, donkeys) from big cats, the presence of stripes on both zebra and tigers is irrelevant, and potentially misleading. On the other hand, stripes are the key to distinguishing horses from zebra.
Understanding features in the context of a classification is central to the concepts of monothetic and polythetic classes recognised in the fields of taxonomy and knowledge organisation. Monothetic classifications are based on universal attributes: there is at least one necessary and sufficient attribute for class membership. Polythetic classifications are instead based on combinations of attributes, none of which are sufficient in isolation to indicate membership and, potentially, none of which are necessary. Carl Linneaus, inventor of the binomial nomenclature for species and "father of taxonomy," recognised that natural orders could not be defined monothetically. lacking features that were unique and constant over families and that, until such features could be found, such classifications were necessarily polythetic (Hjörland, 2017: Stevens 1998).
Figure Iillustrates Linnaeus' system for classifying plants, which relies on polythetic classifications and is still in use today. Consider, for example, that we can distinguish A from B by the number of filaments, but not

---

Under review conference paper at ICLR 2022
I from O we can distinguish from C based on whether there are split anthers (the ends), but not C from A. and so on. To recognise classes it is necessary to consider their attributes in the context of other attributes.
Threshold functions are a related concept in Boolean algebra. A threshold function evaluates positively if weighted sum of binary inputs crosses some threshold. Lacking a preferred feature basis, we can identify monothetic classifications with threshold functions, threshold functions that are polythetic in one basis, such as logical (R(r, y). may be re- cast in basis where they monothetic e.g. bi- nary OR (z, y) evaluates as the unary MIN + y. 1). We can compare the frequency of monothetic and polythetic classifications in this general case. The number of binary inputs of length total number of Boolean functions is the number of binary labellings of these inputs, 22". whereas the number of threshold functions grows only singly ex- ponentially (s 2n²) ), and therefore monothetic clas- sifications represent a vanishingly small proportion of the total (see Gruzling (2007): Irmatov (1993).
2n and the
M Section of Methodus Plantarum Sexualis, Georg Ehret, illustrating Linnaeus's Systema Naturae, 1735. Class defining attributes need not be exclusive or universal, and useful classifications may be contextual.
Figure I:
This work explores meta-learning for polythetic classification. Specifically, we
consider the limitations of widely used threshold classifiers, such Prototypical Networks (Snell
ct al. 2017). and how they are able to learn and approximate non-threshold functions in practice;
show that simple alternatives based attention, such Matching Networks (Vinyals et al. 2016).
are polythetic by default but susceptible to misclassification due to excessive sensitivity; characterise the challenge of spurious correlations irrelevant features for attentional classifiers; and propose a simple solution to this challenge that is non- non-parametric and based on self-attention.
Throughout, we evidence these findings and the effectiveness of our proposals with experiments on synthetic and real-world few-shot learning tasks.
2 BACKGROUND
Problem formulation. We are interested in few-shot classification: provided with small number of labelled points S = {2>,, ys tiek, the support set, with feature vectors æ; E R and labels yi (1,..., K). we want to predict the labels of the query set Q = { {æy jelg- S₃ denotes the set of support elements with label and I, is the index set of the subscript e.g. Is the index set of the support. The label space arbitrary and potentially unique to a task (also referred to as an episode) - both the number of classes, k, and assigned labels may vary over tasks and, importantly, examples that share labels under one categorisation will not necessarily share labels under another. We will refer to classification functions over features simply classifications. and in meta-learning we are often interested classifications that only depend on some features, and not the remainder, 3 n a.
Classifiers. Deep neural models for problems of this kind are usually equipped with either threshold classifier or an attentional classifier. Threshold classifiers are based, as the name suggests, on using thresholds partition a space into regions associated with each class. Prototypical Networks (Snell et al.) 2017) are example of such a model. This model embeds examples using a learned neural function, fe. finds the average embedding for each class in the support, Cg2 and classifies queries based their proximity (measured with
2

---

Under review conference paper at ICLR 2022
fix.
1r..g) v) -ND(r.
(s(s.3) -XOR(z.))
Atteese S
PHAaEge -
Proragge Ielifast
Alrelkon =
Figure 2: Confidence heatmaps and decision boundaries prototype and attention classifiers on 2- variable Boolean functions. The attention classifier shown 'harden' the classification and produce decision boundaries more closely aligned with the axes (see Appendix B). As XOR(x, (æ, y) is not threshold function, simple prototypes fail produce correct classification scheme, in this case the prototypes are equal (= (0, 0)) and there is no decision boundary.
temperature of I, lower temperatures
distance function d) to these class prototypes:
exp(-d(fo(z)).om ck)) Poly >o()=)/>>> = Σx exp(-d(fo cw))
Ck = SR
Σ fa(x)) VETsR
(1)
The key advantage of such an approach is that salient class features preserved when forming the prototype while irrelevant aspects of particular examples are washed out. Attentional classifiers instead use a similarity function directly compare queries with each example in the support. For example, Matching Networks (Vinyals et al., 2016) also learn embedding functions, but each query is compared with every member of the support to weight sum their labels, which can write simply as ý Eiers af In the popular terminology of transformers we may write the embedded queries as Q, the embedded support as keys K and their labels as values V, and dot-product attention classification with temperature 7
DotAttn (Q, K, V, 7) softmax(=QK")V E R|
(2)
Attentional classifiers are more sensitive to variations within class at the cost of additional computation.
Boolean tasks. In comparing these classifiers we make repeated use of tasks based on Boolean functions. and on exclusive-OR (XOR) in particular. For a binary feature vector x {-1, the number (x)) Thea I; is the parity function or exclusive-or (XOR) over the bits (æ;) We write parity function of bits XOR,, The set of parity functions over n. bits form a linearly independent basis (O'Donnell 2014) and, as such, being able to model the partition functions guarantees that one can model any other Boolean function over that domain. Put another way, the decision boundaries of XOR are at least complicated as those for any other a- variable Boolean function: there is one between every possible pair of feature vectors. For these reasons, XOR is our polythetic function of choice in derivations, examples, and experiments.
3 CHALLENGES IN META -ETA-LEARNING POLYTHETIC CLASSIFICATIONS
Threshold and attentional classifiers have their own strengths and, in meta-learning polythetic classifications, their own weaknesses. Threshold classifiers are insufficiently flexible and attentional classifiers are prone to misclassification.
Threshold classifiers. Figure 2 shows the decision boundaries formed by prototypical threshold classifier and attentional classifier for selection of 2-variable Boolean functions, highlighting the problem with using threshold classifiers for polythetic classification: logical XOR(æ, y) is not a threshold function and the prototypes fail to produce useful decision boundary. This is the perceptron problem identified by Minsky &
3

---

Under review conference paper at ICLR 2022
Prranopiial
XOR
(0,
(1,1)
(0, 1)
(L,
Seguenee bengls 4
L1,
(0,0)
(1, 0)
(b) (0, 0,
Eambekking dimoession
Figure 3: A non-threshold function of 2-variables, the pseudo-variable solution, and Prototypical Network performance on the X problem. (a) XOR(x, 2)) which does not have a threshold solution in 2 dimensions. (b) Appending the aseudo-variable XOR(x, y) gives a 3-dimensional embedding in which all 2-variable Boolean functions have threshold solutions. XOR(2 (z, y) is a sseudo-variable in that it is determined by the other variables and cannot freely vary, for example the hatched circle at (1, 0, 0) cannot occur. Right: Accuracy of prototypical networks for the XOR2 problem over sequence length and embedding dimension. Mean over 1000 tasks, [|| 40. See Appendix F for details.
Papert (1969). However, deep networks using threshold classifiers can learn XOR. This is possible because the network can learn additional pseudo-features for the non-threshold functions it observes. Figure 3soows an example for 2-variables: one can embed the corners of the square at the corners of tetrahedron with coordinates (z, (z, y)) and produce linear thresholds solutions for every 2-variable Boolcan function.
There are two problems with this approach, both of which are exacerbated in the meta-learning context: i) the required number of pseudo-variables grows as (*) ~ O(n)) to account for all combinations of a active components, and fi) the method does not generalise to unseen non non-threshold functions (see Appendix [A]. The
right plot in Figure demoonttattss these shortcomings for XOR;; the required number of aseudo-vanablle is (?) ~ ((n²) and initially we find that a quadratic embedding is able to maintain performance, but for longer sequences the number of threshold-functe unseen in training grows and performance degrades.
Attentional classifiers. Attentional classifiers avoid these problems - the required embedding dimension is linear in the number of features and the classifier generalises to unseen classifications but suffer from over-sensitivity to irrelevant features. This results in misclassification, which we quantify in the case of Boolean functions to understand the scaling properties of this problem generally.
Consider classifications over binary feature vectors E {-1, 13", with the class determined by XOR. over a elements with the remaining 3 = n a being irrelevant. Assume each of the 2 variations of the active elements are present with equal frequency, r, for support set S of size [S] = r2". and that the remaining elements follow Bernoulli distribution with probability p. Using an attention classifier of the form V Es softmax; (a(ă, ax:)) yi. where u( xx) is a measure of the similarity of a and how likely is it that we misclassify query drawn from the same distribution as S?
Without loss of generality, we can focus on the positive examples (with label = 1) for which positive output gives the correct classification. Using dot dot-prrduct attention, the mean and variance of the classifier output are, with p2 (1 2)² and
== r(e --- (ne qe -e-1)²
-r(e- -r(e- [-)--_)))
(3)
0= r(6² 22-
(pe² + g)=
(pe qe 9)-1) 28
+pe+ge-1)
r(("" + 22- -201.
(4)
4

---

Under review conference paper at ICLR 2022
introducing e and d for compactness. The mean is positive, as desired, but we interested in the rate of misclassification, which may be interpreted using the ccale-free and dimensionless sefffiientt-f-variattt (the ratio of the standard deviation to the mean) where greater variation indicates greater rate of misclassification. From Equations
3 and we have
ve " 1 Vi e-
8
1/2
(5)
Starting with the leftmost term, increasing the number of repetitions, r. reduces the relative variability. This aligns with intuition and limits, where empty support set, 0. provides basis which to make predictions and at the other extreme, r >> 2 the support set is likely to span the input domain reducing classification to look-up. The term raised to a is approximately 3.2, and so the variability increases with the number of active elements. An intuitive explanation is that the number of immediate neighbours of each point grows as (") a and this reduces the confidence with which the point is classified, so the barrier to misclassification reduced. Finally, > c² and so the rightmost term positive and grows exponentially in 3, meaning that misclassification increases with the number of irrelevant features. A full derivation, including alternative attention functions, is provided Appendix C
The problem of misclassification due to over-sensitivity in attentional classifiers was recognised in the work of Luong al. (2015) on sequence processing. There the problem was addressed by attending only to a subset of elements within some distance of the target position. However, sets do not have such an ordering and so we instead propose a feature-seled selection method to resolve the problem more generally.
4 ATTENTIONAL FEATURE SELECTION
A key challenge of meta-learning is that not features are relevant in all tasks and that the support unlikely to span the input domain. The model must choose, using incomplete information, what to focus and what to ignore by detecting the salient features within and between classes. For monothetic classifications this straightforward: by definition, averaging highlights necessary features whilst diminishing irrelevant features. Prototype methods rely on this process. In the polythetic case, attentional classifiers have the advantage of being able learn non threshhrehll functions without the need for aseudo-variable, but do not benefit, prototypes do, from "washing-out' irrelevant features through averaging. Indeed, attentional classifiers are susceptible even in the monothetic setting misclassifying on the basis of closely matching features that
100, 1.00
L.LG 0.55
LO0LOE,, 0.30
L.,,, L.0, LOOL
Figure 4: Feature values and attention coefficients during the feature-selection self-attention (n {0, 5, 10, 25}) within a class of XOR3. Nodes depiet examples of the support set: the colour of the left halves represents the active features; the right halves represents the magnitude of the irrelevant features. Edge width and opacity indicate the attention strength between a pair of nodes. The red, green, blue and white groups, different variants, automatically segregate which preserves their active features while the irrelevant features converge, as shown in the feature scores beneath each plot. this way, the active features identify themselves.
5

---

Under review conference paper at ICLR 2022
are not relevant the problem (putting zebra with the big-cats because its stripes match those of a tiger in the support set, for example). Misclassification occurs when irrelevant features overwhelm the signal from the active elements. As this a problem of highlighting the salient patterns within a set, we propose a self- self-attention based mechanism for feature selection, presented in Algorithm and illustrated in Figure 4
Intuitively, the process exploits the over-representation of patterns within features that are relevant to the classification as compared to patterns within the irrelevant features. We first standardise the features to prevent those common to the entire support, which are not diseriminating, from dominating (Line T and stabilise, 6, to prevent weakly activated features from being excessively scaled-up. We then repeatedly self-attend within each separate class k of the support set, using dot product attention with scale т, Xx + DotAttn(X;; XR, X T). Self-attention maps elements of set of vectors to the interior of their convex hull. If every member of a class has some feature in common, the convex hull in that dimension a point and the features do not change. In the polythetic case it patterns of features that matter, and by attending more strongly between elements of the support set with such feature-patterns, these too are preserved. Figure for example, shows polythetic variations within class of XOR; with three active and three irrelevant features (a 3 = 3). The patterns in the active features self-reinforce, forming cliques of strongly connected elements, whilst the irrelevant features decay. Finally, features are scored by their dispersion over the support set (Line [5) which indicates how well they have been preserved through the self-attention iterations, and thus how relevant they are.
Algorithm I: Self-attention feature scoring. Scores can be used for rescaling or masking. Note that the n-nomaalisation is over the entire support set whilst the self-attention is within classes. The choice of dispersion measure is of secondary importance and discussed in the main text.
Input Support set S = [x., M: ke with class labels E {L...., K} and features x; E RF,
F, S₃ denoting the subset of S containing all samples with y = k and X E [|S5 an arbitrarily ordered matrix of feature vectors belonging to Sy; small numerical constant, e attention temperature. - repetitions, R.
Dutput::eature scores. f E R²...
X, < (x: - (xx)/(xx +
// standardise
repeat R times
3
for k 1 K do
// for each class // softmax is row-wise
X 5 < softmax (T x
5 f (xspersion((x]iies]
// men--absolute-
std. dev etc.
The scores can be used directly to rescale features across the support and query sets before applying the classifier, as in Figure or in top-k selection. We focus on rescaling as the method that makes the fewest assumptions about the underlying classification, but using top-k is highly effective when the number of active elements is known, as shown in Appendix D
5 EXPERIMENTS
We compare the proposed method (FS) with Prototypical Networks (PN) (Snell al. 2017). a threshold classifier, and Matching Networks (MN) (Vinyals et al.] 2016). an attentional classifier without feature- selection, in a sequence of increasingly complex synthetic and real-world few-shot learning problems. As our approach is non-parametric and operates directly on high-level features, it is agnostic the choice of feature extractor, and we are free to choose as appropriate e.g. a convolutional neural network for images or a lulti-layer perceptron for tabular data, and in all experiments we use the same embedding model for all methods (see experimental details in Appendix F).
Binary strings. We consider meta-learning Boolean functions of a + f variables. Labels for inputs a E 1) are generated by computing the XOR of random subset of components of size a. Each variation in the active elements occurs 5 times in the support, [S] 5. 2 The subset of active components
6

---

Under review conference paper at ICLR 2022
Direetly op imputs
with soft fenture selection
Aucrrrey
0.72
0.5
-04
oxutppees
oxsmppees
cuangles.
Figure 5: Attention classification of the XORa problem over variant frequency, r, and number of inactive components 3. Increasing r assists feature selection, in agreement with the derived misclassification distribution. Soft feature-selection rescales features according to their scores, as determined by the proposed self-attention procedure. This greatly improves performance even at low repetitions, for example 2 repetitions and 3 inactive components the change in accuracy is +38pp. Neither method is effective at high 3 with low r.
Table I: Binary strings. Accuracy by embedding dimension for sequences of length 5 and n 10. Mean and standard error calculated over 1000 tasks.
V 5
n 10
Model
Emb.
XOR2
XOR;
XOR,
XOR;
XORs
XOR,
07.6 +: 0.5
55.3 = 0.5
0.7
50.2 0.4
50.1 0.2
PN
1
73.6 + 0.5
70.3 + 0.7
91.4 0.4
56.7 0.4
50.1 0.2
50.4 0.2
n²
90.4
- 0.3
77.8 0.7
100.0 0.0
6..1
50.3 0.2
50.6 0.2
FS+MN
n
99.6 1: 0.2
100.0 1: 0.0
100.0 1 0.0
75.9 1: 1.3
82.6 1 1.1
96.3 1. 0.5
is unknown to the meta-learner. Appendix D shows a concrete example of an XOR2 task. Table I summarises the performances Prototypical Networks and our approach. PN accuracy decreases sharply with sequence length n and the number of embedding units required to effectively solve this problem grows rapidly with as shown previously in Figure 3, This suggests that PN are indeed learning pscudo-variablee and demonstrates the limitations of threshold classifiers in solving polythetic problems.
We evaluate the ability of the models to jointly extract high-level features and identify polythetic patterns. We build tasks (episodes) using MNIST digits (LeCun et al.. 2010), where example consists of 4 coloured digits (RGB). An example task and further details are provided in Appendix J For monothetic tasks, a single high-level feature (e.g. colour of the top-right digit) distinguishes classes. For polythetic tasks, class membership derives from XOR interactions over a subset of features, and the remainder are -irrelevant. Table shows the performances on three versions of the polythetic MNIST dataset: clean (excluding non-disee non-discriminative digits), colourless mereeirrelevant digits but no colour), and full (both task- irrelevant digits and colour). The models are trained on monothetic tasks and evaluated both on monothetic and polythetic tasks. Protonets excel at identifying monothetic features and ignoring non-discriminative features, but have a close to random performance on polythetic tasks. Conversely, matching networks, which are polythetic classifiers by default, are highly sensitive to task-irrelevant features. The proposed approach (FS) can simultaneously detect salient features and perform polythetic classifications. Furthermore, as shown in Figure 6. we found our classifier be robust to the rate of polythetic tasks seen during training in second experiment.
Polythetic MNIST.
Omniglot. The Omniglot dataset (Lake al. 2011 consists of handwritten characters from 50 writing systems with 20 hand drawn examples of each character. Training tasks are formed using examples from 30 of the alphabets and test tasks draw from the other 20. We compare our method to PN, MN, Infinite Mixture
7

---

Under review conference paper at ICLR 2022
Table 2: Polythetic MNIST. Evaluation accuracy on monothetic and polythetic tasks in three settings. Mean and standard error calculated over 1000 tasks.
Clean
Colourless
Full
Model
Monothetic
Polythetic
Monothetic
Polythetic
Monothetic
Polythetic
PN
97.9 0.1
50.6 + 0.3 57.6 + 0.4
92.8 + 0.3
49.9 + 0.3
94.5 + 0.3 70.1 # 0.7
49.8 - 0.2 56.6 = 0.5
MN
7
69.7 = 0.4
61.0 0.5
FS+MN
96.8 96.8 0.1
98.3 + 0.0
94.5 + 0.2
98.0 0.0
75.0 + 0.7 60.4 + 0.7
Monothetic test
Polythetic test
Table Omniglot. 20-way, 5-shot characters; 3-way alphabets. Mean and standard error on 1000 tasks.
1009
1005
Toll. dimee accummay
Characters
Alphabets
PN
0.0
83.4 + 0.3
5025
3096
91.1 0.1
78.4 + 0.3
FS+MN
96.2
94.2 + 0.2
Pobthette dariug trintig
Polythetee duing
IMP
98.6 0.0
96.0
MAML
9..
89.9 + 0.3
Figure 6: Polythetic MNIST (colourless) by polythetic proportion during training. FS matches outperforms the other models at all training proportions and is far less affected by the training mix. Mean and standard deviation over 1000 tasks at each proportion.
MN
9.
81.3 +
NN
98.3
95.7 + 0.3
FS+MN
98.1
96.0 + 0.2
FS+NN
98.3 0.0
96.0 + 0.2
Prototypes (IMP) (Allen et al. 2019). and MAML (Finn et al. 2017) with a threshold classifier (Triantafillou et al. 2020 We train the models for character recognition, and additionally evaluate performance 3-way alphabet recognition (inherently polythetic). Our approach can be used in conjunction with most few-shot learning approaches- we specifically apply FS prior to MN and single-nearest-neiihbbur (NN), corresponding to MN with softmax converging to argmax (see Appendix B). Table 3 shows the results of this experiment. The end-to-end trained model (FS+MN) is competitive with PN and MM,, while performing better than MN and MAML in character recognition. In alphabet recogmition, FS+MN performs better than other methods, while being competitive with IMP. We further evaluate the performance when using a pre-trained feature extractor (methods marked with *), obtained by training a PN threshold classifier on character recognition. FS+MN improves over MN in both tasks. Compared to PN, the accuracy of FS+MN* and # in character recognition is reduced by at most 0.5pp; yet improved in alphabet recognition by 12.6pp, while performing similarly to the more complex IMP.
Titeed TieredImageNet (Ren et al. 2018 is subset of ILSVRC-12 (Russakovsky al. 2015)
with polythetic characteristics, with classes grouped into categories corresponding to higher-level nodes in the ImageNet hierarchy. There are 34 categories of 10 to 30 classes each. We compare our method (FS)
Table 4: TeeredI Model accu- racy by categories (C) and groups (G) over 500 tasks.
Table 5: TieredImageNet. Head-to-head comparison 500 tasks by categories (C) and subgroups (G). Bold indicates significance at the < 0.001 level.
C= 2
C=4
C=
C=2
C=4
C 8
FS PN
83,5 = 0,3
4.2
G
X
Y
X/Y
(tie)
X/Y
(tie)
X/Y
(tie)
82.7
0.3
0.2
0.3
0.3
4.14
5
FS
MN
230
162
(108)
329 101
(70)
343 113
(44)
FS
PN
319
136
(45)
339 142
(19)
239 /27
(14)
FS
83.6 = 0.3
65.8 : 0.3
0..
10
MN
82.9 0.3
6.3
0.10
FS
MN
258
185
(57)
357 108
(35)
413/ 68
(19)
PN
81.1
0.3
0.3
0.1
FS
PN
374 101
(25)
396/
99
(5)
296
191
(13)
8

---

Under review conference paper at ICLR 2022
PN and MN classifiers. We use a publicly available pre-trained ResNet-12 (Zhang al. 2020 pre-trained using the training classes in TieredImageNet, as the feature extractor for all models. Table presents the aggregate accuracy while [Sblos]ssows the head-to-heed he-hea results of this experiment. FS leads to significant improvements in performance (except C-8/G-5, where the difference between PN and FS is not significant) this full scale, naturally polythetic problem, particularly the "more polythetic" case with 10 subgroups.
6 RELATED WORK
We characterise meta-learning approaches for few-shot classification. In addition to evaluating their ability to generalise to unseen classes, we investigate how well they can adapt to polythetic tasks (generalisation to unseen ways of catcogorising). Adaptability in few-shot settings has been studied through different paradigms such fast weights (Ba et al. 2016), learnable plasticity (Miconi et al. 2018) and meta-learning. Recent work on meta-learning for fhw-shot classification includes approaches that are able to quickly adapt through various mechanisms such as recurrent architectures (Mishra et al.) 2018 Santoro et al.] (016) for learning parameter updates (Ravi & Larochelle| 2017). Other more general optimisation-aases approaches (Finn et al. 2017 Nichol al. 2018 Rusu al.) 2019), tackle these tasks by explicitly optimising the model's parameters. These, however, are typically madel-ag and commonly used in conjunction with threshold classifiers Triantafillou et al. 2020). inheriting their limitations in a polythetic scenarios.
Our work aligns more closely with metric-learning approaches for few-shot classification (Chen et al. 2019) that apply distance functions between queries and the support in common embedding space (Allen et
2019; Oreshkin et al. 2018 Snell et al. 2017 Sung et al. 2018 Vinyals et al. 2016 Zhang et al. 2020).
Methods that construct class-wis wise prototypes from the support (Allen et al., 2019 Ren al. 2018 Snell
et al. 2017) can successfully tackle monothetic tasks, but can struggle with task-adaptiveness in polythetic
context. Attentional meta-classifiers (Hou et al., 2019: Jiang et al. 2020 Kim al., 2019) Vinyals et al.
2016) adapt to polythetic tasks but lack crucial mechanisms for focusing exclusively on relevant features.
Attending over datapoints has been considered previously. Luong et al. (2015) introduced dot-product attention in the context of attending over sequences. Vinyals et al. (2016) considered ÿ = Liers a(ē, x.) x))y; and provided the conditions under which such a model carries out kernel density estimation or k-nearest- neighbours classification. Vaswani al. (2017) used explicit scaling in hidden attention layers, but scaling a iaaation, classifying softmax by a temperature parameter dates to the work of Boltzmann and later Gibbs (1902). |Pöötz (N3) block recovers a soft-attention weighting when
& Roth (2018) note that their nuural-nerrst-neighhee the number of neighbours is set to 1, and deploy their model on an outlier-detecton set-reasoning task.
7 CONCLUSION
In this work we have articulated the difference between monothetic and polythetic classifications and considered the limitations of standard meta-learning classifiers in the polythetic case. We have shown that threshold classifiers require an embedding space that is exponential in the number of active features and that attentional classifiers are overly sensitive and susceptible to misclassification. To address this, we have proposed an attention based method for feature-selectin and demonstrated the effectiveness of our approach in several synthetic and real-world few-shot learning problems. Our approach simple and can be used in conjunction with most few-shot meta-learners. We expect polythetic meta-learners to find real-world application domains where data is typically scarce and complex, such as healthcare or bioinformatics. For example, we envision use in classifying rare diseases - there are around 7000 rare diseases, affecting 1~17 the worldwide population - from DÑA sequences, where mutations often lead to different phenotypes. In such scenarios, few-shot learning approaches able to generalise over unseen combinations of mutations (i.e. ways of categorising), in a similar vein to our binary strings experiment, may lead to better performance in diagnosing rare diseases and shed new insights into their molecular mechanisms.
9

---

Under review conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
In Section 3 we discuss the challenges of polythetic classification and the limitations of current work on meta-classifiers. Our proposed self-attention feature scoring algorithm is described in detail Section 4, and in particular, Algorithm 1 (and Appendix H). To ensure the work is readily reproducible, besides descriptions the experimental setup provided in Section 5 and the supplementary material (Appendices F G
we
also provide code used for producing the results in the paper.
REFERENCES
Kelsey R. Allen, Evan Shelhamer, Hanul Shin, and Joshua B. Tenenbaum. Infinite mixture prototypes for few-shot learning. In Proceedings of the 36th International Conference Machine Learning, (CML2019) 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp.
PMLR, 2019.
Jimmy Ba, Geoffrey E. Hinton, Volodymyr Mnih, Joel Z. Leibo, and Catalin lonescu. Using fast weights to attend the recent past. In NIPS, pp. 4331-4339, 2016
Wei-Yu Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at few-shot classification. In 7th International Conference on Learning Representations, ICLR 2019. New Orleans, LA, USA, May 6-9, 2019, 2019.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Doina Precup and Yee Whye Tch (eds.). Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1126-112 1 135. PMLR, 06-11 Aug 2017.
J Willard Gibbs. Elementary principles in statistical mechanics. Charles Scribner's Sons, 1902.
Nicolle Gruzling. Linear separability of the vertices of an n-dimensional hypercube. 2007. doi: 10.24124/ 2007/bpgub464.
Birger Hjørland. Classification. Knowledge Organization, 44(2):97-128, 2017. URL http:://wwk isko
otgg/cycla/claasifficttt
Ruibing Hou, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen. Cross attention network for few-shot classification. Advances Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8- 14, 2019, Vancouver, BC, Canada, pp. 4005-4016 2019.
Anwar A Irmatov. On the number of threshold functions. Discrete Mathematics and Applications, 3(4): 429 432, 1993.
Zihang Jiang, Bingyi Kang. Kuangqi Zhou, and Jiashi Feng. Few-shot classification via adaptive attention. CoRR, abs/2008. 024665, 2020.
Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. In International Conference on Learning Representations, 2019. /RLL:tttps////eeeeeiiee http://wwwweeeefteeueeeeeeeeeeee d=SkE6P JC9KX
Brenden Lake, Ruslan Salakhutdinov, Jason Gross, and Joshua Tenenbaum. One shot learning of simple visual concepts. In Proceedings of the annual meeting of the cognitive science society, volume 33, 201 1.
10

---

Under review conference paper at ICLR 2022
Yann LeCun, Corinna Cortes, and CJ Burges. MNIST handwritten digit database. ATT Labs [Online]. Available: http://aan..ecun.com//eeee
2, 2010.
Thang Luong, Hicu Pham, and Christopher D. Manning. Effective approaches to attention-based neural ma- chine translation. In Lluís Mårquez, Chris Callison-Burche Jian Su, Daniele Pighin, and Yuval Marton (eds.), Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pp. 1412 1421. The Association for Computational Lin- guistics, 2015. doi: 10. 1/655/1 166.. URL httpt:////doo.. wtw//.o.o.......o 18653/v1/1111.... 1166
Thomas Miconi, Kenneth Stanley, and Jeff Clune. Differentiable plasticity: training plastic neural networks with backpropagation. Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 3559- 3568. PMLR, 10-15 Jul 2018.
Marvin Minsky and Seymour Papert. Perceptrons. 1969.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. simple neural attentive meta-leamnr. In International Conference on Learning Representations, 2018.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms, 2018.
Ryan O'Donnell. Analysis of Boolean functions. Cambridge University Press, 2014.
Boris N. Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. TADAM: task dependent adaptive metric for improved few-shot learning. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 719-729, 2018
Tobias Plötz and Stefan Roth. Neural nearest neighbors networks. In Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018 URL http:tttt////oceeeee ngs mewrips.cc/ppeee 2018/F F: le 3f fa5 Paper .
aataala8333300 7373df
Sachin Ravi and H. Larochelle. Optimization as a model for few-shot learning. ICLR, 2017.
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B. Tenenbaum, Hugo Larochelle, and Richard S. Zemel. Meta-learning for semi-supervised few-shot classification. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30- May 3, 2018, Conference Track Proceedings, 2018
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211 21-222,, Dee 2015. ISSN 1573-1405.
Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-lcarning with latent embedding optimization. In International Conference on Learning Representations, 2019.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta-learning with memory-augmente neural networks. In Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 1842-186 1850, New York, New York, USA, 20 22 Jun 2016. PMLR.
11

---

Under review conference paper at ICLR 2022
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learn- ing. In I. Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and Garnett (eds.). Advances in Neural Information Processing Systems, volume 30. Curran
Associates, Inc., 2017. URL https: httttps://prooeedinns http.r/s....o//ppp//////////// 088866777461123312ee2222222 Pawp://bbccaaaaprp..o.po Paper.
P.F. Stevens.
Linnaeus, Carl von (1707-78) 1998.
doi: 10442299780411522166009999
URL
https: http::://www.wz/ www.:/wwwwwwww.t............. cwm/ cctp://art.aclee///oorpppaallll
Flood Sung. Yongxin Yang. Li Zhang, Tao Xiang, Philip H.S. Torr, and Timothy M. Hospedales. Learning compare: Relation network for few-shot learning. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1199-1200 1208, 2018.
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evei, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, and Hugo Larochelle. Meta-dataset: dataset of datasets for learning to learn from few examples. In International Conference on Learning Representations, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, F. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. I. Guyon, U. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL URL/sttppss//pprceediooo pdf
nwwrips cteeurpss.c/pape..
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, koray kavukcuoglu, and Daan Wierstra. Match- ing networks for one shot learning. In D. Lee, M. Sugiyama, U. Luxburg. I. Guyon, and R. Garnett (eds.). Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL /tttp:://proceeinnss http://ss...aaaaoccccepeee////// 900235211600012203332766----2222
Chi Zhang, Yujun Cai, Guosheng Lin, and Chunhua Shen. Deepemd: Few-shot image classification with differentiable earth mover's distance and structured classifiers. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
12

---

Under review conference paper at ICLR 2022
A PARTITIONED XOR PERFORMANCES
We hypothesise that, in order to solve polythetic tasks, prototypical networks need to create pseudo-variable the embedding space (e.g. XOR for each pair of components for binary strings tasks). To test this, we train a prototypical network binary strings tasks (where pairs of active components are always chosen from the 5 first components) and evaluate the performance on unseen combinations of components (i.e. combinations of the 5 first components not seen during training). Table 6 shows the accuracies for sequences of length " = 15++ + 3 (where/) / is the number of variables that are always inactive). We attribute the better-than-chance performances (i.e. for 100-dimensional embeddings) to the high-dimensionality of the embeddings by chance, large numbers of non-linear features (c.g. output by a random feature extractor) will include some features that are useful for the task of interest (i.e. similar to extreme learning machines). Overall, these results highlight the inability of prototypical networks to generalise to unseen combinations and support our hypothesis.
Table 6: Accuracy of prototypical networks on unseen non-threshold functions by embedding di- mension. We use sequences of length n + 3, where the number of components that are always inactive. The labels are derived from XOR2 combinations over the first 5 components and the sets of combinations seen at train and test times are disjoint. In other words, the combinations of variables (0, 1), (0, 2), (1, (1,3), (2, 4), (3, 4) are only seen active at train time, whereas, the combina- tions (0, 3). (0, 4), (1, 2), (1, (2 are only seen at test time, with r = 5 repetitions for each XOR combination. Each of the first variables has the same expected frequency of being active at train and test times. The inability to generalise in this scenario suggests that protonets need pseudo-variables (i.e. XOR functions applied each pair of components) solve the binary strings task.
Emb.
B=
48.741 + 0.276
48.855 + 0.282
49.558
0.284
10
50.350 + 0.282
49.397 + 0.286
49.479
+ 0.265
20
+ 0.271 50.705
51.057 0.247
50.144 0.255
100
52.246
= 0.256
52.390 = 0.265
52.573 + 0.258
B TEMPERATURE IN ATTENTION CLASSIFICATION
The softmax in attention mechanisms permits temperature scaling that interpolates between argmax and uniform (and argmin.) This is controlled by T as
exp( (Bx;))
softmax, B) = Σ, exp (3x,)
with softmax converging argmax as 3 + co, and to uniform (i.e. all elements equal to the reciprocal of the length of the vector) as 3 + 0.
For attention classifiers, decrease in temperature increases the model confidence and can cause decision boundarics to move. As 3 > oo and the softmax converges to argmax, the classifier tends to the single nearest neighbour classification scheme; for 3 = 0 the classifier returns the support set class balance. For Boolean functions, changes in temperature effect the degree to which decision boundaries are axis-aligned. For example, Figure 2 shows the decision boundary for f(r y) AND(x, y) using softmax temperature of
13

---

Under review conference paper at ICLR 2022
7
1/3
1/5
Figure 7: Changes in confidence and decision boundary the attention classifier with temperature for AND(2,W) v)-
at In (tanh r), which we derive as
p(class 1) = p(class 0)
exp (B(x + y)) exp (3(-x - v)) + exp (3(-x + v)) + cxp (3(x - y))
Σ exp(...)
Σ Eexp(...
exp (3(x + v)) exp (-3(x + +y)) exp (8(-x v)) exp (3(x -y))
sinh(3(x + y)) cosh(3(2 y))
sinh(3x) (cosh(By) + sinh(Sy)) cosh(3z) (cosh(3y) sinh(3y))
tanh(3x) cosh + sinh(By)
cosh sinh (By)
tanh(Sx) exp( --3y)
2,3 In (tanh (Bz)).
The effect of decreasing the temperature on the decision boundary is shown in Figure] 7
MISCLASSIFICATION DISTRIBUTION FULL DERIVATION
We first present more detailed derivation for the case of dot-product attention. We will make repeated use of the Binomial theorem
12
Σ 11 2
(x y)"
x
(6)
k
Recall that we consider classifications over binary feature vectors E 111,1}", with the class determined by XOR, over a elements with the remaining 3 = n a being irrelevant. Assume each of the 2a variations of the active elements are present with equal frequency, for support set S of size [S] = r2 and that the remaining 3 elements follow Bernoulli distribution with probability p. There are (9) strings over the active elements that differ from a given example in 8 positions. XOR flips the classification with each difference, so if 5 is even the class is the same, if is odd the class is different (parity).
14

---

Under review conference paper at ICLR 2022
The class is determined by the sign of the sum of contributions at an even distance subtract those at an odd distance. Putting those into single sum we have
Σx(-1) exp (a 26)
class = sign
6=0
We can factor out the number of repetitions, r, and it positive doesn't 't change the sign, We can then rearrange match the Binomial theorem form and recover the form in the main text:
(-1)) sign ((e-o=t))=+ -
class = sign
Σ 8=0
a ca- 8
(8)
with p p² (1 p)2 2p² 2p + Something that is important the behaviour of the mean and variance later, and breaks the usual symmetry of and = (1 - p). is that 2 0.5 and is quadratic in defined by the three points (p, p) = {(0,1) 1), 0.5), (1, 1)]- 4 has the usual definition p. so q S 0.5 and so on. The
Next we give the Binomial distribution of the irrelevant feature contribution as (B
contribution at difference d is then X(6) exp (ar - 25 - 2B(3, B). The expectation is computed in the usual way
B
B[ = Σ P(X = 2T)x; = ((B(3, p) =b) exp (a 28 + 26 3)
6=0
exp (a 228)
p q
exp (2b 3) 3
8
= exp (a 26)
(pe) (fe) ( ())
exp (a =
26) (p + ge ge-1))
b=0
Finding the variance in the traditional way, Var[X( [[ E[x()) first [[ following the derivation for [d)]
E E[x(8) Σ P(X x;)aa
P(B(3,P) exp (2a 45 46 23)
(9)
6=0
exx (2a - 40)
Σ
1 (((
exp (46 - 23)
(10)
6=0
exp (2a 40) (se² + Ge 2)-
(11)
From this we can write the variance
Var[X(0)] exp (2a- 46) Go=)+ - (ee + ge (1=+)
-(pe
28
(12)
Next we want to find the expectation the sum of the contributions at each difference 8. As pointed out there are (2) many strings at difference of We apply E [ X.I = Σ, E[X,] and Var(X- - Y| Var X + VarlY),
15

---

Under review conference paper at ICLR 2022
and, remembering to change the sign with each increase in ò,
r( 1() αK
E
B[X(0)]
(13)
6=0
=r (pe² + qe 3) Σ
c
()- 1)
-1
(14)
8
8=0
= r (re² + qe 2) (e (e-e-1 (e-e-1)
(15)
and
(0)
a
Var EiEs( (-16(_1)_ - Var EiES X(6:)
= -Σ'
r
Var Var[x (60]
(16)
0
6=0
=r "((pe² + qe 30-2)² 13
)
-(pe
(pe (pe+ ge-t) 20
Σ
a
a
(e2) a-8
(e
8
(17)
8
6=0
=r ((De² + qe 7)
29
pe + qe
(e² + e 3)a
(18)
C.I. DIFFERENT ATTENTION MECHANISMS
For dot-proee not-product and cosine-similarity attention, 'angular' difference mechanisms, the input variables. This is because we want the score to be greater when the variables are the same and lesser when they are opposed (if we were to use (1,0) we'd have 0 A 1 x and 0 1 0 x 0). With these methods we get
(+; -) encode
(a)(6) = a - 28, Scan (5)
(19)
20
(20)
a
The change for cosine-similarity introduces a factor of exp (1/ar) to the mean and exp (2/a) to the variance, and the overall picture doesn't 't change
(ies(-19) 1111):xXuu X (0:)) = [pe² 1e2 ( (etha-e - -16))
E
(21)
Var )1es(-1)6x x =r "e
ge (m==+ =20) - (īre + qe ge-1)20)
20 (e +e -
a )%
(22)
For squared Euclidean distance attention (which coincides with the 'Laplace attention' LI norm in this encoding), we encode the inputs as (1,0) we get f12(0)- = -(v))² = -5, introducing a factor of exp (-a) to the mean and exp (-2a) to the variance, which also doesn't change the overall picture.
16

---

Under review conference paper at ICLR 2022
D TOP-K FEATURE SELECTION
Top-k feature selection masks out all but the & highest scoring features. That is +- m @ a with
m, {
if raan(s)) S k otherwise
(23)
As compared to soft feature selection which rescales as a
or
3 x with normalised s'.
Directly on inputs
with soft feature selection
0.51
0:52
0.52
0:33
0.54
0.54
0.52
0.55
0.50
0.36
0.53
0.57
0.00
2:62
0:54
2.62
0:72
0.77
0.82
6.0
0.50
0.14
MM
01-2
11-11
Inactive componeets..
8
0.50
0.53
D-30
50
0.53
0.91
2
6
0.77
0.7
0.50
0.47
1.00
3.00
1.00
1004
0.00
9.00
0.22
10.91
00-7
0.72
1:40
1.00
1.000
2
0.50
0.42
0:26
0.89
01.0
66.0
0..00
0.99
0.93
0.6
01:01
1.60)
3.00
1.00
3.00 1.00
3.00
1.00
3.00
0.6
6.00
m.00
0.90
6.00
0.98
0.30
01.00
1:00)
8:00
1.00
1.0
1.00
1.00
4
5
6
7
8
10
5
6
7
8
9
10
with top-k selection
(A:
Attop-k, soft-feature)
10
0.00
10
0AY
01:19
0.24
0.30
0..24
10.2V
0.07
0.21
0.33
0.117
0.20
Inactive components /
8
6.00
0.29
0.9V
0.50
0.30
0..00
1.000
80
0..0
0.-0
0..6
10.00
11:00
00.02
0.00
0190
0.50
0.50
11:1
0.01
0..8
5
0.10
d
0.7
3
0.05
2
1.00
5.00
1.00
5.00
0.00
0.20
5.00
0.00
0.00
5.00
100
1.00
1.00
1.00
1.00
1.00
.1.0
2.00
0.20
0.00
0.00
D.00
0.00
5
6
7
6
7
Variant examples,
Variant examples,
Figure 8: Comparing soft and top-k feature selection in the same setting as Figure classification of XOR, over variant frequency. r, and number of inactive components 3. Examples of variant satisfy the XOR in the same way, i.e. the active components are equal, but may not have the same inactive components. The top-k version uses binary mask leave the & highest scoring features unchanged and zeroing the rest. This produces significant improvements over even the soft feature selection method at high values B, but requires knowledge the number of active elements.
17

---

Under review conference paper at ICLR 2022
E CONVERGENCE OF FEATURE VECTORS ACROSS SELF-ATTENTION ITERATIONS
Figure D illlssraaee how feature vectors converge the XOR2 task across self-attention iterations of the feature selection mechanism.
10.78, 0.80, 0.87
10.81, 0.83, 0.67]
00.89, 0.88, 0.49
0..99. 0.31.
[1066 0.97.0
Figure 9: Feature vectors converging under iterated self-attention on XOR; classification of vectors uniformly sampled over the sphere, in 3D (top) and down the z-axis (bottom). Colours indicate classes. The vectors quickly align by zy-quaarant and the variation is 'washed-out, also seen in the feature selection scores (mean- aoocan-aboouut-devvitito [x, y, 2].
18

---

Under review conference paper at ICLR 2022
F EXPERIMENTAL DETAILS
We use the same feature extractor architecture and train loop for all the baselines.
Feature extractor. We leverage convolutional neural network with 4 blocks feature extractor. Each block consists of a convolutional layer (64 output channels and 3 x 3 filters), followed by batch normalisation (momentum 0.01), a ReLU activation, and 2 x 2 max pooling:
Conv2d(64, x 3) + BN + ReLU + MaxPool(2 x 2)
Then, we flatten the output and apply a linear layer to map the data into a 64-dimensional embedding space (unless otherwise stated). As explained in the main manuscript, each method then manipulates these embeddings in different way.
Train loop. We train all models in an episodic manner. At each training iteration, we follow these steps:
First, we sample spec-fie support and query sets. For polythetic MNIST, the support set has 96 samples (2 classes, groups per class, and 24 group-specific examples per group). The query set consists of 32 samples (2 classes, 2 groups per class, and 8 group-specific examples per group). For Omniglot, the support set consists of 5 examples for 20 classes (20-way, shot). The query set consists of 15 examples per class.
Second, we compute emheddings using the feature extractor and produce class probabilities for the query points. The way in which these probabilities are computed depends the method (c.g. attentional classification for matching networks or softmax over prototype distances for prototypical networks).
Finally, we compute the cross entropy for the query examples and optimise the feature extractor via gradient descent. We employ Adam optimise with learning rate 0.001.
We train the models for 10,000 iterations (i.e. tasks) for all experiments, except for full polythetic MNIST (100,000 tasks). We then compute the performances on a held-out dataset and average the results across 1,000 tasks.
G MULTI-CATEGORICAL PRE-TRAINING
In this experiment we first train a classifier in the multi-categorical setting for the full polythetic MNIST task. In this there are task-irrelevant digits or colours the label describes all four digits with their colours: there are four 10-way labels for the digits (E R4x11) and four 3-way labels for the colours (E R4x3) for a combined multi-hot output vector 52 The model architectures match that used in the other experiments, see Appendix FF] other than using variation of the MLP head that takes the flattened output from the convolutional network. The variation has two hidden layers with 512 units each and ReLU activations, before linear layers with softmaxes for each of the label heads. The model is pre-trained over 800 batches of 16 examples drawn at random from the label combinations. In the multi-categorical pre-training. the model achieved a validation accuracy of 95.6% on the digit labels and 100% on the colour labels over 1600 validation examples.
The pre-trained model was then used with prototypical and attentional classifiers in the polythetic MNIST few-shot classification setting discussed in Section 5 and detailed further in Appendix [F) We compared performance using the multi-headed softmax activations the model was pre-trained with and elementwise sigmoid for both the monothetic and polythetic settings. The results are presented in Table 7 and conform the trend perform no better than chance for polythetic tasks. Attentional classifiers are weaker in the monothetic setting, but more than make up for this defeict in the polythetic setting.
in other experiments: threshold classification has the advantage in monothetic tasks but
19

---

Under review conference paper at ICLR 2022
Table Polythetic MNIST problem with multicategorical pre-training. Mean over 1000 tasks.
Softmax
Sigmoid
Model
Monothetic
Polythetic
Monothetic
Polythetic
Proto.
97.95
50.18
94.64
50.21
Attn.
93.48
93.40
88.16
86.24
H AN ILLUSTRATIVE EXAMPLE OF THE METHOD
Figure provides a more comprehensive overview of the proposed method. In this example, we are interested in distinguishing between big-cats and equids (horses, donkeys, and in the case zebra). We first extract features from all samples in the support and query sets. Here, we imagine these features as corresponding some general 'cat' properties, patterns (such stripes, dots etc.) and general 'equid' properties. Next (as presented in lines 2 in Algorithm[[] we perform repeated self-attention with respect to the separate classes of the support set, which yields updated features for each support sample.
We then aggregate the resulting support features with an appropriate dispersion metric, e.g. mean absolute deviation standard deviation, (line in Algorithm [ to obtain vector of feature scores. These scores quantify the relevance of each feature in a given task. Next, we rescale both the query and (initial) support features, i.e. multiplying by the feature scores, dilute the task-irrelevant and (potentially) misleading features. In this particular example, this would correspond to diluting the 'patterns' feature, since is irrelevant when distinguishing between cats and equids. Finally, produce class probabilities via an attentional classifier.
L24
support set
query set
Figure 10: Diagram the proposed approach. Note how misleading features stripes and spots, which may cause misclassification, diluted through the feature selection process.
20

---

Under review conference paper at ICLR 2022
I CONSTRUCTION OF BINARY STRINGS TASKS
For binary strings tasks, we construct training tasks by sampling [S] 5-2" support examples and [Q] = 5 2 query examples. Each example consists of a + 3 bits. The location of the a active bits is completely random and consistent between the support and query sets within the same task. The labels computed as the XOR over the active components. The remaining 3 noisy components are randomly sampled from a Bernoulli distribution with protability 0.5. Figure 11 shows example of binary strings task with 2 active components, 8 noisy bits, and r = repetitions.
Support set
Query set
+
+
+
7
+ +
+ 7
+
Figure 11: Example of XOR, task with a 2 active components (3rd and 5th bits), noisy bits, and repetitions for each combination of active components. The support and query sets contain r2a examples each.
J CONSTRUCTION OF POLYTHETIC MNIST TASKS
For polythetic MNIST tasks, the support set consists of 96 samples, with 48 samples for class O and 48 samples for class 1. Each class is further divided into 2 groups of 24 samples and each group is defined by specific set of traits. The groups are complementary between classes, e.g. red ones and blue zeros for class O: and blue ones and red zeros for class The set of traits sampled randomly for each task. The query set sampled in the same manner, with 2 groups per class and 8 samples per group. Tables B (Snad] summarise the details of the support and query sets and Figures [3] 14 and show the whole support set for the clean, colourless, and full versions of polythetic MNIST.
e 4
0 5
6 H
4 7
3 9
7 3
2 /
3 3
8 2
7
9
8 6
4
7
2
9
g
6
3
Clase::
Class:
Class
Clss: D
Clsss:
Cluss:
Class: 1
Cloos:
Clase: 1
Class: 1
Case:
Class
2 7
2 8
9 g
9 g
7 6
0 9
8
O 6
J 5
5 5 6 5
g 6 6
4 6
6
6
0
6
6
6
6
9
6
6
5
6
3
Figure 12: Example of an MNIST polythetic task. Examples from class 0 have either digit (in any colour) in the bottom-left corner with a red digit in the bottom-right corner (top row); or digit 6 in the bottom-left corner with a green digit in the bottom-right (bottom row). Examples from class can have either digit 1 in the bottom-left with a green digit in the bottom-right (top row); or digit 6 in the bottom-left with red digit in the bottom-right right corner (bottom row).
21

---

Under review conference paper at ICLR 2022
Support set, Polytheric
Class:
Class:
Class:
Class:
Class:
Class:
Class:
Class:
Class:
Class:
Class:
Class:
Class:
Class:
Class: 0
Class:
Class
Class: a
Class:
Class:
3
3
3
3
3
Class:
Class:
Class: 0
Class:
Class:
Class:
Class:
Class:
3
3
3
3.
3
3
3
3
3
3
3
3
7
3
Class:
Class:
Class:
Class:
Class:
Class:
Class:
Class:
3
3
3
3
3
3
3
Class:
Class
Class:
Class:
Class:
Class:
Class:
Class:
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
Class:
class:
Class:
Class:
Class:
Class: 1
Class:
Class:
Class
Class:
Class
Class:
Class:
Class:
Class
Class:
Figure 13: Example of the support for polythetic MNIST task (clean).

---

Under review conference paper at ICLR 2022
Support set, Polytheiic task
Class:
Class:
Class: 0
Class:
Class:
Class: 0
Class: 0
Class:
4
8
8
9
8
8
8
8
3
8
3 8
9 g g
5
8
6
8
8
6
8
4
8
8
8
S
8
6
Class:
Class:
Class:
Class:
Class:
Class:
Class:
Class:
41 8
5
8
9 8
2 !
4
8
/
8
9
4
2
g 8
9
8
9 8
4
8
8 8
5
8
4
8
9
8
2
Class
Class 6
Class: 0
Class:
Clacs
Class: a
Class: 0
Class:
S
8
8
4 8
0 8
3
8
5
g
8 8
5
8
4
8
4 8
5
4
0
8
6
8
7
8
4
7 8
Class:
Class:
Class:
Class:
5
1
4
/
O / 0 2
7
/ 2
7 3
5
7
/ 2
0 l 2
9
2
2_
7
2
3
2
4
2
Class:
Class: o
Class: 0
Class:
Class:
Class: 0
Class: 0
Class:
5 / 2
/
3 f
7
/
S
/
3
1
4 2 2
3
/ 2
3
2
2
2
9 2
2
2
S 2
9
6 8
1
4
!
/
0 1 2
3
/ 2
9
6
2
0
2
9
6
/
7
2
7
2
4
2
Class:
Class:
Class:
Class:
Class:
Class:
Class:
Class:
7 I
3
1
4
/
5
/
6
1
8
/
9
6
}
8
6
8
6
8
3
8
6
8
8
8
0
8
7
8
Class:
Class:
Class: 1
Class:
Class:
Class: 1
Class: 1
Class:
7 8
7
4
8
7
9
/
O
/ 8
2
/
g
4
8
3
8
4
8
7
8
7
3
8
/
8
Class:
Class:
Class:
Class:
0 Z
/
8
|
2
6
/ 8
/
2
/
4
|
&
6
8
5
4
S
8
4
8
9
8
5
8
Class:
class:
Class: 1
Class:
Class:
Class: 1
Class: 1
Class:
8
%
9
8
Z
8
t
8
8
4
8
O
8-
2
2
8
2
4
9
2
/
>
2
0
2
0
2
class:
3 g
4 8 2
7 8
5 8
6 8
4 8
2 8
8
8
6 2
O 2
3
7
7 2
3
2
9
2
Class
class
Class 1
Class
Class
Class 1
Class 1
Cass
5 8
2 8
5 d
3 8
4 8
2 8
6
8 2
O
$ 2
6 2
6 2
5 2
2 2
2 2
/
2
b
Figure 14: Example of the support set for a polythetic MNIST task (colourless).

---

Under review as conference paper at ICLR 2022
Support set, Polythetic Lask
Class:
Class: 0
Class: 0
Class:
Class:
Class: 0
Class: 0
Class:
8
7
8 4
8
O
8
X
8
8 8
8
4
8
6
8-
7
8
8
5
B
5
8
2
8
Class:
Class:
Class:
Class:
Class:
Class:
Class:
Class:
g+ 8
0
8 7
g
7
8 2
8 3
8
7
8 S
8
9 ?
2
8
9
8
9
8
1
8
4
8
6
8
Class
Class 6
Class: 0
Class:
Class:
Class: 0
Class: 0
Class:
8 8
4 4
8 9
8 9
8
7
8 3
8 2
8 7
8 O 9
0
8
2
8
7
8 7
8
5
8
7
8
Class:
Class:
Class:
Class:
2
3 3
2 7
29 8 6 7
2
7
2
1
2
2 /
> 6
2 9
6
6 6
6
9
6
1
6
7
5
Class:
Class 0
Class: 0
Class:
Class:
Class: 0
Class: 0
Class:
2 2
2. O
2 7
2
/
2 6
2
6
2 6
2
(
6
9
6 8
6
4
6
6
9
6
4
6 6
6 3
Class
Class
Class
2
3
2 O
8'
2
2
6
7
2 7
2 2
6
8
6
6
6
6
6
7
6
6
6
7
Class:
Class:
Class:
Class:
Class:
Class:
Class:
Class:
8 6
g
8
9
8 4
g
2
8
6
8
9
8 3
8 9 2
4
6
2
6
7
6
2
6
9
6
6 3
6
Class:
Class:
Class: 1
Class:
Class:
Class: 1
Class: 1
Class:
8
4
8 2
8 6
4
8 O
8 7
8
8
2
?
7
6
2
6 9
6
3
6
9
6
6
7
6
g
Class:
Class:
Class:
Class:
Class
8
O
8
(
8 q
8
6
8 0
8 5 3
g 9
8 4
6
H
6
S
6
6
6
/
6
6
6
6
7
Class: 1
Class: 1
Class: 1
Class:
Class:
Class: 1
Class: 1
Class:
2 7
2
8
2. 6
2
3
2
?
2
5
2
2 3 2
8 4
8
4
8
2
8
2
8 9
8 4
8
2
8
Class:
class:
Class:
Class:
2
(
2, 8
2 3
2 3
2
S
2 3
3
9
2 3
8 9
8 3
8 6
6
4
8 7
8 4
8 9
8
S
class
Class 1
Class 1
Class
Class
Class 1
Class 1
Class
2
4
2
2
6
2
4
2 c
2 7
9
2 3
8
/
8
2
8
8
9
8 7
8
6
8
8
8
Figure 15: Example of the support set for a polythetic MNIST task (full).

---

Under review conference paper at ICLR 2022
Examples
Groups
Examples per group
Example of groups
Class
48
2
24
green ones and blue threes blue ones and green threes
Class
48
2
24
Table 8: Support set details
32
Examples
Groups
Examples per group
Example of groups
Class
16
8
green ones and blue threes blue ones and green threes
Class
16
8
Table 9: Query set details
25


Reviews:
[{"review_id": "qN_kajIiqxh", "reviewer": "Reviewer_q98z", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1.\tStrong motivation and introduction of monothetic and polythetic classifications, drawing from prior work from other fields\n2.\tInteresting analysis and excellent visualizations\n3.\tUses simple Boolean tasks to explain background conceptually\n4.\tWell-designed toy datasets and experiments to verify hypotheses of the nature of ProtoNets and Matching Networks in the context of the paper\n5.\tWell-written\n\nCons:\n1.\tPractical value isn’t clearly demonstrated; improvements over matching networks for the only non-toy dataset (TieredImageNet) is marginal at best.\n2.\tLimited baselines in experiments\n\nMore detailed comments:\n\nThis paper examines few-shot learning in the context of monothetic and polythetic classifications. This perspective sheds some light on how ProtoNets and Matching Networks perform classification of features, and there is some interesting analysis to back these hypotheses up. Concepts from the paper are well-explained with several simple examples, and toy settings are designed to illustrate them empirically. Furthermore, the analysis leads to a simple self-attention-based solution to improve features for classification. Overall, this paper reads well, and the analysis leads to some interesting insights into few-shot learning.\n\n\nEmpirically, the results are a little more disappointing. While several toy experiments (e.g. XOR binary strings, polythetic MNIST) are well-designed to illustrate the advantages of considering polythetic patterns, it’s not as clear how much the real-world exhibits these characteristics. The proposed method strongly outperforms the baselines in the toy settings, but improvements over the Matching Networks baseline on TieredImageNet is marginal at best. As such, as nice as these insights are, it’s not clear if it’s practically useful. Additionally, while I understand the focus was comparing with ProtoNets and Matching Networks as representatives of threshold-based and attentional classifiers, it would have been nice to have included more baselines in the experiments.\n\n\nQuestions:\n1.\tWhile conceptually easy-to-understand, I was previously unaware of categorizing classifiers as either attentional or threshold-based. Are there any other kinds, or must a non-attentional classifier be threshold based (and vice versa)?\n2.\tHow do other few-shot methods fit into this taxonomy? For example, what are kinds of features (monothetic vs polythetic) are optimization-based methods (e.g. MAML) learning? What about simply training a classifier, or SVM?\n\n\n\n=====Post-Discussion=====\n\nI thank the authors for all their effort during the discussion phase to clarify various questions about their paper and for running additional baselines. After reading the other reviews and seeing the authors' responses, my recommendation remains mostly the same. The new perspective on meta-learning provided by the authors is an interesting one, but the practical benefits of this approach can still be more concretely demonstrated. Additional experiments in such use cases (e.g. the DNA example mentioned in one of the discussions, if such a dataset exists) would significantly strengthen this paper.", "summary_of_the_paper": "This paper discusses monothetic and polythetic classifications in the context of few-shot learning: distinguishing similar classes often require reasoning with combinations of certain features, which can be especially challenging when only a few training examples are available. Two common types of few-shot methods, ProtoNets and Matching Networks, are shown to be threshold classifiers and attentional classifiers, respectively, each with their advantages and disadvantages. The authors introduce a simple method based on self-attention as a solution to these challenges, with experiments on several toy tasks and the more real-world TieredImageNet.", "summary_of_the_review": "As stated in the main reviews, the empirical results aren’t particularly impressive, but overall the paper does provide some interesting analysis that may encourage new ways of thinking about few-shot learning problems. As such, I think this paper may be of interest to the ICLR community.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "iF4ZPGSGvFn", "reviewer": "Reviewer_PpwW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\nThis paper discusses meta-learner from a very unique perspective, with the shortcomings of both ProtoNet-liked threshold meta-learners and MatchingNet-liked attentional meta-learner discussed solidly with intuitive examples and convincing derivation of misclassification rates. \n\nThe proposed attentional feature selection is simple yet effective, and can potentially guide and stimulate many follow-up improvements to meta-learning. \n\n**Weaknesses**\n\nMy main concern is that this paper tends to isolate itself from the entire research of meta-learning, and focus on a corner instead. \nProtoNets (representing threshold meta-learners) and MatchingNets (representing attentive meta-learners) are insufficient to cover the entire research of meta learning. There are many other directions of meta-learning that are completely overlooked in the discussion. \nThe most representative case is the inner-loop optimization-based meta-learner like MAML [1], where the inner-loop adaptation to the feature extractor can potentially solve the challenge of 'not all features are relevant in all tasks and that the support is unlikely to span the input domain' pointed out in this paper. \n\nAnd many other directions and methods, e.g., methods based on Hebbian rules [2] might share a similar spirit with the proposed method, and the presentation can be further improved by incorporating more comprehensive discussion to the latest progress of meta-learning and the connections of the proposed method to others. \n\n**Minor**\n\nThe experimental settings included in this paper seem a bit weak. More common benchmarks on real-world meta-learning (classification) tasks, like miniImageNet, and the challenging cross-domain settings can better support the discussions.\n\nThe overall writing is good, but some further improvements are expected. For example, is the very first sentence of this paper grammatically incorrect? I believe it's better to say 'that need to be neither universal nor ...'\n\n[1] Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks, ICML2017\n\n[2] Differentiable plasticity: training plastic neural networks with backpropagation, ICML 2018", "summary_of_the_paper": "In this paper, the authors present a discussion on two main-stream meta-leaners with attentional classifiers and threshold meta-learners under a unique view of polythetic classification. And to address the limitations of both, attention-based feature selection is introduced. \nImproved performance is demonstrated by both synthetic and real-world few-shot learning tasks.\n", "summary_of_the_review": "I give an initial recommendation of score 6 mainly for appreciating this novel view to mainstream meta-learners, and solid discussions supporting the points. However, to meet and standard of ICLR and demonstrate a clear contribution to the research of meta-learning, I believe further efforts on comprehensive discussions are highly expected. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "Wmir7yL0ROR", "reviewer": "Reviewer_Hoh6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n\n(1) The explanation about the challenges of threshold (Prototypical networks) and attention classifiers (Matching network) is interesting.\n\n(2) The motivation on the polythetic classification is well-motivated. \n\n(3) Well-written and easy to follow.\n\n\nWeaknesses:\n\n(1) Lack of technological innovation. The proposed feature-selection mechanism is too simple and the only technological innovation.  \nThey only use self-attention to obtain a better representation rather than directly using average pooling (Prototypical networks) or all the support data (matching network).\n\n(2) How to choose repetitions R?  It is a hyper-parameter, is the bigger the R the better? Does it have no upper bound?\n\n(3) Lack the experiments for comparison with threshold classifiers and attentional classifiers in the main paper. It is crucial to show the problems of threshold classifiers and attentional classifiers.\n\n(4)  Self-attention also has some parameters for the transformation, why is the proposed method non-parametric?\n\n(5) The related work is too little. Maybe consider adding some self-attention work?\n\n\n", "summary_of_the_paper": "This paper first considers the limitations of threshold and attentional classifiers. They proposed an attention-based method for feature selection to address the problems of threshold classifiers and attentional classifiers.  The experiments on  several synthetic and real-world few-shot learning tasks seem good. ", "summary_of_the_review": "The implementation details and related work of the proposed paper should be further added.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Hh-eaecbZdA", "reviewer": "Reviewer_f8L4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "STRENGTHS:\n\nThe few-shot polythetic classification problem is interesting and novel, and the shortcomings of baseline approaches are readily apparent. Work is well-grounded in older prior literature and the XOR_alpha task is extensively analyzed both theoretically and empirically. Writing is clear and concise (though not always easy to understand, see below). The proposed method performs well in its intended setting. \n\nWEAKNESSES/ISSUES (in no particular order):\n\n- The XOR_alpha meta-learning task is never formally described (it is only described as a function of bits on page 3), and so subsequent sections become difficult to follow. Please state clearly in section 2 that XOR_alpha also refers to the collection of (presumably?) all alpha-variable XOR “tasks” in a given binary n-space with n>=alpha, with a (presumably?) random partition of train and test tasks. Perhaps also consider using different notation when referring to the meta-learning problem (i.e. caption for Fig.3 right) vs the task (i.e. second to last paragraph of pg.4). Either way, the problem setup for this and other experimental tasks more broadly (i.e. the non-standard tiered-ImageNet task) should be elaborated up front and more clearly. As is I would have a difficult time re-implementing some of these benchmarks without the provided code. \n- Contrary to claims in the abstract and conclusion, the paper does not show that the embedding space of a threshold classifier must grow exponentially with the number of features. Pg.4 discusses embedding growth but has it at O(n^alpha), which for given alpha is only polynomial in the number of features. The embedding space does grow exponentially with task complexity alpha, which is itself O(n), but they are not the same and these claims should be clarified (assuming that this is in fact authors’ intended argument). \n- On a much broader level, I have a hard time understanding the motivation for this work. What is the envisioned use case scenario for this kind of approach? The synthetic examples, while interesting, are very contrived (which is fine since they’re synthetic) but the paper does not provide a strong motivation for the “real world” examples (omniglot and tiered-ImageNet) either. These are also somewhat contrived: while the fine-to-coarse generalization task under discussion is interesting, it’s the opposite coarse-to-fine generalization that has a clear use case. What is the envisioned use-case for a polythetic meta-learner? The introduction suggests certain kinds of fine-grained classification, but if this is the case then it should be investigated directly, such as with CUB or tiered meta-iNat benchmarks. \n- The first paragraph of page 2 is difficult to parse, both because sentences are somewhat long and mostly because it is not immediately clear how conclusions are following from premises (i.e. the described 45degree rotation/”change of basis” of the OR function was initially confusing to me because this geometric operation does not translate intuitively into pure Boolean algebra; the rates of growth (2^2^n and 2^n^2) are not immediately obvious). This section could use some elaboration and clarification. \n- I do not find Appendix A convincing. Appendix A claims to be a demonstration that protonets do not generalize to unseen variable combinations. While it is true that train and test combinations do not overlap, by my understanding the chance performance here is clearly due to the fact that training time noise features are test time signal features, and vice versa, and the network has simply (and correctly!) learned to suppress the “relevant” noise features. As such, no learning-based classifier should be able to solve this problem as presented; it is less a demonstration of the limits of protonets and more a demonstration of the no free lunch theorem. A much more useful and interesting set of results would be to partition the combinations of active variables randomly over train and test, so that combinations do not overlap but features are equally active in expectation at both train and test time. By my understanding this is exactly the experiment in Fig.3 right, though, so perhaps this ought to just be removed entirely. \n- Authors use a pre-trained ResNet-18 for the tiered-ImageNet experiment, but the provided source for the model is PyTorch itself. How was this model pretrained? If the model was pretrained on ImageNet, then it has been trained on the test images and these results are not valid. \n- Typo in Algorithm 1 Input: the an arbitrarily ordered matrix -> an arbitrarily ordered matrix\n", "summary_of_the_paper": "Authors propose the general problem of few-shot polythetic classification, where class membership is determined by the combinations of present and absent features and the salient features and combination patterns change at test time. Authors demonstrate that prototypical approaches with linear decision boundaries respond poorly to these highly non-linear problems, while attention-based soft nearest neighbors paradigms work well but overfit to specious cues. An attention-based feature refinement technique is proposed and evaluated, beating both baselines on specifically polythetic few-shot tasks. ", "summary_of_the_review": "The paper proposes an interesting problem, and based on theoretical and empirical analysis provides a neat solution. Issues stem from the perhaps overly concise writing; many aspects of the paper are in need of elaboration. These include the motivation, problem setups for the various benchmarks, and the reasoning behind certain conclusions. Currently I am scoring the paper as if my understanding is correct and authors’ intended arguments are simply wrong, but if these issues can be clarified I’d be happy to raise my score. \n\nPOST-DISCUSSION:\n\nAuthors mostly address the above issues, in discussion and in the revised manuscript. I have some lingering concerns, shared with reviewer q98z, regarding the lack of _demonstrated_ practical benefit, and the fact that accuracy gains in the more “real-world” benchmarks shrink substantially relative to the motivating XOR problem. However, the revised paper is much improved, conceptual novelty remains high, and on the whole the paper is of sufficient quality for acceptance. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper analyzes problems of existing threshold meta-learners and attentional meta-learners for few-shot learning in polythetic classifications. The threshold meta-learners such as prototypical networks require exponential number of embedding dimensionality, and the attentional meta-learners are susceptible to misclassification. The authors proposed a simple yet effective method to address these problems, and demonstrated its effectiveness in their experiments. This paper discusses meta-learning from a very unique perspective as commented by a reviewer, and clearly explained problems of widely-used meta-learning methods. However, this paper focus on prototypical networks and matching networks even though there have been proposed many meta-learning methods. Some existing methods seem not to have the problems of prototypical networks and/or matching networks. In addition, the practical benefits of the proposed approach are not well demonstrated. Although the additional experiments in the author response addressed some concerns of the reviewers, they are not enough to demonstrate the effectiveness of the proposed method.
Decision: Reject