Few-shot Learning by Focusing on Differences
Authors: Muhammad Rizki Maulana
Lee Wee Sun
Venue:  (2020)
Keywords: Deep learning
few-shot learning

Abstract: Few-shot classification may involve differentiating data that belongs to a different level of labels granularity. Compounded by the fact that the number of available labeled examples are scarce in the novel classification set, relying solely on the loss function to implicitly guide the classifier to separate data based on its label might not be enough; few-shot classifier needs to be very biased to perform well. In this paper, we propose a model that incorporates a simple prior: focusing on differences by building a dissimilar set of class representations. The model treats a class representation as a vector and removes its component that is shared among closely related class representatives. It does so through the combination of learned attention and vector orthogonalization. Our model works well on our newly introduced dataset, Hierarchical-CIFAR, that contains different level of labels granularity. It also substantially improved the performance on fine-grained classification dataset, CUB; whereas staying competitive on standard benchmarks such as mini-Imagenet, Omniglot, and few-shot dataset derived from CIFAR.

Under review as a conference paper at ICLR 2020
FEW-SHOT LEARNING BY FOCUSING ON DIFFERENCES
Anonymous authors Paper under double-blind review
ABSTRACT
Few-shot classification may involve differentiating data that belongs different level of labels granularity. Compounded by the fact that the number of available labeled examples are searce in the novel classification set, relying solely on the loss function to implicitly guide the classifier to separate data based on its label might not be enough; few-shot classifier needs to be very biased to perform well. In this paper, we propose model that incorporates simple inductive bias: focus- ing on differences by building a dissimilar set of class representations. The model treats class representation among closely related class representatives. It does so through the combination of learned attention and vector orthogonalization. Our model works well on our newly introduced dataset - CIFAR-Hard - that contains different levels of labels granularity. It also substantially improved the performance on fine- lne-grained classi- fication dataset, CUB; whereas staying competitive on standard benchmarks such as mini- Mumi-lmagenet, Omniglot, and few-shot dataset derived from CIFAR.
vector and removes its component that is shared
INTRODUCTION
Progress in artificial intelligence (AI) has been rapid. AI agents have been outperforming humans in an increasing variety of tasks, such in recognizing images on ImageNet (He et al. (016)) and in the ancient game of Go (Silver et al. 2016). However, challenges remain - systems that outperform humans usually require learning from very large-scale data. In contrast, humans only require few examples to be able to rapidly adapt to a novel task; humans are still better learners. Few-shot learning methods which learn classes from few labeled examples aim to bridge this gap.
In learning a classification algorithm from a few labeled examples, one may train the algorithm with different set of abundantly labeled data (base set); before adapting it to the unknown examples. However, might be the case that the available labeled examples are of different granularity level. For example, it is possible that it is only trained to differentiate between on differentiating different breeds of dogs. The set of labeled examples is also very limited; in the extreme case, only one labeled example is provided for each class (called one-shot classification). A few-shot learning algorithm can learn to identify features that are important for doing well on the base set these can be adapted to classify the few labeled examples as long as the domain remains similar. But with so few examples provided and possible differences in the task granularity, few-shot learning algorithms need be very biased to perform well. The question then: what kind of bias is reasonable?
and dogs, but is tested
Our work. In this paper, we propose model that performs classification in a novel task by fo- cusing on the differences between closely related classes of its support set. Our bias is loosely inspired by how scientists often work (Mill's method of difference): in looking for potential causes of a phenomenon, a scientist would often focus the differences in the circumstances (features) in the instance in which the phenomenon occurred and the circumstances in instances for which the phenomenon did not occur (Mill 1875). Our method focuses on the differences by removing components in each class representative that are shared with closely related classes.
Our contributions.
1. We introduce a method which incorporates a simple yet effective inductive bias: focus- ing differences, and show that it works well classifying closely related classes. Our

---

Under review as a conference paper at ICLR 2020
results show that the method achieves better performance standard fine- grained classi- fication benchmark (CUB dataset) and on our proposed benchmark consisting of a mix of fine and coarse- geriined classification tasks, CIFAR-Hard On other commonly used bench- mark datasets, CIFAR-FS, mini-ImageNet, and Omniglot, its performance is competitive with existing methods.
2. We propose methodology to build harder few-shot learning datasets without requiring very large hierarchically labeled datasets. Using this methodology, build CIFAR-Hard a mix of fine-grained and coarse-grained few-shot classification dataset derived from CIFAR-I00. Our empirical evaluation shows that the currently existing methods well equipped to handle this scenario.
DISSIMILARITY NETWORK
2.1 FEW-SHOT LEARNING
In few-shot learning. we are given base set B and a novel set N. The base set contains labeled examples from a large number of classes while novel set contains classes not found in the base set. The objective of few-shot learning is to train a classification algorithm P on the training set Xtrain 1B, in such way that generalizes to the elements of the novel N. Some methods may also train on the small number of labeled examples from the novel set. In that case, the training becomes Xtrain = B U Nabeled, with labeled novel set Nabeled C. N. In one-shot learning, the novel set only contains one labeled example for each class, while for k-shot learning, the novel set contains & examples for each class.
We use the episodic training proposed by Vinyals et al. 2016) to make sure that the training and test condition match. At every step, we sample some examples to form an episode C Xtrain to train the classification algorithm P. Each episode T consists of small set of N-labeled examples (called support set) S = {(x,, y/4) } and a set of M examples to be labeled (called query set) Q } f-m where æ; E R is a D-dimensional feature vector with label y: (for computing the loss on the query set), simulating the conditions for learning the novel set N. We denote the set of labeled examples from the support set with class {1..., K} in an episode as S₁.
Our method adopts the approach of similarity learning. Instead of learning distance or similarity function, we learn a space (embedding) that works well with fixed similarity- classifier in that space. Specifically, our model learns construct space that is optimized to separate data that belong to different classes for a classifier that uses dot- dot-product as its similarity function. We define two levels of embedding based how the embedding utilizes task information:
Global task embedding learns an embedding function that is optimized for all episodes that it is trained upon. The assumption is that the produced embedding will learn meaningful and general representation that is sufficient to separate data points on the novel task without knowing what classes appear in it.
Task-aware embedding removes the assumption that the learned global embedding is sufficient for the novel task. Instead, takes the possible classes of the novel task into account. On the novel task, it will embed the query set conditioned on the support set that is aware of its member, giving the full context to the prediction. Our model builds an explicit task-aware embedding that separate a class from the weighted average of its closely-related classes. It is explicit in the sense that our model explicitly encodes such an inductive bias into its function (architecture). In contrast, implicit task-aware embedding only relies upon its loss function to adjust its function as to induce separation between classes.
2.2 MODEL
For each class, our model which we call the Dissimilarity Network - computes class representa- tive, called a protorype, that is used to classify new instance of data by comparing its similarity to the prototype. through an inner product. For 1-shot classification, each training example in the novel task is transformed into the prototype, while for k-shoo classification, the mean of the k instance representations that belong to each class is transformed into the prototype.
2

---

Under review as a conference paper at ICLR 2020
Wi
f
W2
self-atta
44
Ck
Wk
2x
Figure 1: Dissimilarity Network architecture
The Dissimilarity Network computes prototypes that are dissimilar to one another, in the sense that the components of the class prototypes are orthogonal to the direction of the weighted average of closely-related classes. This lifts the burden of the classifier as the classes are more easily distin- guishable since what remains are their differences. The model works by iteratively enhancing the representation of the prototypes through set of learned transformations (embedding functions). The first transformation builds a global task embedding through a learned cameenionnaliy-reduutiie function. The global embedding retains features that useful for the training tasks but does take into account the classes that are present the novel task. The second transformation transforms the prototypes into a task-aware embedding using self-attention networks, taking into account other classes that are present in the task. Finally, the last transformation computes the class prototypes that are dissimilar, by locally orthogonalizing the representations the weighted average of other closely related class representations.
When presented with new point to classify, computes the global embedding for that point, trans- form it using the tware embedding, locally orthogonalize the point for each possible class, and select the most similar prototype's class as the class the new point. Figure illustrates how the model constructs class representations as well as predict the new point.
2.2. GLOBAL EMBEDDING
We learn feature extraction function fy TRD > HR for the images to reduce their representation // 44dimensional vector. We used deep convolutional neural networks (Krizhevsky et 2012) which captures the local interaction of neighboring pixels and builds a hierarchical representation of them. This feature extractor constructs our first level of embedding. It is trained to extract information that is useful for all the training episodes but is not specialized to particular novel task. We use the global embedding to compute the class mean (or prototype) e₁ of the H H--imenniiona representation of the support points,
[x:))
(EE
2.2.2 SELF-ATTENTION
By seeing the prototypes conditions the member the set to every other member within the set. This gives task-awareness to our prototypes; each member is now aware of the other class representatives of the given task.
of vector: C {c
learn set-to-set function that
Our set-to-set operation based on the self-attention mechanism introduced by Vaswani et al. 2017). Given query, attention mechanism learns "attend" by means of weighted aver-

---

Under review as a conference paper at ICLR 2020
a to different parts of the set depending how relevant is to the query. In self-attention, the query is the member of the set itself. Our self attention uses embedding functions for query hg. key kg, and value ht. Each embedding function is parameterized by neural network that computes a mapping K - RK*H For simplicity we assume that for any input in the form of a set of vector A. it will automatically be cast into matrix A E RRKH whereas the output will be cast back into a set. The self-attention mechanism sel fattn R K +
12 xxx is formulated as follows.
he ho(C))x ) (C)(C)
self-attn(C)) = softmax
(2)
VH
Intuitively, the self seff-attention computes a weighted average of the element of the input matrix C' E RK*", representing set of prototypes C. This operation has the effect of averaging out noisy components from global embedding that may be relevant for other tasks but are irrelevant to the set of classes in the current novel task. The weights are obtained using the learned attention function, which in this case, is parameterized by lig and lg.. In our case, our prototype C learns to aware of the other class representatives C \ Cj by incorporating some of their components.
We use bidirectional LSTM (BLSTM) (Hochreiter & Schmidhuber, 1997) for our attention em- bedding function for query he and key lg. while using either identity function or BLSTM on hy. BLSM computes a concatenation of two sequence of opposing- direction by sequen- tially applying the element æt E RH of its imput = [#1,...,,y into LSTM, h2 uz LST M h1-1, 11) The computation yields a sequence of vector that each are conditioned on its neighhoring elements. Our BLSTTM uses context sharing between key attention embedding h& and query attention embedding ha.
We are aware of the sequential nature of BLSTM, which can be counter-intuitive we are model- ing a set-to-set operation which should not have any preference for ordering. However, we found empirically that this setup offers more performance gain compared the use of traditional linear function as attention embedding function. The BLSTM may learn to ignore the unimportance of set ordering due to the nature of episodic training, which exposes it to many permutations of the possible class-orderings. Moreover, the attention also gives global context of the member of the set, which could further alleviate the ordering issues (if any).
2.2.3 FOCUSING ON DIFFERENCES
We encode our inductive bias in the form of neural network architecture, in which we remove the components that are shared among other class representatives, thereby giving the model the ability to focus only on the inter-class differences. Since we treat the prototypes as a vector, one natural way to achieve that is by making the prototypes to be locally orthogonal to the components that are shared among other classes. We learn to find such components by using dot-product attention.
For --dimensional prototype ca E C of class k. it will have corresponding task-aware vector prototype wi E W = self-attn(C) following the method described in the Section 2.2.2 Let W' = W \ Wk. the components shared among the other classes k' + k that are locally orthogonal to the vector prototype wa is computed using attention function attn RH x RK*H
#
Wk W
attn W') = softmas
W
(3)
VH
Essentially, through weighted averaging, it selects components from the other class prototypes based on how similar the embedded vector prototype wy is to them.
We make the prototype Wi to be orthogonal the shared components of W we by remov- ing its projection to the shared components. Specifically, we use the shared components b, = attn(wx, W \ wx) that belongs class k as the basis of the projection proj by) as follows:
We bb bE
proj by)
b
Intuitively, it maps the vector of prototype wy to represent its direction using the given basis by (i.e.. projecting it into the basis). It produces components that are linearly dependent on the basis, thereby removing it: 2 = wR proj wt. b,) will produce a new prototype representation 2n that is orthogonal to the weighted average of the components of other closely-related prototypes.
4

---

Under review as a conference paper at ICLR 2020
2.2.4 CLASSIFICATION AND LEARNING
To classify new point L. we follow similar transformations computing the locally orthogonal prototypes. First global embedding is computed for the point é = fy (@). For consistency, we follow the transformation given by sel f -attn. However, since we only have a single point é, we construct a set of length K by duplicating the point K times as the input: W feat-a (())
If the attention value embedding by (Section 2.2.2) is an identity function (or any other element- wise function), any V e = è since its taking a weighted average of a set of identical vectors. When by a BLTSM (or function that operates on a set of elements), by transforms the vector through a mtati-stage non-linear processing.
For a task-aware embedding uZ E W. we compute the vector that is locally orthogonal to the set of prototypes of class k * k by computing &g Vvx proj((ux. by) with basis given by by= = attn(ww, W \ wx) from the Section 2.2.3
Given unlabeled data è. the transformations gives a set of locally orthogonalized vector { i= for comparison with the locally orthogonal prototypes { ]=m, Dissimilarity Network then com- putes a distribution over classes for point d using Softmax over inner product:
(xt,
ply = k()) = Σn expl (Zk; )x))
(5)
Learning is done using the cross-entropy loss with the label of the instance.
3 RELATED WORKS
There are many works on few-shot learning, which was started on the assumption that currently, learned tasks can help in making prediction a new task (Fei-Fei et al., 2006). It soon gained interest from many researchers, which introduced many interesting techniques that contribute to huge strides of progress in few-shot learning. We will quickly review some of the recently proposed methods and delve deeper into metric learning-based methods that are more related to our work.
Transfer cearning-based methods follows the standard transfer learning procedure (network pre- training& & fine-tuning). Gidaris & Komodakis (2018) 8); Qi et al. (2018); Chen et al. (2019) propose to directly predicting weights of the classifiers on a novel task.
Initialization based methods address few-shot learning by finding way to better initialize model. Ravi & Larochelle (2016) uses LSTM as a meta-optimizer to rapidly adapt neural network on the novel task, whereas Munkhdalai & Yu (017) uses external memory update weights. Another line of works is concerned about finding a good Initialization, as such that finetuning can be done using fewer steps (Finn al. 2017 Nichol & Schulman 2018 Rusu al. 2018
Metric and similarity learning-based methods assumes that representation produced by some model on task share some similarities with those that are produced by another task. Essentially, the goal is to learn a comparison model that can distinguish different classes on the novel task by measuring its distance or similarity to some representation produced by the support set.
Our proposed method is similar to the prototypical networks (Snell et al. 2017) and subsequently Mensink al. (2013) -in its use of mean representation of class (or prototypes). The similarity stops there, as the prototypical networks directly perform classification by comparing the distance of the new input to each prototype. They assume that the embedding function that produces the prototypes can sufficiently capture useful and general enough representation that is transferable to the novel set; only computes global task embedding. As shown our results, their assumption breaks down when there changes in class granularity the label is of fine granularity. In contrast, our method does not directly classify on the prototypes; instead, it transforms the prototypes, producing task-aware embeddings that are locally orthogonal to the shared components belonging to different classes. Thus, classification is performed by computing a Softmax over dot product between the new point and the task- prototypes.
The Dissimilarity Network uses context embedding similar to the full context embedding (FCE) ex- tension of the matching network (Vinyals et al. 2016). However, there are some glaring differences
5

---

Under review as a conference paper at ICLR 2020
in how they operate. The matching network carries the entire support for the prediction. It predicts the label of an unknown point by computing the linear combination of the label of its support set; the support set grows, the memory increases linearly with it. Their full context embedding ditions the prediction on the entire support set; computing the task-aware embedding is quadratic in the number of elements of the support set. Moreover, they do not construct explicit reference for the classes to condition into, it is less clear how reasonable separation of point belonging to different classes can be maximized. As pointed out by Snell et al. (2017). the FCE extension of the matching network does not make that much difference.
Our model only maintains set of prototypes and classifies a new point based on how orthogonal its representation to the prototypes. Since we condition the prediction only on the prototypes, it will not grow with the size of the support set. Moreover, computing the task-aware embedding is only quadratic in the number of prototypes (i.e., labels)
opposed to the number of support set. We also explicitly computes the representations to be dissimilar; lifting the reliance on learning sufficiently separable inter-class representations only to the loss function.
Our similarity function can be set differently. One possible extension is to use learned similarity or distance function similar to RelationNet (Sung et al. 2018), which we leave for future work.
4 EXPERIMENTS
Datasets & scenarios. We evaluate all models on the standard dataset that is widely used in few- shot learning: omniglot, minilmageNer, and CUB. Apart from that, we also evaluate all models on CIFAR dataset with the two splits: hard and normal split (which we will elaborate later).
Evaluation. Our evaluation follows Chen et al. (2019), that is by sampling N-class to form N-way classification (with N=5 unless otherwise stated). For k-shot task, we pick &o labeled instances for each class to be the support set and 16 instances for query set. All results are averaged over 600 experiments which follow the above settings. We evaluate all models on 1-shot and 5-shot setting, which is the most common setting adopted few-shot learning.
Implementation details. All methods are trained using Adam optimizer Kingma & Ba (014) with the initial learning rate of 10-3, which we cut half every 2000 episodes. We apply the following standard data augmentation
all datasets (except CIFAR); random crop, right-left flip, and color jittering. Following Snell et al. (2017). ). we use a four-layer convolution backbone (Conv-4) with an input size of 84x84 as feature extractor for all methods. We use the open-source implementation of Chen et al. (2019) for other methods that we reported. We pick the best performing model based on the validation for meta-learning methods, whereas for baseline and baseline++ Chen et al. (2019), we follow the recommended settings prescribed in their paper. We trained our model on 800 epochs. Our best performing model on all datasets, based on validation set, uses identity function for the value attention embedding by except on the CUB dataset, which uses BLSTM.
4.1 STANDARD BENCHMARKS
Omniglot (Lake et al. 2011 dataset consist of 1623 handwritten digits from 50 different alphabets.
There are 20 examples per character which is drawn by different people. We follow VVnyalls et al. 2016) procedure for evaluation.
mini-ImageNet a 100 classes subset of ImageNet (Deng et al., 2009) dataset (ILSVRC-12 dataset), which was first proposed by VViyyals et al. (2016). It consists of 600 images per class. Recent works follow the setting proposed by Ravi & Larochelle (2016), which consist of randomly selected 64 base, 16 validation, and 20 novel classes.
CUB or Caltech-UCSD Birds 200-2011 I dataset (Wah et al. 201 is a fine-grained classification dataset which consist of 200 classes (bird species) and 11, 788 images in total. We follow Hilliard et al. (2018) setting which is composed of 100 base, 50 validation, and 50 novel classes.
CIFAR-FS dataset is derived from CIFAR dataset (Krizhevsky et al. 2009) consist of 60,000 32x32 color images with 100 classes (belonging to 20 superclasses), with 600 images for each class. Our split consists of randomly sampled 40 base, 15 validation, and 45 novel classes (detail appendix).

---

Under review as a conference paper at ICLR 2020
4.2 HARDER BENCHMARKS
General setting, Our approach requires the dataset that we derived from to have at least two differ- levels of class granularity. For example, CIFAR dataset which has levels of labels granular- ity. ImageNet labels also form a hierarchy which - which-through this method - can be derived into several hard few-shot classification datasets. the case where different labels of granularity are absent, one may be able to construct new labels by exploring the natural hierarchy which may present.
Method. Given a J-labeled dataset IC = {( Df y/ (ffine).... .....,(e (æ Vfrarse, sy where the labels comes from a two-level hierarchy: coarse-grained label yfuurse E Kecourse and fine-grate fine-grained label y/ (superclass) E Kcaarse
fine E K fine, fine denote subset of labels K. Krine that belongs to coarse-grained label
For all coarse-grained label yf E IKcourse, select some y gf'ee E Kiinee that is the subclasses of yfoerse (i.e., (oooess) producing a set of fine-grained labels from all superelasses fine Con-
base"
struct the base set: B = {( (f yfoorse, y/ iee E D2 y fine E //ine) J. The novel set be built by taking the rest of unused data and pair them with their fine-grained labels:
N= = {< yr ffine) (2,, y. y ( E D, "n' & [x,e ). Validation set is constructed the same way as novel set - we leave out the detail of its construction for simplicity.
This approach (illustrated Figure 2 is advantageous as on each task, the labels can vary from being fine-grained to moarse-grained depending on the random selection. As such, the methods that rely on the awareness of the overall novel tasks will likely to fail as builds a general embedding that works all but not optimized for the current task. On the other hand, dynamic method that conditions the prediction on the support set of the given task will likely to perform better.
Supercios
Dog
S aameoe
Buskdng
Parml
Peraien
Poodle
Sub-
dass
Rogoot
Boxer
Spanow
Beagle
Eagle
Sprins
Figure 2: Illustration on construction of harder dataset. The base dataset uses the superclass taxon- omy (e.g., cat, dog, and bird) instead its subclass (e.g., siamese, boxer, and owl).
CIFAR-Hard is derived from CIF,R-100, using the aforementioned method. We derive our harder benchmark from CIFAR- 100 because has two different level of labels granularity. The size of the dataset is also not too big, making it suitable for use as a benchmark. 'he following is the detail for each split (more detail the appendix). 20 coarse-grained base classes from 40 fine-grained classes (derived from the entire 20 superclasses, 2 classes each). 15 validation classes (derived from 5 superclass, classes each). 45 novel classes (derived from 15 superclass, 5 classes each).
4.3 RESULTS & DISCUSSIONS
Table [ how our method fares against others. Our method performs the best on line-grained classification task such as CUB and improved on wide margin on its 1-shot classification task. As we have suspected, there is an increasing need to focus on the difference between classes when the classification task becomes increasingly fine-grained. Despite also being trained on fine-grained classification tasks the CUB dataset, our inductive bias still seems to be helpful in classifying similar-looking classes as further separate class representation by explicitly removing latent fea- tures shared among those classes. At glance, it appears if the 1-shot performance improvement our method is significantly higher than its improvement in 5 5-shot. However, this due to the fact that the Baselinee++ uses retraining, which relies heavily on the availability of labeled data the novel
7

---

We have proposed Dissimilarity Network for few-shot learning based on the idea of focusing on differences in the class representation. Our approach directly addresses the failure modes of some few-shot classifiers that do not explicitly take into account the classification task hand, yielding non-satisfactory results on some task such as fine-grained novel classification with coarse-grained base classification task. To demonstrate the necessities of building task-aware embedding for such task, we came up with a challenging dataset, Hard, which we have shown to be harder than the CIFAR-FS. Dissimilarity Network introduced an architectural inductive bias which removes the shared components among classes in the prototypes by orthogonalizing them (i.e., removing their projected components) to their leave-self-out weighted local average. Our method performs Under review as a conference paper at ICLR 2020
Table I: 5-shot and 1-shot classification accuracy on all datasets. Four-layer convolution backbone (Conv-4) with an input size of 84x84 is used as feature extractor for all methods for a fair compar- ison. No finetuning is performed on CIFAR-H and CIFAR-FS, all the hyperparameters are the one used in the mini-ImageNet dataset. All accuracy results are averaged over 600 test episodes and are reported with 95% confidence intervals. *Results reported by Chen et al. (2019).
5-shot classification accuracy (%)
Method
CIFAR-H1
CUB
CHFAR-FS
mini-ImageNett
Omniglot
Average
Dissinilarity (Ours)
68.45 0.74
81.23 + 0.63
+..7
65.40 + 0.61
95.27 + 0.100
77.00
0.36
MatchingNer (Vinyals el al. 2016
63.34
0.78
72.56 1 0.70
61.14
0.77
63.48 1 0.06**
98.37
U.11
73:24
0.60
ProtoNet (Snell ai. 2017
64.30 1 0.81
70.77 0.669
69.96
0.77
64.24 1: 0.72*
99.15
1: 0.12
73.68 1 0.62
MAML Finn ot 2017
62.87 =
72.09
0.76*
63.98
0.81
62.71 + 0.711
99.53 0.08*
72.61 = 0.63
Relation [Sung
2018
+
76. 11 + 0.68*
68.87
0.76
66.00 + 0.69*
99.30
0.
74.81
+ 0.61
Baselinet+ Chen et 2019
0.77
79.34
0.80
66.43 0.63*
98.38
72.45
I-shot classification accuracy (%)
Method
CIFAR-H
CUB
CIEAR-ES
mini-ImageNet
Omniglot
Average
Dissimilarity (Ours)
51.02 1 0.89
65.82 1 0.94
54.66 0.82
49.34 1. 0.78
97.30
0.25
63.75 1 0.74
MatchingNer Ner (Vinvals et
2016
50.42 0.92
61.16 00.80
53.92 + 0.92
48.14 + 0.18*
9r.18
0.30
62.28 + 0.78
ProtoNet Snell mi. 2017
47.16 0.900
51.31 0.91
50.08 1 0.88
44.42 1 0.84
98.01
1. 0.300
58.20 1 0.77
MAML Fina el al. 2017
48.64 4:4::00000 0.93
55.92 + 0.95*
51.78 + 0.94
46.17 +: 0.82*
98.57 1: 0.19
60.28 +: 0.77
Relation [Sune
2018
50.78 #09 0.95
62.45 + 0.98*
51.21 0.93
19.31
+ 0.85*
97.22
0.333
62 + 0.81
Raseline++ 12n 2019
30.30
00.53
43.38 0.73
48.24 0.775*
95.41 0.39
57.37 0.68
set. If we take a look at the rest of the methods which are based on meta-learning (i.e., optimization- based and metric learning), they all suffer equally on both I-shot and 5-shot; especially on 1-shot as it is of higher variance. The difference between our method and the other methods averaged is consistent on both shots, which around 7%.
Our method also significantly surpassed competing methods on the harder benchmark, CIFAR-H, Its improvement is slightly higher on 5-shot compared to the 2nd best performing model. It expected as the higher the number of support set, the more likely our method finds the most repre- sentative prototypes; as the variance of the samples for the mean decreases. Due to the nature of how the dataset is constructed, to perform well on this dataset, methods should be able to dynamically adapt its classification function based on the level of granularity presented. As our method explicitly induce dissimilar prototypes, it's able to fare significantly better compared to others. Overall, there is an average drop of 3.7% in terms of method's performance between CIFAR- H and CIFAR-FS con- firming CIFAR-H be in fact, harder. On CIFAR-FS, the normal variant of the CIFAR dataset, our method performed slightly better on 5-shot classification. On mini-Image our method performed comparably with the rest, while being slightly worse the Omniglot dataset. This is expected, as adding more inductive bias may only hurt its asymptotic performance.
To summarize, our method performed the best in its intended scenario when the granularity of the labels are fine enough (or changing), as such that the model has to be able to dynamically adapt to it. In the case when the granularity of the labels is fixed, or when the labels are quite coarse, our method will perform comparably to the others. When accuracy is already high, our method might fail to reach optimum asymptotic performance due to the additional constraint that we impose.
5 CONCLUSION
comparably to the state-of-the-an methods standard benchmarks such Omniglot and mini- ImageNet, and substantially outperform other methods on CUB dataset and on the newly constructed CIFAR-Hal dataset.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Wei-Yu Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Wang, and Jia-Bin Huang. A closer look at few-shot classification. In International Conference on Learning Representations, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Li Fei-Fei, Rob Fergus, and Pietro Perona. One-shot learning of object categories. IEEE transactions on pattern analysis and machine intelligence, 28(4):594-611, 2006.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1126-1135. JMLR. org, 2017.
Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4367- 4375, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Nathan Hilliard, Lawrence Phillips, Scott Howland, Artöm Yankov, Courtney D Corley, and Nathan O Hodas. Few-shot learning with metric-agnostic conditional embeddings. arXiv preprint arXiv:1802.04336 2018.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735-1780, 1997.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6995 2014.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Brenden Lake, Ruslan Salakhutdinov, Jason Gross, and Joshua Tenenbaum. One shot learning of simple visual concepts. Proceedings of the annual meeting of the cognitive science society, volume 33, 2011.
Thomas Mensink, Jakob Verbeek, Florent Perronnin, and Gabricla Csurka. Distance-based image classification: Generalizing to new classes near-zero cost. IEEE transactions on pattern anal- ysis and machine intelligence, 35(11 25(11):2624-266 2637, 2013.
John Stuart Mill. A system of logic, ratiocinative and inductive: Being connected view of the principles evidence, and the methods of scientific investigation, vol. 1875.
Tsendsuren Munkhdalai and Hong Yu. Meta networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pp. 2554 JMLR. org, 2017.
Alex Nichol and John Schulman. Reptile: a scalable metalearning algorithm. arXiv preprint arXiv:1803.02999 2, 2018.
Hang Qi, Matthew Brown, and David G Lowe. Low-shot learning with imprinted weights. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5822- 5830, 2018.
Sachin Ravi and Hugo Larochelle. Optimization model for few-shot learning. 2016.

---

Under review as a conference paper at ICLR 2020
Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osin- dero, and Raia Hadsell. Meta-learning with latent embedding optimization. arXiv preprint arXiv:1807.05960, 2018.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schritwieserr, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484, 2016.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, pp. 4077-4087, 2017.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE Con- ference Computer Vision and Pattern Recognition, pp. 1199-1208, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in neural information processing systems, pp. (630-3638, 2016.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
A APPENDIX
A.1 SPLITS FOR CIFAR
Base:
forest, house, television, wolf, cloud, sweet pepper, dinosaur, tank, caterpillar, cup, sun- flower, whale, can, bottle, road, crocodile, woman, bear, otter, willow snail, aquarium girl, trout, bowl, worm, pear, streetcar, castle, flatfish, lobster, turtle, poppy, orchid, man, seal, lamp, lawn_mower, beetle, clock
Validation: oak_tree, kangaroo, mushroom, porcupine, squirrel, lizard, train, spider, keyboard, maple.tree, bicycle, orange, lion, rabbit, motorcycle
Novel: fox, boy, skyscraper, bridge. mouse, shrew, plain, possum, tiger, tulip. wardrobe, sea, couch, mountain, Icopard, camel, shark, plate, dolphin, table, bee, pickup_truck, palm_tree, beaver, baby, bus, butterfly, ray, apple, cattle, crab, pine tree, raccoon, tractor, chair, rose, telephone, chimpanzee, snake, bed, hamster, skunk, cockroach, rocket, elephant
A.2 SPLITS FOR CIFAR-HARD
aquatic mammals, fish, flowers, food containers, fruit and vegetables, house- hold electricaa devices, household furniture, insects, large cargeecarnivorss large_man- man- uadd_ooutoorraco thoings,, large_naid natural outdoor sandooorscceess.. scenes, medium_mammen mammals, vehicles.2
Base:
aargeoomnivores herbivore, people, reptiles, small malmmls,,, trees, vehicles.1,
Validation: rabbit, hamster, bed, house, kangaroo, lamp, skyscraper, squirrel, castle, table, chim- panzee, telephone, television, wardrobe, elephant
Novel:
baby, beaver, beetle, bicycle, bottle, bus, butterfly, can, caterpillar, crocodile, cup, dol-
phin, flatfish, forest, girl, lion, lobster, man, mountain, oak_tree, orange, orchid, otter, pear, bockup-truck pine_tree porcupine, possum, ray, rocket, rose, sea, shark, skunk, snail, snake, street- car, sweet.pepper, tank, tiger, tulip, turtle, willow.tree, wolf, worm
10


Reviews:
[{"review_id": "ryx1J6kTFB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper the authors propose a metric based model for few-shot learning. The goal of the proposed technique is to incorporate a prior that highlight better the dissimilarity between closely related class prototype. Thus, the proposed paper is related to prototypical neural network (use of prototype to represent a class) but differ from it by using inner product scoring  as a similarity measure instead of the use of euclidean distance. There is also close similarity between the proposed method and matching network \n\n\noverall, the paper does not  highlight the novelty of their proposed method especially prototypical network and matching network. Thus, the related work session is so general and does not tackle the close models in details. The experiments do not provide convincing evidence of the correctness of the proposed approach.\n\nSeveral parts are unclear/incomprehensible:\n(1) The Introduction is confusing and does not demonstrate the problem that the paper is trying to solve. Specifically, the described intuition (Mill’s method of difference) is not convincing \n(2) the first sentence of  the section “Our work.”  (page 1) is long and unclear …  “In this paper, we propose a model that focuses on the differences in the support set of closely related classes in assigning the class label to a new instance in the novel task.”\n(3) the use of the two level of embedding is confusing and not clear. Figure 1 is also confusion and not clear.\n\nthe correctness of the proposed approach is not proved by the conducted experiment and does not provide convincing and fair comparison with SoA techniques:\n(1) The experiments do not provide the details of the used architecture compared to your baseline.  (how many layers are used in both embedding systems)\n(2) In Table 1 you are using the results reported by Chen et al. (2019) did you use his framework (Resnet or 4 layers CNN)\n\n**Minor comments** \nThe definition of the embedding function f = (f_g o f_f)  (in line 1 page 5) is not consistent with the domain of each function  f_g is defined on R^H x R^H.\n\nK is not defined (last line page 3)\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1l8TWFRFS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The stated contributions of the paper are: (1) a method for performing few-shot learning and (2) an approach for building harder few-shot learning datasets from existing datasets. The authors describe a model for creating a task-aware embedding for different novel sets (for different image classification settings) using a nonlinear self-attention-like mechanism applied to the centroid of the global embeddings for each class. The resulting embeddings are used per class with an additional attention layer applied on the embeddings from the other classes to identify closely-related classes and consider the part of the embedding orthogonal to the attention-weighted-average of these closely-related classes. They compare the accuracy of their model vs others in the 1-shot and 5-shot setting on various datasets, including a derived dataset from CIFAR which they call Hierarchical-CIFAR.\n\nOverall, while we like the concepts/ideas and the problem is definitely important, we were not enthusiastic about the paper. First, we found the write up to be cryptic, involving very long unclear statements. It read as if the authors were writing for themselves and not for ICLR general audience. Beyond the writing style, we found the paper to have:\n\n* Inadequate description of the model, including mathematical inaccuracies.\n* Inadequate description of Hierarchical-CIFAR, motivation, and evaluation.\n\nDescription of model:\n------------------------------\nThe manuscripts describe the presented approach as metric learning but make no use of a distance function in the different spaces they map to. Instead, the manuscript defines an inner product over embeddings to compute similarities.\n\nThe manuscript describe their “self-attention operation” as a dynamic set-to-set operation. While the usual definition of self-attention is permutation invariant, the definition presented here is not and thus cannot be accurately described as a mapping between sets. Specifically, in the usual presentation of self-attention the only sharing of information between different set elements is during the outer product of the key and query vectors. The use of a BLSTM between “neighboring elements” of the set of prototype vectors violates this assumption and induces a lack of permutation invariance. This makes the method sensitive to permutations of classes, which does not make sense for predicting unordered classes.\n\nIn sections 2.2.3 and 2.2.4 the material is presented twice but slightly differently. For example, the definition of $b_k$ inline before equation 3 differs from equation 6 later in the text.\n\nEquation 7 is incorrect and should not exclude the current class from the denominator.\n\nThe description of how to classify new points after equation 6 is poorly explained. The description of what happens when $h_V$ is a “BLTSM” [sic (should be BLSTM)] is noninformative.\n\nThe manuscript describes two dimension sizes $H$ and $M$ but the definition of $attn$ requires that $H = M$.\n\nDescription of new dataset and evaluations:\n------------------------------------------------------------\nOne of the stated contributions of the manuscript is a methodology to build harder few-shot learning datasets. Section 4.2 is the only place in the text that appears to address this point, but is unclear where either new finer-grained or coarser-grained labels are coming from (new manual annotation or otherwise). The manuscript “leave[s] out the detail of its construction for simplicity”, but it is unclear what is being done here in the first place.\n\nThe manuscript does not detail tuning competing methods on the new dataset and so it is unclear whether it is a fair comparison.\n\nThe manuscript presents evaluations without any discussion of differences in performance between datasets or the 1-shot/5-shot settings. For example, their method is significantly better on CUB on 1-shot but not so much on 5-shot, on the other hand it is not significantly better on 1-shot for H-CIFAR and CIFAR-HS but then becomes better than the rest with 5-shot.\n\nAdditional comments/corrections\n---------------------------------------------\nThere were numerous typos and grammatical errors that were present in the manuscript that did not directly impact this evaluation but should be fixed in the future.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1esrDQ0YS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a new neural network model, called as Dissimilarity Network, to improve the few-shot learning accuracy.\nOverall the idea is well motivated that by emphasizing the difference among classes, the model can achieve more accurate predictions for classes where only limited data points are available for training.\nHowever, the paper is not quite well written.\nFirstly, much of the work is built upon previous work including attention mechanisms, episodic training for few-shot learning. Such components are the core of this work because the attention mechanisms implement the class-awareness, and the episodic training facilitates the LSTM structure. Yet these are not well explained and not much context is provided, thus making the paper hard to follow.\nSecondly, some terms are fairly overloaded, or not clearly defined. For example, the “prior” as mentioned in both the abstract and the introduction doesn’t refer to the commonly interpreted term as in the Bayesian settings, but rather as a hand-waiving term to indicate the model design. Also, the terms, “score”, “metric”, “dissimilarity” are mentioned in the paper but the paper is not really learning the metric, to my understanding. Thus the details of the paper is quite hard to grasp. \nLastly, the idea of designing the global embedding and the task aware embedding is interesting but shouldn’t really be restricted to few-shot learning. It would be interesting to test the idea on general classification tasks, for example in a simple cross validation settings.\nThus I think the paper would be stronger if the above are addressed and it’s not ready for publishing yet in its current form.\n\nBelow are some more detailed comments:\n1)\tIn the abstract, the “newly introduced dataset H-CIFAR” is not precise to me; my understanding is that the paper proposes such an experiment design for testing how well a classifier can predict the labels with hierarchy. The current writing refers to that the authors comprises a completely new dataset with new labels.\n2)\tIn the last sentence of the second paragraph in Introduction, the question is asked “what prior” should be reasonable. Since the authors didn’t really add any priors in a Bayesian settings but rather designed an architecture, I suggest to reword something like “how to explicitly encode hierarchies into the model structure”.\n3)\tIn Section 2.1, some more description for “episodic training” would be nice: why should it be used? How is it used and why it makes sense in the few-shot learning context?\n4)\tIn Section 2.2, it would be nice to add the mathematical definition of “prototype”.\n5)\tIn Section 2.2.1, it would be nice to define “H”.\n6)\tIn Section 2.2.2, is M required to be fixed given it’s episodic training? Also it would be nice to add more details about the attention mechanism.\n7)\tIn the result section, it would be nice to discuss when the proposed method is doing better than other methods, for example RelationNet, as well as when it’s worse since different datasets show different results.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Main content:

[Blind review #3] The authors propose a metric based model for few-shot learning. The goal of the proposed technique is to incorporate a prior that highlight better the dissimilarity between closely related class prototype. Thus, the proposed paper is related to prototypical neural network (use of prototype to represent a class) but differ from it by using inner product scoring  as a similarity measure instead of the use of euclidean distance. There is also close similarity between the proposed method and matching network.

[Blind review #2] The stated contributions of the paper are: (1) a method for performing few-shot learning and (2) an approach for building harder few-shot learning datasets from existing datasets. The authors describe a model for creating a task-aware embedding for different novel sets (for different image classification settings) using a nonlinear self-attention-like mechanism applied to the centroid of the global embeddings for each class. The resulting embeddings are used per class with an additional attention layer applied on the embeddings from the other classes to identify closely-related classes and consider the part of the embedding orthogonal to the attention-weighted-average of these closely-related classes. They compare the accuracy of their model vs others in the 1-shot and 5-shot setting on various datasets, including a derived dataset from CIFAR which they call Hierarchical-CIFAR.

--

Discussion:

All reviews agree on a weak reject.

--

Recommendation and justification:

While the ideas appear to be on a good track, the paper itself is poorly written - as one review put it, more like notes to themselves, rather than a well-written document to the ICLR audience.
Decision: Reject