Out-of-distribution Detection in Few-shot Classification
Authors: Kuan-Chieh Wang
Paul Vicol
Eleni Triantafillou
Chia-Cheng Liu
Richard Zemel
Venue:  (2020)
Keywords: few-shot classification
out-of-distribution detection
uncertainty estimate

Abstract: In many real-world settings, a learning model must perform few-shot classification: learn to classify examples from unseen classes using only a few labeled examples per class.
Additionally, to be safely deployed, it should have the ability to detect out-of-distribution inputs: examples that do not belong to any of the classes.
While both few-shot classification and out-of-distribution detection are popular topics,
their combination has not been studied. In this work, we propose tasks for out-of-distribution detection in the few-shot setting and establish benchmark datasets, based on four popular few-shot classification datasets.  Then, we propose two new methods for this task and investigate their performance.
In sum, we establish baseline out-of-distribution detection results using standard metrics on new benchmark datasets and show improved results with our proposed methods.

Under review as a conference paper at ICLR 2020
OUT-OF-DISTRIBUTION DETECTION IN FEW-SHOT CLASSIFICATION
Anonymous authors Paper under double-bling review
ABSTRACT
In many real-world settings, a learning model must perform few-shot classification: learn to classify examples from unseen classes using only a few labeled examples per class. Additionally, to be safely deployed, it should have the ability to detect out-of-distribution inputs: examples that do not belong to any of the classes. While both few-shot classification and nut-of-distribution detection are popular topics, their combination has not been studied. In this work, we propose tasks for out- of-distribution detection in the few-shot setting and establish benchmark datasets, based on four popular few-shot classification datasets. Then, we propose two new methods for this task and investigate their performance.
In sum, we establish baseline out-of -i-or--diitribution detection results using standard metries on new benchmark datasets and show improved results with our proposed methods.
INTRODUCTION
Few-shot learning, at a high-level, is the paradigm of learning where model is asked to learn about new concepts from only a few examples (Fei-Fei et al., 2006; Lake et al., 2015). In the case of few- shot classification, a model must classify examples from novel classes, based on only a few labelled examples from each class. The model has to quickly learn (or adapt) classifier given this very limited amount of learning signal. This paradigm of learning is attractive for the fundamental reason that it resembles how an intelligent system in the real-world has to behave. Unlike the traditional supervised setting, in most real-world settings we would not have access to millions of labelled examples, but would benefit if few-shot classifier could be deployed, for example, recognize the facial gestures of a new user, in order to improve human-computer interaction for individuals with motor disabilities (Wang et al., 2019).
For an intelligent system to be deployed in the real-world, not only does it have to do well on the designated task, but perhaps more importantly it should defer its actions when faced with unforeseen situations. In particular, when an input is invalid, or does not belong to any of the target classes, the system should identify the input as -ut-of-distributtin. Successfully detecting out-of-distribution examples is crucial in a safety critical environment. In the supervised setting, out-of-distribution detection has been studied from many different angles (Hendrycks Gimpel, 2016; Nalisnick et al., 2018), but this task has not been investigated in the few-shot setting.
Worryingly, the current state-of-the-arr learning systems, deep neural networks, are known to be unreasonably confident about inputs unrecognizable to humans (Nguyen et al., 2015), and their predictions can be manipulated with imperceptible changes in input space (Szegedy et al., 2013). In general, the behavior of deep nets is not well specified when the test queries are out-of-distribution.
standard practice when studying duttof-distribution detection is to evaluate the detection per- formance when examples from other datasets mixed into the test set (Hendrycks & Gimpel, 2016). Here we refer to this type of out-of-distrrbutie input as out-of-dataset (OOS)' inputs. In the few-shot setting, within each episode, what is in-distribution is specified based on a few labeled examples, known as the support set. Hence, there naturally exists another type of out-of-distribution input, the inputs that belong to the same dataset but come from classes not represented by the support
We denote out-of-dii cuttof-diitrbuttion and out-of-dataset with the acronyms OOD and OOS, respectively.

---

Under review as a conference paper at ICLR 2020
set. We refer these out-of-episode (OOE) examples. These different types of uut-of-distribution
examples are illustrated in Figure
Being able to detect out-of-distribution amples is critical for improvements many other important applications, including semi- supervised learning and continual learning. In the case of semi-supervised learning methods, was shown that if the unlabelled set is pol- luted with only 25% -utto--distributionn exam- ples, then using the unlabeled data actually has a negative effect on performance (Oliver al., 2018). In the natural continual learning frame- work, where a model has learn new concepts while not forgetting old ones, detecting when examples do not belong to any previously-leamed class a fundamental problem.
Support Examples
Queries Out-af-Frissdee
Iistisribttion
Out-of-
C
' g
Unrom/ Nole
7 4p
CIFARTO DD-DW
gamn 5
D2-
NOMMNNSST
Figure 1: Examples of the support set, in- distribution, OOE and OOS inputs in one episode.
Hence, in this work, we focus on this core problem of out-of-distribttion detection in the few-shot setting.
Contributions
We develop benchmark datasets for out-of-distribution detection, both OOE and OOS, based on four standard benchmark datasets for few-shot classification: Omniglot, CIFARIO0, minImageNet, and rieredImageNet.
We establish baseline results for both the OOS and OOE tasks for two popular few-shot Pessifiers-PPoootyyae
Networks and MAMI- these datasets.
We show that simple distance metric-based approach dramatically improves the perfor- mance on both tasks.
Finally, we propose learned scoring function which further improves both tasks the most challenging new benchmark datasets.
OVERVIEW OF FEW-SHOT CLASSIFICATION
In few-shot classification, model tasked to classify unlabeled 'queries' Q ] one of Nc classes from a set Clest This setup differs from standard 'supervised' classification in that only few labeled examples are avail- able from each class e E Crest referred to as that class" support set S. {(x,, y.)) Fol- lowing the standard terminology, we refer to the number of classes Ne the 'way" of the task and the number of support examples per class Ns the "shot" of the task. We also use the term episode to refer to a classification task defined by support and query
Ng [-) into
Episode 1
Traning Set
Test Quarles Q
S
8
Mata-Test Time Training Set
While we assume that little data is available for each such test classification episode, the model has access to a (possibly large) training set be- forehand that contains examples from different set of classes Ctrein disjoint from Clest The key is therefore to figure out how to exploit this seemingly -irreeeetr data at training time in or- der to obtain a model that is capable of learning episode at test time using only its small support set a way that performs well on clas- sifying its corresponding query
Figure 2:
Standard episodic set-up. A test episode in standard few-shot classification consists of few training (or support) examples from novel classes, and test/query examples from those classes. Many systems not only evaluate on episodes but also train episodically, i.e., looping episodes as opposed to over typical mini-batches.
2

---

Under review as a conference paper at ICLR 2020
Most recent approaches for this adopt the design choice of creating episodes from the training of classes too, and expressing the training loss for each episode in terms of performing well on its query examples, after having 'learned' on its small support set. The intuition is to practice learning episodes that have the same structure as those that will be encountered at test time. At training time, these episodes are created by randomly sampling Nc classes (from the training set of classes), Ns examples of each of those classes to form the support set, and some different examples of each of them form the query set. We refer to this type of training as 'episodic training' (see Figure 2). Different methods are distinguished by the manner in which learning is performed on the support set. We now give an overview of two popular approaches few-shot learning: Prototypical Networks (Snell et al., 2017) and MAML (Finn et al., 2017).
Prototypical Networks. Prototypical Networks (Snell et al., 2017) are simple but effective instance of the above framework where the 'learning procedure' that the model undergoes based on the support set has closed form. More concretely, it consists of a parameterized embedding function fø (typically a deep net) and a distance metric d(.--) on the embedding space. Given support sets of the chosen classes, the Prototypical Network computes the prototype M. of each class in the embedding space:
Σ fa(x:).
He
S x,ES.
Then query xin is classified based on its distance to the class prototypes:
exp(-d(Soxx")) Mc))
Polv clx²)) E. cxp(-d(fo(x")) me))
During training episodes, the parameters of fe are updated according to the Prototypical Network loss:
(oN(o:(5,1) =- Σ
logpoly c(x"").
(2)
(X*,
Algorithm 2 (in Appendix B) is description of standard episodic training of Prototypical Network.
Meta-learning. MAML (Finn et al., 2017) is another popular model of this episodic family that is parameterized by a representation function and linear classification layer on top, where jointly we denote the weights tb. Training unfolds over sequence of training episodes, episode, the weights z are adapted via a few steps of gradient descent (denoted as SGD (L)) the support set, resulting in
usual. In each
to minimize the cross entropy loss over the Ne-way classification updated weights & which are then used to classify the queries in the given episode. Over a number of episodes, the aggregated loss then used to update s/ again with gradient descent.
Φi = SGD & (CrossEntropyLLooss (S*)))
M
L (St, Q* (
Σ CrossEntropyLooss(o; ( Q(1)
The model is thus encouraged to learn a global initialization v of weights such that a few steps of adaptation on new episode's support set suffice for performing well on its query set.
3 OVERVIEW OF OUT-OF-DISTRIBUTII DETECTION
The term than that of the training data. Hendrycks & Gimpel (2016) used different benchmark datasets as sources of out-of-distribution examples. For example, when a network is trained on MNIST, the out-of- -iittr--isttiittionn examples can come from Omniglot, black-and- common evaluation setup to treat data from the same dataset- but from different classes than those under consideration as out-o--distribution. These have been referred to as same manifold (Liang al., 2017), unobserved class (Louizos & Welling, 2017) nut-of-distribution examples.
refers to input data that is drawn from a different generative process
CIFAR10, etc. Another
3

---

Under review as a conference paper at ICLR 2020
Problem Set-up. Out-of-distributioo detection is a binary detection problem. At test-time, the model is required to produce score, s8(x) E IR, where x is the query, and 8 is the set of learnable parameters for the detection task. We desire so (x"" examples are higher than that of out-of-distribution examples. Typically for quantitative evaluation, threshold-free metries used, e.g., the area under the receiver-operating curve (AUROC) (see Section 5 for details).
sx(x""",, ie, the scores for in-distribution
Approaches. The main approaches distribution detection can be categorized into one of the three families: 1) scores based on the predictive probability of classifier; 2) scores based on fitting a density model to the inputs directly; and 3) scores based fitting density model to representations of pretrained model (e.g., classifier). These are illustrated in Figure 3.
out-of-
Feature Net
Classifier
ply x)
Score Function
1. Predictive probability Recall that classification of the in- in-distribution data is done using Py clx")) where o represents the classifier parameters. Commonly used scores include softmax prediction probability (SPP),
p(x)
p(z)
Figure 3:
Schematic of OOS ap-
s(x"; 6) max.' Pb = e' xx" (Hendrycks & Gimpel, 2016) and negative predictive entropy (NPE), s(x Φ))
proaches.
Ee Poly = c' [x')) poyp = é' d\x")). Note that scores operate on top of pretrained classifiers.
the notation ((:() to emphasize that these
popular extension to use Bayesian classifiers, i.c., Bayesian Neural Networks (BNNs), and improve the scores by looking at the aggregated score based on the model posterior.
2. Input density- Another natural approach detecting out-of-distribution examples fit density model on the data and consider examples with low likelihood under that model to be OOD, However, this approach not as competitive when the input domain is high- dishndimensional images. Nalisnick et al. (2018) showed that deep generative models (c.g.. flow-based models (Kingma & Dhariwal, 2018) or auto-regressive models (Salimans et al., 2017)) can assign higher densities to out-of-distribution examples than in-distribution examples.
3. Representation density While fitting a density model on the inputs directly has not proven useful for OOD detection, fitting simple density models on learned classifier representations has. Lee et al. (2018b) fit dixture-of-Gaussee classifier activations of the training set. Intuitively. this approach fits density model in space where much of the variation in the imput has been filtered out, which makes it an easier problem than learning density model in the input space.
(MoG) density with shared diagonal covariance on the
4 OUT-OF-DISTRIBUTUUE
DETECTION IN THE FEW-SHOT SETTING
In this study, we focus on two types of out-of distribbitiitto detection problems, described below. In both cases, denote the of in-distribution and out-of-distribution examples by Q {xi Ji=1" and A = {xr=),_ Jim] where No is the number of examples. Note that we use Nę to denote the number of queries in an episode, and the number of iiissrruttion/ouuo-o-dddsttrubui the same thing depending on context. One could consider different numbers of OOD examples from in-distribution ones, but this omitted for presentation clarity.
No
examples, as they mean
Out-of-Episooe (OOE). OOE examples come from the same dataset, but from classes not in the current episode. In other words, the current episode consists of classes Cepisode we sample OOE examples R as follows:
Co00 +- RANDOMSAMPLE( ctrain
Copisode .Nc)
(3)
+ RANDOMS
.o2)
Here, De denotes the set of all examples of classes set C", is the set difference. This type of out-of-distribution detection is easily motivated. Taking the example where we want to build
4

---

Under review as a conference paper at ICLR 2020
Algorithm Episodic training with OOE inputs. Modified steps are highlighted in blue.
1:
while not converged do
2: Cepisode 6- RANDOMS AMPLECCSAML ,Nc)
randomly select classes D for each class
3:
for in Cspisode do
4:
S. + RANDOMSAMPLE(D)c)) (c). Nss
D select the support set & select the query set
5:
Qc <- RANDOMSSAMPEE(D Se, NQ)
6:
end for
7:
Cooe <- RANDOMSAMPLE( NQ)
Cepisade Nc)
prepare OOE queries
8:
R RANDOMS
9: 0 a (VoLPN (5, +XV LOOE (0: (S)Q.R))
10: end while
customized facial gesture recognizer for user, when the system sees the user's face performing gesture that is not registered (i.e., not in the support set), we would like the system to know that the gesture out-of-distribution, and perform an inappropriate action.
Out-of-Datasee (OOS). OOS examples come from a completely different dataset. For example, if the in-disrribution is Omniglot, then the OOS examples can come from black-and-white CIFAR1O. The motivation for this type of out-of-distribttion example is also straightforward: system should defer its actions when faced with something completely different from what it was trained on.
Generally, we use s()) to denote the scoring function for out-of-distribution detection, which expresses the model's "confidence" that an example is in-distribution. Hence, we desire that s(x"") s (xx")) for any in-distribution query xt and out-of-distribution example x n
4.1 PROPOSED FEW-SHOT OUT-OF-DISTRIRUTE DETECTION METHODS
In what follows, we propose two novel methods: 1) a parameter-free method that measures the distance in the learned embedding of few-shot classifier, and 2) a learned scoring function on top of the embedding of few-shot classifier.
(1). Minimum Distance Confidence Score (-MinDist). illustrate why standard softmax pre- diction probability (SPP) fails in the few-shot setting, consider the classifier learned by Prototypical Network. The original Prototypical Network formulation makes decisions based on a softmax over the negative distances in the embedding space. However, when query embedding far away from all prototypes (as we may expect for OOS examples), converting distances to probabilities can yield arbitrarily confident predictions (for details Appendix E). This makes SPP unsuitable for OOS detection. We propose an alternative confidence score, based on the negative minimum distance from query to any of the prototypes:
s s(x"; Φ) min d(fo(x"), pc)
(5)
Episodic Optimization with OOE Inputs. When training our backbone, we can also add term to our loss to encourage it to accurately detect OOE examples, in addition to accurately performing the episode's 's classification task. Intuitively, adding this term changes the embedding in such way that the optimized confidence score performs well on the OOE task. This new term is the following:
Loo (S R))
Σ logo (s(x) x"EQ
Σ log(1 (s(x"""::)) R
where s(:; Φ) here can be any of the parameter-free scores, and a(.) is the logistic function. Algo- rithm 1 description of episodic training with OOE examples.
(2). Learnable Class BOundary (LCBO) Network. We introduce parametric, class-conditional confidence score that takes query x and a class c, and yields a score indicating whether x belongs to class

---

Under review as a conference paper at ICLR 2020
The LCBO takes input: the support embeddings for particular class, and 2) query embedding. The LCBO outputs real-valued score representing the confidence that the query belongs to the corresponding class.
Sp (xx Sc) +
Aggregation. The LCBO outputs coanitsconiiionall confidence scores (e.g.. the confidence that query belongs to specific class). To obtain a final score for in-distribution OOS for each query, we aggregate the class-conditional scores. We take the maximum confidence of all the classes:
80 xx" max (x Se)
eEC
Intuitively, (x(x", So) computes the distance between a query embedding and a prototype, and the max() aggregation function says that query is an inlier if it belongs to at least one class. By design, this is strictly more powerful than -MinDist since it is parameterized by a new set of weights 6, but could also recover simple distance between x" and μa, i..., -Msn The difficulty of designing good uncertainty estimate based on a trained classifier leads us to believe that adding capacity to the confidence score using learnable parameters be beneficial.
Implementation Details. We parameterize the learned confidence score sa by an MLP with two hidden layers of dimension 100, that takes as input the concatenation [pgi fa (x"")) where μ. is the class prototype and fø fx(")) the query embedding. Note that LCBO always operates on top of the backbone (-). ), so this dependency is omitted for notational simplicity.
However, instead of training the aggregated score, we use the following binary cross-entropy objective on the score before aggregation:
Training the LCBO. We train the LCBO episodically.
LLCBOlÓ,6 8; ( (S,Q.R) = Σ
logo (x (Se)=
E
log(1-o Se)))
(c
x E R, ef ComitaC)))
(9)
For the OOE queries x we assigned them a label drawn from the uniform distribution of the in-distribution classes.
EXPERIMENTS
In this section, we: 1) establish the OOE and OOS detection performance of standard few-shot methods as well as a novel variant, and 2) show that both our proposed methods improve substantially over these baseline approaches.
To enable fair comparisons, for the experiments in this section we use the same network configuration, standard 4-layer ConvNet architecture that is well-established in the few-shot literature (Snell et al., 2017). None of the methods discussed here sacrifice in-distribution classification accuracy.
Evaluation Metrics. We evaluate the OOE and OOS detection performance using the area under the receiver-operating curve (AUROC). This is a simple metric that circumvents the need to set a threshold for the score. The base-rate (i.e., a completely naïve scoring function) for all of our experiments is 50%. A scoring function that can completely separate s(x"") from s(x""t) would achieve an AUROC score of 100%. Following standard practice (Hendrycks Gimpel, 2016; Liang et al., 2017; Lee et al., 201 8a), also report scores for under the precision and recall curve (AUPR), and false positive rate (FPR) (Table 1). All results are evaluated using 1000 test episodes, i.e., episodes that contain classes never seen during training. Please refer to Appendix C for descriptions of the in-distribution and OOS datasets.
5.1 CUT-OF-DIITTRBUTIION DETECTION WITH BASELINE METHODS
We first evaluate the out-of-e and distrii-iitiiitto detection performance of three few-shot classifiers, using the standard SPP confidence score. The results summarized in Table 1. We note that not only are these classifiers similar in their distribution classification accuracy (Chen et al., 2019), but their ability detect out-of out-of-distribution examples also similar.

---

Under review as a conference paper at ICLR 2020
AUROC T
AUPR
FPR90
in-distribution
task
PN
MAML
ABML
PN
MAML
ABML
PN
MAML
ABML
OOE
90.6
88.4
85.5
90.3
89.3
86.3
27.5
38.1
46.1
Omniglot
00S
63.8
89.5
88.8
70.8
90.3
89.1
53.7
35.1
38.3
O0E
60.3
61.6
59.4
63.0
63.6
61.1
85.1
84.0
85.3
CIFARI00
00S
57.4
58.1
65.8
63.8
62.6
68.1
86.7
86.0
79.3
OOE 56.6
56.8
54.6
58.5
58.1
56.0
87.0
86.9
88.4
minilmageNer
00S 50.4
65.8
63.8
61.5
68.2
65.4
0.4
78.5
80.0
00E 59.4
57.3
63.3
61.3
58.6
66.3
85.1
86.5
80.7
tieredlmageNet
00S 66.4
68.3
64.2
72.6
70.3
65.4
79.0
75.7
81.4
Table I:
Baseline results for various few-shot classifiers. All numbers in percentages, evaluated in the 5-way 5-shot setting for all 4 datasets, using the standard Conv4 backbone and SPP confidence score. The reported OOS numbers the means over all the OOS datasets used for the corresponding in-distribution dataset.
AUROC
AUPR t -MinDist
FPR90 + -MinDist
in-distribution
task
SPP
-MinDist
LCBO
SPP
LCBO
SPP
LCBO
OOE
89.5
98.3
96.4
88.6
98.2
92.5
28.3
3.8
7.3
Omniglot
OOS
35.8
100
58.4
45.6
100
58.5
80.4
0.0
42.1
OOE 60.1
68.0
73.3
61.0
67.2
71.5
84.3
73.1
62.8
CIFAR100
00S 55.8
86.1
79.6
58.3
86.2
79.3
87.6
31.9
52.5
00E 56.7
61.9
65.6
56.8
61.1
63.1
86.8
80.2
73.1
minilmageNet
00S 51.8
61.0
74.7
54.0
64.0
75.2
89.2
60.1
61.0
0OE 59.0
62.4
65.0
60.0
61.4
62.8
85.1
79.0
74.4
fieredImageNet
00S 53.6
51.4
70.7
56.2
59.9
73.1
88.5
66.0
70.4
Table 2: LCBO, -MinDsst results for ProtoNet. All numbers in percentages, evaluated in the 5-way 5-shot setting for all 4 datasets, using the standard Conv4 backbone. The reported OOS numbers the means over all the OOS datasets used for the corresponding in-distribution dataset. For detailed OOS results, see Appendix D
Bayesian methods provide alternative that may help in OOD detection, by quantifying uncertainty in predictions. We evaluate a recent method that shows strong calibration results: the Amortized Bayesian Meta-Learning (ABMI)) algorithm (Ravi & Beatson, 2019) which realizes Bayesian MAML following the hicrarchical variational Bayes formulation of Amit & Meir (2018). However, ABML did not significantly improve over MAML, least according to our implementation (since Ravi & Beatson (2019) did not release code, in App. F we discuss details of our best effort to reproduce this method). Next we show that ofitt--distributtin performance can be greatly improved.
5.2 OUT-OF-DISTRIBUTTH DETECTION WITH -MINDIST & LCBO
Few-shot classification be evaluated in many different (way, shof) settings, e-g., 5-way 5-shot, 10-way 1-shot, etc. Due to lack of space, we report only 5-shot 5-way results in this section. Full results for (5, 103-way X { 5h-h settings on CIFAR- 100 are provided in Appendix I.
Table 2 shows the results of SPP, -MinDist, and the learned LCBO score on all four of the datasets. Across the board, on both OOE and OOS tasks, either -MinDist LCBO outperformed the baseline method. Interestingly, it seems to confirm our hypothesis that -MnnDist might not be the most suitable confidence score for all embedding spaces. For more detailed discussion of -MinDist and its connection to a similar method proposed in the supervised setting (Lee et al., 2018b), please see Appendix E. On the largest datasets, i.e., both versions of the ImageNet dataset, LCCOO outperformed -MinDist on both OOE and OOS tasks. This was somewhat surprising, since might expect that parameter- functions like -MinDist can generalize better to OOS datasets that are very different from the in-distribution data. This still true CIFAR 100, but not ImageNet datasets.
One major difference between CIFAR100 and ImageNet was the image size (32 x 32 vs 84 x 84), which resulted in different embedding dimensions (256 1600). This suggests that as we scale up the dimensionality of the embedding space, it becomes increasingly difficult to design suitable parameter-free confidence score. Hence, learnable such LCBO becomes critical.

---

Under review as a conference paper at ICLR 2020
Effect of different backbones. We also investigated the effect of the backbone network, fa. OOE and OOS detection, Recently, Chen et al. (2019) trained larger backbones like ResNet without using episodic training. We include results with these larger backbones in Appendix I.
Effeet on example downstream application.
Ren al. (2018) proposed to study few-shot semi- superviiee learning (FS-SSL) where each episode is augmented with an unlabelled set. To make it more realistic, there are also 'distractors' present.
In previous FS-SSL studies, only OOE examples are considered for both training and testing phases. This somewhat unrealistic, as there can be unforeseen distractors in the test episodes. In Table 3 we show that when evaluated in this more realis- tie setting, the method of Ren et al. (2018) suffers. Here, we do not claim that LCBO improves upon semi-supervised learning methods. Still, especially in the case when dis- tractor inputs are OOS instead of only OOE examples, baseline semi-supervised methods significantly degrade the classification accuracy (see Appendix G for more details on this task).
Model
OOE Uniform Gaussian
Supervised
47.5
47.7
47.4
Soft k-means (Ren et al., 2018)
+1.4
-9.4
-11.4
with LCBO
-0.2
+0.1
+0.0
Classification accuracy (in percentages) of semi- supervised learning results on fieredImageNet. Column headings indicate type of distractor used at test-time. '+', and "-" denote the lack and presence of degradation. 'Supervised" refers to training without unlahelled set.
Table 3:
6 RELATED WORK
As few-shot out-of-distriburs detection is new problem, here we discuss recent attempts to study uncertainty in the few-shot setting, and previous approaches that worked well for ofist--dist detection in the supervised setting.
Bayesian Few-Shot Classifier. A number of papers investigated Bayesian extensions of MAML. Compared with other works (Grant et al., 2018; Finn et al., 2018; Yoon et al., 2018) 1) on extending the MAML framework to the Bayesian setting, ABML maintains uncertainty on the global initialization v). Furthermore, as a way to analyze the uncertainty estimation of ABML, Ravi & Beatson (2019) studied the few-shot out-of-distribution detection of the ABMI, framework. However, they did not report quantitative evaluations we did.
Other contof-DDistribution Approaches. ODIN (Liang et al., 2017) consists of 2 innovations: 1) it performs temperature scaling to calibrate the predicted probability (Guo et al., 2017); and 2) when doing
detection, it adds virtual adversarial perturbations (VAP) to the input. Intuitively, VAP will have larger effect on the in in-iistribution input compared to the out-of- distribution input. Lee et al. (2018b) showed that this approach can be complementary to fitting a Gaussian density the activations of the network. Our preliminary experiments showed that ODIN did not have a big impact in the few-shot setting. Outlier exposure, another recent method (Hendrycks al., 2019), also did not show significant effect. We included these results in Appendix J.
For a while, methods using the predictive probability were the dominant approach in out-of- distribution detection. Nalisnick et al. (2018) pointed out that the community had been using the learned density model incorrectly by directly looking at the p(x) scores, and instead should use measure of typicality (Nalisnick al., 2019). Ren al. (2019) proposed to train a separate "background" model and use the likelihood ratio as the score. Generativv/density models have not been extensively studied in the few-shot setting. We believe that this due to the lack of a good task/quantitative evaluation, and that the tasks we study might facilitate research done on such models. Another topic similar to ours is generalized zero-shot recognition (Mandal et al., 2019). The main difference in our setting is that not only are the OOD examples unseen, but the in-distribution examples/classes at evaluation time are also unseen, and only defined by a support set.
7 CONCLUSION
To the best of our knowledge, this is the first study to investigate both OOS and OOE tasks and report results using commonly-used metrics in the few-shot setting. We showed that existing confidence scores developed in the supervised setting (i.e., setting with a fixed number of classes) are not suitable when used with popular few-shot classifiers. Our proposed confidence scores, -MinDist and LCBO, substantially outperformed the baselines on both tasks across four staple few-shot classification datasets. We hope that our work encourages future studies on quantitative evaluation of duttof-distribution detection and uncertainty in the few-shot setting.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Ron Amit and Ron Meir. Meta-learning by adjusting priors based on extended PAC-Bayes theory. In International Conference on Machine Learning (ICLR), pp. 205-214, 2018.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. arXiv preprint arXiv:1505.05444, 2015.
Wei-Yu Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Wang, and Jia-Bin Huang. A closer look at few-shot classification. In International Conference on Learning Representations (ICLR), 2019.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3606-3663,,, 2014.
Li Fei-Fei, Rob Fergus, and Pietro Perona. One-shot learning of object categories. IEEE TPAMI, 28 (4):594-611, 2006.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Madel-agnostic meta-learning for fast adaptation of deep networks. arXiv preprint arXiv:1703.03400 2017.
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-learning arXiv preprint arXiv: 180.00887..2 2018.
Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradient-
based meta-leaming hierarchical Bayes. In International Conference Learning Representa- tions (ICLR), 2018. URL https: /wopoppeeeviieeeeeee wwwnnreviee..... net./ forum? ddnd-_idBBBBB---
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. arXiv preprint arXiv: 00061700664999 2017.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint 1rXiv:1610.02/153 2016.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. In International Conference Learning Representations (ICLR), 2019. URL https: h //openreview net forumiideee
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible Ixl convolutions. In Advances in Neural Information Processing Systems, pp. 10215-10224, 2018.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Brenden Lake, Ruslan Salakhutdinov, Jason Gross, and Joshua Tenenbaum. One shot learning of simple visual concepts. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 33, 2011.
Brenden Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 350(6266); 15006266) 2015.
Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence-culibrated classifiers for
detecting duitt--diitributiin samples. In International Conference on Learning Representations (ICLR), 201 Sa. URL ht : / wwwnopeeeeeiveeee net. Foo.mndd=rryii22ea....
Kimin Lec, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In Advances in Neural Information Processing Systems (NIPS), pp. 7167-7177, 2018b.
Shiyu Liang, Yixuan Li, and R Srikant. Enhancing the reliability of out-of-distribution image detection neural networks. arXiv preprint 0rXiv:/706.026680 2017.
Christos Louizos and Max Welling. Multiplicative normalizing flows for variational Bayesian neural networks. arXiv preprint arXiv: 1703.0990 2017.

---

Under review as a conference paper at ICLR 2020
Devraj Mandal, Sanath Narayan, Sai Kumar Dwivedi, Vikram Gupta, Shuaib Ahmed, Fahad Shahbaz Khan, and Ling Shao. cut-of-distributioo detection for generalized zero-shot action recognition. In Conference Computer Viston and Pattern Recognition, pp. 9985-9999) 2019.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't know? arXiv preprint arXiv:1810.09136, 2018.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, and Balaji Lakshminarayanan. Detecting out-of-distribution inputs to deep generative models using test for typicality. arXiv preprint arXiv:1906.02994. 2019.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 427-436, 2015.
Avital Oliver, Augustus Odena, Colin Raffel, Ekin Cubuk, and Ian Goodfellow. Realistic evaluation of deep semi-supervised learning algorithms. arXiv preprint arXiv: arXiv:1804.00 1804.09170, 2018.
Sachin Ravi and Alex Beatson. Amortized Bayesian meta-learning. In International Conference on Learning Representations (ICLR), 2019. URL https: /ttp///pppw/ee.eeeee/ee/eeeeeeee net forum?id-
rkgpy3C5tX.
Jie Ren, Peter J Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark A DePristo, Joshua V Dillon, and Balaji Lakshminarayanan. Likelihood ratios for out-of-distribution detection. arXiv preprini arXiv:1906.00 090699996955555 2019.
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B Tenenbaum, Hugo Larochelle, and Richard S Zemel. Meta-learning for semi-supervised few-shot classification. arXiv preprint 1803.080000000 2018.
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. PixeICNN+:: Improving the PixelCNN with discretized logistic mixture likelihood and other modifications. arXiv preprint arXiv: 2017.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems (NIPS), pp. 4077-4087, 2017.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties neural networks. arXiv preprint arXiv: 1312.6799, 2013.
Oriol Vinyals, Charles Blundell, Tim Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in Neural Information Processing Systems (NIPS), 2016.
Kuan-Chich Wang, Jixuan Wang, Khai Truong, and Richard Zemel. Customizable facial gesture recognition for improved assistive technology. 2019.
Pingmei Xu, Krista A Ehinger, Yinda Zhang, Adam Finkelstein, Sanjeev Kulkarni, and Jianxiong Xiao. Turkergaze: Crowdsourcing saliency with webcam based eye tracking. arXiv preprint 004.0:5504.06555.. 2015.
Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian model-agnostic meta-learning. In Advances in Neural Information Processing Systems (NIPS), pp. 7332-7342, 2018.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. LSUN: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 5555000663335. 2015.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralb.. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(6): 2017.
10

---

Under review as a conference paper at ICLR 2020
A NOTATION
Symbol
Meaning query, support, disstractors/outoo--isttiiii sets
Q.S. R Clest Ctrain Copisade
classes in the test, train set
classes in an episode the set of all examples belonging to class Cepisade number of way/classes, number of shots per episode generic image input
Depissuee Ne, Ns
X
x x
query, and out-of-distribution examples the set of support examples of class c embedding/hackkone network prototype confidence score
S.
f
s(.)
Table 4: Description of the functions used throughout this paper.
B EPISODE CONSTRUCTIONS
Training. Algorithm/ 2 is a description of episodic training of classifier. Here, De denotes the set of all examples of classes in set C". RANDOMSAMPLER n) randomly selects n elements from the set
Algorithm 2 Episodic training.
1: while not converged do
2: Copisode <- RANDOMSAMPLE((TrrA ,Nc) for e Cspisode do
sample classes b. for each class sample support >> sample query set
3:
4: S. RANDOMSAMPPLE(Dd) Ns)
5: Qa < RANDOMSAMPPL((Pi) \2,
6: end for
7:
a(V (VVLppl(;;(s00
8: end while
Evaluation. Algorithm a description episodic evaluation of the OOE task. Note that now both the in-distribution and out-of-episode classes are drawn from unseen classes. Both the in- distribution and -ut-of-episodes scores are accumulated over 1000 episodes, and evaluated using a metric such AUROC. In the OOS setting, one would modify lines to 12 to sample from the OOS set, e.g., R + RRANDOMSAMPLE((DOS) No NQ).
C DATASETS
Omniglot. The Omniglot dataset (Lake al., 2011) contains 28 28 greyscale images of hand- written characters. This is the most widely adopted benchmark dataset for few-shot classification. We use the same splits as in (Snell et al., 2017). Each class has 20 samples, and there are total of 1200 x 4 training classes and 423 x 4 unseen classes.
The CIFAR 100 dataset (Krizhevsky, 2009) contains 32 32 color images. It similar to the CIFARIO dataset, but has 100 classes of 600 images each. We used 64 classes for training, 16 for validation, and 20 for test.
CIFAR100.
minilmageNet. The minilmageeet dataset is another commonly used few-shot benchmark (Snell et al., 2017: Vinyals et al., 2016). It consists of 84 84 colored images. It also has 100 classes, and 600 examples each. Similarly, we used 64 classes for training. 16 for validation, and 20 for test.
11

---

Under review as a conference paper at ICLR 2020
Algorithm 3 Episodic Evaluation (OOE).
1: Sin <- 2
2: Sout 4- Ø
3: for 1000 do
4: Copisode
f- RANDOMSAMPLEEC ,Nc)
sample in-distribution examples
5: for c in Copisade do
6: Se +- RANDOMSAMPLE(D Ns)
7:
Qc f RANDOMSAMPLE(D Ses .Q))
8:
end for
9:
Cone <- RANDOMSAMPLE(( Crrpisne .Nc
Copisnde
>>
sample OOE examples
10:
for in Cone do
11:
R. f- RANDOMSAMPLLE(())) NQ)
12:
end for
13:
for x in Q do
Sin Sin {s(x";; o0)
15:
end for for n in R do
17: Sout & Sout U {s(xx ;0)
end for end for Metric(Smn: Sout)
19:
20:
magred The migred dataset is very similar to the minilmageNet dataset. Proposed by Ren al. (2018), 8), has 608 classes instead of 100."
Out-of-Dataset. The OOS datasets were adopted from previous studies including those by Hendrycks al. (2019); Liang et (2017). Since we experimented with in-distribution datasets of different scales, the OOS inputs were scaled accordingly.
Noise: We used uniform, Gaussian, and Rademacher noise of the same dimensionality as the in-distribution data (e.g., 3 x 32 x 32 uniform noise as OOS data for CIFAR-100)
notMNIST consists of 28 28 grayscale images of alphabetic characters from several typefaces.
CIFARI0bw is simply grayscale version of CIFARIO.
LSUN a large-scale scene understanding dataset (Yu al., 2015).
iSUN is a subset of SUN consisting of 8925 images (Xu et al., 2015).
Texture is a dataset with different real world patterns (Cimpoi et al., 2014).
Places another large scale scene understanding dataset (Zhou al., 2017).
SVHN refers to the Google Street View House Numbers dataset (Netzer et al., 2011).
Tinylmagenet consists of 64 x 64 color images from 200 ImageNet classes, with 600
examples of each class.
D EXPANDED TABLE
All the results in this section are in the 5-way, 5-shot setting, and were obtained using the 4-layer convolutional backbone.
We follow the instructions otcppps//ggtiibbocc/////eggggggr www.setttsssssssseee -publ ic.
12

---

Under review as a conference paper at ICLR 2020
D.1
OMNIGLOT
Metric
AUROCT
AUPR
FPR90J
Method
SPP
-MinDist
I.COO
SPP
-MinDist
LCBO
SPP
-MnnDist
LCBO
OOE
89.5
98.2
96.4
88.6
98.3
92.5
28.3
3.8
7.3
Gaussian
17.4
100.0
82.9
34.3
100.0
67.2
95.5
0.0
17.7
uniform
86.5
100.0
98.2
84.6
100.0
98.5
36.2
0.0
2.5
notMNIST
28.4
100.0
12.2
37.4
100.0
33.2
87.7
0.0
88.5
cibw
29.5
100.0
28.7
37.7
100.0
37.5
86.9
0.0
71.7
MNIST
17.1
100.0
70.1
34.2
100.0
56.1
95.6
0.0
30.1
OOS MEAN
35.8
100.0
58.4
45.6
100.0
58.5
80.4
0.0
42.1
MEAN
44.7
99.7
52.8
64.2
71.7
0.6
36.3
Table 5: Expanded Omniglot results
D.2
CIFARI00
Metric
AUROC1
AUPR -MinDist
FPR90_
Method
SPP
-MinDist
LCBO
SPP
LCBO
SPP
-MinDist
LCBO
OOE
60.1
68.0
73.3
61.0
67.2
71.5
84.3
73.1
62.8
Gaussian
47.2
100.0
89.0
49.7
100.0
88.5
93.3
30.9
Uniform
63.7
96.2
82.8
67.6
97.1
83.9
83.8
7.5
48.8
Rademacher
47.3
100.0
87.2
49.6
100,0
86.5
93.0
0.0
34.7
Texture
54.5
89.5
80.4
54.8
87.5
78.6
87.9
26.1
48.9
Places
56.5
88.9
78.1
58.4
89.0
87.9
56.8
SVHN
64.0
48.4
67.7
67.0
50.7
68.2
82.2
93.1
75.5
LSUN
57.3
91.4
79.5
59.2
91.7
80.3
87.5
25.7
56.7
iSUN
55.6
90.1
78.7
57.4
90.1
78.6
88.4
28.9
56.4
TinyImagenet
56.2
88.9
79.7
58.0
88.5
79.5
88.1
31.4
53.7
OOS MEAN
55.8
88.2
80.3
58.0
88.3
80.2
88.0
27.3
51.4
MEAN
56.2
86.1
79.6
58.3
86.2
79.3
87.6
31.9
$2.5
Table 6: Expanded CIFARI00 results
D.3 miniMMAGENET
Metric
AUROCT
AUPR -MnnDist
FPR90
Method
SPP
-MinDist
L.BO
SPP
LCBO
SPP
-MinDist
LCBO
OOE
56.7
61.9
65.6
56.8
61.1
63.1
86.8
80.2
73.1
Gaussian
37.4
100.0
64.3
41.7
100.0
64.7
95.8
0.0
68.0
Uniform
54.4
99.8
87.8
56.3
99.8
87.3
87.5
0.0
34.4
Rademacher
39.0
100.0
64.0
100.0
95.7
71.1
Texture
52.7
49.9
74.6
53.7
45.9
73.3
88.8
77.5
60.2
Places
57.7
46.6
76.6
59.0
47.7
77.3
86.1
91.3
61.6
SVHIN
51.1
5.6
74.5
$4.0
31.2
76.2
91.0
100.0
65.8
LSUN
59.2
51.3
76.1
53.6
85.2
66.4
iSUN
57.9
49.7
78.1
59.4
50.2
78.7
85.6
89.5
59.4
TinyImagenet
56.4
46.5
75.9
57.7
47.2
76.0
86.9
90.1
62.0
OOS MEAN
51.8
61.0
74.7
54.0
64.0
75.2
89.2
60.1
61.0
MEAN
52.2
61.1
73.8
54.2
63.7
88.9
62.1
62.2
Table 7: Expanded minilmageNet results.
13

---

Under review as a conference paper at ICLR 2020
D.4 rieredlMAGENET
Metric
AUROCT
AUPR -MinDist
FPR901
Method
SPP
-MinDist
LCBO
SPP
LCBO
SPP
-MinDist
LCBO
OOE
59.0
62.4
65.0
60.0
61.4
62.8
85.1
79.0
74.4
Gaussian
38.4
100.0
76.2
41.5
100.0
77.7
95.6
57.6
Uniform
41.6
99.1
90.4
42.8
98.9
92.0
94.0
2.0
32.5
Rademacher
40.0
100.0
77.3
42.3
100,0
78.1
95.1
0.0
54.2
Texture
55.5
34.5
61.9
56.8
40.6
63.5
88.6
93.3
81.8
Places
61.7
27.9
66.9
64.7
40.6
70.0
84.6
99.7
80.4
SVHN
54.9
10.6
57.8
58.9
32.2
61.5
90.5
99.9
90.0
LSUN
66.8
30.6
71,4
70.0
43.8
75.0
79.9
99.8
77.2
iSUN
62.7
29.4
67.9
65.1
41.4
71.0
82.8
99.6
79.9
Tiny Tingene
60.7
30.8
66.5
63.3
42.0
69.2
85.2
99.4
80.0
OOS MEAN
53.6
51.4
70.7
56.2
59.9
73.1
88.5
66.0
70.4
MEAN
54.1
52.5
70.1
56.5
60.1
72.1
88.1
67.3
70.8
Table 8: Expanded tiereddmageNet results.
E -MINDIST
In Tables 5 and 6, we show that -MinDist improved both OOE and OOS detection results under all metrics. The improvement on the OOS task was very pronounced due the fact that baseline scoring functions based on p()]x"") behaved erratically for x! far away from the empirical distribution of the in-distribution embedding. For example, when the embedding network is trained on CIFARIOG an embedded point based on image of Gaussian noise has an L2-norm 10x larger than the average embedding of an in-distribution input. This resulted in very confident SPP score (see paragraph below). This effect was eliminated by using -MinDist, and any embedded point far away from the class prototypes was assigned low confidence. This intuition seemed apply to most of the OOS tasks. On the more challenging task of OOE detection, -MinDist improved over the baselines, but not by as large a margin when in-distribution dataset was easy (e.g., Omniglot). The improvement on the OOE task was more substantial when the in-distribution dataset was CIFAR100.
Toy example of when 'softmax of distance' breaks down. Note that when the input to the softmax, our logits, the negative distances each of the prototypes:
exp(-d(Jo(x"") Mc)
( -lx = Σ. expp(dd(f((")) (2))
(10)
the softmax function invariant a constant additive bias in the logits. This makes anything outside of the convex hull formed by the prototypes equivalent to being right on the boundary of the convex hull. In the case that we have I-dimensional embedding, and only 2 prototypes located at 0 and anything within the range of 0 and 1 would give reasonable probability, and the point 0.5 would give maximum entropy. However, our intuition says that anything that is very far away from both prototypes, say the point of 100, should also have maximum entropy. Yet, due the invariant to constant additive bias, anything outside of the range and would have the undesirable behavior that as one moves away from this range, the output of the softmax decreases in entropy while we desire it to increase in entropy. In higher dimensions, a similar phenomenon happens, hence confidence functions that operate in the predicted probability space are not suitable for the outtof-distribution data.
good connection between -MinDitt and the method in (Lee et al., 2018b) can be made. However, Lee al. (2018b) fit a full covari- ance Gaussian to each of the classes, and use the Mahalanobis distance as score, which requires computing the inverse covariance of the support embeddings. This approach faces fundamental difficulty in the few-shot setting: because the number of training ex- amples (i.e., 25 for the 5-way 5-shot setting)
Table 9: AUROC comparison (Lee al., 2018b)
Dataset Model
CIFAR100
minilmageNet
OOE
00S
OOE
0OS
-MinDist
68
86
62
61
LCBO
73
80
66
75
Mahalanobis (tied) (Lee et al., 2018h)
57
86
53
59
Mahalanobis
54
86
56
42
(Lee et al., 2018h)
14

---

Under review as a conference paper at ICLR 2020
In [9]
In [10]: F.softmax(torch. (attrtrccnhFtttttttssrrrrr-aaaa: 0)
Out [10] tensor([0.5000 0.5000])
In [11]: 100
In [12]: Fis FloctTeensor((o-00o11)) .pw(()) 0)
Out [12]: tensor([1., 0.])
Figure 4: The toy example in PyTorch. is our embedded query, and we have prototype at O, and another at When a 55, SPP is 0.5. When 100, SPP is 1, which is undesirable.
is smaller than the dimension of the embedding space (i.e. 256), the covariance matrix is singular. Early in our project we found that the most natural adaptation of (Lee et al., 201 2018b), which learns Gaussian with diagonal covariance per class, performed worse than -MinDist (Table 9).
F ABML
The setup of few-shot ABML consists of prior p(t)) the global initialization v), and prior p( on episode-specific model weights for each episode. The training objective to learn a posterior distribution & which maximizes variational lower bound of the likelihood of the data.
In each episode, with model weights prior p(olv) ()))), the ABML algorithm performs standard Bayes by Backprop (Blundell et al., 2015) on the support set obtain the variational posterior distribution for In practice, the initial variational parameter for Φ is set to & to reduce the total number of parameters, while the performance did not seem be negatively affected empirically (Ravi & Beatson, 2019). Furthermore, based on the assumption that the variance in t should be low due to training over large number of episodes, Ravi & Beatson (2019) simplify the inference of ş to a point estimate, and update vb by the usual gradient descent with gradients aggregated over a sequence of episodes, analogous to the MAML setting.
Following the description in (Ravi & Beatson, 2019), implemented ABML based on the MAML implementation we got from http::://gtttbb
Suggested in Ravi & Beatson (2019)
Ours
Inner LR
0.1
0.01
Outer LR
0.001
0.001
SGD steps (training)
5
5
SGD steps (testing)
10
10
Num posterior samples (train-inner)
5
Num posterior samples (train-outer)
2
Num posterior samples (test)
10
10
an for hyper-prior by for hyper-prior Inner KI. weight Outer KI. weight
2
.2
.2
?
0.01
?
0.1
Table 10: Hyperparameters used for ABML. The last two rows, the KL weights, are not described in (Ravi & Beatson, 2019) explicitly, but only described as 'down-weighed their text. We chose what empirically works best for us.
Since in general it difficult to measure how properly Bayesian a method is, we also performed the calibration experiment from the original paper, and found similar trend (see Figure 5). Combined with similar classification accuracy, we believe that we have a somewhat meaningful implementation of ABMI..
15

---

Under review as a conference paper at ICLR 2020
MAML
ABML posterior sample)
ABML (10 posterior sample)
[GG: 3.119)
40%
22
0.7
0.4
076
114
112
170
Figure 5: Calibration results. ABML with 10 posterior samples (ECE=0.40%) have better calibration than ABML with 1 posterior sample (ECB=1.16%), and MAMI. (ECE=3.61%) 1.). ECE is the expected calibration error (Guo et al., 2017).
G SEMI-SUPERVISED FEW-SHOT CLASSIFICATION
First studied by Ren et al. (2018), there has been recent surge of interest in semi-supervised few-shot learning. Each episode has an additional unlabeled set U = { Examples from this set act as additional learning signals in each episode, much like the role of the support set. However, there two differences: I) we are not given label information, and 2) it contains 'distractor' classes, i.e., data that do come from target classes of interest. In (Ren al., 2018), their 'distractor' inputs are exactly what we refer to as OOE inputs here.
known, at least the supervised setting, that when the unlabelled dataset polluted with aut-of-distribution examples, semi-supervised methods can sometimes even degrade the classifier accuracy (Oliver et al., 2018). Similarly, in (Ren et al., 2018), without the more sophisticated methods that implicitly mask out distractors, soft k-Means with the unlabelled dataset barely has effect.
Here, we propose simple semi-supervised inference method with Prototypical Networks based on LCBO. Since naturally, we can think of Pi,e A (1; Sc)) as the probability an unknown input u; belonging to class we simply perform soft k-Means to obtain our new prototypes using Pr.c as the responsibilities:
P. Exies. fo(x) + Em.r foPr.eSo
(11)
1Sal + En.ev PiDi.
maxe Ps.c
p. Σ, maax, Pr.c
(12)
Pr. = ReLU ReLUUpi,e
(13)
and classification in this semi-supervised setting done based on these updated prototypes μ. We use Pi.e instead of pi,c because Pi.c was optimized so that point on the boundary of being diitt---itti would have Pi.c of .5, whereas in this soft clustering scheme, we want those points to have 0 weight.
Model
OOE
Uniform
Gaussian
Supervised
47.5
47.7
47.4
Baseline soft k-means (Ren al., 2018)
48.9
38.3
36.0
Ours
47.3
47.8
47.4
Table 11:
Classification accuracy (in percentages) of semi-supervised learning results on fieredImageNet. Column headings indicate type of distractor used.
Here, we do not claim that LCBO improves upon semi-supervised learning methods. Especially in the case when distractor inputs are OOS, instead of only OOE examples studied by Ren et al. (2018), baseline semi-super methods significantly degrade the classification accuracy. 3 Yet, since LCBO can detect ditt-f--iittibuutoon examples, it prevents this harmful effect. This
TThis method refers the haseline formulation of soft k-means in Ren al. (2018).
16

---

Under review as a conference paper at ICLR 2020
justifies empirically motivation that improvements in the out-of-distributtin detection can benefit downstream applications.
H TEST ACCURACIES
Model
5w1s
5w5s
10w1s
10w5s
Protonet
53.0
70.4
40.6
57.9
MAML
51.4
69.8
40.5
55.1
ABML
44.8
63.7
34.6
51.5
Table 12: Test accuracy for different architectures on CIFAR-100 using Conv4
I ADDITIONAL RESULTS FOR CIFAR-100
1.1 DIFFERENT shot-/way- SETTINGS
The overall trend that LCBO and -MinDist are better than SPP holds for different few-shot evaluation
settings.
Metric
AUROCT
AUPR -innDist
FPR90 -MinDist
Method
SPP
-MinDist
LCBO
SPP
LCBOO
SPP
LCBO
5wls OOE
54.8
65.4
65.6
55.6
64.2
64.3
87.6
75.3
74.5
5w1s OOS
54.6
80.5
71.6
56.8
80.3
73.1
88.3
49.2
66.8
5w5s OOE
60.1
68.0
73.3
67.2
71.5
84.3
73.1
62.8
5w5s OOS
55.8
88.2
80.3
58.0
88.3
80.2
88.0
27.3
51.4
10w Is OOE
54.0
61.1
60.7
54.9
60.2
59.8
88.6
80.1
79.9
10w1s 0OS
56.5
80.1
62.7
58.7
79.9
66.2
86.5
47.5
84.5
10w5s OOE
57.7
63.0
66.4
62.5
63.9
86.0
79.3
71.8
10w5s OOS
52.0
87.9
66.2
56.6
88.0
66.0
87.8
27.7
75.0
Table 13: Comparison of OOE and OOS detection performance several way and shot settings for CIFAR-100, using the Conv4 backbone.
1.2 DIFFERENT "BACKBONE"
Now for better classification accuracies, researchers moving to larger network architectures, referred to as "backbone" by Chen et al. (2019). Here we show results using the standard Conv4 network and ResNet18 trained with and without data-augmentation (Table 14).
Metric
AUROCT
AUPR
FPR90
Method
SPP
-MinDist
LCBO
SPP
-MinDist
LCBO
SPP
-MinDist
LCBO
Conv4 OOE
60.1
68.0
73.3
61.0
67.2
71.5
84.3
73.1
62.8
Conv4 OOS
55.8
88.2
80.3
58.0
88.3
80.2
88.0
27.3
51.4
ResNet18 OOE
61.1
66.1
72.2
60.3
65.0
71.6
85.4
75.2
67.5
ResNet18 OOS
61.4
80.4
81.1
61.2
80.5
82.3
84.8
47.3
53.1
ResNet1 8+aug 0OE
61.8
73.3
77.1
58.8
72.3
74.3
82.4
65.0
54.3
ResNetl8+aug OOS
65.0
87.6
84.8
62.0
88.6
84,9
80.6
37.5
42.1
Table 14: Comparison of the Conv4 and ResNet18 backbones, in the 5-way 5-shot setting.
17

---

Under review as a conference paper at ICLR 2020
J OTHER RELATED APPROACHES THAT DID NOT MAKE A BIG DIFFERENCE
J.1 OUTLIER EXPOSURE (HENDRYCKS ET AL.,, 2019)
We also investigated the effect of outlier exposure (OE) (Hendrycks al., 2019) for training the LCBO network. We denote LCBO trained with OE as LCBO+OE. Note, however, that this setup differs from that studied by Hendrycks et al. (2019). They do have a learnable confidence score like LCBO. They simply have regularization term to encourage the backbone network to output uniform distribution for OE inputs. We not train the backbone with OE as they do, but use OE inputs additional we modify the second term in Equation to include queries from the auxiliary dataset, D, along with the usual OOE queries R:
examples to train our LCBO network. To train LCBO+OE,
Σ (ex*))Q
log (5g (x*", S.))
1og(1-o((5) (x S,)
x E c.D.
(14)
The test-time aggregation for LCBO+OE is identical that described in Section 4.1.
We investigated two auxiliary dataset settings D for LCBO+OE: 1) using the Tiny Images dataset as suggested by Hendrycks al. (2019); and 2) using combination of Tinylmages and the three OOS noise distributions we consider (Gaussian, uniform, and Rademacher noise).
Metric
AUROC
AUPR
FPR90
OOE
12.7
70.7
63.5
Gaussian
95.2
94.2
12.5
Uniform
74.0
72.0
62.2
Rademacher
95.2
94.1
12.4
Texture
75.1
72.8
59.6
Places
69.9
68.4
SVHN
75.0
74.8
62.1
LSUN
71.4
71.3
69.6
iSUN
69.4
68.5
71.3
Tinylmagenet
73.9
72.4
63.6
OOS MEAN
77.8
76.7
53.5
MEAN
77.3
76.1
54.5
Table 15: Conv4 backbone, 5w5s, LCBO+0E (Tinylmages)
Metric
AUROC
AUPR
FPR90
OOE
72.3
69.9
63.3
Gaussian
99.9
99.9
0.2
Uniform
100.0
100.0
0.0
Rademacher
99.9
99.9
0.2
Texture
88.9
86.0
26.8
Places
70.0
61.0
SVHN
79.3
79.9
56.5
LSUN
72.4
70.4
61.2
iSUN
74.1
71.4
57.7
Tinylmagenet
75.8
73.2
54.8
OOS MEAN
84.8
83.4
35.4
MEAN
83.5
82.1
38.2
Table 16: Conv4 backbone, LCBO OE Iminylmages, Gaussian, uniform, Rademacher]
J.2 ODIN
ODIN (Liang et al., 2017) is shown to perform well in the supervised setting. However, as we discussed, SPP does not work well with the Prototypical Network. Below is Table showing
18

---

Under review as a conference paper at ICLR 2020
attempt to ODIN in our setting. It slightly improves over SPP, but the improvement is not substantial when compared to -MinDist. We then tried to, like ODIN, perform virtual gradient perturbation. Instead of computing the gradient of the perturbation by probability, we tried perturbing based on the distance the embedding, so we could improve over -iinDist. However, this approach not effective in initial attempts.
Metric
AUROC1
AUPRT
FPR90L
Method
SPP
ODIN
-MinDist
SPP
ODIN
-MinDist
SPP
ODIN
-MinDist
OOE
90.2
89.8
98.6
90.0
89.9
98.8
28.3
30.4
5.2
Gaussian
17.9
20.9
100.0
35.6
37.4
100.0
94.5
94.1
0.0
uniform
85.7
88.3
100.0
90.2
92.2
100.0
37.1
31.6
0.0
notMNIST
27.6
32.6
100.0
39.7
43.4
100.0
87.4
86.6
0.0
cifarl0bw
30.2
34.4
100.0
39.4
41.9
100.0
85.8
85.0
0.0
MNIST
16.9
20.2
100.0
35.9
38.9
100.0
95.9
94.2
0.0
19


Reviews:
[{"review_id": "S1eZ1xo0KH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work investigates a new problem setting that combines few-shot image classification and out-of-distribution detection. The main procedure for the task still follows a standard few-shot classification task, but in each episode, the data from a different distribution may be presented together with the query images. The evaluation focuses on the performance of out-of-distribution detection, which relies on a scoring function to assign a high score for the normal query images while having a low score for the out-of-distribution images. The paper evaluates with three different scoring functions on four few-shot classification datasets and nine out-of-distribution datasets.\n\nWe recommend a weak rejection, although the proposed new problem setting might be interesting to the community, due to three major concerns. The first is insufficient description for reproducing the evaluation. For example, the text mentions “All results are evaluated using 1000 test episodes” without information of how the in-distribution data and out-of-distribution data (especially the OOS data) is chosen in each episode. Second, the experiments and discussions do not provide enough insights for readers to understand the impact of combining the two problems. Some questions should be addressed and are listed below. The last is the writing style which has a noticeable fraction of content not directly related to the proposed problem setting. Such a style can confuse readers and require additional passes of reading to understand. \n\nSome questions to be addressed for the second concern:\n1. What’s the impact of doing out-of-distribution under the few-shot setting? Is it harder than a normal out-of-distribution detection setting? How does the “N-way X-shot” setting affect the difficulty of the problem?\n2. The paper proposes to use -MinDist and LCBO for the scoring functions instead of the methods that are commonly seen in a standard out-of-distribution detection paper. Why not use those previous methods (ex: ODIN, Mahalanobis, ensemble strategies, etc.) for the evaluation? If those previous methods do not work well with the proposed new setting, what are the possible causes?\n\nExamples of the third concern include: (1) The MAML in Section 2 has a whole paragraph that could be summarized in a few sentences. (2) Figure 3 draws the schematics of out-of-distribution detection, but its connection to the proposed setting is not clearly described. (3) The introduction mentioned semi-supervised learning and continual learning, which does not strengthen the argument of why few-shot learning and out-of-distribution detection should be combined. (4) Lastly, the FS-SSL paragraph in the experiment section has no conclusion.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HylaiguaFr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThe paper investigates both Out of Distribution (OOS) and Out of Episode (OOE) tasks. The distinction is that OOE samples are from same dataset but come from classes not represented by the support set. The paper shows that existing confidence scores developed in the supervised setting are not suitable when used with popular few-shot classifiers. The paper proposes two new confidence scores, -MinDist and LCBO. \n\nStrengths\n\nThe paper proposes benchmark datasets for out-of-distribution detection of few-shot classification. These datasets are Omniglot, CIFAR100, miniImageNet, and tieredImageNet.\n\nThe paper presents baseline results for two popular few-shot classifiers — Prototypical Networks, and MAML.\n\nThe paper shows that a simple distance metric-based approach improves the performance on both tasks.\n\nIt also proposes a parametric, class-conditional confidence score that takes a query x and a class c, and yields a score indicating whether x belongs to class c.\n\nWeaknesses\n\nThe paper is very specific to two few shot learning baselines. The question is how representative these two baselines are. Will the distance metric help for other few-shot classifiers?\n\nWhat is the motivation to detect OOS and OOE in the few shot setting given the accuracy is already low?\n\nThe contribution is mainly the metrics. \n\nOverall, the paper does not have enough interesting results for acceptance.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Bkl-tCaK5S", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method for out-of-distribution (OOD) sample detection for cases that only a few samples are available from positive classes. The authors bridge the gap between novelty detection and few-shot classification models. The paper addresses an interesting task in machine learning but in point of view of the out-of-distribution detection task, the proposed method seems to be an incremental task and far from the state-of-the-art.\n\n--The authors claim that the combination of one-class classification and few-shot classification has not been studied before, while this task is implicitly mentioned in previous work. For instance, these topics are explored in “Out-of-Distribution Detection for Generalized Zero-Shot Action Recognition” for a specific task.\n--The authors divided the out-of-dataset (OOS) into three categories, while the most famous state-of-the-art methods for one class classification task especially those that are devised based on reconstruction error or adversarial training schemes do not fall in any of these categorized.\n--Detecting the OOS samples task is very similar to recognizing the membership attack task, but the authors do not mention anything about them.\n--The proposed model has access to the negative samples (OOD samples) during training. First, this assumption is not realistic, as in all such applications, the irregular (OOD) samples are either poorly sampled or not sampled at all. Second, most of the previous works on OOD assume that OOD samples are not available during training. This makes them not directly comparable to the proposed methods that its settings are more-or-less advantaged. \n--Page 4 – what is D_c’ or C’ in Eqs. (3) and (4)?\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a method for out-of-distribution detection under the condition of access to only a few positive labeled samples. The main contribution as summarized by reviewers and authors is the new proposed benchmark and problem statement. 

All reviewers are in agreement that this paper is not ready for publication in its current form. The main concern is around the validity of the problem statement. The reviewers seek more clarity motivating the proposed scenario. Though the authors argue that as few-shot recognition is very difficult and may benefit from strategies like active learning, it is not directly clear how out of distribution detection is the best approach. In addition, R3 seeks clarification on the similarity to existing work. 

Considering the unanimous opinions of the reviewers and all author rebuttal text, the AC does not recommend acceptance of this work. We encourage the authors to focus their revisions on the explanation and motivation of this new benchmark and submit to a future venue. 

Decision: Reject