Bayesian Few-Shot Classification with One-vs-Each Pólya-Gamma Augmented Gaussian Processes
Authors: Jake Snell
Richard Zemel
Venue: ICLR 2021 Poster (2021)
Keywords: few-shot learning
gaussian processes
bayesian deep learning
uncertainty estimation

Abstract: Few-shot classification (FSC), the task of adapting a classifier to unseen classes given a small labeled dataset, is an important step on the path toward human-like machine learning. Bayesian methods are well-suited to tackling the fundamental issue of overfitting in the few-shot scenario because they allow practitioners to specify prior beliefs and update those beliefs in light of observed data. Contemporary approaches to Bayesian few-shot classification maintain a posterior distribution over model parameters, which is slow and requires storage that scales with model size. Instead, we propose a Gaussian process classifier based on a novel combination of Pólya-Gamma augmentation and the one-vs-each softmax approximation that allows us to efficiently marginalize over functions rather than model parameters. We demonstrate improved accuracy and uncertainty quantification on both standard few-shot classification benchmarks and few-shot domain transfer tasks.

Published as a conference paper at ICLR 2021
BAYESIAN FEW-SHOT CLASSIFICATION WITH ONE-VS-EACH PÓLYA-GAMMA AUGMENTED GAUSSIAN PROCESSES
Jake C. Snell
Richard Zemel
University of Toronto
University of Toronto
Vector Institute
Vector Institute
jsnel 1 11@cs toronto.
Canadian Institute for Advanced Research
zeme 1 cs toronto..dd.
ABSTRACT
Few-shot classification (FSC), the task of adapting a classifier to unseen classes given small labeled dataset, is important step on the path toward human-like machine learning. Bayesian methods are well-suited to tackling the fundamental issue of overfitting in the few-shot scenario because they allow practitioners specify prior beliefs and update those beliefs in light of observed data. Contem- porary approaches Bayesian few-shot classification maintain posterior dis- tribution over model parameters, which is slow and requires storage that scales with model size. Instead, we propose Gaussian process classifier based a novel combination of Pólya-Gamma augmentation and the one-vs-each softmax approximation (Titsias 2016) that allows us efficiently marginalize over func- tions rather than model parameters. We demonstrate improved accuracy and un- certainty quantification on both standard few-shot classification benchmarks and few-shot domain transfer tasks.
INTRODUCTION
Few-shot classification (FSC) is a rapidly growing area of machine learning that seeks to build classifiers able to adapt to novel classes given only a few labeled examples. It an important step towards machine learning systems that can successfully handle challenging situations such as personalization, rare classes, and time-varying distribution shift. The shortage of labeled data in FSC leads uncertainty the parameters of the model, known model uncertainty epistemic uncertainty. If model uncertainty not handled properly in the few-shot setting, there is a significant risk of overfitting. In addition, FSC is increasingly being used for risk-averse applications such as medical diagnosis (Prabhu 2019) and human-computer interfaces (Wang et al. 2019) where it is important for a few-shot classifier to know when it uncertain.
Bayesian methods maintain a distribution over model parameters and thus provide natural frame- work for capturing this inherent model uncertainty. In a Bayesian approach, prior distribution is first placed over the parameters of a model. After data is observed, the posterior distribution over parameters computed using Bayesian inference. This elegant treatment of model uncertainty has led to surge of interest in Bayesian approaches to FSC that infer a posterior distribution over the weights of neural network (Finn et al. 2018 Yoon et al. 2018 Ravi Beatson 2019).
Although conceptually appealing, there are several practical obstacles to applying Bayesian infer- ence directly to the weights of neural network. Bayesian neural networks (BNNs) are expensive from both computational and memory perspective. Moreover, specifying meaningful priors in pa- rameter space is known to be difficult due to the complex relationship between weights and network outputs (Sun et al. 2019).
Gaussian processes (GPs) instead maintain a distribution over functions rather than model parame- ters. The prior is directly specified by a mean and covariance function, which may be parameterized by deep neural networks. When used with Gaussian likelihoods, GPs admit closed form expressions for the posterior and predictive distributions. They exchange the computational drawbacks of BNNs

---

When applying GPs to classification with softmax likelihood, the non-conjugacy of the GP prior renders posterior inference intractable. Many approximate inference methods have been proposed to circumvent this, including variational inference and expectation propagation. In this paper we inves- tigate a particularly promising class of approaches that augment the GP model with a set of auxiliary random variables, such that when they are marginalized out the original model is recovered (Albert & Chib 1993; Girolami & Rogers 20066 Linderman et al. 2013). Such augmentation-bans -based ap- proaches typically admit efficient Gibbs sampling procedures for generating posterior samples which Published as a conference paper at ICLR 2021
for cubic scaling with the number of examples. In FSC, where the number of examples is small, this is often an acceptable trade-off.
when combined with Fisher's s identity (Douc et al. (0144) can be used to optimize the parameters of the mean and covariance functions.
In particular, augmentation with Pólya-Gamma random variables (Polson et al. 2013) makes in- ference tractable logistic models. Naively, this is useful for handling binary classification, but in this paper show how to extend Pólya-Gamma augmentation one-vs-each softmax approximation (Titsias) 2016). which can be expressed as a product of logistic sigmoids. We further show that the one-vs-each approximation can be interpreted as composite likelihood (Lindsay) 1988 Varin et al. 2011). a connection which to our knowledge has not been made in the literature.
multiple classes by using the
In this work, we make several contributions:
We show how the one-vs-each softmax approximation (Titsias) 2016) can be interpreted as a composite likelihood consisting of pairwise conditional terms.
We propose a novel GP classification method that combines the one-vs-each softmax ap- proximation with Pólya-Gamma augmentation for tractable inference.
We demonstrate competitive classification accuracy of our method standard FSC bench- marks and challenging domain transfer settings.
We propose several new benchmarks for uncertainty quantification in FSC, including cali- bration, robustness input noise, and out-of- detection.
We demonstrate improved uncertainty quantification of our method on the proposed bench- marks relative to standard few-shot baselines.
RELATED WORK
Our work is related to both GP methods for handling non-conjugate classification likelihoods and Bayesian approaches to few-shot classification. We summarize relevant work here.
2.1 GP CLASSIFICATION
Non-augmentation approaches. There are several classes of approaches for applying Gaussian processes to classification. The most straightforward method, known as least squares classifica- tion (Rifkin & Klautau| 2004). treats class labels real-valued observations and performs infer- ence with Gaussian likelihood. The Laplace approximation (Williams & Barber 1998) constructs Gaussian approximate posterior centered at the posterior mode. Variational approaches (Titsias) 2009 Matthews et al. 2016) maximize a lower bound on the log marginal likelihood. expectation propagation (Minka 2001 Kim & Ghahramani 2006 Hernandez-Lobato & Hernandez-Lobato 2016), local Gaussian approximations to the likelihood are fitted iteratively to minimize KL diver- gence from the true posterior.
Augmentation approaches. Augmentation--asee approaches introduce auxiliary random vari- ables such that the original model is recovered when marginalized out. Girolami Rogers (2006) propose Gaussian augmentation for multinomial probit regression. [inderman et al. (2015) uti- lize Pólya-Gamma augmentation (Polson et al. 2013) and stick-breaking construction to decom- pose multinomial distribution into product of binomials. Faju- et al. (2020) propose logistic- likelihood for classification and uses Gamma and Poisson augmentation in addition to Pölya-Gammm augmentation in order perform inference.
2

---

Published as a conference paper at ICLR 2021
2.2 FEW-SHOT CLASSIFICATION
Meta-lcarning. A common approach to FSC is meta-learning, which seeks to learn a strategy update neural network parameters when faced with a novel learning task. The Meta-learner LSTM (Ravi & Larochelle, 2017) learns a meta-level LSTM to recurrently output a new set of pa- rameters for base learner. MAML (Finn et al. 2017) learns initializations of deep neural networks that perform well on task-specific losses after one or a few steps of gradient descent by backpropa- gating through the gradient descent procedure itself. LEO (Rusu et al. 2019) performs meta-learning in learned low-dimensional latent space from which the parameters of a classifier are generated.
Metric learning. Metric learning approaches learn distances such that input examples can be meaningfully compared. Siamese Networks (Koch 2015) learn shared embedding network along with a distance layer for computing the probability that two examples belong to the same class. Matching Networks (Vinyals et al. 2016) uses nonparametric classification in the form of atten- tion over nearby examples, which can be interpreted as a form of soft k-nearest neighbors in the embedding space. Prototypical Networks (Snell et al. 2017) make predictions based on distances to nearest class centroids. Relation Networks (Sung et al. (2018) instead learn a more complex neural network distance function on top of the embedding layer.
Bayesian Few-shot Classification. More recently, Bayesian FSC approaches that attempt to infer a posterior over task-specific parameters have appeared. Grant et al.] (2018) reinterpret MAML as approximate empirical Bayes algorithm and propose LLAMA, which optimizes the Laplace approximation to the marginal likelihood. Bayesian MAML (Yoon et al. 2018) instead uses Stein Variational Gradient Descent (SVGD) (Liu & Wang 2016) to approximate the posterior distribution over model parameters. VERSA (Gordon et al. (019) uses amortized inference networks to obtain an approximate posterior distribution over task sask-specific parameters. ABML (Ravi & Beatson 2019) uses a few steps of Bayes by Backprop (Blundell et al., 2015) on the support set to produce an approximate posterior over network parameters. CNAPs (Requeima et al., (209)) modulate task- specific Feature-wise Linear Modulation (FiLM) (Perez et al. 2018) layer parameters as the output of adaptation network that takes the support set as input.
GPs for Few-shot Learning. There have been relatively few works applying GPs to few-shot learning. TTossou et al. (2020) consider Gaussian processes in the context few-shot regression with Gaussian likelihoods. Deep Kernel Transfer (DKT) (Patacchiola et al.) 2020) uses Gaussian processes with least squares classification to perform few-shot classífication and learns covariance functions parameterized by deep neural networks. More recently, Titsias et al. al. (2020) applies GPs to meta-learning by maximizing the mutual information between the query set and a latent representa- tion of the support set.
BACKGROUND
In this section we first review Pólya-Gamma augmentation for binary classification and the one-vs- each approximation before we introduce our method in Section
3.1 PÓLYA-GAMMA AUGMENTATION
The Pólya-Gamma augmentation scheme was originally introduced to address Bayesian inference
in logistic models (Polson et al., 2013). Suppose we have a vector of logits vp E R with corre- sponding binary labels (0, 1J The logistic likelihood
fe yu:
p(yl+))
II a ( (v:))
II
I + ee
where (-) is the logistic sigmoid function. Let the prior over 1b be Gaussian: p(ı) (Vblu, E).
In Bayesian inference, we are interested in the posterior p(tbyy) x p(y|{))(<>) but the form of
(1) does not admit analytic computation of the posterior due to non-conjugacy. The main idea of Polya-Gamma augmentation is to introduce auxiliary random variables w to the likelihood such
that the original model is recovered when is marginalized out: kyyl=b) = p(w))pyy(b w) dw.
3

---

Published as a conference paper at ICLR 2021
Conditioned (G 0), the batch likelihood is proportional to diagonal Gaussian (see Section A for a full derivation):
N II
plym w) x
e 22 x N "k 1/, (0,52-1)
i=1
where Ri = 1/1 - 1/2 and = diag(w). The conditional distribution over ab given y and w is now tractable:
p( plvly,, w) x ply pb ()p()))))))))))))))))))))) x ( Σ(Σ ki), Σ),
(3)
where Σ = (E-1 + (2) ! The conditional distribution of w given &b and can also be easily computed:
3
p(wiluc vh) x PG(w:|1, -w.0// 0G & (G(
(4)
where the last expression follows from the exponential tilting property Pólya-Gamma random variables. This suggest Gibbs sampling procedure in which iterates w(t) p(wly, <b(t=1)) and 2b(() p(t)) X. y, (()) are drawn sequentially until the Markov chain reaches its stationary distri- bution, which is the joint posterior p(1b, w/y). Fortunately, efficient samplers for the Pólya-Gamma distribution have been developed (Windle et al. 2014) facilitate this.
3.2 ONE-VS-EACH APPROXIMATION TO SOFTMAX
The one-vs-cach (OVE) approximation Titssis 2016) was formulated as a lower bound to the soft- max likelihood in order to handle classification over a large number of output classes, where com- putation of the normalizing constant is prohibitive. We employ the OVE approximation not to deal with extreme classification, but rather due to its compatibility with Pólya-Gamma augmentation, as shall see. The one-vs-each approximation be derived by first rewriting the softmax likelihood as follows:
ply i
(5)
E, 1+ I+ E
where f A (S1y..., Sc) the logits. Since in general (1+1ap 2 (1+E ak) for a 2 0. the softmax likelihood (5) can be bounded as follows:
ply =
II jti
o( fz), jfi
(6)
which the OVE lower bound. This expression avoids the normalizing constant and factorizes into a product of pairwise sigmoids, which is amenable to Saaya- augmentation for tractable inference.
ONE-VS-EACE PÓLYA-GAMMA GPs
In this section, we first show how the one-vs-each (OVE) approximation can be interpreted as pairwise composite likelihood. We then we introduce our method for GP-based Bayesian few-shot classification, which brings together OVE and Pólya-Gamma Gamma augmentation a novel combination.
4.1
OVE AS COMPOSITE LIKELIHOOD
Titsias (016) showed that the OVE approximation shares the same global optimum as the softmax maximum likelihood, suggesting a close relationship between the two. We show here that in fact OVE be interpreted
pairwise composite likelihood version of the softmax. Composite likelihoods (Lindsay 1988 Varin et al. 2011) are a type of approximate likelihood often employed when the exact likelihood intractable or otherwise difficult to compute. Given a collection of marginal or conditional events (E,....,E) Ex} and parameters f, a composite likelihood is defined as:
K
(a y)) A
II Cx y)
(7)

---

Published as a conference paper at ICLR 2021
where (f(f) y) p(y Eg and we 0 are arbitrary weights.
In order to make the connection to OVE, it will be useful to let the one-hot encoding the label be denoted as y E {0. 1). Define set of C(C 1)/2 pairwise conditional events Egj, one each for all pairs of classes * j. indicating the event that the model's output matches the target label for classes i and j conditioned all the other classes:
ply E Eg f) A p(v., Vy Y--j. f),
(8)
where -ij denotes the set of classes not equal to either or j. This expression resembles the pseu- dolikelihood (Besag 1975). but instead of single conditional event per output site, the expression in (8) considers all pairs of sites. Stochr & Friel (2013) explored similar composite likelihood gen- eralizations of the pscudolikelihood in the context of random fields.
Now suppose that ye for some class & {i.j}. Then plus, yj ly-ij f) = 1 due to the one-hot constraint. Otherwise either y/ 1 or Vj 1. In this case, assume without loss of generality that
y: = and y = O and thus
el
ply Mi Y-igh f)
ef + ef, oosi-(a1i -s)--
The composite likelihood defined in this way with unit component weights therefore
LOVE (f y) = IIII Plu. Volyss,, = IIIII - -ffi-1))* i .i
(10)
Alternatively, we may simply write LOve = I1 (fi - fi), which is identical to the OVE bound (6)
4.2 GP CLASSIFICATION WITH THE OVE LIKELIHOOD
We now turn our attention to GP classification, Suppose we have access to examples E R with corresponding one-hot labels Y E 10 where C' is the number of classes. We consider the logits jointly as single vector
(f1 SN. f2
fKY
(11)
and place an independent GP prior on the logits for each class: fx) ~ GP(m k(x,x') x')). Therefore we have X) (f K), where = m)x;) and K block diagonal with KG = k(x,, x)) for each block K
The Pólya-Gamma Gamma integral identity used to derive (2) does not have a multi-class analogue and thus
direct application of the augmentation scheme the softmax likelihood nontrivial. Instead, we propose to directly replace the softmax with the OVE-based composite likelihood function from (T0) with unit weights. The posterior over when using OVE as the likelihood function can expressed as:
N
p(f[X, & x)
II II
-t') ild'6
(12)
which 16lya-Gamma augmentation can be applied as we show in the next section. Our motiva- tion for using a composite likelihood therefore differs from the traditional motivation, which is to avoid the use of a likelihood function which is intractable to evaluate. Instead, we employ a com- posite likelihood because it makes posterior inference tractable when coupled with Polya-Gamma augmentation.
Prior work on Bayesian inference with composite likelihoods has shown that the composite posterior consistent under fairly general conditions for correctly specified models (Miller 2019) but can produce overly concentrated posteriors (Pauli et al. 201 Ribatet et al. 2012) since each component likelihood event treated as independent when in reality there may be significant dependencies. Nevertheless, show in Section 5 that in practice strong calibration relative to baseline few-shot learning algorithms. We leave further theoretical analysis of the OVE composite posterior and its properties for future work.
method exhibits competitive accuracy and
5

---

Published as a conference paper at ICLR 2021
Compared to choices of likelihoods used by previous approaches, there several reasons to prefer OVE. Relative to the Gaussian augmentation approach of Girolami & Rogers (2006). Pólya-Gamma Gamma augmentation has the benefit of fast mixing and the ability of a single value of w capture much of the marginal distribution over function valuef The stick-breaking construction of Linderman et al. (2013) induces a dependence on the ordering of classes, which leads to undesirable asymmetry. Finally, the logistic-softmax likelihood of Galy-Fajou et (2020) requires three augmentations and careful learning of the mean function avoid priori underconfidence (see Section F.1 for more details).
4.3 POSTERIOR INFERENCE VIA GIBBS SAMPLING
We now describe how we perform tractable posterior inference in our model with Gibbs sampling- Define the matrix A - OVE-MATRIX(() to be CN x CN sparse block matrix with C' row partitions and C' column partitions. Each block Ace' is a diagonal N x N matrix defined as follows:
Acc diag(Y.z) .2) =[c==(] [cc=cl. I
(13)
where Y.,. denotes the e'th column of Y. Now the binary logit vector zb - Af E RCN will have entries equal to J.i - - for each unique combination of e and i, of which there are CN in total. The OVE composite likelihood can now be written as L(
2 I (1b;), where the 2 term arises from the N cases in which , = 0 due to comparing the ground truth logit with itself.
Analogous to (2) the likelihood of 2b conditioned on w and Y is proportional to diagonal Gaussian:
NC II #=1
C( Y. w
22 x N( kx/ S2 12-1)
(14)
where Ki 1/2 and = diag(w). By exploiting the fact that vA = Af, we can express the likelihood in terms of f and write down the conditional composite posterior as follows:
X( Y, w) x (S2 " *| (f x W(ff)(K * A k), Σ),
(15)
where Σ = (K- 1 + A ATAA-' - which is an expression remarkably similar to (3) Analogous to
(4) the conditional distribution over w given f and the data becomes p(wly, f) PG 1, Af).
The primary computational bottleneck of posterior inference lies in sampling f from (15) Since
Σ is a CN x CN matrix, a naive implementation has complexity OCCNN*) By utilizing the matrix inversion lemma and Gaussian sampling techniques summarized in (Doucet 2010 this can be brought down to (CC3). Details may be found in Section
4.4 LEARNING COVARIANCE HYPERPARAMETERS FOR FEW-SHOT CLASSIFICATION
We now describe how we apply OVE Pölya-Gamma augmented GPs to few-shot classification. We assume the standard episodic few-shot setup in which one observes a labeled support set S (X, Y). Predictions must then be made for a query example yx). We consider a zero-mean GP prior over the class logits f" (x(x ~ GP(O, Ko (x.xx')) x')), where O are learnable parameters of our covariance function. These could include traditional hyperparameters such as lengthscales or the weights of deep neural network as in deep kernel learning (Wilson et al. 2016).
We consider two objectives for learning hyperparameters of the covariance function: the marginal likelihood (ML) and the predictive likelihood (PL). Marginal likelihood measures the likelihood of the hyperparameters given the observed data and is intuitively appealing from a Bayesian perspec- tive. On the other hand, many standard FSC methods optimize for predictive likelihood on the query set (Vinyals et al. 2016: Finn et al. 2017] Snell et al. (2017). Both objectives marginalize over latent functions, thereby making full of our Bayesian formulation.
The details of these objectives and how we compute gradients can be found in Section C Our learning algorithm for both marginal and predictive likelihood may be found in Section D Details
of computing the posterior predictive distribution p(y,|x_, Y.w) may be found Section Finally, details of our chosen "cosine" kernel may be found in Section
See in particular Appendix C of [inderraan al. 2015 for detailed explanation of this phenomenon.

---

Published as a conference paper at ICLR 2021
5 EXPERIMENTS
In this section, we present our results on few-shot classification both in terms of accuracy and un- certainty quantification. Additional results comparing the one-vs-cach composite likelihood the softmax, logistic softmax, and Gaussian likelihoods may be found in Section F
One of our aims is to compare methods based on uncertainty quantification. We therefore developed new benchmark evaluations and tasks; few-shot calibration, robustness, and out-of-episode detec- tion. In order to empirically compare methods, we could not simply borrow the accuracy results from other papers, but instead needed to train each of these baselines ourselves. For all baselines except Bayesian MAML,, ABMI, and Logistic Softmax GP, we ran the code from (Patacchiola iet al., (2020) and verified that the accuracies matched closely their reported results. We have made PyTorch code for our experiments publicly available
5.1 FEW-SHOT CLASSIFICATION
For our few-shot classification experiments, we follow the training and evaluation protocol of Patac- chiola et al. (2020). We train both 1-shot and 5-shot versions of our model in four different settings: Caltech- UCSD Birds (CUB) (Wah al.] [01I], mini-Imagenet with the split proposed by Ravi & Larochelle 2017). well as two cross-domain transfer tasks. The first transfer task entails training on mimi-immaa Iagee and testing on CUB, and the second measures transfer from Omniglot (Lake et al.) 2011) to EMNIST [Cohen et al., 2017). Experimental details and overview of the baselines used can be found in Section G Classification results are shown in Table and [ We find that our proposed Pölya-Gamma OVE GPs yield strong classification results, outperforming the baselines in five of the eight scenarios.
Table 1: Average accuracy and standard deviation (percentage) on 5-way FSC. Baseline results (through DKT) are from Patacchiola al.) (2020). Evaluation is performed on 3,000 randomly generated test episodes. Standard deviation for the remaining methods are computed by averaging over batches of 600 episodes with different random seeds. The best results are highlighted in bold.
CUB
mini-ImageNee
Method
1-shot
5-shot
1-shot
5-shot
Feature Transfer
46.19 0.64
68.40 + 0.79
39.51 0.23
60.51 = 0.55
Baseline+++
61.75 0.95
78.51 + 0.59
47.15 + 0.49
66.18 = 0.18
MatchingNet
60.19 1.02
75.11 + 0.35
48.25 = 0.65
62.71 = 0.44
ProtoNet
52.52 1.90
75.93 0.46
44.19 1.30
64.07 = 0.65
RelationNet
62.52 + 0.34
78.22 0.07
48.76
0.17
64.20 + 0.28
MAMI
56.11 +.11
74.84 0.62
45.39
0.49
61.58 + 0.53
DKT + Cosine
63.37 0.19
77.73 + 0.26
48.64
+ 0.45
62.85 + 0.37
Bayesian MAML
55.93
72.87
44.46
0.30
62.60 = 0.25
Bayesian MAML (Chaser)
53.93
71.16 0.32
43.74 1 0.46
59.23 = 0.34
ABML
48.80
70.91 + 0.32
40.88 = 0.25
58.19 = 0.17
Logistic Softmax GP Cosine (ML)
60.23 + 0.54
74.58 + 0.25
46.75 + 0.20
59,93 = 0.31
Logistic Softmax GP Cosine (PL)
60.07 0.29
78. 14 + 0.07
47.05
0.20
66.01 + 0.25
OVE PG GP Cosine (ML) [ours]
63.98 0.43
77.44 + 0.18
50.02 0.35
64.58 = 0.31
OVE PG GP Cosine (PI.) [ours]
60.11 + 0.26
79.07 + 0.05
48.00
0.24
67.14 + 0.23
5.2 UNCERTAINTY QUANTIFICATION THROOGH CALIBRATION
We next turn uncertainty quantification, an important concern for few-shot classifiers. When used in safety-critical applications such as medical diagnosis, it is important for machine learning sys- tem to defer when there is not enough evidence to make decision. Even non-critical applications, precise uncertainty quantification helps practitioners in the few-shot setting determine when class has an adequate amount of labeled data or when more labels are required, and can facilitate active learning.
hetpps//github.co.
lolov-eeelllaaaae patee-ppppaaaammaaaaa gamma gp

---

Published as a conference paper at ICLR 2021
Table 2: Average accuracy and standard deviation (percentage) on 5-way cross-domain FSC, with the same experimental setup as in Table Baseline results (through DKT) are from (Patacchiola al. 2020
Omniglot=>EMMn
mini-ImageNet-:cUU
Method
1-shot
5-shot
1-shot
5-shot
Feature Transfer
64.22 + 1.24
86.10
0.84
32.77
0.35
50.34 + 0.27
Baseline-++
80.01 = 0.92
39.19 + 0.12
57.31
0.11
MatchingNet
75.01 = 2.09
87.41 1.79
36,98 0.06
50.72 0.36
ProtoNet
72.04 0.82
87.22 1.01
33.27
52.16
RelationNet
75.62 + 1.00
87.84 + 0.27
37.13 0.20
51.76 1.48
MAML
72.68 + 1.85
$3.54
34.01 1.25
18.83
DKT + Cosine
73.06 + 2.36
88.10 0.78
40.22 + 0.54
55.65
Bayesian MAML
63.94 0.47
65.26 0.30
33.52 + 0.36
0.16
Bayesian MAML (Chaser)
55.04
$4.19
36.22 0.50
5
ABML
73.89
87.28 7
3
47.80
Logistic Softmax GP Cosine (ML)
62.91 1001-000 0.49
83.80 + 0.13
36.41 0.18
50.33 +.13
Logistic Softmax GP Cosine (PI.
70.70 0.36
86.59
36.73 +
+
OVE PG GP Cosine (ML) [ours]
68.43 0.67
86.22 0.20
19.66
55.71 0.31
OVE PG GP Cosine (PL) [ours]
77.00 = 0.50
1.522
37.49 0.11
57.23 0.31
We chose several commonly used metrics for calibration. Expected calibration error (ECE) Guo ctt al. 2017) measures the expected binned difference between confidence and accuracy. Maximum calibration error (MCE) is similar to ECE but measures maximum difference instead of expected difference. Brier score (BRI) (Brier 1950) a proper scoring rule computed as the squared error between the output probabilities and the one-hot label. For recent perspective on metrics for uncertainty evaluation, please refer to Ovadia et al. (2019). The results for representative approaches on 5-shot, 5-way CUB can be found in Figure across the metrics.
Our OVE PG GPs are the best calibrated overall
04
30
0.
0.
0.
03
2.44
34
04
04
04
PCB:
8-250
HCE: 8-200
MCE:
9-133
HEES
8-155
MEE: 8-335
REE: 0-460
0.
0.
02
044
02
M
00
02
0.4
OX
08
C2
CA
5.5
06
02
0.4
0E
08
C3
ES
14
1.0
MAML
Sayesie M5AM, Chhasr
Sww.accceoooo GP
Sufrrcc GO
CP P
90
EC2:
R02
CCE: 847
10
FCE:
92
03
S00:
320
BKZI 8.301
3421
0.436
BRC
8.550
3821
0. 166
1621
8.317
0821
9-264
Figure 1: Reliability diagrams, expected calibration error (ECE), maximum calibration error (MCE), and Brier Score (BRI) for 5-shot 5-way tasks CUB (additional calibration results can be found in Appendix TT. Metrics are computed on 3,000 random tasks from the test set. The last two plots are our proposed method.
5.3 ROBUSTNESS TO INPUT NOISE
Input examples for novel classes in FSC may have been collected under conditions that do not match those observed at training time. For example, labeled support images in medical diagnosis application may come from different hospital than the training set. To mimic simplified version of this scenario, we investigate robustness to input noise. We used the Imagecorruptions package Michaelis et al. 2019) to apply Gaussian noise, impulse noise, and defocus blur to both the support set and query sets of episodes at test time and evaluated both accuracy and calibration. We used corruption severity of 5 (severe) and evaluated across 1,000 randomly generated tasks the three
8

---

Published as a conference paper at ICLR 2021
datasets involving natural images. The robustness results for Gaussian noise shown in Figure 2 Full quantitative results tables for each noise type may be found in Section We find that in general Bayesian approaches tend to be robust due to their ability to marginalize over hypotheses consistent with the support labels. Our approach is one of the top performing methods across all settings.
CUB
min-Imagennt
mini-imagonel
CUB
Accurecy
40
Beseinett
20
Matching Networks
Prototypical Network
RalaforNaa
DKT
Bayeslan MAXL
5.0
1,0
Bayesian (Chaser)
Logalicc Safmaa ML. Cosine
Biier
0.5
0.5
Logistic Safmax PL Cosina
OVE ML Cosine OVE PL Cosine
Impurse
Impulse
Dafocus
GAussion
Impuse
Figure 2: Accuracy (t) and Brier Score (L) when corrupting both support and query with Gaussian noise 5-way 5-shot tasks. Quantitative results may be found in Appendix D
5.4 OUT-OF-EPISODE DETECTION
Finally, we measure performance on out-of-episode detection, another application in which uncer- tainty quantification is important. In this experiment, we used 5-way. 5-shot support sets test time but incorporated out-of-episode examples into the query set. Each episode had 150 query examples: 15 from each of randomly chosen in-episode classes and 15 from each of randomly chosen out- of-episode classes. We then computed the AUROC of binary outlier detection using the negative of the maximum logit as the score. Intuitively, if none of the support classes assign a high logit the example, it can be classified as an outlier. The results are shown in Figure Our approach generally performs the best across the datasets.
CUB
mini-imagenat
Omnigial - EMNIST
mini- - CUB
Baselnes+
Matching Networks
Protetypical Natworkss
ReistorNet
0.7
Bayesian MAML
0.6
Bayssia MAML, (Craser)
Logste Sofmar
Cosine
0.5
Logiste Softmax GP PL Cosine
OVE GP PL Cosine
Figure 3: Average AUROC (f) for out-of-episode detection. The AUC is computed separately for each episode and averaged across 1,000 episodes. Bars indicate a 95% bootstrapped confidence interval.
6 CONCLUSION
In this work, we have proposed a Bayesian few-shot classification approach based on Gaussian pro- cesses. Our method replaces the ordinary softmax likelihood with a one-vs-each pairwise composite likelihood and applies Polya-Gamma augmentation to perform inference. This allows to model class logits directly function values and efficiently marginalize over uncertainty in each few-shot episode. Modeling functions directly enables our approach to avoid the dependence on model size that posterior inference weight-space based models inherently have. Our approach compares favorably to baseline FSC methods under variety of dataset and shot configurations, including dataset transfer. We also demonstrate strong uncertainty quantification, robustness to input noise, and off-es-e detection. We believe that Bayesian modeling is powerful tool for handling uncertainty and hope that our work will lead to broader adoption of efficient Bayesian inference in the few-shot scenario.

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGMENTS
We would like thank Ryan Adams, Ethan Fetaya, Mike Mozer, Eleni Triantafillou, Kuan- Chinh Wang, and Max Welling for helpful discussions. JS also thanks SK T-Brain for supporting him on an internship that led precursors of some ideas in this paper. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute (https : //wwwww.w///mmm////////// vector inrrriinttttt.e.
ai ai/partners). This project supported by NSERC and the Intelligence Advanced Research Projects Activity (LARPA) via Department of Interior/Interior Business Center (Dol/IBC) contract number D16PC00003, The U.S. Govermment authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. Disclaimer: The views and conclusions contained herein those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of TARPA, DoI/IBC, or the U.S. Government.
REFERENCES
James H. Albert and Siddhartha Chib. Bayesian analysis of binary and polychotomous response data. Journal of the American Statistical Association, 88(422):669-679, June 1993.
Julian Besag. Statistical analysis of non-lattice data. Journal of the Royal Statistical Society: Series D (The Statistician), 24(3):179-1955 1975.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. In International Conference Machine Learning, 2015.
Glenn W. Brier. Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1):1-3, 1950.
Wei-Yu Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. closer look few-shot classification. In International Conference Learning Representations, 2019.
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and André van Schaik. EMNIST: Extending MNIST to handwritten letters. In 2017 International Joint Conference on Neural Networks
(IJCNN, pp. 2921-2926, 2017.
Randal Douc, Erie Moulines, and David Stoffer. Nonlinear Time Series: Theory, Methods and Applications with R Examples. CRC Press, 2014.
Arnaud Doucet. A Note on Efficient Conditional Simulation of Gaussian Dis-
tributions. 2010.
URL https: /tt////www//. cs. www.cc..bc................. wwt.:aaaaaaaaaaceeeeaaaaaa
pdf
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning. volume 70 of Proceedings of Machine Learning Research, pp- 1126-1135, International Convention Centre, Sydney, Australia, August 2017. PMLR.
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-learning. In S. Ben- gio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31, pp. 9516-9527. Curran Associates, Inc., 2018.
R. A. Fisher. The use of multiple measurements in taxonomic problems. Annals of Eugenics, 7(2): 179-188, 1936.
Théo Galy-Fajou, Florian Wenzel, Christian Donner, and Manfred Opper. Multi-class Gaussian process classification made conjugate: Efficient inference via data augmentation. In Ryan P. Adams and Vibhav Gogate (eds.), Proceedings of the 35th Uncertainty in Artificial Intelligence Conference, volume 115 of Proceedings of Machine Learning Research, pp. 755-765, Tel Aviy, Israe, July 2020. PMLR.
10

---

Published as a conference paper at ICLR 2021
Mark Girolami and Simon Rogers. Variational Bayesian multinomial probit regression with Gaus- sian process priors. Neural Computation, 18(8):1900-1117 August 2006.
Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard Turner. Meta- learning probabilistic inference for prediction. In International Conference Learning Repre- sentations, 2019.
Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradient- based meta-learning hierarchical Bayes. In International Conference on Learning Representa- tions, 2018.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp- 1321-1330, International Convention Centre, Sydney, Australia, August 2017. PMLR.
Daniel Mernandez-Lobbll and Jose Miguel Hernandez-Lobato Scalable Gaussian process classifi- cation via expectation propagation. In Arthur Gretton and Christian C. Robert (eds.), Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, volume 51 of Pro- ceedings of Machine Learning Research, pp. 168-176, Cadiz, Spain, May 2016. PMLR.
Nathan Hilliard, Lawrence Phillips, Scott Howland, Artëm Yankov, Courtney D. Corley, and Nathan O. Hodas. Few-Shot Learning with Metric-Agnostic Conditional Embeddings. arXIv:1802.04376 [cs, stat], February 2018.
Yehuda Hoffman and Erez Ribak. Constrained realizations of Gaussian fields-A simple algorithm. The Astrophysical Journal, 380:L5-L8, 1991.
Hyun-Chul Kim and Zoubin Ghahramani. Bayesian Gaussian process classification with the EM-EP algorithm. IEEE Transactions on Pattern Analysis and Machine Intelligence, 18(112)194881955 2006.
Diederik Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations, 2015.
Diederik P. Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparame- terization trick. In C. Cortes, N. Lawrence, Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 28, pp. 2585-2583. Curran Associates, Inc., 2015.
Gregory Koch. Siamese Neural Networks for One-Shot Image Recognition. Master's Thesis, Uni- versity of Toronto, 2015.
Brenden Lake, Ruslan Salakhutdinov, Jason Gross, and Joshua Tenenbaum. One shot learning of simple visual concepts. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 33, 2011.
Scott Linderman, Matthew J Johnson, and Ryan P Adams. Dependent multinomial models made easy: Stick-breaking with the Pólya-Gammi Gamma augmentation. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems, vol- ume 28, pp. 3456-3464. Curran Associates, Inc., 2015.
Bruce G. Lindsay. Composite Likelihood Methods. Contemporary Mathematics, 80:221-239, 1988.
Qiang Liu and Dilin Wang. Stein variational gradient descent: general purpose bayesian inference algorithm. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29, pp. 2378-2386. Curran Associates, Inc., 2016.
Alexander G. de G. Matthews, James Hensman, Richard Turner, and Zoubin Ghahramani. On sparse variational methods and the Kullback-Leibler divergence between stochastic processes. In Arthur Gretton and Christian C. Robert (eds.), Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, volume 51 of Proceedings of Machine Learning Research, pp. 231-239, Cadiz, Spain, May 2016. PMLR.

---

Published as a conference paper at ICLR 2021
Claudio Michaelis, Benjamin Mitzkus, Robert Geirhos, Evgenia Rusak, Oliver Bringmann, Alexan- der S. Ecker, Matthias Bethge, and Wieland Brendel. Benchmarking Robustness in Object De- tection: Autonomous Driving when Winter is Coming. In NeurlPS 2019 Machine Learning for Autonomous Driving Workshop, 2019.
Jeffrey W. Miller. Asymptotic normality, concentration, and coverage of generalized posteriors. vrXiv:1907.09611 [math, stat], July 2019.
Thomas Peter Minka. A Family of Algorithms for Approximate Bayesian Inference. PhD thesis, Massachusetts Institute of Technology, 2001.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your models uncertainty? Evalu- ating predictive uncertainty under dataset shift. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Sys- tems, volume 32, pp. 13991-14002 Curran Associates, Inc., 2019.
Massimiliano Patacchiola, Jack Turner, Elliot J. Crowley, Michael O'Boyle, and Amos Storkey. Bayesian Meta-Learning for the Few-Shot Setting via Deep Kernels. Advances in Neural Information Processing Systems, 2020.
Francesco Pauli, Walter Racugno, and Laura Ven. Bayesian composite marginal likelihoods. Statis- fica Sinica, pp. 17, 2011.
Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron Courville. FiLM: Visual reasoning with general conditioning layer. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), April 2018.
Nicholas G. Polson, James G. Scott, and Jesse Windle, Bayesian inference for logistic models using Pólya-Grel Gamma latent variables. Journal of the American Statistical Association, 108(504): 1539-1349) December 2013.
Viraj Uday Prabhu. Few-Shot Learning For Dermatological Disease Diagnosis. Master's Thesis, Georgia Institute of Technology, 2019.
Sachin Ravi and Alex Beatson, Amortized Bayesian meta-learning. In International Conference on Learning Representations, 2019.
Sachin Ravi and Hugo Larochelle. Optimization as model for few-shot learning. In International Conference on Learning Representattons, 2017.
James Requeima, Jonathan Gordon, John Bronskill, Sebastian Nowozin, and Richard E Turner. Fast and flexible multi- task classification using conditional neural adaptive processes. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32, pp. 7959-7970. Curran Associates, Inc., 2019.
Mathieu Ribatet, Daniel Cooley, and Anthony C. Davison. Bayesian inference from composite likelihoods, with an application to spatial extremes. Statistica Sinica, 2012.
Ryan Rifkin and Aldebaro Klautau. In defense of one-vs-all classification. Journal of Machine Learning Research, 101-11: 2004.
Andrei Rusu. Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. In International Conference on Learning Representations, 2019.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (cds.), Advances in Neural Information Processing Systems, volume 30, pp. 4077-4087. Curran Associates, Inc., 2017.
Julien Stochr and Nial Friel. Calibration of conditional composite likelihood for Bayesian inference on Gibbs random fields. In International Conference on Artificial Intelligence and Statistics, 2015.
12

---

Published as a conference paper at ICLR 2021
Shengyang Sun, Guodong Zhang, Jiaxin Shi, and Roger Grosse Functional Variational Bayesian Neural Networks. In International Conference on Learning Representations, 2019.
Flood Sung, Yongxin Yang. Li Zhang, Tao Xiang, Philip H.S. Torr, and Timothy M. Hospedales. Learning to compare: Relation network for fho-shot learning. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition (CVPR), June 2018
Michalis Titsias. Variational learning of inducing variables in sparse Gaussian processes. In David van Dyk and Max Welling (eds.), Proceedings the Twelth International Conference on Artificial Intelligence and Statistics, volume 5 of Proceedings of Machine Learning Research, pp. 567-574, Hilton Clearwater Beach Resort, Clearwater Beach, Florida USA, April 2009, PMLR.
Michalis K. Titsias. Onc-vs-each approximation to softmax for scalable estimation of probabili- ties. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garntt (eds.), Advances in Neural Information Processing Systems, volume 29, pp. 4161-4169. Curran Associates, Inc., 2016.
Michalis K. Titsias, Sotirios Nikoloutsopoulos, and Alexandre Galashov. Information Theoretic Meta Learning with Gaussian Processes. arXiv:2009.03222 [cs, stat), October 2020.
Prudencio Tossou, Basile Dura, Francois Laviolette, Mario Marchand, and Alexandre Lacoste. Adaptive Deep Kernel Learning. 1905. [cs, staf], December 2020.
Cristiano Varin, Nancy Reid, and David Firth. An overview of composite likelihood methods. Insti- lute of Statistical Science, Acudemia Sinica, 2011.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap. Koray Kavukcuoglu, and Daan Wierstra. Match- ing networks for one shot learning. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29, pp. 3630-3638. Curran Associates, Inc., 2016.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The Caltech- UCSD Birds-200-2011 Dataset. Technical Report CNS-TR-2011-001, California Institute of Technology, 2011.
Kuan-Chich Wang, Jixuan Wang, and Khai Truong, Customizable Facial Gesture Recognition For Improved Assistive Technology. In ICLR Al for Social Good Workshop. 2019.
Yeming Wen, Paul Vicol, Jimmy Ba, Dustin Tran, and Roger Grosse. Flipout: Efficient Pseudo- Independent Weight Perturbations on Mini-Batches. In International Conference Learning Representations, 2018
Christopher K. Williams and D. Barber. Bayesian classification with Gaussian processes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(12):1342-1351, 1998.
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P. Xing. Deep kernel learning. In Arthur Gretton and Christian C. Robert (eds.), Proceedings of the 19th International Con- ference on Artificial Intelligence and Statistics, volume 51 of Proceedings of Machine Learning Research, pp. 370-378, Cadiz, Spain, May 2016. PMLR.
Jesse Windle, Nicholas G. Polson, and James G. Scott. Sampling Polya-Gan Gamma random variates: Alternate and approximate techniques. arXiv:1405.055: [stat], May 2014.
Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian mgnel-agnostic meta-learning. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Baaaa-Biacch,,,, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31, pp. 7332-7342, Curran Associates, Inc., 2018.
13

---

Published as a conference paper at ICLR 2021
A DERIVATION OF PÓLYA-GAMMA AUGMENTED LOGISTIC LIKELIHOOD
In this section, we show the derivation for the augmented logistic likelihood presented in Section 3.1 First, recall the logistic likelihood:
N II (1-0)
(c 1 1e
p(y
(16)
where (-) is the logistic sigmoid function. We have a Gaussian prior p(ı) Ne Σ) and introduce Pólya-Gamma auxiliary random variables w to the likelihood such that the original model recovered when w is marginalized out: p(y|ı) = p(w)p(y 1b, w) dw.
The Pólya-Gamma distribution distributions:
cG can be written as infinite convolution of Gamma
1)
(17)
2T (k-
The following integral identity holds for b 0:
(1 (1+e7 e - / 000
(e
-wm 12 p(w) du,
(18)
where x a- b/2 and w PG 0). Specifically,, when = yndnd b 1, we recover an individual term of the logistic likelihood (16)
)e 1 1 eτ 2 e (-000/2p(w) du,
p(yl))
(19)
where k = 1/2 and w PG(1,0). Conditioned on w, the batch likelihood is proportional to diagonal Gaussian:
N
p(ylb, w) x
II
/2 xx N(2 'xs 1/1,,,22 (2,52-1)
(20)
f=1
where n; = 4/ - 1/2 and = diag(w). The conditional distribution over zb given y and w is now tractable:
p(bly,w) x ply|v w)p(() & ( E + +c), Σ), +()=++++>))
(21)
where Σ = -E-1 11
B EFFICIENT GIBBS SAMPLING
The Gibbs conditional distribution over f is given by:
= "
(22)
where Σ (K-1 - A ATAA-- Naively sampling from this distribution requires O(C compu- tation since E is CN x CNN matrix. Here we describe method for sampling from this distribution that requires 3 computation instead.
First, we note that (22) can be interpreted as the conditional distribution p(flz = S -k) resulting from the following marginal distribution p(f) and conditional p(z(f):
p(f) = N,, K)
(23)
p f) = N(x)AAr
(24)
where we have made implicit the dependence on X, Y, and w for brevity of notation. Equivalently, the distribution over f and z can be represented by the partitioned Gaussian
NN
(1
K
KAT AKA
+ 1)-
(25)
Aμ
AK
14

---

Published as a conference paper at ICLR 2021
The conditional distribution p(flz) is given
p(f = N( 4 A St)). Σ)
(26)
where -=
(K-1 + A (2A) Note that pf 52- k)) recovers desired Gibbs conditional
distribution from (22)
An efficient approach conditional Gaussian sampling is due to Hoffman & Ribak [991 and described in greater clarity by Doucet (2010). The procedure as follows:
1. Sample fo p(f) and 20 p(z(f).
2. Return = fo + KA (AKA + + (2z) Zo) as the sample from p(f]z).
K is block diagonal and thus sampling from p(f) requires O(CN2) time. Af can be computed in OCCN) time, since each entry is the difference between f!" and ff for some i and Overall, step requires O(CNS) time.
We now show how to compute f from step 2 in OCCN3)) time. We first expand (AKA) +
(AKA + 12-1)-1 Ω (AAKK- A Ω
(27)
We substitute into the expression for f:
f fu KA (Ω QAKK A (AAA A ' 2o)
(28)
fu KA 2 kk zu) KA KAT ATRA) -T T -i zo)
(29)
= fo + Kv - KA T [A(KK + A DAA 'v,
(30)
where we have defined & A 12 20)
20).
Now let d & (d...... dly. df. .f......d. ....... df.... ..... dT where di YYc Ee Define Y to be the C. N matrix produced by vertically stacking (Y.g. and let Wi be the CN x N matrix produced by vertically stacking diag((wf, ..) A ΩA may then be written as follows:
A nA
SPST, where
(31)
D = Ω + diag(d),
(32)
S Y
w
(33)
P ON
In
(34)
IN
ON
Substituting (11
into (30)
fo + Kv KA QAKK +D 1D-SPST) SPS 'v.
(35)
Now we expand (K + D SPS )-
KK D SPS 1) -
ES
(36)
where
KK- K(K + -D1---D--- is block-diagonal matrix that can computed
in (C time, since D is diagonal and K is block diagonal. Now, substituting (36) back into
(35)
f= f + Kv
KA SAEV + KA TAASS(S ES P P-1)-1s-- Ev. Ev..
(37)
Note that TS ES P-1)-1 is 2N 2N matrix and thus can be inverted in O(N3) time. The overall complexity is therefore (CC.3).
C MARGINAL LIKELIHOOD AND PREDICTIVE LIKELIHOOD OBJECTIVES
Marginal Likelihood (ML.). The log marginal likelihood can be written follows:
(0 X, Y) A log (Y = log p(w) (\W, X) dw
log ( p(w) / C(FlY, ()) df du
(38)
15

---

Published as a conference paper at ICLR 2021
The gradient of the log marginal likelihood can be estimated by posterior samples pe (e X, Y). In practice, we use a stochastic training objective based on samples w from Gibbs chains. We use Fisher's identity (Douc et al. 2014) to derive the following gradient estimator:
f po V) logre (Y w. X) dw RJ
V LLLML
Σ
Ve log po Y wlm)
(39)
where (()) w (M) are samples from the posterior Gibbs chain. As suggested by Patacchiola al.) (2020), who applied GPs FSC via least-squares classification, we merge the support and
query sets during learning to take full advantage of the available data within each episode.
Predictive Likelihood (PL). The log predictive likelihood for a query example x, is:
(0; X, Y, y.) Po x., X. Y) = log p(w)po (y, X, Y. w) dw.
(40)
We use an approximate gradient estimator again based on posterior samples of we
VoLm 2 / p Y)Vo log X, Y) dw
M
Vo
log Po, X, Y (())..
M
(41)
We note that this is not an unbiased estimator of the gradient, but find it works well in practice.
D LEARNING ALGORITHM
Our learning algorithm for both marginal and predictive likelihood is summarized in Algorithm
Algorithm One--s-Faaee Pólya-Gamma GP Learning
Objective L E {LMI. Lpr.. Task distribution T, number of parallel Gibbs chains M, number of steps T, learning rate n).
Input:
Initialize hyperparameters 0 randomly.
repeat
Sample S = (X, Y), Q Y.)
if L LMn then
X XUX., + YUY.
end if
A + OVE OVE-MATRIX(Y)
for m = to do
(m) PG(1,0), (m) (o(
w
for 1 to do
(m))
Af(
w (m)
PG(L, wb) (m)
p(l)) Pe Y, w (m)) (m))
end for end for
if L LmL then
0 0+
m==l Ve logpe Y/X, (. (m))
M
else
Σ E, Vo logopoly,, X.j, S, (m))
0 0
end if until convergence
E POSTERIOR PREDICTIVE DISTRIBUTION
The posterior predictive distribution for a query example x, conditioned on is:
p(y./x., X, Y. /(x.//E.)/(.xxx.. X, Y, w) df.,
(42)
16

---

Published as a conference paper at ICLR 2021
where f. the query example's logits. The predictive distribution over f. can be obtained by noting that zb and the query logits are jointly Gaussian:
[
v
AKA T -2-1 AK. (AK.)
K., 1).
f.
(43)
where K, is the NC C' block diagonal matrix with blocks Kx.X,x.) and K.. is the C' C diagonal matrix with diagonal entries Ki (o,,,, x.)). The predictive distribution becomes:
pff. X., X, Y, Y.W) w) (f. (4, E.), where μ.. = (AK.)
+ +0 1-' 12- Σ, = K. - (A.K(AA,)
'K and
(44)
+
With p(f,[x,, W, hand, the integral in (42) can easily be computed numerically for each class by forming the corresponding OVE linear transformation matrix A and then performing ID Gaussian- -eriimt quadrature on each dimension of (f A
A.E,A"))
F DETAILED COMPARISON OF LIKELIHOODS
In this section seek to better understand the behaviors of the softmax, OVE, logistic softmax, and Gaussian likelihoods for classification. For convenience, we summarize the forms of these likelihoods in Table 3
Table 3: Likelihoods used in Section F
Likelihood
[(fly c)
exp(Sc)
Softmax
Ee exp(fer)
Gaussian
II NI 1[c' = =c- 11-1pp- 0²
(fe)
Logistic Softmax (LSM)
Ee o(fe)
Onc-vs-Each (OVE)
olfe - fe)
6'
F.1 HISTOGRAM OF CONFIDENCES
We sampled logits from fc NN 1) and plotted histogram and kernel density estimate of the maximum output probability max ply e f) for each of the likelihoods shown in Table where === 5. The results shown in Figure 4 Logistic softmax is priori underconfident: it puts little probability mass on confidence above 0.4. This may be due to the use the sigmoid function which squashes large values of f. Gaussian likelihood and OVE are priori overconfident in that they put a large amount of probability mass on confident outputs. Note that this is not a complete explanation, because GP hyperparameters such the prior mean Gaussian likelihood variance may be able to compensate for these imperfections to some degree. Indeed, we found it helpful to learn constant mean for the logistic softmax likelihood, mentioned in Section G.2
F.2 LIKELIHOOD VISUALIZATION
In order to visualize the various likelihoods under consideration, we consider a trivial classification
task with single observed example. We assume that there are three classes (C 3) and the single example belongs to the first class (y) 1). We place the following prior f (f1, f2, fa)
0
0 0
p(f) = N (p 0
101=0011
Σ
10101
(45)
0
1000
17

---

Published as a conference paper at ICLR 2021
Logistic Softmax Gousdian
0.2
1.0
Confidence
Figure 4: Histogram and kernel density estimate of confidence for randomly generated function samples fe (0, 1). Normalized output probabilities were computed for C 5 and a histogram of max.p(y = cf was computed for 50,000 randomly generated simulations.
In other words, the prior for fi and f'2 standard normal and f' is clamped at zero (for ease of visualization). The likelihoods are plotted in Figure 5 and the corresponding posteriors are plotted
Figure
0.040
0.64
1.05
0.30
6.60
0.90
0.35
0.48
0.00
0.45
41
0.000
0.025
0.40
0.24
0.32
0.45
-2-
0.30
0.005
8000
-2
0.08
-2
0.30
0,00
(a) Softmax
(b) Gaussian
(c) Logistic Softmax
(d) One-vs-Each
Figure 5: Plot of L(f 1), where f3 is clamped to 0. The Gaussian likelihood penalizes configu-
rations far away from (f1, f2) = (1, -1). Logistic softmax is much flatter compared to softmax and has visibly different contours. One-vs-Each visually similar the softmax but penalizes (fi,f2) near the origin slightly more.
0:75
0.28
0150
0.50
0.15
0.125
0.20
0.12
0.100
0:16
0.75
0.055
0.500
0.08
-2
0.33
0.000
2
0.00
0.000
0.00
(a) Softmax
(b) Gaussian
(c) Logistic Softmax
(d) One-vs-Eace
Figure 6: Plot of posterior p( y = 1), where fs is clamped to The mode each posterior distribution is similar, but each differs slightly in shape. Gaussian more peaked about its mode, while logistic softmax is more spread out. One-vs-Eade -ach is similar to softmax, but is slightly more elliptical.
18

---

Published as a conference paper at ICLR 2021
F.3 2D IRIS EXPERIMENTS
We also conducted experiments 2D version of the Iris dataset (Fisher 1936), which contains 150 examples across 3 classes. The first two features of the dataset were retained (sepal length and
width). We used zero-mean GP prior and an RBF kernel (xx, x') exp d(xxx')) x')²) where d(x
d(.,-) :) is Euclidean distance. We considered training set sizes with 1, 2. 3, 4, 5, 10, 15, 20, 25, and 30 examples per class. For each training set size, we performed GP inference on 200 randomly generated train/test splits and compared the predictions across Gaussian, logistic softmax, and one- vs-each likelihoods.
Predictions a test point x. were made by applying the (normalized) likelihood the posterior predictive mean f.. The predictive probabilities for each likelihood is shown in Figure 7 for randomly generated train/test split with 30 examples per class. Test predictive accuracy, Brier score, expected calibration error, and evidence lower bound (ELBO) results across various training set sizes are shown Figure
The ELBO is computed by treating each likelihood's posterior xf Y) approximation to the softmax posterior p(f]X, Y).
ELBO(g) E Dog p([] + E.llogp(( Eo (f|X, Y)
log r(x) (L[g Y) (ffX, Y)).
Even though direct computation of the softmax posterior y) is intractable, computing the ELBO is tractable. A larger ELBO indicates a lower KI. divergence to the softmax posterior.
One-vs- eaeevs-Each performs well for accuracy, Brier score, and ELBO across the training set sizes. Gaus- sian performs best on expected calibration error through 15 examples per class, beyond which one- vs-each is better.
Clack
Clack
(a) Gaussian
(b) Logistic Softmax
(c) One-vs-Each
Training points (colored points) and maximum predictive probability for various like- lihoods on the Iris dataset. The Gaussian likelihood produces more warped decision boundaries than the others. Logistic softmax tends to produce lower confidence predictions, while one-vs-each produces larger regions of greater confidence than the others.
Figure 7:
(a) Accuracy
(b) Brier
(c) ECE
(d) ELBO
Figure 8: Comparison across likelihoods in terms of test predictive accuracy, Brier score, expected calibration error (computed with 10 bins), and ELBO. Results are averaged over 200 randomly generated splits for each training set size (1, 2, 3, 4, 5, 10, 15, 20, 25, and 30 examples per class). Error bars indicate 95% confidence intervals.
19

---

Published as a conference paper at ICLR 2021
G FEW-SHOT EXPERIMENTAL DETAILS
Here we provide more details about our experimental setup for our few-shot classification experi- ments, which are based on the protocol of (Patacchiola et 2020).
G.1 DATASETS
We used the four dataset scenarios described below. The first three are the same used by Chen et al. 2019) and the final was proposed by Patacchiola et al. (2020).
CUB. Caltech- UCSD Birds (CUB) (Wah et al. 2011) consists of 200 classes and 11,788 images. A split of 100 training, 50 validation, and 50 test classes was used (Hilliard al.
2018 Chen et al. 2019).
mini-Imagenet. The mini-Imagened dataset (Vinyals et al., 2016) consists of 100 classes with 600 images per class. We used the split proposed by Ravi & Larochelle (2017). which has 64 classes for training. 16 for validation, and 20 for test.
This cross-domain transfer scenario takes the training split of mini Aini-mmaenntt and the validation & test splits of CUB.
Mini--magennt-iii
Omniglot + EMNIST. We use the same setup as proposed by Patacchiola et al. (2020)
Omniglot Lake al., (211]) consists of 1,623 classes, each with 20 examples, and is aug- mented by rotations of 90 degrees to create 6,492 classes, of which 4,114 are used for training. The EMNIST dataset (Cohen al. 2017). consisting of 62 classes, is split into 31 training and 31 test classes.
G.2 FEW-SHOT CLASSIFICATION BASELINES
Here we explain the few-shot baselines in greater detail.
Feature Transfer (Chen et al. 2019) involves first training an off-line classifier the training classes and then training a new classification layer on the episode.
Baseline+ (Chen et al. 2019) is similar to Feature Transfer except it uses a cosine dis- tance module prior the softmax during fine-tuning.
Matching Networks (Vinyals al. 2016) be viewed soft form of k-nearest neigh- bors that computes attention and sums over the support examples to form predictive dis- tribution over classes.
Prototypical Networks (Snell et al. 2017) computes class means (prototypes) and forms a predictive distribution based on Euclidean distance to the prototypes. It can be viewed as a Gaussian classifier operating in an embedding space.
MAML (Fin al. 2017) performs one or a few steps of gradient descent on the support set and then makes predictions on the query set, backpropagating through the gradient descent procedure. For this baseline, we simply quote the classification accuracy reported by (Patacchiola et al. 2020).
RelationNet (Sugg et al. 2018) rather than using predefined distance metric as in Match- ing Networks or Prototypical Networks instead learns a deep distance metric as the output of neural network that accepts input the latent representation of both examples. It is trained to minimize squared error of output predictions.
Deep Kernel Transfer (DKT) (Patacchiola et al. 2020) relies on least squares classifica- tion (Rifkin & Klautau 2004) to maintain tractability of Gaussian process posterior infer- ence. In DKT, a separate binary classification task is formed for each class in one-vs-rest fashion by treating labels in +11 as continuous targets. We include the results of DKT with the cosine kernel as implemented by Patacchiola et al. (2020), which is parameterized slightly differently from the version we used (47)
(x)) T go (x')
Kum (x, 0,, v) = softplus(a) softplus(v).
(46)
[[ (x)xx]] |oo (x')l
20

---

ap- Published as a conference paper at ICLR 2021
Bayesian MAML Yoon et al. 2018) relies on Stein Variational Gradient Descent (SVGD) (Liu & Wang 2016) to get an approximate posterior distribution in weight-space. We com- pare to both the non-chaser version, which optimizes cross-entropy of query predictions, and the chaser version, which optimizes mean squared error between the approximate pos- terior the support set and the approximate posterior the merged support & query set. The non-chaser version is therefore related to predictive likelihood methods and the chaser version is more analogous to the marginal likelihood methods. For the non-chaser version, we used 20 particles and step of adaptation at both train and test time. For the chaser version, we also used 20 particles. At train time, the chaser took step and the leader additional step. At test time, we used 5 steps of adaptation. Due to the slow performance of this method, we followed the advice of Yoon et al. (2018) and only performed adaptation on the final layer of weights, which may help explain the drop in performance relative to MAML. The authors released Tensorflow code for regression only, so we reimplemented this baseline for classification in PyTorch.
Amortized Bayesian Meta-Learning (ABML) (Ravi & Beatson 2019) performs a few steps of Bayes-by-backpen (Blundell al., 2015) in order to infer fully-factorin proximate posterior over the weights. The authors did not release code and so we imple- mented our own version of ABML PyTorch. We found the weighting on the inner and outer KL divergences to be important for achieving good performance, We took the neg- ative log likelihood to be mean cross entropy and used an inner KL weight of 0.01 and outer KL weight of 0.001. These values were arrived upon by doing a small amount of hyperparameter tuning on the Omniglot-> EMNIST dataset. We used a = 1.0 and 3 0.01 for the Gamma prior over the weights. We only applied ABML to the weights the network; the biases were learned as point estimates. We used 4 steps of adaptation and took samples when computing expectations (using any more than this did not fit into GPU memory). We used the local reparameterization trick (Kingma et al. 2015) and flipout (Wen et al., 2018) when computing expectations in order to reduce variance. or- der to match the architecture used by Ravi & Beatson (2019), we trained this baseline with 32 filters throughout the classification network. We trained each 1-shot ABMI model for 800 epochs and each 5-shot ABML model for 600 epochs as the learning had not converged within the epoch limits specified in Section G.3
Logistic Softmax GP GGaly-Fajou et al.) 2020 is the multi-class Gaussian process classi- fication method that relies on the logistic softmax likelihood, Galy-Fajou et al. (2020) did not consider few-shot, but we use the same objectives described Section 4.4 to adapt this method to FSC. In addition, we used the cosine kernel (see [ection[H]oor a description) that we found to work best with our OVE PG GPs. For this method, we found it important to learn a constant mean function (rather than a zero mean) in order to improve calibration.
G.3 TRAINING DETAILS
All methods employed the commonly-used Conv4 architecture (Vinyals et al. 2016) (see Table 4 for detailed specification), except ABML which used 32 filters throughout. All of our experiments used the Adam (Kingma & Ba 2015) optimizer with learning rate 10 10-3, During training, all models used epochs consisting of 100 randomly sampled episodes. A single gradient descent step on the encoder network and relevant hyperparameters is made per episode. All 1-shot models are trained for 600 epochs and 5-shot models are trained for 400 epochs, except for ABML which was trained for an extra 200 epochs. Each episode contained 5 classes (5-way) and 16 query examples. At test time, 15 query examples are used for each episode. Early stopping was performed by monitoring accuracy on the validation set. The validation set was not used for retraining.
We train both marginal likelihood and predictive likelihood versions of our models. For Pólya- the PyP6lyaGamma package During training, we use a single step of Gibbs (T=1). For evaluation, we run until T = 50. In both training and evaluation, we use M = 20 parallel Gibbs chains to reduce variance.
Gamma sampling
21

---

Published as a conference paper at ICLR 2021
Table 4: Specification of Conv4 architecture. Conv2d layers are 3 3 with stride 1 and same padding. MaxPoole 2d layers are 2 x 2 with stride 2 and val id padding.
Output Size
Layers
Output Size
Layers
28 28
Input image
84 84
Input image
Conv2d
Conv2d
BatchNorm2d
BatchNorm2d
x 14 14
64 x 42
ReLU
ReLU
MaxPool2d
MaxPoo12d
Conv2d
Conv2d
BatchNorm2d
BatchNorm2d
64 X 7
64 x 21 x 21
ReLU
ReLU
MaxPoo12d
MaxPool2d
Conv2d
Conv2d
Rat.chNor22d
BatchNorm2d
64 x 3
64 x 10 10
ReLU
ReIU
MaxPool2d
MaxPoo12d
Conv2d
Conv2d
BatchNorm2d
BatchNorm2d
64 1 x
64 5
ReLU
ReLU
MaxPool2d
MaxPool2d
64
Flatten
1600
Flatten
(a) Ommiglot-IEMNIST dataset.
(b) All other datasets.
H EFFECT OF KERNEL CHOICE ON CLASSIFICATION ACCURACY
In this section, examine the effect of kernel choice on classification accuracy for our proposed One-vs-Each Gamma OVE GPs.
Cosine Kernel. In the main paper, we showed results for the following kernel, which we refer to the "cosine" kernel due to its resemblance cosine similarity:
[x) go (x)
h (x, 0'; (x) = exp(a)
(47)
|ge x go (x')]
where g-(-) a deep neural network that outputs a fixed-dimensional encoded representation of the input and a is the scalar log output scale. Both 0 and a are considered hyperparameters and learned simultaneously as shown in Algorithm We found that this kernel works well for a range of datasets and shot settings. We note that the use of cosine similarity is reminiscent of the approach taken by B method (Chen et al. 2019), which computes softmax over cosine similarity to class weights.
Here we consider three additional kernels: linear, RBF, and normalized RBF.
Linear Kernel. The linear kernel is defined as follows:
1 TJ exp(a)go(x) ge(),,
A Jin (x. 0))
(48)
where D is the output dimensionality of (x).. We apply this dimensionality scaling because the dot product between (x( and go (x') may be large depending on D.
RBF Kernel.
The RBF (also known as squared exponential) kernel can be defined as follows:
Lab (x, x'.x'',,a, E) exp(a) exp (- 2D exp Ie [x) (x) - (x()))
(49)
where & is the log lengthscale parameter (as with a, learn the l alongside 0).
22

---

Published as a conference paper at ICLR 2021
Normalized RBF Kernel. cosine kernel:
Finally, we consider normalized RBF kernel similar in spirit to the
tbf-noomm (x, xx = exp(a))en) exp (- (-2exp(( (1)²
go (x)
g (x')) 1²)
go (x('))
(50)
2
1190 (x)]
The results our Pólya-Gamma OVE GPs with different kernels can found in Tables and In general, find that the cosine kernel works best overall, with the exception of Omniglot Binglot=+EMNISI.. where RBF does best.
Table 5: Classification accuracy for Pólya-Gamma OVE GPs (our method) using different kernels. Cosine is overall the best, followed closely by linear. RBF-based kernels perform worse, except for the Daniglott+EENISS dataset. Evaluation is performed on 5 randomly generated sets of 600 test episodes. Standard deviation of the mean accuracy also shown. ML Marginal Likelihood, PL Predictive Likelihood.
CUB
mini-ImageNNt
Kernel
Objective
1-shot
5-shot
1-shot
5-shot
Cosine
ML
63.98 0.43
77.44 0.18
50.02 0.35
64.58 = 0.31
Linear
ML
62.48 + 0.26
77.94 + 0.21
50.81 = 0.30
66.66 = 0.45
RBF
ML
58.49 0.40
75.50 0.18
50.33 = 0.26
64.62 0.37
RBF (normalized)
ML
62.75 + 0.32
78.71 + 0.08
50.26 = 0.31
64.84 + 0.39
Cosine
PL
60.11
79.07 0.05
48.00 = 0.24
67.14 = 0.23
Linear
PL
60.44
78.54
47.29 0.31
66.66 = 0.36
RBF
PL
56.18 + 0.69
77.96
48.06 0.28
66.66 = 0.39
RBF (normalized)
PL
59.78 0.34
78.42 + 0.13
47.51 0.20
66.42 + 0.36
Table Cross- classification accuracy for Pölya-Gamma OVE GPs (our method) using dif- ferent kernels. The experimental setup is the same as Table 5
Maniglott+EMNIIS
mini-ImageNet CCUB
Kernel
Objective
1-shot
5-shot
1-shot
5-shot
Cosine
MI
68.43 + 0.67
86.22 0.20
39.66
0.18
55.71 + 0.31
Linear
ML
72.42 0.49
88.27 + 0.20
39.61 0.19
55.07
0.29
RBF
ML
78.05 + 0.38
88.98 + 0.16
36.99 =
+ 51.75 0.27
RBF (normalized)
MI
75.51 0.47
88.86
0.16
0.16 38.42
54.20
Cosine
PL
77.00 + 0.50
87.52 + 0.19
37.49
0.11
57.23 + 0.31
Linear
75.87 0.43
88.77 + 0.10
36.83 0.27
56.46 + 0.22
RBF
PL
74.62 + 0.35
89.87 + 0.13
35.06 0.25
55.12 0.21
RBF (normalized)
PL
76.01 0.31
89.42 0.16
37.50 = 0.28
56.80 + 0.39
I ADDITIONAL CALIBRATION RESULTS
In Figure 9
we include calibration results for mini- Imag-emt and Maniglot-+EMMNII They follow
similar trends to the results presented Section 5.2
23

---

Published as a conference paper at ICLR 2021
Reckun Troedda
Ayolimes
Praneyrool Milaeh
C
64
PCE: 14641
MEB: 150
PCL: 9.845
MEB: 0.158
MEE: 9.358
REE: 0.446
0.100
9.577
080
04
04
PO8:
840
MCE: 8-871
POL: 8.458
MEB: 8- 843
MCE: 8.820
REE: 9-825
0.0
02
04
0F
60
02
M4
05
06
02
0F
02
5.A.
03
02
08
09
C2
65
1.
Onnige - EMNIST
Cackur Trandfar
Basalooe
Vartling Nehaccs
Prostppic Mitwars
DXT Coala
04
8.22:
8-004
92
CCE: 0-100
02
0C2.
0-357
02
CCE: 0-118
02
FCE: 8.552
C2
CCE: 0.413
64.1 460
9.200
0-100
M421 9.518
0821 9.345
ICheee
14
04
PCD:
4-32-
ME61 160
PCX
9-124
ME61 265
MC61
1.0
MED 240
110
02
52
52
28
1.
min--lmagent: CUB
Taskur
Desdlopr
hobvoda
DKT
08
02
BCD:
DEE:
02
DCB1 0:000
62
FEE: 9:254
MCE:
0.728
8.850
0,730
0.679
02
Dayostan MAMS
Zeytslum MANM, Xchaes
Soltan ML. Cabl
Lagazt Suftexx Oedh
OVE OP VL Cante
OVE CT Coare
EC22 8- 962
2185 8. 1m0
8-222
C183 0. 922
018 925 158 9,340
6.30
Caridbroa
Caridiic
Figure 9: Reliability diagrams, expected calibration error, maximum calibration error, and Brier scores for 5-shot 5-way tasks mini-Imagenet, Omniglot->EMNNN and mini- Iman--maageett-CUBBB. Metrics are computed on 3,000 random tasks from the test set.
24

---

Published as a conference paper at ICLR 2021
J QUANTITATIVE ROBUSTNESS TO INPUT NOISE RESULTS
In this section we include quantitative results for the robustness to input noise results presented in Figure 2 Results for Gaussian noise shown in Table 7 impulse noise in Table g and defocus blur in Table
Table 7: Accuracy (%) and Brier Score when applying Gaussian noise corruption of severity 5 to both the support and query set of test-time episodes. Results were evaluated across 1,000 randomly generated 5-shot 5-way tasks.
CUB
mini-Imagell
mini-ImageNet -eNNNNet-UCC CCUB
Method
Acc. (t)
Brier (L)
Acc. (*)
Brier (L)
Acc. (t)
Brier (L)
Feature Transfer
30.45
0.775
22.58
0.799
22.75
0.799
Baseline-++
22.60
0.798
23.82
0.797
24.13
0.797
MatchingNet
26.72
0.803
24.80
0.797
23.59
0.804
ProtoNet
32.28
0.778
29.97
0.781
32.30
0.779
RelationNet
25.23
0.799
23.69
0.800
20.00
0,800
DKT Cosine
29.54
0.779
27.78
0.792
31.94
0.782
Bayesian MAML
22.79
0.905
20.52
0.963
20.46
0,949
Bayesian MAML (Chaser)
20.20
1.133
20.41
1.118
21.39
1.039
LSM GP + Cosine (ML)
27.92
0.787
22.43
0.798
22.36
0.799
LSM GP Cosine (PL)
31.21
0.772
31.77
0.768
34.74
0.754
OVE PG GP Cosine (ML) [ours]
32.27
0.774
29.99
0.776
29.97
0.784
OVE PG GP
Cosine (PI.) [ours]
33.01
0.771
33.29
0.760
31.41
0.764
Table 8: Accuracy (%) and Brier Score when applying impulse noise corruption of severity 5 to both the support and query set of test-time episodes. Results were evaluated across 1,000 randomly generated 5-shot 5-way tasks.
CUB
mini-ImageNet
mini-ImageNet--CC
Method
Acc. (1)
Brier (L)
Acc. (*)
Brier (L)
Acc. (t)
Brier (L)
Feature Transfer
30.20
0.776
23.54
0.798
22.87
0.799
Baseline-++
28.05
0.790
23.72
0.798
25.58
0.795
MatchingNet
28.25
0.790
23.80
0.803
23.21
0.811
ProtoNet
32.12
0.774
28,81
0.783
32.70
0.775
RelationNet
25.23
0.799
23.13
0.800
20.00
0,800
DKT Cosine
29.74
0.778
29.11
0.789
32.26
0.781
Bayesian MAML
22.76
0,903
20.50
0,970
20.56
0,950
Bayesian MAML (Chaser)
20.25
1.172
20.51
1.116
21.45
1.022
LSM GP + Cosine (ML)
28.18
0.787
21.82
0.799
23.64
0.797
LSM + Cosine (PL)
32.10
0.769
30.22
0.776
35.09
0.751
OVE PG GP Cosine (ML) [ours]
31.41
0.778
29.66
0.778
30.28
0.783
OVE PG GP Cosine (PL) [ours]
33.36
0.772
33.23
0.761
32.06
0.762
25

---

Published as a conference paper at ICLR 2021
Table 9: Accuracy (%) and Brier Score when applying defocus blur corruption of severity 5 to
both the support and query set of test-time episodes. Results were evaluated across 1,000 randomly generated 5-shot 5-way tasks.
CUB
mini-ImageNet--CC
Method
Acc. (t)
Brier (L)
Acc. (*)
Brier (L)
Acc. (t)
Brier (L)
Feature Transfer
38.03
0.734
33.06
0.791
33.47
0.792
Baseline +-
42.55
0.710
35.89
0.761
39.88
0.740
MatchingNet
44.43
0.682
34.43
0.754
35.95
0.741
ProtoNet
46.78
0.676
36.92
0.737
41.45
0.714
RelationNet
40.81
0.759
30.11
0.790
25.69
0.794
DKT + Cosine
45.34
0.695
38.29
0.737
45.17
0.703
Bayesian MAML
42.65
0.697
30.63
0.808
37.32
0.736
Bayesian MAML (Chaser)
40,66
0,881
29.93
1.121
31.33
1.125
LSM + Casine (ML)
45.37
0.706
34.10
0.769
39.66
0.753
LSM GP + Cosine (PL)
48.55
0.690
39.46
0.737
43.15
0.714
OVE PG GP Cosine (ML) [ours]
46.46
0.701
37.65
0.775
43.48
0,723
OVE PG GP Cosine (PL) [ours]
49.44
0.695
38.95
0.780
43.66
0.720
26


Reviews:
[{"review_id": "mUExdbxFn8E", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n***\n***\nUpdate after author response: \nThe authors have addressed my comments and my recommendation remains unchanged.\n***\n***\n\n##################################################\n\nSummary:\nThe paper proposes a novel Bayesian method for few-shot classification. The proposed classifier makes use of the commonly-used Polya-gamma augmentation, but with likelihood replaced by a one-vs-each softmax approximation. The one-vs-each softmax approximation allows efficient computation in the posterior inference. The authors demonstrate better accuracy and uncertainty quantification in benchmark datasets. \n\n##################################################\n\nPros:\n1. The paper is nicely implemented and the proposed method is clearly motivated from existing methods and show promising performance.\n\n\n##################################################\n\nCons:\n1. While the robustness to input noise is evaluated, it is unclear why the proposed method can handle the situation where training and testing data contain different noise levels or generated from different mechanism.\n\n2. It may be better to highlight more the challenges specifically in few-shot classification problems and motivate the OVE approximation in this context. \n\n3. More justification of the the one-vs-each likelihood in a Bayesian setup is needed. It is unclear why it should be preferred except for computational reasons.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "TMUxtO45Bwk", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n--------\nThe authors considers the problem in Few-shot classification and addresses the need for uncertainty management (calibrated output uncertainty, robustness to input noise, and out-of-episode detection) while maintaining high accuracy. To this end the authors propose a novel approach based on Gaussian process classification with Polya-gamma augmentation, one-vs-each Softmax posterior approximation and with a novel cosine *similarity* kernel (in composition with deep kernels). The latent variables from the Poly-gamma augmentation and latent GP function are Gibbs sampled and GP hyperparameters (including the parameters of the NN making up the deep kernel) are optimized using gradient decent based on the samples. The approach is validated in comprehensive comparative empirical experiments involving multiple datasets, and is demonstrated to be top performing in both accuracy and uncertainty management. \n\n\nStrong points\n-------------\n1. Well written paper, addressing an important problem in FSC with a well motivated and promising novel approach, filled with technical and methodology detail for completeness.\n2. The approach combine high accuracy with calibrated output probabilities.\n3. The performance is consistently strong at both robustness to input noise and out-of-episode detection.\n4. The experiments are extensive w.r.t. the competitive approaches and have wide coverage given the different data sets used.\n\nWeak points\n-------------\nNothing obvious to me.\n\nReason for score\n----------------\nA well written paper, with several well motivated and empirically validated contributions, on an important topic. The paper seem to be technically correct and well placed in the literature. Especially the contributions on uncertainty quantification (both benchmarks and the proposed method) i believe are valuable and important for the FSC field, as well as for ICLR at large.\n\nMinor comments\n--------------\nIt would be better if Figure 2 can be made larger. Maybe by sharing the legend with Figure 3 and shortening one or two sentences slightly to make room?", "extra_scores": {}, "extra_text": {}}, {"review_id": "F82Ng-0C2Vl", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work studies the GP-based few-shot classification problem with one-vs-each softmax approximation and polya-gamma augmentation. It points out the existing problems of the current schemes: (1) the non-conjugacy of GP classification, which is solved by the augmentation of Polya-gamma random variables; (2) the incompatibility of Polya-gamma augmentation with softmax link function, which is addressed by the one-vs-each softmax approximation. The theoretical analysis is solid with only some minor typos. Also, a lot of experimental comparisons are conducted between the proposed model with alternatives w.r.t classification accuracy, uncertainty quantification, noise robustness and out-of-episode detection. \n\nI recommend acceptance of the paper for the reasons below. Although the GP classification with Polya-gamma random variables is not new, the one-vs-each softmax approximation is a new idea as far as I know to reconcile the softmax link function with Polya-gamma augmentation. Another similar work is “Multi-class gaussian process classification made conjugate: Efficient inference via data augmentation, UAI 2019, providing a logistic-softmax likelihood to achieve the same effect, which is cited as LSM-GP by the work. Also, a large number of experiments are conducted and baselines are compared to validate the conclusion.\n\nWeakness: A major concern is the work does not provide enough theoretical analysis about why the OVE likelihood is better than LSM. From the conjugacy perspective, both likelihood functions achieve the conjugate form. Besides, if my understanding is right, the LSM is an accurate “softmax” likelihood while the OVE is an approximation (lower-bound of the real softmax), why does the approximated likelihood (OVE) performs better than the accurate one (LSM) in all experiments? Is there any intuition behind this phenomenon? Although the author provided some evidence in experiments and sec.6 that OVE is better than LSM, some deep theoretical analysis is needed. Also, in almost all experiments, the author stated the proposed OVE “is one of the top performing methods”. I am not interested in such statement but the intuition analysis why it is better or worse. The work should not focus on SOTA-chasing but the delicate analytical solution, even if the performance is not the best. \n\nSome minor concerns: In the comparison of likelihood in Fig.5, it is farfetched to stated the OVE is close to softmax, as it is obviously closer to the Gaussian likelihood which is overconfident as stated in the paper. \nIn sec4.2, it is unclear how to reduce the complexity from C^3N^3 to CN^3. It would be better to add a few words to describe it. \nIn sec4.3, how to marginalize over latent functions in both objectives? I did not find any details in the submission and appendix.\nIn sec5.3, if my understanding is right, the robustness to input noise comes from the GP smooth effect, so again why does the OVE-GP model outperform other GP-based models? Any intuition behind this?\n\n\nTypo: sec4.1 \\vec(f)\\simGP(m,k)   f should be a function not vector\nEq(7) equal -> proportional or add normalization. The right hand side is not a pdf. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "6_XX0EyBPYH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "1. Summary and contributions\nThis paper aims to improve the accuracy and uncertainty quantification in FSC using GP classifier. They use Polya-Gamma augmentation for tractable inference and introduce one-vs-each (OVE) approximation instead of softmax to apply PG’s property to the multi-class scenario. \n\n2. Strengths\n- Although using GP classifier with PG augmentation is not a novel idea, this is the first work tried in FSC. Also, OVE for handling multiple classes in PG augmentation is an interesting contribution.\n- The method is clearly stated. It looks much more efficient than BNN-based algorithms.\n- The authors demonstrated a range of experiments, including uncertainty quantification, noise robustness, and out-of-episode detection. Their OVE PG GP with cosine kernel consistently performs well in every experiment compared to other prevalent methods. \n\n3. Weaknesses\n- It is quite a novel idea to use PG augmentation with OVE approximation, but I think the authors are not well motivated about why their PG augmentation is necessary in FSC. If it were to quantify uncertainty, there are already GP-based algorithms as GPNet or LSM GP. Which property of PG augmentation makes it advantageous to other algorithms?\n- In the experiments section, while the authors made an effort to implement extensive demonstrations, the discussion lacks. Specifically, while their OVE PG GP is consistently better than other GP-based classifiers, metric learning models, or Bayesian NN, the reason why their method outperforms the counterparts is still ambiguous. The authors should put more effort into explaining why OVE PG GP better classifies and captures uncertainty.\n- Table 1 & Table 2. For OVE PG GP, it seems there is no clear winner between ML and PL objectives. For example, in CUB experiment, ML wins PL in 1-shot and PL wins ML in 5-shot. Some results have large deviations between them (e.g., nearly 9% difference in Omniglot->EMNIST 1-shot). Can you give a comment on the results of ML or PL: which is better to choose, and why they make inconsistent results in different experimental settings? \n- sec 6. ‘The OVE likelihood is better suited to classification ~’: Is there a theoretical or intuitive ground on this claim? I understand that OVE likelihood is introduced due to PG’s incompatibility to multiple classification. However, is there a specific reason why OVE is better than LSM with respect to classification ability?\n- In Appendix I, the authors compared several likelihood functions. It seems that OVE is the most similar to Gaussian likelihood. Then, why is OVE likelihood free of the ‘ill-suited nature of applying Gaussian likelihoods to the fundamentally discrete task of classification (sec 6)’? \n\n4. Correctness\n- sec 4.2. eq. (8) n→N, Y_(.c)→Y_(.c') Otherwise, entries of Af is not equal to f_i^(y_i )-f_i^c.\n\n5. Additional questions or feedback\n- sec 4.3. ‘We consider a zero-mean GP ~’: Can you explain why we should consider independent GPs for each class? Is it a common principle in GP classifier?\n- Table 1. In mini-ImageNet 1-shot experiment, ABML result is extremely low and even lower than the original paper (37.65 in Table 1 vs. 45.0 in Ravi & Beatson, 2019.). Is there a significant change in the experimental setting?\n\n6. Recommendation\nThis paper combines a novel idea of PG augmentation and OVE approximation into FSC, but still requires clear placement among the existing methods and more discussion on the reasoning of the results. I expect the authors to answer my concerns stated above during the rebuttal period. Thus, I vote this paper for rejecting (weak reject).\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a Bayesian approach for classification able  to  adapt  to  novel  classes  given  only  a  few  labeled  examples. The models combines a one-vs-each approximation of the likelihood combined with a Gaussian process. This allows to resort to a data-augmentation scheme based on Polya-gamma random variables. 
The paper is clearly written and combines existing techniques in a convincing manner; the experiments demonstrate better accuracy and uncertainty quantification on benchmark datasets. 

I recommend acceptance.
Decision: Accept (Poster)