Meta Learning Low Rank Covariance Factors for Energy Based Deterministic Uncertainty
Authors: Jeffrey Ryan Willette
Hae Beom Lee
Juho Lee
Sung Ju Hwang
Venue: ICLR 2022 Poster (2022)
Keywords: calibration
meta-learning

Abstract: Numerous recent works utilize bi-Lipschitz regularization of neural network layers to preserve relative distances between data instances in the feature spaces of each layer. This distance sensitivity with respect to the data aids in tasks such as uncertainty calibration and out-of-distribution (OOD) detection. In previous works, features extracted with a distance sensitive model are used to construct feature covariance matrices which are used in deterministic uncertainty estimation or OOD detection. However, in cases where there is a distribution over tasks, these methods result in covariances which are sub-optimal, as they may not leverage all of the meta information which can be shared among tasks. With the use of an attentive set encoder, we propose to meta learn either diagonal or diagonal plus low-rank factors to efficiently construct task specific covariance matrices. Additionally, we propose an inference procedure which utilizes scaled energy to achieve a final predictive distribution which is well calibrated under a distributional dataset shift. 

Published as a conference paper at ICLR 2022
META LEARNING Low RANK COVARIANCE FACTORS FOR ENERGY-BASED DETERMINISTIC UNCERTAINTY
Jeffrey Ryan Willette', Hae Beom Lee', Juho Lee & Sung Ju Hwang!
KAIST', AITRICS
(jiillette, haebeom. lee, juholee, jjwwang82) (aapaangg2@Kkiitttt @kaist ac.kr
ABSTRACT
Numerous recent works utilize bi-Lipschitz regularization of neural network layers to preserve relative distances between data instances the feature spaces of each layer. This distance sensitivity with respect to the data aids in tasks such as uncertainty calibration and out-of-distribution (OOD) detection. In previous works, features extracted with a distance sensitive model are used to construct feature covariance matrices which are used in deterministic uncertainty estimation or OOD detection. However, in cases where there is a distribution over tasks, these methods result in covariances which are sub-optimal, as they may not leverage all of the meta information which can be shared among tasks. With the use of attentive set encoder, we propose to meta learn either diagonal or diagonal plus low-rank factors efficiently construct task specific covariance matrices. Additionally, we propose an inference procedure which utilizes scaled energy to achieve a final predictive distribution which is well calibrated under a distributional dataset shift.
INTRODUCTION
Accurate uncertainty in predictions (calibration) lies at the heart of being able to trust decisions made by deep neural networks (DNNs). However, DNNs can be miscalibrated when given out-of- distribution (OOD) test examples (Ovadia et al., 2019; Guo et al., 2017). Hein et al. (2019) show that the problem can arise from ReLU non- non-linearities introducing linear polytopes into decision boundaries which lead to arbitrary high confidence regions outside of the domain of the training data. Another series of works (van Amersfoort et al., 2021; Liu et al., 2020a; Mukhoti et al., 2021; van Amersfoort al., 2021) link the problem to feature collapse, whereby entire regions of feature space collapse into singularities which then inhibits the ability of a downstream function to differentiate between points in the singularity, thereby destroying any information which could be used to differentiate them. When these collapsed regions include areas of OOD data, the model loses any ability to differentiate between in-distribution (ID) and OOD data.
solution to prevent feature collapse is to impose bi-Lipschitz regularization into the network, enforcing both upper and lower Lipschitz bound on each function operating in feature space (van Amersfoort et al., 2021; Liu et al., 2020a), preventing feature collapse. Such features from bi- Lipschitz regualarized extractors are then used to improve downstream tasks such as OOD detection or uncertainty quantification. Broadly speaking, previous works have done this by constructing covariance matrices from the resulting features in order aid in uncertainty quantification (Liu et al., 2020a; Van Amersfoort et al., 2020) or OOD detection (Mukhoti et al., 2021). Intuitively, features from a Lipschitz regularized extractor make for more expressive covariances, due to the preservation of identifying information within different features.
However, empirical covariance estimation is limited when there are few datapoints on hand, such as in few-shot learning. A key aspect of meta-learning is to learn meta- meta-knowledge over a task distribution, but as we show, empirical covariance estimation methods are not able to effectively encode such knowledge, even when the features used to calculate the covariance come from a meta-learned feature extractor (see Figure 6). As a result, the empirical covariance matrices are not expressive given limited data and thus the model loses its ability to effectively adapt feature covariances to each task.
Another obstacle, highlighted by Mukhoti et al. (2021), is that plain softmax classifiers cannot accurately model epistemic uncertainties. We identify contributing factor to this, which is the shift

---

Published as a conference paper at ICLR 2022
Accuracy: 1,00 2 NLL 0.00
Accuraccy: 1,00 M
Accuracy: 0.97 F
Accuracy: 0.91
NLL 0.00
NLL 0.25
NLL 0.54
ENT I0/0DD: 0.00/ 0.01
ENT ID/0OD: 0.01 /..0
ENT ID/000: 0.50 0.550 ECE
ENT ID,00D: 0.37.0.11
(a) PDO
(b) Protonet
(c) ProtoSNGP
(d) Protonet
Accuracy: 1.00
Accura
NLL 0.04
NLL 0.10
NLL 0,38
ENT ID/0OD: 0.07 0.55 ECE ID/00D 0.03 0.20
ENT ID/OOD: 0.12 0.555 ECE ID/00D 0.04 205/00.0 0.19
ENT ID/0OD: 0.43/ 1.14 ECE ID/00D 0.08 0.44
(c) Proto Mahalanobis
(f) Proto Mahalanobis
(g) Proto Mahalanobis
(h) Proto Mahalanobis
Figure I: Top row: Examples of the learned entropy surface of baseline networks, Bottom row: our Proto Mahalamobis models. Each pixel in the the background color represents the entropy given that coordinate in imput space, Baseline networks exhibit high confidence in areas where there has been no evidence, leading to higher calibration eтror when presented with OOD data.
invariance property of the softmax function. Specifically, even if an evaluation point comes from an OOD area and is assigned low logit values (high energy), this alone is insufficient for well calibrated prediction. Small variations in logit values can lead to arbitrarily confident predictions due to the shift invariance. From the perspective of Prototypical Networks (Snell al., 2017), we highlight this problem in Figure 3, although it applies to linear softmax classifiers as well.
In the following work, we first propose method of meta-learning class-specific covariance matrices that is transferable across the task distribution. Specifically, we meta-learn a function that takes a set of class examples as an imput and outputs class-speeific covariance matrix which in the form of either diagonal or diagonal plus low-rank factors. By doing so, the resulting covariance matrices remain expressive even with limited amounts of data. Further, in order to tackle the limitation caused by the shift invariance property of the softmax function, we propose to use scaled energy to parameterize a logit-normal softmax distribution which leads to better calibrated softmax scores. We enforce its variance to increase the minimum energy increases, and vice versa. In this way, the softmax prediction can become progressively more uniform between ID and OOD data, after marginalizing the logit-normal distribution (see example in Figure 1).
By combining those two components, we have an inference procedure which achieves a well calibrated probabilistic model using deterministic DNN. Our contributions are as follows:
We show that existing approaches fail to generalize to the meta-learning setting.
We propose a meta learning framework which predicts diagonal or low-rank covariance factors as function of a support set.
We propose an energy-based inference procedure which leads to better calibrated uncertainty on OOD data.
2 RELATED WORK
Mahalanobis Distance. Mahalanobis distance has been used in previous works for OOD detection (Lee et al., 2018) which also showed that there is a connection between softmax classifiers and Gaussian discriminant analysis, and that the representation space in the latent features of DNN's provides for an effective multivariate Gaussian distribution which be more useful in constructing class conditional Gaussian distributions than the output space of the softmax classifier. The method outlined in Lee al. (2018) provides solid groundwork for method, which also utilizes
2

---

Published as a conference paper at ICLR 2022
(a) ProtoSNGP
(b) ProtoDDU
(c) Proto Mahalanobis
Figure 2: Comparison between covariances leamee in SNGP (Liu et al., 2020a), DDU (Mukhoti et al., 2021) and Proto Mahalanobis (Ours) in the few shot setting mall- 2- 2-way/5-Sa shot). Covariance generated from SNGP close multiple of the identity matrix, while that of ProtoMahalanoois contains significant contributions from ofagonaalnall elements.
Mahalanobis distance in the latent space, and adds a deeper capability to learn meta concepts which can be shared over a distribution of tasks.
Post Processing. We refer to post-processing training and before inference in order to improve the test set performance. In the calibration literature, temperature scaling (Guo et al., 2017) is a common and effective post-processing method. As the name suggests, temperature scaling scales the logits by a constant (temperature) before applying the softmax function. The temperature is tuned such that the negative lielikkelioood (NLL) validation set is minimized. Previous works which utilize covariance (Lee et al., 2018; Mukhoti al., 2021; Liu et al., 2020a) have also applied post-processing methods construct latent feature covariance matrices after training. While effective for large single tasks, these post-processing methods make less expressive covariances in the meta learning setting, demonstrated in Figure
any method which applies some function after
Bi-Lipsehitz Regularization. Adding regularizer to enforce functional smoothness of a DNN is a useful tactic in stabilizing the training of generative adversarial networks (GANs) (Miyato et al., 2018; Arjovsky et al., 2017), improving predictive uncertainty (Liu et al., 2020a; Van Amersfoort et al., 2020), and aiding in OOD detection (Mukhoti et al., 2021). By imposing smoothness constraint on the network, distances which are semantically meaningful w.r.t. the feature manifold can be preserved in the latent representations, allowing for downstream tasks (such as uncertainty estimation) to make use of the preserved information. (Van Amersfoort et al., 2020) showed that without this regularization, a phenomena known feature collapse can map regions of feature space onto singularities (Huang et al., 2020), where previously distinct features become indistinguishable. For both uncertainty calibration and OOD detection, feature collapse map OOD features onto the same feature spaces as ID samples, adversely affecting both calibration and OOD separability.
Meta Learning. The goal of meta learning (Schmidhuber, 1987; Thrun & Pratt, 1998) is to leverage shared knowledge which may apply across a distribution of tasks. In the few shot learning scenario, models leverage general meta-knowledge gained through episodic training over task distribution (Vinyals al., 2016; Ravi Larochelle, 2017), which allows for effective adaptation and inference on task which may contain only limited amounts of data during inference. The current meta-learning approaches are roughly categorized into metric-based (Vinyals et al., 2016; Snell et al., 2017) optimization-based approaches (Finn et al., 2017; Nichol et al., 2018). In this work, our model utilizes a metric-based approach as they are closely related to generative classifiers, which have been shown to be important for epistemic uncertainty (Mukhoti et al., 2021).
3 APPROACH
We start by introducing task distribution p(7) which randomly generates tasks containing support set S = {Sx,, N and query set @ = {xx, W Then, given randomly sampled task (S, Q). we meta-learn a generative classifier that can estimate the class-wise distribution of query examples, pxxly= = e, conditioned on the support set S, for each class c = 1,..., C. A generative classifier is a natural choice in setting due to fact that it utilizes feature space densities which has been shown to be requirement for accurate epistemic uncertainty prediction (Mukhoti al., 2021). 1). Under the class-balanced scenario p(y =
p(y C)) we can easily predict
3

---

Published as a conference paper at ICLR 2022
the class labels as follows.
p(u = 0) xx c,
(I)
p(x|y = (
3.1 LIMITATIONS OF EXISTING GENERATIVE CLASSIFIERS
Possibly one of the simplest forms of deep generative classifier Prototypical Networks (Snell et al., 2017). In Protonets we assume deep feature extractor fo that embeds x to common metric space such that fo(x). We then explicitly model the class-wise distribution p(z = C, S) of the embedding z instead of the raw input x. Under the assumption of regular exponential family distri- bution for po(2|y = C, S) and a Bregman divergence such as Euclidean or Mahalanobis distance, we have pa(2ly = S) class-wise embedding mean computed from S., the set of examples from class c. In Protonets, d is squared Euclidean distance, resulting in the following likelihood of the query embedding z = fx) in the form of a softmax function.
ue)) (Snell et al., 2017), where μac |.. Exes. So (x( is the
exp( || 12ol13)
(y = c) 2e-1 exp(-11z Me (|2)
1. Limitations of fixed or empirical covariance. Unfortunately. Eq. (2) cannot capture a nontrivial
class- conditional distribution structure, as Euclidean distance in Eq. (2) is equivalent to Mahalanobis distance with fixed covariance I for all classes, such that Po(zly c, S) = N(zz; Pe; I). For this reason, many-shot models such as SNGP (Liu et al., 2020b) and DDU (Mukhoti al., 2021) calculate empirical covariances from data after training to aid in uncertainty quantification. However, such empirical covariance estimations are limited especially when the dataset size is small. If we consider the few-shot learning scenario where we have only few training examples for each class, empirical covariances can provide unreliable estimates of the true class covariance. Unreliable covariance leads to poor estimation of Mahalanobis distances and therefore unreliable uncertainty estimation.
Shift invariant property of softmax and OOD cal- ibration. Another critical limitation of Eq. (2) is that produces overconfident predictions in areas distant from the class prototypes. The problem can arise from the shift invariance property of the softmax function (w)) = e e" with w denoting the logits, such that alw + +) = / e''ss = e Ew
14
f2
z
Figure 3: || - M1ll remains the same while z travels along the line, making a pre- diction with unnecessarily low entropy.
a(w) for any shift s. More specifically, suppose we have two classes c = 1,2, and moves along the line extrapolating the prototypes μ and #2 such that z #n + c(p2 11) for I 0 or 1. Then, we can easily derive the following equality based on the shift invariant property of the softmax function:
(o = 1|2, S) = + exp(+1lm2 -nll)
where + corresponds the sign of Note that the expression is invariant to the value of excepp for its sign. Therefore, even z is OOD, residing somewhere distant from the prototypes μ4 and μ2 with extreme values of we still have equally confident predictions. See Figure 3 for illustration.
3.2 META-LEARNING OF THE CLASS-WISE COVARIANCE
In order to remedy the limitations of empirical covariance, and capture a nontrivial structure of the classccondiionnl distribution even with small support set, we propose to meta-learn the dlasswiise covariances over P(T). Specifically, we meta-learn set encoder 96 that takes class set S, as input and outputs a covariance matrix corresponding to the density p(zly = c, S), for each class 1, ... C. We expect 9e encode shared keta-knnowedgge gained through episodic training over tasks from p(T), which, as we will demonstrate in section 4, fills key shortcoming of applying existing methods such as DDU (Mukhoti et al., 2021) and SNGP (Liu al., 2020a). We denote the set-encoder g& for each class e as
Ac, P. (Zc),
Z. (ā- Melē fo(x) and x S.).
(4)
4

---

Published as a conference paper at ICLR 2022
Algorithm 1 Proto Mahalanobis Training
Algorithm 2 Proto Mahalanobis Inference
1: Input: Task distribution p(r). initial and ø
1: Input: Task meta-learned 0 and ø
2: Output: Meta-learned 0 and
2: for 1 to C do
3: while not converged do
3: Me [8c LRES, fo(x)
4: Sample task = (S, 2)
4: Ac, te + (2.
Eq. 4
5: for to C' do
5: Ee t. h + P.
Eq. 5
6: p.. Sc Exes, [x(
6:
Compute M. and [
Eq. 8,9
7: Ar, gol
Eq. 4
end for
8:
Eg + Ag + T
b Eq. 5
8: Eval. po.o S) for (2. E Q
> Eq. 11
9:
Compute E and Ed
b Eq. 8,9
10:
end for
L.
2 Exxlee -6,00) avocc
pa o [ww)
Eq.
12: (0.0)
13:
end while
where Ae E Rdxd is diagonal matrix and 4. Rdxr is rank-r matrix. Now, instead of the identity covariance matrix or empirical covariance estimation, we have the meta-learnable covariance matrix consisting of the strictly positive diagonal and low-rank component for each class c = 1, 1,...,C C.
Σ. A + 2
(5)
easy to see that E. is a valid positive semi-definite covariance matrix for positive Ag. Note that the covariance becomes diagonal when r = 0. A natural choice for Jo is the Set Transformer (Lee et al., 2019) which models pairwise interactions between elements of the input set, an implicit requirement for covariance matrices.
Now, we let Pa.s (zly = c, c, = NN Ncs E.). From Bayes' rule (see Appendix A.1), we compute the predictive distribution in the form of softmax function as follows,
ps. po.o c)
Pe, clz, -0=11 Pe, (zly c)
(6)
expl [z Me) Σg 5-1(2 (z Me) log Eel)
Ee= exp( (z me)) T Σ (z pc) log Exe
Covariance inversion and log-determinant. Note that the logit of the softmax function in Eq. (7) involves the inverse covariance Σ-1 and the log-determinant log |E. In contrast to both DDU and SNGP which propose to calculate and invert an empirical feature covariance during post-proese prospprccsssig. the meta-learning setting requires that this inference procedure be performed on every iteration during meta-training, which may be cumbersome a full (dd) inversion is to performed. Therefore, we utilize the matrix determinant lemma (Ding & Zhou, 2007) and the Cherman-Morrisee formula in the following recursive forms for both the inverse and the log determinant in Equation 7.
E.'0 + Ej D
E, D
E
(8)
1 1 +1 2;
4
det(E, t1 Det1 (1 T E, $ dett E()
3.3 OUT-OF-DISTRIBBUTIE CALIBRATION WITH SCALED ENERGY
Next, in order to tackle the overconfidence problem caused softmax shift invariance (Figure 3), we pro- pose incorporating positive constrained function of energy 1((F) max(c, (o,,,,,,,, log Σ. exp(- F,)), with temperature 7, into the predictive distribution. Energy has been used for OOD detection (Liu et al., 2020b) and density estimation (Grathwohl et al., 2019), and the success of energy in these tasks implies that it can be used to calibrate the predictive distribution (example in Figure 1h). Results in Grathwohl et al. (2019) show improvements in calibration, but their training procedure requires a full input space generative model during training, adding unwanted complexity if the end goal does not require input space generation. Our method makes use our logit values w = (w).....uu wc) to parameterize the mean of logit-normal distribution with the variance given by h ().. In this
5

---

Published as a conference paper at ICLR 2022
Table 1: OOD ECE on models trained on variations the Omniglot and MinilmageNet datasets. The OOD distribution for these models are random classes from the test set which are not present in the support set.
Omniglot OOD Class ECE
MinilnageNet OOD Class ECE
Model
5-way 5-shot
5-way 1-shot
20-way 5-shot
20-way 1-shot
5-way 1-shot
5-way 5-shot
MA
63.14+0.67
53.90:+0.77
56.60:1:598
48.39+t
29.00+t0.67
42.43-+:0.51
Reptile
48.01
41.84.10.98
46.31110.30
L.,49
29.8610.73
38..9
Protonct
68.50=00.99
67.64+00.63
77.58+0.37
72.07+00.63
33.23++1.20
47.06++1.30
Protonet-SN
69.43-+-0.57
67.67++070
77.84+00.44
72.36++0.58
33.2442.14
46.76++1.40
ProtoDDU
69.16=20.63
66.61#1.15
78.14:0.11
71.39:0.74
35.31+2.09
1..82:11.28
ProtoSNGP
65.39+00.44
60.22+#0.11
76.90:200722
68. 68.16+0040
34.38+1.21
45.84+00.1
Ours (Diag)
33.95+00.98
40.52++0.68
40.00:+0.23
1..39++1.84
17.19+1800
32,22+33.12
Ours (Rank
33.19:0.944
39.62++.02
40.04:10.40
1..28
18.78+1.72
34.44:0.64
way, the logit-normal distribution variance rises in conjunction with the energy magnitude, making predictions more uniform over the simplex for higher magnitude energies.
Pa.o (wel S) (Nes (es where je = (z -ma) -. (1.)))) - log Eell
(10)
1 T log Σ exp (-(z mv) T E (z pe))
Intuitively, h(b) is dominated by mine EeE) thereby acting as a soft approximation to the minimum energy magnitude (shortest Mahalanobis distance), which only becomes large when the energy is high for all classes represented in the logits. Then, the predictive distribution becomes
P0. = S) = / ply =
z, S)dw
(11)
M
(mpp(w!m)
Σ M m= Ee exp(w! (m)))'
w(!)) p(welzz,S).
(12)
Meta-training
At training time, we do not sample w and use the simple deterministic approx- imation (v, (y E(w]). Therefore, the loss for each task becomes C,, = Eix,jee log pa (v][B) We then optimize 0 and & by minimizing the expected loss E Ep(r) (0,) ()] over the task distribution p(T) via episodic training.
Energy scaling. Inference with equation 11 can still benefit from temperature scaling of 5 in 10. Therefore, in order properly scale the variance to avoid underconfident ID performance, we tune the temperature parameter T after training. Specifically, we start with 7 = 1 and iteratively increase T by 1 until E log p(y|z, S)] < Ep log ([)])[w])) where - log p(u) [[w]] is the NLL evaluated by using only the deterministic logits [E[w].
3.4 SPECTRAL NORMALIZATION.
Lastly, we enforce a bi-Lipschitz regularization fo by employing both residual connections and spectral normalization on the weights (Liu et al., 2020a), such that Equation 13 is satisfied. Using features Z. the calculation of covariance (Z (Z MM.) and the subsequent mean and variance of 10 both implicitly utilize distance, therefore we require bi-Lipschitz regularization of ʃe. We choose spectral normalization via the power iteration method, also known the Von Mises Iteration (Mises & Pollaczek- Pollaczek-Geiringer, 1929), due to its low memory and computation overhead as compared to second order methods such gradient penalties (Arjovsky et al., 2017). Specifically, for features at hidden layer h()), at depth I, and for some constants On, 022 for all z; and 2j. we enforce:
an |1|lz!"
z (1)
<
|1b(z, - h(z bozz="") - M I az azllz
z (1)
(13)
4 EXPERIMENTS
The goal of our experimental evaluation is to answer the following questions. 1) What is the benefit of each component of our proposed model? Does do produce more expressive covariances than

---

Published as a conference paper at ICLR 2022
0.5
ProtsDUU
Nathed
NAML
Protonet
Reptile
ProtonerSh
(Dlgg
ProteSMUPP
Ours (Ratk-11
0.3
0.2
0.0
Test
Shift Intensity: Omniglot-C (5-way/5-shot)
0.5
0.3
0.2
0.1
0.0
Test
Shift Intensity: Omniglot-C (5-way/1-shot)
0.6
0.4
0.0
Test
Shift Intensity: Omniglot-C (20-way/5-shot)
0.4
0.3
0.2
0.1
0.0
Test
Shift Intensity: Omniglot-C (20-way/1-shot)
Figure 4: ECE results for all models on different variants of the Omniglot dataset. ProtoMahalanobis models show comparable in distribution ECB while significantly improving ECE over the baselines corrupted instances from the dataset.
empirical features? 3) How does the ID/OOD calibration and accuracy compare with other popular baseline models?
Datasets. For few shot learning, we evaluate our model on both the Omniglot (Lake et al., 2015) 5) and MinilmageNNet (Vinyals et al., 2017) datasets. We utilize corrupted versions (Omniglot-C and MinilmageNet-CC) which consists of 17 corruptions at 5 different intensities (Hendrycks & Dietterich, 2019). We follow the precedent set by Snell al. (2017) and test Omniglot for 1000 random episodes and MinilmageNet for 600 episodes. For corruption experiments, the support set is uncorrupted, and corruption levels 0-5 are used as the query set being the uncorrupted query set). We also experiment with multiple toy datasets which include half-moons, and concentric circles for binary classification and random 2D multivariate Gaussian distributions for multiclass classification (Figure 1). On the toy datascts, create task distributions by sampling random tasks with biased support sets, applying random class shufffing and varying levels of noise added to each task. Randomly biasing each task ensures that single task contains information from the whole distribution and therefore, the true distribution must be meta-learned through the episodic training over many such tasks. For a detailed explanation of the exact toy dataset task creation procedure, see the appendix section A.2.
Baselines. We compare our model against Protonets (Snell et al., 2017), A spectral normalized version of Protonets (Protonet-SN), MAMI. (Finn et al., 2017), Reptile (Nichol al., 2018), and
7

---

Published as a conference paper at ICLR 2022
Hotted
0.4
ProtoDDU
HAML
Protone:
Reptie
PeotonetSN
Ours (Dog)
Oura (Mark-11
0.2
0.1
0.0
Test
Shift Intensity: MinilmageNet-C (5-way/5-shot)
0.2
Test
5
Shift Intensity: MinilmageNet-C (5-way/1-shot)
Figure 5: ECE for different variants of the MinilmageNet dataset. ProtoMahalanobis models show improved ECE on corrupted data instances while maintaining comparable performance on in-distribution data.
straightforward few-shot/protonet adaptations of Spectral Normalized Neural Gaussian Processes (ProtoSNGP) (Liu et al., 2020a) and Deep Deterministic Uncertainty (ProtoDDU) (Mukhoti et al., 2021). These models represent range of both metric based, gradient based, and covariance based meta learning algorithms. All baseline models are temperature scaled after training, with the temperature parameter optimized via LBFGS for 50 iterations with learning rate of 0.001. This follows the temperature scaling implementation from Guo et al. (2017).
We provide results for Expected Calibration Error (ECE) (Guo et al., 2017) on various types of OOD data in Figures 4 and as well as Table 1. Accuracy and NLL are reported in Appendix A.8. Meta learning generally presents high correlation between tasks, but random classes from different tasks which are not in the current support set S should still be treated as OOD. In Table we provide results where the query set consists of random classes not S. Pahtanaaalanobis models perform the best in every case except for Omniglot 20-way/1-shot, where Reptile showed the lowest ECE. The reason for this can be seen in Figure 4, where Reptile shows poor ID performance relative to all other models. Under-confidence confidence scores on OOD data, even though the model is poorly calibrated. Likewise we also evaluate models Omniglot-C and MinilmageNNt-CC in Figures 4 and 5. As the corruption intensity increases, ProtoMahalanobis models exhibit lower ECE relation to baseline models while maintaining competitive ID performance. Overall, Reptile shows the strongest calibration of baseline models although it can be underconfident on ID data as can be seen in Figure 4.
Calibration Error:
ID data can lead to better
In our experiments, transductive batch normalization used in MAML/Reptile led to suboptimal results, as the normalization statistics depend on the query set which is simultaneously passed through the network. Passing large batch of cormupted/uncorrupeed samples caused performance degradation on ID data and presented an unrealistic setting. We therefore utilized the normalization scheme proposed by Nichol al. (2018) which creates batch normalization statisties based on the whole support set plus single query instance.
Eigenvalue Distribution. In Figure 6, we evaluate the effectiveness of meta learning the low rank covariance factors with Is by analyzing the eigenvalue distribution of both empirical covariance from DDU/SNGP and the encoded covariance from 9o (Equation 5). The empirically calculated covariances exhibit lower diversity in eigenvalues, which implies that the learned Gaussian distribution more spherical and uniform for every class. ProtoMahhllnnbiis models, on the other hand, exhibit a more diverse range of cigenvalues, leading to non-trivial ellipsoid distributions. We also note that in addition to more diverse range of eigenvalues, the differences between the distributions cach class in S are also amplified in Protanahalanobis models, indicating class specific variation between
8

---

Published as a conference paper at ICLR 2022
class
10
class
10
8
Count
6
Count
4
2
2
0
0
2
3
0.00
0.25
0.50
0.75
1.00
50
100
150
200
(a) Ours E Eigenvals
(b) ProtoDDU E Eigenvals
(c) ProtoSNGP E Eigenvals
Figure 6: Precision matrix eigenvalue distribution for various meta learning model variants. A diverse distribution eigenvalues which varies by class, indicates class specific, non-spherical Gaussian distribution learned. Data comes from Omniglot 5- --aa//5-hhot experiments.
learned covariance factors. Extra figures are reported the Appendix A.7, where can be seen that the cigenvalue distribution becomes less diverse for ProtoMahalanobis models in the one-shot setting.
Architectures. For both Omniglot and MinilmageNet experiments, we utilize layer convolutional neural network with 64 filters, followed by BatchNorm and ReLU nonlinearities. Each of the four layers is followed by a max-pooling layer which results in vector embedding size 64 for Omniglot and 1600 for MinilmageNet. Exact architectures be found in Appendix A.9. Protonet-like models use BatchNorm with statistics tracked over the training set, and MAML-like baselines use Reptile Norm (Nichol al., 2018). As spectral normalized models require residual connections maintain the lower Lipschitz bound in equation 13, we add residual connections to the CNN architecture in all Protonet based models.
4.1 IMPLEMENTATION DETAILS
ProtoSNGP & ProtoDDU Both ProtoSNGP and ProtoDDU baselines are adapted to meta learning by using the original backbone implementation plus the addition of positive constrained meta parameter for the first diagonal term in Equation 8 which shared among all classes. This provides meta knowledge and a necessary first step in applying the recursive formula for inversion to make predictions on each query set seen during during training.
Covariance Encoder Sg- We utilize the Set Transformer (Lee et al., 2019), as the self-attention performed by the transformer is an expressive means to encode pairwise information between inputs. We initialize the seeds in the pooling layers (PMA), with samples from No0,, 1). We do not use any spectral normalization in Iø. as it should be sufficient to only require that the input to the encoder is composed of geometry preserving features. Crucially, we remove the residual connection Q + (QK V as we found that this led to the pooling layer ignoring the inputs and outputting an identical covariance for each class in each task. In the one-shot case, skip the centering about the centroid in Equation 4 because it would place all class centroids at the origin.
CONCLUSION
is widely known that DNNs can be miscalibrated for OOD data. We have shown that existing covariance based uncertainty quantification methods fail calibrate well when given a limited amounts of data for class-specific covariance construction for meta learning, In this work, we have proposed a novel method which meta-learns diagonal or diagonal plus low rank covariance matrix which can be used for downstream tasks such as uncertainty calibration. Additionally, we have proposed an inference procedure and energy tuning scheme which can overcome miscalibration due to the shift invariance property of softmax. We further enforce bi-Lipschitz regularization of neural network layers to preserve relative distances between data instances in the feature spaces. We validated our methods both synthetic data and two benchmark few-shot learning datasets, showing that the final predictive distribution of our method is well calibrated under distributional dataset shift when compared with relevant baselines.
9

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENTS
This work was supported by the Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (No.2019-0-00075. Artificial the Engineering Research Center Program through
Intelligence Graduate School Program(KAIST) the National Research Foundation of Korea (NRF) funded by the Korean Government MSIT (NRF- the Institute Information & communications Technology Planning & Evaluation (ITTP) grant funded by the Korea government (MSIT) No. 2021-0-02068 (Artificial
Intelligence Innovation Hub). and the National Research Foundation of Korea (NRF) funded by the Ministry of Education (NRF202IRI R1F12IIIFAA111116
REFERENCES
Bruno Andreis, Jeffrey Willette, Juho Lee, and Sung Ju Hwang. Mini-batch consistent slot set encoder for scalable set encoding. arXiv preprint arXiv:2103.01615. 2021.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein gan, 2017.
Jiu Ding and Aihui Zhou. Eigenvalues of rank-one updated matrices with some applications. Applied Mathematics Letters,
2007.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leariig for fast adaptation of deep networks. In International Conference on Machine Learning, pp. 1126-1135. PMLR, 2017.
Will Grathwohl, Kuan-Chieh Wang. Jörn-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. arXiv preprint arXiv:1912.03263: 2019.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. WWeinberger. On calibration of modern neural networks, 2017.
Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf. Why relu networks yield high- confidence predictions far away from the training data and how to mitigate the problem, 2019.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv: 12261, 2019.
Haiwen Huang, Zhihan Li, Lulu Wang, Sishuo Chen, Bin Dong, and Xinyu Zhou. Feature space detection. arXiv preprint arXiv:2011.14654 2020.
singularity for
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 14Xiv:1412.6980 2014.
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 13306266):1332-1133 2015.
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set trans- former: framework for attention- based cootmutaaionninvartii neural networks. In International Conference Machine Learning, pp. 3744-3753. PMLR, 2019.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. arXiv preprint arXiv:1807.03888. 2018.
Jeremiah Zhe Liu, Zi Lin, Shreyas Padhy, Dustin Tran, Tania Medrax-Weiss and Balaji Lakshmi- narayanan. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. arXiv preprint arXiv:2006.1000
2020a.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detection. arXiv preprint arXiv:2010.03759, 2020b. 20200.
RV Mises and Hilda Pollaczek-Geiringee Praktische verfahren der gleichungsauffösung. ZAMM- Journal of Applied Mathematics and Mechanics/ZZitschrift für Angewandte Mathematik und Mechanik, 91):58--77, 1929.
10

---

Published as a conference paper at ICLR 2022
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXi::1802.05957, 2018.
Jishnu Mukhoti, Andreas Kirsch, Joost van Amersfoort, Philip H. S. Torr, and Yarin Gal. Deterministic neural networks with appropriate inductive biases capture epistemic and aleatoric uncertainty, 2021.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms, 2018.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua V Dillon, Balaji Lakshminarayanan, and Jasper Snock. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. arXiv preprint arXiv: 1906.02500 2019.
Massimiliano Patacchiola, Jack Tumnrr, Elliot Crowley, and Amos Storkey. Bayesian reta-learning for the few-shot setting via deep kernels. In Advances in Neural Information Processing Systems, 2020.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. ICLR, 2017.
Jürgen Schmidhuber. Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta-... hook. PhD thesis, Technische Universität München, 1987.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. arXiv preprint arXiv:I703.0000 1ee:: 2017.
Sebastian Thrun and Lorien Pratt (eds.). Learning to Learn. Kluwer Academic Publishers, Norwell, MA, USA, 1998. ISBN 0-7923-8047-9,
Joost Van Amersfoort, Lewis Smith, Yee Whye Teh, and Yarin Gal. Uncertainty estimation using a single deep deterministic neural network. In International Conference on Machine Learning, pp. 9690-9700. PMLR, 2020.
Joost van Amersfoort, Lewis Smith, Andrew Jesson, Oscar Key, and Yarin Gal. Improving determin- istic uncertainty estimation in deep learning for classification and regression, 2021.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, al. Matching Networks for One Shot Learning. In NIPS, 2016.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Match- ing networks for one shot learning, 2017.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan Salakhutdinov, and Alexander Smola. Deep sets. arXiv preprint arXiv:1703.061 14, 2017.

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.1 LOss DERIVATION (EQUATION 7)
The full derivation of Equation 7 can be achieved by first applying Bayes' Rule, assuming simple uniform prior over the class labels, p(ux(x;) can be proportionately expressed as,
p(ys(x,) p(xsslu))p((u ((ys)) p(x,)
or p(x; yk p(yx)
(14)
In which case the objective of the model becomes raising the class conditional p(x,/1)). while simultaneously lowering p(x:ju;) V * i. This is in fact equivalent to softmax + cross entropy loss over the class conditional densities which are output from our model. In the softmax case, maximizing p(y][x; for given class can be done by,
p(ui(x;)
(15)
Eg exe
Which them implies that the loss to be minimized is the following, commonly known as the negative log likelihood of the data, or the empirical cross entropy between the true data distribution and the predictive distribution of the model.
LNLL- = EpI- logp(u|>)]
N
- logp(yi(x;)
N
1
Σ
(-2z 2j + log Σ Eexp(2)))
N
i=0
(16)
LOE
Dx px(x) log po (2)xx)xx
(og
N
==0
1
Σ (-2] 2j + log Σ (exp(z)))
N
In our case, assuming a uniform prior over the classes, we can analogously formulate the loss as.
C = Ep | log p(x())p(())
1
(o))( log Σ pxx' ((()))
N
1
N
(17)
logg((x;llk logp(uk) + log p(vs) + log Σ p(x' (1xx)l9x))
N
N
Σ
log p(x:)))))))))))))))))))<))) + log Σ p(x' p(x((lb))
A.2 TOY DATASETS
To add bias to each samples task from our 2D toy datasets, we first randomly choose an axis (X or Y) for each class and then slice the datapoints in half randomly. We then sample the support set from the
12

---

Published as a conference paper at ICLR 2022
chosen biased subset and leave the rest of the remaining points for the query set. Each sampled task calculates the mean and variance from the support set, which are then used to normalize all instances in S and Q.
Dataset
N- Way
K-Shot
Circles
2
Moons
2
Gaussians
10
10
A.2.1 META MOONS
For the Meta Moons dataset, we randomly invert the classes to make sure that the class indices appear in random order for each task. We add random amount of Gaussian noise to cach moon with uniform standard deviation in the range of (0, 0.25).
100
5.
300
0.0
320
438-
Figure 7: Random task samples from the Meta Moons dataset.
A.2.2
META CIRCLES
For the Meta Circles dataset, we randomly invert the order of the classes that the inner circle and the outer circle are not guaranteed to appear in the same order on every task. We inject a random amount of Gaussian noise into the data, with uniformly random standard deviation in the range of (0, 0.25). We also randomly choose the scale factor between the size of the inner circle and the outer circle, which is uniformly random in the range of (0, 0.8]
teb:1
%.
:
-20
Figure 8: Random task samples from the Meta Circles dataset.
A.2.3 META GAUSSIANS
The task construction of the Meta Gaussians dataset requires that we construct random positive semidefinite covariance matrices for each class. We first uniformly sample N 2 x 2 matrices in the range U-,1 1) and perform a QR decomposition to extract orthonormal matrices Q. We then sample random diagonal D U(0,) 1), and construct the final matrix QDQ which is positive
13

---

Published as a conference paper at ICLR 2022
Table 2: Accuracy, ECE, NLL, and OOD AUPR different n-way k-shot classification problems on Omniglot-C C dataset which contains 17 different corruptions at different intensity levels. MAML/Reptile both utilizee Reptile Norm' instead of transductive BatchNorm
Accuracy T
N
Model
S-way
S-way
I-shot
way
20-way
S-way S-sbot
5-way
20-way
20-way
MAML
65.02217.91
61.721 16.96
23.17
11.35::22.7
3.658:12121
1.528:00. 369
5.96213.822
3.66811 .73
Repoile
61.29:118.61
60.01 17.63
46 23-755
43.42::22.13
2 1.300
1,571 796
3.957+2.357
2.8641: 1 380
Protonct
6091+19.13
58.15+194 19.6
26-20-25.21
43.11=25.55
6.526+3-800
6.539444003
10.70++5..60
9.119-444
Protones-SN
6008+19:47
57.64+
7.1894-4.261
6.541+39900
11.200+6.166
8.446-+-45590
ProtDDC
60.31:19919
19.03:19:57
45.75-25.35
43.90::25:22
10:45:47,143
10.428::71196
18.01419,740
17,039 935
ProtoSNiP
19.18++99.66
57.03+19991
46 26.42-25.12
44.37=24.85
2.534++1.302
2,015=0:967
6.409+3.116
4:151 982
(Diag)
60.77411912
33.98-25.33
43.05.:25.5
1.20510.547
2.16611.141
Ours (Rank 1)
59.88119955
19.15
15:17
43.12:225.5
1000010.088
1.312:0 595
2.511:1.2112
3.56/11 621
Our (Rank 2)
19.68:19.61
56.61
45. 45.53=2555 52
43.28::25.45
1045:10.405
L.31420.590
1.14
3.787 1.760
Our (Rank 4)
60.28119.51
59,141199
05.87 153722550 25
42.38.225.00
1008.10.510
1.26310.550 580
2.52841. 1.16
4,01911 1877
39.69119.69
19.21119963
15.93
43.53:25542
1005:10.501
1.32110613
1.16
3/474116600
BCE L
OOD AUPPR
MAML
28.10=15.25
11.74
33 22.64:117 17.35
13.95
0.602+000060
0,683+0075
0.440=200032
0.484
Reptile
19.98-19.77
0.672100005
Protonet
34.08117.022
17.69
13.57-20.88
43.47120069
0.86710.166
0.853:00 163
0.8761101172
0.863100
Protonei-SN
35.12:17.46
17.27
43.75
42.06:120.10
0 1.869110.166 168
0.857110 164
0.875:10.171
0.862.10 166
ProteDDU
27.84214.19
33.79116.61
18.62-18.16
40.83=:19.55
0.6754:00083
0.55210.044
0.64710.074
0.57910
ProtoSNGP
29.261111.33
14.00
10:12=19..66
16.13
0.87511.112
0.85510.163
0.87810.173
Ours (Ding)
5.87+2.55
11.15=4993
9.81=4.36 36
21.99=9..5
0.809--00.11 169
0,855+0 164
0.876+00.11 172
0.863+0
Ours (Rank 1)
6.10:22.87
11811:996
11 5,
20.26:18.66 68
0.869.10.16
0,850:0 162
08744:0.11 171
0,86310 166
(Rank
662:4:3.09
12:38-5.11
01:4229444
0.853+00
0.874-0.11
0.862.00
(Rank-u 4)
7.10+3.40
11.17=495
1087-44.755
0.568-+0.
0.851+0 162
0.874+0.117
0.86440
Ours (Rank 9)
7.0543.42
12.89 60
10.61:44.84
20.56:49.22
0.868:+0.169
0,851
0.876:b0.172
0.863+0.106
This leads to the distribution of each class being an elliptical multivariate Gaussian distribution.
Figure 9: Random task samples from the Meta Gaussians dataset.
A.3
EXTRA RESULTS
We provide extra results on the MinilmageNet-C C and Omniglot-C dataset here. Tables 2 and contain results averaged over the whole corrupted dataset, including the natural test set and all 5 levels of corruption
A.4 SET ENCODING RELATED WORKS
Set encoding functions require special end-to-end design considerations such as obeying permutation invariance w.r.t. the input set f([X1,Xy X,. = Xr(),
for any random permutation of indices π (.). Likewise, the intermediate latent representations must satisfy permutation cquivariance such that ((X+(1),Xxrr) ..... (x(n))) {f (X), f=(2) (X), .... fx(1) (X)).
X,,)
Deepsets (Zaheer et al., 2017) first proposed basic adaptations of linear and convolutional neural networks which obey the above required properties and have the addition of sum decomposable (permutation invariant) pooling function and decoder to match the requirements of the given task. As sets can have complex interactions between elements, it may be beneficial to model pairwise interactions between set elements. The Set Transformer (Lee et al., 2019) uses a transformer architecture with self attention model such pairwise interactions between set elements. As
14

---

Published as a conference paper at ICLR 2022
Table 3: Accuracy, ECE, NLL, and AUPR for different n-way k-shot classification problems on the MinilmageNet-C dataset which contains different corruptions at different intensity levels. MAMl/Reptile both utilize 'Reptile Norm' instead of transductive BatchNorm
Accuracy
NLL
Model
5-way I-shot
5-way 5-sbot
S-way L-shot
5-way 5-shol
MAML
$1.9447.85
30. 10..0222..9
L.720=0.242
1.748=004422
Reptile
13.07:17885
19. 12.84
L.50220.15
1.58 339
Prosonet
33.43:28.0 49
14,05
L80120.323
0.123
Protonat-SN SN
32. 02.79+8.55
40.95-14.21
1836+-0.44
2,112=0..80
Pro
33 82.6228822
41,46114,35
1990510.655
2,180L0,,122
ProoSNNGP
33.6653.05
41.20=13.06
1.899:0,6244
68
1556610,158
4830
Ours
33. 33.1998..4
40:89:13.00
1.77560.171
1.09
Our (Rank 2)
40
3.87
1.571+0.177
1.859-04456
Ours (Rank 4)
32.52.18445
41.24111..8 13,84
1.59110.19
1896610:466
Our (Ruak 89
32 41.55.55
40.33-133.77
1.581=0.175
L.644=00442
BCB
AUPR
MAML
19:62.19.02
24.03.11.22
0,53610,083
0.628L0,
Regtle
12.92:43384
17,60=8.55 98
0.75620.151
0.749:00.12
20.78:+8.06
17.75-13.66
0.637=0.004
0,579=00077
21.66:19.26
0.572L0.088
ProsDDUU
69
13.13-13.71
0.500=0:041
ProvSNGP
20.30:1..28
25.96: 12.19
0.63620.080
0.653-00089
Ous (Diag)
8.13:23.72
15,57=7.32
0.636220.191
0,5
Durs (Rank I)
0.02:33.33
17.27:8842
0.625-0.087
0.574: 0.067
Ours (Rank 2)
9.27.1..02
16.2717.33
0.63210.087
0.581
Our (Rank 4)
9.87-5.22 29
75.85
0,629=0.085
0.562=0.058
Ours (Rank 8)
9.27.14.38
15.9017.50
0,63710,092
0,504
Table 4: Accuracy, ECE, NLL, and AUPR for different n- n-way k-shot classification problems on the Omniglot dataset. All metrics are measured the natural test set except AUPR/AUROC which is measured using random classes which are different from the classes the support set. Our model maintains competitive performance for ID on all metrics. MAML/Reptile both utilize 'Reptile Norm" instead transductive BatchNorm
Accuracy
Model
S-way S-sbot
S-way I-shot
20-way S-sbol
20-way 1-shot
S-way S-shot
S-way I-bhol
20-way S-shot
20-way 1-shot
99.51=0006
0.23
91.97:10.27
0.015:20002
0.104::0 0016
0.078:20.115
0.289:00.013
Reptile
98.55
95.72
96.50:10.07
90.95110.47
0.054:10.002
0.1501000010
0.142:10.002
0.365110.115
Protonct
99.65-0002
96.21-0.16
99. 29.0
97. 97.1710009
0.013:10.002
0.059:100007
0.027:10.006
0.087:10.008
SNotonet-SN
99.67=00.04
98 98.26=0.12
99. 260006
97.51
0.013:10.00:
0.0611000077
0.029:100006
0.086:10.011
ProtoDDE
99.70=00.5
98. 01-1011
99.28-00.05
97.54++0.16
0.010=00.002
0,08++0.010
0:027-+0.005
0.085-+0.010
ProteSNGP
99.65:00.07
98.23=0008
99.23-00.06
97.41:2:011
0.012:+0.003
0.0542:0.004
0.029:10.006
0.085
Ours (Diag)
99.64200.06
98.21-0023
99. 20.266::000
97.49:10.09
0.020:10.002
0.064200.005
0.032:4-0.00 002
0.089-00.006
Ours (Rank
99.63-0006
28.11:0.12
99 20:20900008
97.61
0.020=000002
0.067:0.005
0.031-10.000
0.086-10.007
Ours (Rank 2)
99.62=0.06
98. 08.30-00.23
99. 30--0.066
97.56-2-.110
0.020++0.002
0.064:00.010
0.031
0.087
Ours (Rank 4)
99 99.66:20.04
98 07
99, 00007
97.56:20.17
0.019:00002
0,060:20.005
0.032:20:003
0.08
99.64120002
96.35
99.32.0.04
97.63.10.16
0.01910.001
0.05910.004
0.030:10.003
ECE
OOD AUPR 1
MAMI.
0,05=0003
1.06--0.12
1.39-0.89
4.93-0.68
0.856-+0.006
0.799+0.010
0.622-+00
0.578-+0.007
Reptile
1.64::008
3.95::044
3.21=0.11
8.73-0.12
0.801
0.813=000155
0.591:10.002
0.579.10.00
Protonet
0.09100022
0.54100.09
0.19110.04
0.35
0.99411.001
0.9771-00001
0.990110.00
0.974110.001
Protonct-NN
0.09::0.44
0.51:00.17
0.2100003
0.39-0.11
0.99110.000
0.977100002
0.990.10.000
0.975:10.000 0.002
ProtoDDU
0.07::0.02
0,475:10.004
0.496:1:0000
0.496:10.002
ProteSNGP
0.09:00044
0.15::0.04
0.14=00.0
0.04
0.994-+00001
0.977=000003
0.989-+0.001 001
0.972-+0.002
(Diag)
1.07=-0.12
2.02+0.12
L11=:0.05
1.58-0.26
0.994-00.00
0.976--0.000
0.990-+0.00 DO0
0.974--0.001
0.71
0.994:000001
0.9762-0.002
0.990=10.001
0.974-20.001
Ours (Rank 2)
1.07::0.10
2.11::0.38
1.13::0.00
1.47-0.28
0.994-0001
0.977:000004
0.990 DO1
0.975-20.002
Ours (Rank 4)
1.00--0.11
2,00-540.45
1.16=0.06
1.52-0.12
0.994+0.000
0.977=90.002
0.990-H0: D01
0.975-+0.001
1.8810017
1.10:0.10
1,4=0.111
0.994:20000
0,97820.002
0.990:20.001
15

---

Published as a conference paper at ICLR 2022
Table 5: Accuracy, NLL, ECE, and AUPR different n-way k-shot classification problems on Mini- ImageNet dataset. All metrics are measured on the natural test set except AUPR/AUROC which is measured using random classes which are different from the classes in the support set. Our model maintains competitive performance for ID data on all metrics. MAMI /Reptie both utilize 'Reptile Norm' instead of transductive BatchNorm
Accuracy
NLL
S-way 1-shot
5-way
1-shot
5-way
46.1311:19
1.297:000055
0.001:0.017
Reptile
47.292121
1.29740:023
0.967=0.011
Photenet
+0.1
0.830-0.008
Proconet
47.47.10.00
68.00.00.99
1.27910.017
0.82010.016
ProcoMMUU
42.5740153
(4.31200.30
1.24620.006
0.836=00.066
ProtSNNP
40 00.55+00.00
e6. 00.80:0.88
1:200:0.012
0.840
Ours (Diag)
48.31
06.12.11.76
1.27740.023
0.857=0.044
Dara (Rink)
48.57 48.57+00.56
66, 046666
1.267+00013
0.899-0.016
Ours (Rank 2)
18.08.10599
67.15
1.27 10.021
0.853.0.017
Our (Raak
47.26x:0..2
1.274=0.030
0.808=0.0010
1.658
1,272:0,000
0.873:0.038
BCB
AUPR 1
MAML
3.39::1.00 L.0
2.79=0045
0.508-00000
0.544=11.005
Reptile
4 66-500.76
1.64:0.23
0.517+0.006
0,542::0.00 0.00
Prosonet
5.6211.00 1.8
4.0910.86
0.0010.0000
0.596.20.006
Protond-SNN
6.77+1181
3.22=0.96
0.08--00011
0.502=00002
ProedDDU
7.3311.44
3,4610.55
0.47310.006
0,4791000001
ProcSSNGP
4.0620084
0.021:00.07
0,667::0.004
6.32::2.66
0.607:0.011
0.600:0.0013
8.9911.00
6.7720053
0.60500:007
0.611:0.008
Oara (Rank 2)
7.50+2.30
7.92-1.34
0.610+0.000
Ours (Rank 4)
7.5712.45
8.02..2.19
0.800.10.005
0,509 0.000...00
Ours (Reak 89
10.09:5:.04
8.80=
0.009::0:010
0,500:10:005
transformers have a quadratic complexity w.r.t. input set length, it may not be possible to process a large set with transformer and maintain permutation invariance, if the set will not fit into memory. Therefore, recent works have also further explored how to make an attentive set encoder which can process sets in batches (Andreis et al., 2021) while maintaining the above requirements of set functions.
For our model, we chose to use the set transformer architecture, as it models pairwise interactions between elements which is an implicit requirement of construction a Gaussian covariance matrix. Therefore, it has the proper inductive biases needed to satisfy our requirement of predicting low rank covariance factors given an input set of features.
A.5 EXTRA TOY RESULTS
In Figures 10, 11, 12, 13, 14, 15, and 16 we provide extra qualitative results on toy dataset covariances and entropy surfaces. In Tables 6, 7, and 8 we provide tabular results of all toy experiments, showcasing the differences between in-distribution data and random uniform OOD noise.
In Distribiution NLL 1
Out of Distribiution AUPR 1
Model
Accuracy t
ECE 1
ECE
AUROC
Protonet
97.02
0.171
2.21+1.40
08.19990.9
0.99910.000 0.00
0.931=0.012
ProtonetSN
97.31+1.54
0.140++0008
2.38+11.88
48.40+40.64
0.999=0.000
0.932+00.006
Proto DDU
96.04-12.74
0.158
2.43
49.43
0.97610.000
0.119100013
Proto SNGP
97.22+1.119
0.046
4.81+2.63
45.74+4.19
0.995+0.001
0.684+00066
Ours (Diag)
96.82-11.09
0.167:10.06
5,0911.18
15.66:12.65
0.99910.000
0.937-00006
Ours (R
9..86+1.442
0.157
4.21+1.77
20.60+2.24
0.999+0.000
0.934+0.008
Ours (Rank-2)
9..5
0.162++0.02
5.13+1..3
17.7442.76
0.999=0.000
0.939+0.006
Ours (Rank-4)
96.90+1.15
0.157+00033
4.69:11.2
18.49442.62
0.999=0.000
0.937-00.006
Ours (Rank-8 8)
96.6911.33
0.17110.047
4.7411148
19.75+3.53
0.99910.000
0.035
Ours (Rank-16)
96.73++1.28
0.161+0.057
3.80+1.87
18.47422.0
0.999+0.000
0.939
Ours (Rank-32) 32)
96.73:11.28
0.170:1.002
4.49.1101
18.2243.26
0.99910.000
0.009
Ours (Rank-64)
9..73+1.55
0.159+0.044
4.43+00.9
17.90+2.49
0.999+0.000
0.008
Table 6: Tabular results from the meta-moons toy experiment
16

---

Published as a conference paper at ICLR 2022
Accuracy: 1.00
NLL 0.04
0.07 IDI00D
(a) Meta Circles (ProioMalalanobisFCC) From left to right: entropy surface, covariances class 1-2
Accuracy: 0.96
0.10
ID/OOD: 0.12 ECE ID/00D 0.04 10.19
(b) Meta Gaussians (ProtoMahalanobissFCC From left right: entropy surface, covariances for class 1-2
Figure 10: ProtoMahalanobisFCCre model performance on the meta-moons and meta- meta-circles toy datasets.
In Distribiution
Out of Distribiution
Model
Accuracy 1
N
ECE |
ECE
AUPR 1
AUROC
Protonet
89.12:1..92
0.343
4.33:10.83
81.84110.78
0.9991100000
0.950:10.009
ProtonetSN
88.93=4.112
0.342:10.098
4.23+:0.1
82.42+00.7
0.999=0.000 0.00
0,951=0.008
Proto DDU
90.45
0.267+0.061
3.52:10.95
84.48:10.41
0.969=0.001
0.175=+0.003
Proto SNGP
89.63
0.323
7.394:2.71
62.60:16.57
0.99910.000
0.916+0.014
Ours (Diag)
90.61
0.27910.075
7.2411.57
42.55.10.98
0.99910.000
0.95410.005
Ours (Rank-1)
91.01+2..65
0.271+0.062
7.12+1.00
42.77+1.62
0.999+0.000 0.000
0.954+0.005
Ours (Rank-2)
91.31
0.26910.064
7.5911134
1.91
0.99910.000
0.955+0.004
Ours (Rank-4)
90.96::2.76
0.272++0006
6.68
1.83
0.999#0.000
0.955+00004
Ours (Rank-8)
90.99:12.52
0.268-10.065
6.74:11.2
43.14:11.49
0.999100000
0.955+00004
Ours (Rank- 16)
91.01+2.88
0.264::0.065
6.96111.2
42.93:+1.44
0.99910.000
0.955+0.005
Ours (Rank-32)
90.45++3.22
0.267+0.069
6.35+0.27
42.89++1448
0.999+
0.956
Ours (Rank-66 64)
90.83
0.264-10.064
6.7910.83
42.67
0.99910.000
0.95610.004
Table 7: Tabular results from the meta-Gaussians toy experiment
In Distribiution
Out of Distribiution
Model
Accuracy 1
NLL
ECE +
ECE 4
AUPR 1
AUROC 1
Protonet
94.45
0.195
3.49.11.677
49.19-10.25
1.000-10.000
0.952-10.007
ProtonetSN
94.53:12.49
0.185-10.098
3.03:11.58
49.17:0.21
1.000=0.00 0.00
0.952+0.007
Proto DDU
95.02-11.79
0.165
3.624:2.27
48.87-10.18
0.972-10.00
0.072+00.008
Proto SNGP
94.49:22.09
0.192:10.071
6.05133.56
45.14:1..08
0.99211.001
0.68310.055
Ours (Diag)
94.24+33.8
0.215+0.133
4.11+0.62
14.64+44.32
1.000+0.000
0.954+0.011
Ours (Rank-1) I)
94.08-14.62
0.214110.5$
4.3411.46
19.0411.21
1.00010.000
0.953-10.013
Ours (Rank-2)
94.53++4.27
0.192+0.114 148
3.54+1.52
18.224:3.72
1.000=0.000
0.954+00013
Ours (Rank-4)
94.12:14.68
0.209
4.27+1183
19.51
1.00010.000
0.9541100013
Ours (Rank-8) 8)
94.00=4..69
0.194++..13.
3.80+1.45
19.37+4.42
1.000
0.955+0.014
Ours (Rank-10 16)
94.12:14.40
0.205
4.12111.93
20.59+5.447
1.000=0.000
0.955-0.013
Ours (Rank-32)
93.84414.66
0.193+0.141
3.48+11.54
20.50++5.56
1.000+
0.954+00011
Ours (Rank-64)
94.16.14.61
0.196:10.466
3.46111.11
19.3316.30
1.00010.000
0.95510.014
Table 8: Tabular results from the meta-circles toy experiment
17

---

Published as a conference paper at ICLR 2022
Accuracy: 0:57 NLL 0.35
Figure 11: ProtoMahalanobisSCC model performance the Gaussians toy dataset. From the top left: Entropy surface, covariances for clases 1-10
Accuracy: 1.00
Accuracy: 1.00 +-
ID:00D: 0.00
ENT ID/00D: 0.46 ECE ID/0OD 0.18 0.30
ECE ID/OOD 0.50
(a) Meta Circles (DDU). From left right: entropy surface (distance), entropy surface (softmax sample), covariances class 1-2
Accuracy: 0.97
Accuracy: 0.97
ID/00D: 0002 ECE ID/DOD 0. 6.50
ID/OOD: 0.50 ECE ID/0OD 0.18 0.25
(b) Meta Moons (DDU). From left right: entropy surface (distance), entropy surface (softmax sample), covariances class 1-2
Figure 12: Proto DDU model performance the two toy meta learning datasets.
18

---

Published as a conference paper at ICLR 2022
0.84
NLL.0.44 0.44
NLL 1,84
ENT ID/O0D: 0.36 0..1
ENT ID:poD: 2.26. 1-29 10:000
Figure 13: Proto DDU model performance on the Meta Gaussians dataset. From the top left: Entropy surface (distance), entropy surface (softmax sample), covariances for clases 1-10
Accuracy: 1.00
Accuracy: 0.96
Accuracy: 0.81
NLL 0.00
NLL 0.15
NLL 0.34
ID/000: 0.01
0.37/ ID/0G0
A
Figure 14: Protonet model performance the three meta-toy datasets. From left to right: Meta- Circles, Meta-Moons, Meta-Gaassians.
Accuracy: 0.99
Accuracy:
ID/DOD: 0.47 ECE ID/OOD 0.18/ 0.32
ENT ID/DOD: 0.04 00.04 ECE ID/0OD 0.01 0.49
(a) Meta Circles (SNGPProtoFC), From left right: entropy surface (softmax sample), entropy surface (distance). covariances class 1-2
Accuracy: 0.97
Accuracy:
ID/DOD: 10.50
ENT ID/DOD: 0.18
10.19 ECE ID/OOD 0.05 0.44
ECE ID/OOD 0.19 0.25
(b) Meta Moons (SNGPProtoFC), From left right: entropy surface (softmax sample), entropy surface (distance). covariances class 1-2
Figure 15: SNGPProtoFC: model performance on the meta-moons and meta-circles toy datasets.
19

---

Published as a conference paper at ICLR 2022
Accuracy: 0.53 R
Accuracy: 0.52 +5
NLL 1,04
NLL 0.48 0.48
10/000:
ID1DOD
Figure 16: SNGPProtoFC model performance the meta- vett-Gaassiias toy dataset. From the top left: Entropy surface (softmax sample), entropy surface (distance), covariances for clases 1-10
A.6 FURTHER IMPLEMENTATION DETAILS
Positive Diagonal Constraint In order to constrain the diagonal A Proto Mahalanobis models (Equation 5) to be positive mentioned in Section 3.2, utilize truncated sigmoid function = m01 (2)). We truncate the values in order to avoid extreme values during the inversion.
SNGP DDU Both SNGP (Liu et al., 2020a) and DDU (Mukhoti et al., 2021) were originally designed under the assumption that entire dataset would be used in the final pass to construct feature covariance matrix. Given that few-shot-learning contains a limited number of samples for each task, compose the feature covariance a positive constrained (via softplus) meta learned parameter. A can be seen as a shrinkage estimation (&A (1 6) < for low sample size, with meta learned mixing coefficient 6.
diagonal + low-rank factor A Φ T where A is
In order to extend SNGP to work in the few shot learning scenario under the prototypical network Snell et al. (2017) framework, had to modify the original algorithm by replacing the last linear layer with the embedding layer and centroids used by prototypical networks. Empirically, we found that using the SNGP logit-normal inference procedure led to severe performance decrease, therefore our results utilized Mahalanobis distance instead.
OOD AUPR/AUROC In order to evaluate the OOD AUPR/AUROC metrics in the supplementary tables, utilize the method proposed by Liu et al. (2020b). Specifically, the total energy in the logits log Σ. exp(22 2;) as the score when evaluating AUPR/AUROC.
Optimizers
All models are trained with the Adam (Kingma & Ba, 2014) optimizer
A.7 ADDITIONAL EIGENVALUE DISTRIBUTIONS
The eigenvalue distributions highlighted in section 4 exhibit the most diverse of eigenvalues. However, the eigenvalues ProtoMahalanobis precision matrices become less diverse in the one-shot setting which also where unable to mean center the respective features by class.
20

---

Published as a conference paper at ICLR 2022
1.0
PrawODU
RAML
0.9
Protonel
Beztll
Accuracy
0.8
ProtusiGG
PratonetSM
Ours (Nanic
Ours (D
0.7
0.6
0.5
0.4
Test
Shift Intensity: Omniglot-C (5-way/5-shot)
0.9
Accuracy
0.8
0.7
0.6
Test
Shift Intensity: Omniglot-C (5-way/1-shot)
1.0
Accuracy
0.8
0.2
Test
Shift Intensity: Omniglot-C (20-way/5-shot)
1.0
Accuracy
Test
Shift Intensity: Omniglot-C (20-way/1-shot)
Figure 18: Accuracy boxplots for different variations of the Omniglot dataset
Figure 17: From left to right: covariance, precision, and cigenvalue distribution for ProtoMahalanobis precision matrix Omniglot 20-way/1-shot (left) and 20-way/5-shot (right) experiments.
21

---

Published as a conference paper at ICLR 2022
(Dlag)
ProteShiGP
Ouns (Ranle3)
10
5
Shift Intensity: Omniglot-C (5-way/5-shot)
20
NLL
10
Test
Shift Intensity: Omniglat-C (5-way/1-shot)
20
15
Test
Shift Intensity: Omniglot-C (20-way/5-shot)
Shift Intensity: Omniglot-C (20-way/1-shot)
Figure 19: NLL boxplots for different variations of the Omniglot dataset
22

---

Published as a conference paper at ICLR 2022
0.7
Mubro
Prsuapu
Protonet
Reatll
Accuracy
ProtometSN
Ours (D
Prokettcc
(Ranic3)
0.5
Test
Shift Intensity: MinilmageNet-C (5-way/5-shot)
Accuracy
0.40
0.35
Test
Shift Intensity: MinilmageNet-CC (5-way/1-shat)
Figure 20: Accuracy boxplots for different variations of the MinilmageNet dataset
Veived
PresDDU
WANL
Protee::
Deptle
Protone:SN
Ours (Dlog)
ProubbiG
Durs (aakk1))
Test
Shift Intensity: MinilmageNet-C (5-way/5-shot)
3.0
2.0
Test
Shift Intensity: MinilmageNet-CC (5-way/1-shot)
Figure 21: NLL boxplots for different variations of the MinilMageNet dataset
23

---

Published as a conference paper at ICLR 2022
Table Convolutional architecture used for MAML/Reptile Omniglot
Layers
Conv2d(1, 64, pad=1, stride=2) - BarchNom((rpiilooomm/nn))) - ReLU
Conv2d(64, 64, pad=l, stride=2) Baich/Vom//rppllcmmmm//// ReLU
Conv2d(64, 64, pad=1, stride=2) BachhNommiepiilenomm-nnnu ReLU
Conv2d(64, 64, pad=1, stride=2) BauchNoomfreppiinoon--Tuuue ReLU AveragePool(2) FC(644, nway)
Table 10: Convolutional architecture used for MAML/Reptile MinilmageNet. Reptile uses 64 filters instead of 32.
Layers
Conv2d(1, 32, pad=1,
BatchNommkeeplleennneeTnnnu - ReLU MaxPool2d(2)
Conv2d(32, 32, padel, stride=2) - BatchNormeeppieeoom--TTe - ReLUU MaxPool2d(2)
Conv2d(32, 32, pad=1, stride=2) BachhNommdeppllknomeennu ReLU MaxPool2d(2)
Conv2d(32, 32, pad=1, stride=2) Buchhoorddeppilcoom==Tte - ReLU MaxPoolDd(2)
Flatten
FC(1600, away)
A.8 ADDITIONAL BOXPLOT RESULTS
Figures 18, 20 show extra boxplot results for accuracy while Figures 19, 21 show negative log likelihood.
A.9 ARCHITECTURE DETAILS
Tables and 10 show the backbone architectures for MAML/Reptile Omniglot and MinilmageNNet respectively. Table 11 shows the backbone architecture for all Protonet based models.
A. RUNTIME ANALYSIS
In Tables 12, and 13 we provide a runtime analysis of different variants of our models and baselines. Linear models are evaluated by using the mean and standard deviations from 50 iterations of both training and inference on the MetaMoons dataset. Convolutional models are likewise evaluated on 50 iterations of the Omniglot dataset. All models were evaluated on single GeForce GTX 1080 Ti GPU. SNGP/DDU also utilize the matrix inversion outlined in Equation 8.
Mahalanobis models show slightly better (Linear) similar (CNN) latency to SNGP/DDU for diagonal and rank-1 variants. Latency increases as the rank goes higher due to more factors and more iterations required for inversion and log-determinant calculations. Comparing Protonet, Protonet-SN, and other variants which need to construct a covariance, we can see that constructing the covariance matrix adds a cost which is roughly equivalent spectral normalization.
Table 11: Convolutional architecture used for Protonet Models, Plain Protonets use spectral normalization
Layers
ConttralNoemm(Cny22111 64, pad=t, stride=l), residul=TTue, c=3) - BatchNorm( - ReLU + Drapout)) - AveragedPol12d12)
SpectrlNormm/onv222 64, pad=1, stride1)), residual-True. c=3) BatchNormo + ReLU + Dropoutt) - AveragePool2d
SpeatralNormi(Cnn Conve211 64, pad=l, stride=)), residualETrue, c=3) BatchNormO ReLU - Dropout) AveragePool2dd2)
CottalN 64, pad=1, strid=1), residual-True, c=3) - BatchNormO - ReLU + Dropoutt) - AvcragePool2d(22
Flatten()
FCIfeatures, away)
24

---

Published as a conference paper at ICLR 2022
Model
Train Iteration (ms)
Eval Iteration (ms)
ProtoMahalanobis-FCC diag
10.33++040
2.95+00.20
ProtoMabalannois-FCC Rank-1
10.96:0..3
3.11
ProtoMaablannois-FCC Rank-5
13.06:t0.44
3,84
ProtoMahalanobis-F Rank-10
15.65=+0.35
4.63:10.20
ProtoDDU-FC
11.86=t0.4
3.97
ProtoSNGP-FC
12.32:L0.38 2.83 6.78-2034
4.02.10.22
Protonet-FC
0.86:10.09
Protonet-FCC SN
1.91
Table 12: Runtime analysis of linear variants of models
Model
Train Iteration (ms)
Eval Iteration (ms)
ProtoMahalanobis Diag
11.84:10.69
3.61::0.35
ProtoMahalanobis Rank-1
12.5
3.85:10.31
ProtoMahalanobis Rank-5
15.04=0..69
4.55=10.36
ProtoMahalanobis Rank-10
17.72:10.71
5,42:10.45
ProtoDDU
11.39:10.81
3.80+0.29
ProtoSNGP
11.43:+0.65
3.68++0.30
Protonet
3.55 8.03+10.40
1.14+:0.2
Protonet SN
2.42:10.18
Table 13: Runtime analysis of CNN variants of models
A.11 FURTHER EIGENVALUE EXPERIMENTS
In Table 14, we perform further experiments and analysis into the behavior the low rank covariance encoder outlined in Section In this experiment, we first obtain the predicted precision matrix and perform an eigendecomposition J where
analyze the significance of the eigenvalues of the precision matrix.
A = QAQ-AQQ-- E R N We then construct a set of alternate precision matrices S {A') each set element is recomposition A = QA'Q-1 where A has one eigenvalue reset to 1. We then compute the final Accuracy, NLL, and ECE once for each matrix S. If the predicted eigenvalues are due to arbitrary етror noise, then we would expect to see that the test statistics would arbitrarily improve for some precision matrices S.
Instead, in Table 14 we see that the precision matrix which is predicted from the Set Transformer gives the best results on test set cases, showing that the predicted values are necessary for the given solution. This experiment utilizes Omniglot 5-way/5-shot and the ProtoMahalanobis Rank-1 I variant.
25

---

Published as a conference paper at ICLR 2022
Neted
(2)ag)
OKT-bnces
(Ranic))
Accuracy
0.8
0.6
Test
Shift Intensity: Omniglot-C (5-way/5-shot)
0.30
0.25
0.20
0.10
0.05
Shift Intensity: Omniglot-C (5-way/5-shot)
1.0
0.9
Accuracy
0.8
0.7
Test
Shift Intensity: Omniglot-C (5-way/1-shot)
0.5
0.3
0.1
Shift Intensity: Omniglot-C (5-way/1-shot)
Figure 22: Extra results comparing Deep Kernel Transfer (Patacchiola al., 2020) the Omniglot dataset. In our experiments, DKT showed a large variance in performance between tasks. In the 5-way'l-shot case, calibration on corrupted data comes the expense of underconfidence on in distribution data.
26

---

Published as a conference paper at ICLR 2022
Σ Matrix
Accuracy
NLL
ECE
better%
predicted level 0 E
99.5510.04
0.02:10.00
1.21
100%/100% %/100% %
modified level 0 Σ
97.98:10.23
0.09:10.01
3.31
0%/0%/0%%
predicted level Σ modified level | Σ Σ-1
63.43::1.49
0.97
5.22
58.44:1:..1
1.30
9.06
0%%0%%0%%
predicted level 2 Σ modified level 2 Σ
+
1.04
6.92 1.43
100 110%%%%%0000%
51.17=2.12
1.54
11.92:2.04
0%/0%/0%%
predicted level -11 modified level 3 Σ Σ-1
1..45++133
1.21
6.62:1
100 100%/100 % 9/100% %
1..3
1.72:10.74
13.51
0%/0%/0%
predicted level modified level 4 Σー
45.07=11004
1.37-10.02
8.34
100 % %/100 %
19.70
1.97=11.13
15.85+2.11
0%/0%/0%%
predicted level Σ modified level 5 Σ-1
40.7310.70
1.46:10.02
10.25
100%/10%% /1000000 % 0%/0%/0%%
36.5411.91
2.09
17.3442.06
Table 14: Analyzing the predicted precision matrix against of modified precision matrices with perturbed eigenvalues. The predicted precision matrix performs better in every instance, showing that the precision matrix arbitrary. This data comes from Omniglot 5-way/5-shot and utilizes the ProtoMahalanobis Rank- variant
27


Reviews:
[{"review_id": "KTjcu_80jQ4", "reviewer": "Reviewer_eieK", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written and easy to follow. Some specific comments / suggestions:\n1. It would be great if the authors could expand the discussion of bi-Lipschitz regularization in the related work section. Some discussion and literature review of set-input functions, e.g., Set Transformer, Deep Sets will also be helpful.\n2. What are the practical ways to ensure that Lambda_c matrix is non-negative, without making too many entries of Lambda_c to be zero?\n3. Is there any benchmark on the computational requirement of the proposed methods, compared to competing methods? The covariance inversion and log-determinant will add computational cost to the methods, and it would be good to add some analysis to show the tradeoff between computational cost (i.e., number of iterations) versus performance.\n4. In the numerical studies, it seems that the proposal performed comparatively better in the low-way low-shot setting. Is this the correct observation? If so, is there any intuition behind this?", "summary_of_the_paper": "In this paper, the authors proposed methodologies to estimate the covariance matrix of embedded features under the meta-learning setting. The authors first showed that the empirical covariance matrix is inappropriate in this setting and that due to the shift invariant property of softmax function, ProtoNet could produce overly confident predictions. \n\nThe authors instead propose to learn the covariance matrix through a Set Transformer structure, where the covariance matrix can be decomposed into the sum of a diagonal component and a low rank component. The authors used scaled energy and bi-Lipschitz regularization to make the methodology more performant. \n\nNumerical studies are performed to evaluate the calibration performance as well as the eigenvalue distribution of several methods.", "summary_of_the_review": "The paper is well-written with clear motivations and easy to follow presentations of the methodology. The proposal is relatively novel, and could be a meaningful contribution to the meta-learning community.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AfFsFJrV0t", "reviewer": "Reviewer_Zz5v", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThe overall presentation of this paper is both clear and straightforward.\n\nThe figures and equations precisely facilitate the reading. \n\nNo noticeable typos were spotted. \n\nThe topic that the authors selected is important and largely overlooked by the research of meta-learning and few-shot learning. The effectiveness of the proposed method is reflected by the quantitative comparisons. \n\nWeakness:\nMy major concern to this paper mainly arises from the use of an additional meta-learned set encoder to predict the task-specific covariance. \nIt has been shown in previous work that some meta-learners have natural advantages in quantifying uncertainty. \n\nFor example, relying on kernel methods, Gaussian processes based meta-learners [1,2] have demonstrated strong uncertainty calibration without leveraging any auxiliary components. \n\nAdditionally, I might be wrong, but I'm curious that if this class-level covariance can be obtained through aggregating sample-wise information so that an additional set encoder can be removed. \nPlease refer to Eq.6 to Eq. 9 of [3] for an example. Although the motivations are different, I believe there is some shared wisdom. \n\n[1] Deep kernel transfer in gaussian processes for few-shot learning, NeurIPS 2020\n\n[2] Bayesian Few-Shot Classiﬁcation with One-vs-Each Pólya-Gamma Augmented Gaussian Processes, ICLR 2021\n\n[3] Amortized Bayesian Prototype Meta-learning: A New Probabilistic Meta-learning Approach to Few-shot Image Classification, AISTAT 2021", "summary_of_the_paper": "The authors explore the uncertainty calibration problem in meta-learning few-shot classification, which is a relatively new topic that was seldom discussed before. \nTo solve the challenges that the few samples in few-shot learning are usually insufficient for providing reliable estimation to the covariance matrices, the authors introduce meta-learning covariance matrices parametrized by a set encoder (implemented as set-transformer). And the authors further introduce scaled energy to parameterize a logit-normal softmax distribution for improving the uncertainty calibration of the softmax classification layer.", "summary_of_the_review": "Overall this paper attacks a very important problem of meta learning and few-shot learning. And considering the quantitative results, the proposed method does achieve better uncertainty calibration. \nI expect further discussion on justifying the necessity of adopting an additional set encoder to achieve this capability. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7IUU9ukg-jk", "reviewer": "Reviewer_QrCN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This submission is empirical: the paper is well-motivated, and the paper adapts existing, popular techniques to engineer a solution to the standing problem (using quadratic discriminant analysis, modeling the covariance matrix as a sum of a diagonal matrix and a low-rank psd matrix, etc.).\n\nOn the downside, there are several theoretical questions that are left unanswered:\n1. Why is the Set Transformer a good choice? On which ground can we justify that the Set Transformer can generate sensible values of Lambda and Phi to construct the covariance matrices?\n2. There are a lot of methods from statistics to estimate covariance matrices from low sample size dataset (linear/nonlinear shrinkage, sparse, etc.). Why can't they be used in this application? Is it possible to show that the Set Transformer is better than these traditional approach (for any possible criteria)?\n3. In page 8, why should the eigenvalue distribution be diversified? It is well known that the eigenvalue can be sensitive to small perturbations of the off-diagonal entries of the matrix. Thus, what we observed (diversity of the eigenvalues) can be just an artifact of the numerical errors from the Set transformer network. Diverse eigenvalues also imply that the condition number is large, which leads to higher numerical errors to the log-determinant term. Thus, I am not fully convinced that a diverse set of eigenvalues is of any advantage.\n\n\nMinor:\n- What is f(E) and min(E) after equation (10)?\n- The abstract and intro focus heavily on bi-Lipschitz regularized NN, however, the exposition in section 3 (except for Section 3.4 which is 10 lines long) relies minimally on this architecture. This emphasis may be misleading in the first read.", "summary_of_the_paper": "This submission focus on improving the performance of uncertainty quantification and OOD detection using bi-Lipschitz regularized neural networks. There are two involved challenges here: (i) we are required to estimate the class-conditional covariance matrix of the latent features. This estimation is challenging under low sample size. (ii) generative classifiers using softmax activation function suffer from shift invariance, and thus may become over-confident for OOD samples.\n\nTo resolve these issues, the paper focus on two innovations:\n\n(i) a meta-learning model to estimate the class-conditional covariance matrices,\n(ii) a modification of the inference procedure with the energy function to detect OOD samples.\n\nThe paper concludes with numerical results showing the performance of the proposed solution approach.", "summary_of_the_review": "In my opinion, the paper is a good engineering attempt to solve a specific problem. It lacks, unfortunately, concrete scientific contributions.\n\n\n======== Post-rebuttal ========\nThe authors have provided further empirical results to justify the use of the ST to estimate the covariance matrix. I have raised my evaluation from 3 to 5. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers all found the work well-motivated in addressing uncertainty, a topic that has not seen much focus in meta-learning and few-shot learning. They describe the challenges well: small sample sizes and OOD shift. They then propose a solution they find works well empirically to overcome these challenges based on a set encoder and an energy function respectively.

The proposal is largely one of engineering components that have been found to work well in the literature. I'm sympathetic to this style of research (particularly in today's neural network research), although the reviewers raise a primary concern about whether the choices leading to the proposal are justified. In particular, two Reviewers argue that there are no clear ablations compared to alternative simpler approaches, and so the approach of selecting a Set Transformer is rather arbitrary. My perspective is that theory provides one sufficient but not necessary angle to do this, and I do find the authors' replies to the two reviewers convincing. In particular, they add a baseline to estimate covariances suggested by Reviewer Zz5v and they describe how the current baselines do in fact use the shrinkage suggestion by Reviewer QrCN.

I recommend the authors use the reviewers' feedback to enhance their submission's clarity and overall quality.
Decision: Accept (Poster)