Task Calibration for Distributional Uncertainty in Few-Shot Classification
Authors: Sungnyun Kim
Se-Young Yun
Venue:  (2021)
Keywords: few-shot learning
meta-learning
uncertainty estimation

Abstract: As numerous meta-learning algorithms improve performance when solving few-shot classification problems for practical applications, accurate prediction of uncertainty, though challenging, has been considered essential. In this study, we contemplate modeling uncertainty in a few-shot classification framework and propose a straightforward method that appropriately predicts task uncertainty. We suppose that the random sampling of tasks can generate those in which it may be hard for the model to infer the queries from the support examples. Specifically, measuring the distributional mismatch between support and query sets via class-wise similarities, we propose novel meta-training that lets the model predict with careful confidence. Moreover, our method is algorithm-agnostic and readily expanded to include a range of meta-learning models. Through extensive experiments including dataset shift, we present that our training strategy helps the model avoid being indiscriminately confident, and thereby, produce calibrated classification results without the loss of accuracy.

Under review as a conference paper at ICLR 2021
TASK CALIBRATION FOR DISTRIBUTIONAL UNCERTAINTY IN FEW-SHOT CLASSIFICATION
Anonymous authors Paper under double-blind review
ABSTRACT
As numerous meta-learning algorithms improve performance when solving few- shot classification problems for practical applications, accurate prediction of un- certainty, though challenging, has been considered essential. In this study, we con- template modeling uncertainty in a few-shot classification framework and propose a straightforward method that appropriately predicts task uncertainty. We suppose that the random sampling of tasks can generate those in which it may be hard for the model to infer the queries from the support examples. Specifically, measuring the distributional mismatch between support and query sets via class-wise simi- larities, we propose novel meta-training that lets the model predict with careful confidence. Moreover, our method is algorithm-agnose include a range of meta-learning models. Through extensive experiments includ- ing dataset shift, we present that training strategy helps the model avoid being indiscriminately confident, and thereby, produce calibrated classification results without the loss of accuracy.
and readily expanded to
INTRODUCTION
With the great success of deep learning over the last decade, there has been growing interest in investigating methods that are more intuitive to mimic human intelligence. One of the desirable characteristics of human cognition is the ability to learn new information quickly. Few-shot learning is problem that requires machines to learn with few examples, and it is usually solved by meta- learning algorithms. Through training over a number of few-shot tasks, the meta-learning model learns to acclimate rapidly to data and perform desired tasks with the help of prior across-task knowledge. Noteworthy approaches include learning metric space (Koch et al., 2015; Snell et al., 2017; Sung et al., 2018), learning update rule (Ravi & Larochelle, 2017; Andrychowicz al., 2016), or learning an initialization (Maclaurin et al., 2015; Finn et al., 2017)
Mcanwhile, calibration is critical in real-life because the model should correctly inform humans or other models of its degree of uncertainty. Misplaced confidence or overconfidence of deep network can result in dramatically different outcomes in the decision-making process, such as during autonomous driving (Helldin et al., 2013) or medical diagnoses (Cabitza et al., 2017). Calibration even more crucial few-shot learning, for given few numbers of data hand, the model is likely to put wrong confidence into the predictions of the unknown data. Also, existing calibration methods, while effective in general classification, are not readily applicable the few-shot learning- Therefore, we present a novel method to measure task-level uncertainty and exploit it to make well-calibrated model.
When the model is meta-trained to solve the few-shot classification, it generates task by randomly choosing classes and sampling support and query examples of corresponding classes. However, be generated, and there some tasks in which it suppose in this way variety of tasks may be hard for the model to infer the queries from the support examples. Common meta-learning approaches force the model to learn anyway, lacking the discussion about task generation. In this study, we design an algorithm-agnostic calibration method for a few-shot classification model by diminishing the learning signal from those ill-defined tasks. This novel training lets the model predict with careful confidence, ultimately obtaining better calibration ability.
To summarize, contributions are follows:

---

D(F) Under review as a conference paper at ICLR 2021
After adaptation to the support examples (blue), inference is made from the query examples (red), which guides the following meta- update. While Task comprises well-distributed Dapt and D Day. Task 2 holds a significant support-query dis- crepancy. We limit the feedback flow from Task 2.
Figure 1:
adaptatton
We propose simple and straightforward method by which few-shot classification model learns the ill-defined tasks in a timid way, so as not to place the wrong confidence.
We extend our approach from an optimization-bant firming that our method is versatile and can be easily expanded to include a range of meta- learning models.
to a metric-based framework, con-
We demonstrate that we can design uncertainty-aware model that is suitable for calibra- tion and robustness.
To the best of knowledge, there has been work that used weighted meta-update in reinforce- ment learning literature (Xu et al., 2019) to prioritize and give different weights for each trajectory. However, our work is the first one that quantified task uncertainty and applied in the few-shot classification framework. Moreover, we validate our work through a dataset shift experiment, which set be more challenging and hardly ever addressed due the degradation of the estimation ability (Snoek et al., 2019).
2
PRELIMINARIES
2.1 PROBLEM SETUP
meta-learning problem is composed of tasks from task distribution p(r), and for each meta- training task T ~ p((T) there is a training set and test set. We denote each as a support set, spr and query set, D(r) respectively. In typical few-shot classification scenarios, each task is comprised of data from N randomly selected classes and k support examples per class, with a total of kNN spr This called N N-way k-shot problem. Also, there is disjoint
examples, {(x(7) [,(²)) ((( == ()())
(7) (/ ) == D Meta-training is an iterative process of optimization on Dary after being seen Pept- This setup resembles learning novel task with only few examples of training data. The adaptation on Pept corresponds to training with the novel task data. Given few examples, meta-learner expected to perform predictions on unseen data, which in setup is Dayy That is, the meta-learner evolves by acquiring general information across the tasks by optimizing on Day.
query set, ((x)
2.2 UNCERTAINTY IN FEW-SHOT CLASSIFICATION
good meta learner should not only perform well on given tasks but should also make reasonable decisions based on aleatoric or epistemic uncertainty (Der Kiureghian & Ditlevsen, 2009). While aleatory irreducible uncertainty from data, including class overlap, observation noise, data ambiguity, episteme arises from the stochasticity of the model itself, and can be reduced by obtaining sufficient data. For example, aleatory is the case when 'dog' images have inherently high noise (e.g., blurred and corrupted), and thus, have the appearance of other class such as 'cat' (known-unknown), Episteme arises when the model not sufficiently trained for it to match the images 'dog'.
There is another source, distributional uncertainty, which is ascribed to mismatch between training and test data (Malinin Gales, 2018). In few-shot classification, this corresponds to the discrep- ancy between support and query set, often caused by intra-class variations (Chen al., 2019). For example, distributional uncertainty may arise when the 'dog' class includes support examples as 'bulldogs' while by chance, the query is 'chihuahua", resulting in large uncertainty of mapping 'chi- huahua' to any class Lunknown-unknoww Typical meta-learning algorithms do not consider this mismatch, which call support-query discrepancy, and naïvely adapt to every observed support
2

---

Under review as a conference paper at ICLR 2021
example. Appendix A provides the formalized description of distributional uncertainty and its lation to the few-shot learning. In this work, we propose a straightforward method of training an uncertainty-awar awwre model (Figure 1), where measure the distributional mismatch within task and constrain the feedback from uncertain tasks.
TASK CALIBRATION METHOD
Despite the importance of distributional uncertainty modeling, the problem setup, which restricts the number of data, renders the model extremely sensitive to the distribution of given data. There- fore, we focus on task-level distributional uncertainty instead of considering that of individual data and models. We suggest that this can be implemented by non-Bayesian approach with appro- priate measure. We propose task calibration model-agnostic meta-learning (TCMAML), a variant of MAML (Finn al., 2017), that estimate task distributional uncertainty during meta-training and utilize to modulate the optimization of global parameters. Thereafter, we extend our approach the metric-based learning model.
3. BRIEF REVIEW MAML
Before introducing our new approach, we provide brief review of model-agnostic meta-learning (MAML) (Finn al., 2017) to promote readers' understanding. MAML learns a proper initializa- tion of globally shared that is quickly adjusted to each task T by a few gradient descent steps. After being adapted support examples, MAML. updates 0 by optimizing on the query set, mini- mizing task-specific loss C, with task-adapted parameter d,, This is summarized the bi-level optimization.
inner-level:
Φ, - --o- (T))
(1)
- Σ V Vol, (Φ, (o,; D()) T
outer-level:
0 0 0
(2)
The meta-update (2) aggregates gradients over multiple tasks, so that the tasks are learned simultane- ously. If we can measure the task uncertainty, adaptive weighting of task-specific losses is possible, ie., L Σt= (Ø,, where T is the number of tasks in meta-batch. Each weight reflects the uncertainty of a specific task. If task 7 has a large uncertainty component, the small value of w, will decrease the contribution of VoL,.
3.2 TCMAML
We suggest that class-wise wise similarity would be good metric to estimate task uncertainty. As the similarity between classes increases, it becomes harder to distinguish one from another. The simi- larity is measured in the embedding space mapped by h the feature extractor before the classifier.
With the pairs of query inputs and labels (ml" 1/4 (T)), we collect the average features for every class h F7 Σ E(gnh"=e) he. (zz(²)), where h maintains feature dimensions. hg. contains the information about the support and query data because the average features of the query inputs are parameterized by the task parameter Φ,, while &, is derived from the adaptation on the support data as (1), Therefore, the relations between support and query data determines h We will go deeper in the next section.
We now define the cosine similarity matrix C,, as for all i.j E (1....,,N)
hs .cs Tepp
[C1s3 = Sc (h ho.
|| hoo.. ceall [ho..
calculates how correlated the two class embeddings are. The diagonal line of C, is always 1, and the rest of the elements are between [-, To map the similarity matrix into a scalar, we use Frobenius norm. For C NxN, the Frobenius norm of the matrix ||C | E Cl From
on, will simply similarity score 8, denote | (lrl..
3

---

Under review as a conference paper at ICLR 2021
In few-shot classification, it is important to carefully consider the task generation. More specifically, we prefer to sample the tasks considering how well the task defined and how much it is reliable to train with. The original meta-loss in (2) assumes equal reliability for uniformly sampled tasks; p(T) = +, Vr. Let ((T)) be our desired task generating distribution. There is no straightforward way sample from ((T), ) but can alternatively use importance sampling.
((T) (0) R B (0) = R p(()) p(T)
+ 1 Σ P(T) p(T)
((()))
= (2()),,0
Now if we sample the tasks randomly from p(T) and compute the importance weight q(7). we can easily get the adaptive empirical risk, which allows training with the prior knowledge of task configuration.
'CMAML estimates ((T) with softmax function over 8, values. Consequently, modify the meta-update rule (2) as follows:
exp(-8./ Wr Σ exp(-8771
0 0 L Vo wr (=(()::
where T' is a scaling factor. Designing the importance weight as above implies large weight on the well-defined tasks, where the measured class-wise similarity is low, and small weight on the ill-defined tasks, where the similarity is high. While the softmax function is an exponential scaling, linear scaling is also possible; w, = (1/s,), E/E 1/==(1/s,, (1/s,). However, we empirically found that the exponential scaling better performs. Refer to Appendix B.2 for linear scaling TCMAML.
3.3 RELATION WITH DISTRIBUTIONAL UNCERTAINTY
The meta-update (5) indicates that for uncertain task, small weight value is multiplied, that its contribution to the updates 0 is weak. In this sense, TCMAML considers task distributional uncertainty by measuring the similarity between the query examples. If we assume a cross- entropy loss, & from task adaptation (1) is an approximation of the maximum likelihood estimate with observed support examples.
Φ arg max
(T) ,),
(ar(T,
D( spi
0
Let Φ.. be the parameter from task where there is distributional uncertainty. This is satisfied when D(F") spt
4.55
has exactly the same distribution as D(F") We aim to maximize p(D
Similantty scares
431
(7") (0) in meta-learning, which is attained when 0 = Φ,,, from (6) and D(=²) ss D(r"' We can assume that make a clear distinction between differ- ent classes the embedded representation vectors should be less correlated. Therefore in task r", the query in- puts' embedded vectors, h..,.cr are less correlated be- cause Φ,- optimally distinguishes D(F")
0:20
4.2
Standard
Poorlyderl
Well-cefined
Figure 2: Average similarity scores with 95% confidence interval. Stan- dard indicates general few-shot tasks. Poorly-defined indicates extremely large support-query dis- crepancy; images from mismatched classes, and Well-defined imitates T² More details Appendix B.4.
In contrast, if there significant distributional mis- match between (
spt and D(r). Φ, is the suboptimal solu- tion to maximize p(D gry 10). because the parameters can- infer D(=)) from ( In the previous example, the parameters adapted from "bulldog" images might not sep- arately distinguish "chihuahua" from another class such as "cat". This implies higher correlation between the embedded vectors hg. Therefore, we can compare the class-wise cosine similarity, Sc, computed using the average features in task 7" and т.
Sc (c cn T ,g)) S Sc Sh V Vi.j
4

---

Under review as a conference paper at ICLR 2021
From (3) and (5), the above inequality leads to smaller weights on the loss function of uncertain tasks. For general task т, significant mismatch between the distribution of D(g) and D( in- creases the inequality gap.
Figure 2 shows the similarity scores of tasks from different distributions with real-world images. In well-defined tasks as r², the adapted model is good at distinguishing the query representations because they are less correlated. On the other hand, in the extremely discrepant case where the classes of support images do not correspond to those of query images, between-class similarities are high because the model cannot infer the query set from observing the support set. General few- shot classification tasks lie between these well-defined and poorly-defiedd ones. TCMAML is thus trained by imposing a prior regarding whether a given input task has high distributional uncertainty or not.
3.4 EXTENSION TO THE METRIC-BASED MODEL.
We emphasize that task calibration approach be easily extended a metric- approach. models such as MAMI,, a metric-based model does not require a process of explicit adaptation to the task. Instead, the model calculates the distances between the query and support data in a non-parametric way. In this study, we use prototypical network, i.e., Pro- toNet (Snell et al., 2017), which is high-performance metric-based few-shot learning method, and propose and apply our task-calibrated version, TCProtoNet.
Unlike optimization-
Using the distance vector of each query to the support prototypes p= [pi...... pN] we collect the average vectors for each class, dø..c F E (aro""=e) d(x!) (T) .p). where (xx)) (=) ..//)) E D and
d(:p) E N hold. The assembly of a similarity matrix using the average vectors and the weight computation are effected similarly as in section 3.2. We empirically demonstrate that TCProtoNet performs better calibration than vanilla ProtoNet. The result implies that in a range of few-shot learning models, our method versatile and applicable to more recent methods such as task depen- dent adaptive metric, TADAM (Oreshkin et al., 2018).
4 RELATED WORK
Uncertainty estimation in deep learning has been persistently studied despite the challenges (Gal, 2016; Lakshminarayanan et al., 2017; Guo et al., 2017: Jiang et al., 2018). Mcanwhile, Depeweg al. (2017) dealt with the decomposition of uncertainty stand and identify their characteristics. Our work also investigates different sources of uncertainty, initially being inspired by Malinin & Gales (2018), which builds Dirichlet Prior Network to explic- itly model distributional uncertainty. We extend their argument to a few-shot learning framework without requiring any additional parameters or networks. In parallel, Sensoy et al. (2018) parameter- ized Dirichlet using the evidence vector to regularize predictive distribution, to penalize divergence from the uncertain state.
Bayesian neural networks to better under-
Few-shot image classification is one of the most challenging tasks for coping with high degrees of uncertainty (Fe-Fei et al., 2003). Estimation and judgment over the uncertainty become challeng- ing when encountered with high-variance data such a large-scale image set. To overcome this issue, several recent studies have relied on designing Bayesian neural networks. These networks use posterior approximation quantify parameter uncertainty, which is a type of epistemic uncertainty. Probabilistic MAML (Finn et al., 2018) is a variant of MAML (Finn et al., 2017), which approxi- mates the distribution over model parameters by optimizing the variational lower bound. Bayesian MAML (Kim et al., 2018) uses the Stein variational gradient descent for more flexible modeling of posterior. Ravi Beatson (2019) and Gordon al. (2019) both amortize approximate inference. Neural Processes (Garnelo et al., 2018) are combination of a stochastic process and neural network that learns the distribution of functions while representing uncertainty, although they are yet to be proven capable of solving large-scale classification problems. The Bayesian models must compute posterior inference, and most approaches use empirical approximation, which usually requires heavy computations. Conversely, vantageous and does not rely on posterior approximation. The TCMAML running time is almost the same as that for vanilla MAMI..
method uses non- Bayesian approach, which is computationally ad-
5

---

Under review as a conference paper at ICLR 2021
5 EXPERIMENTS
5.1
EXPERIMENTAL DETAILS
We mainly used two different datasets for the experiments: mini- Imaieemaeeeet and CUB-200-2011.
mini-ImageNet (Vinyals et al., 2016) dataset contains 100 classes and 600 images per class, We followed the standard protocol which was initially provided by Ravi & Larochelle (2017), splitting (train, validation, test) set to (64, 16, 20) non-overlapped classes.
CUB-200-2011 (Wah et al., 2011) dataset is composed of 200 bird species classes incorporating 11,788 fine-grained images. According to Hilliard et al. (2018), it is divided into (100, 50, 50) classes for (train, validation, test) set.
Thanks to their work conducted on fair comparison in the few-shot classification setup, most of our implementation details follow Chen al. (2019) both in I-shot and 5-shot tasks. Every accuracy reported is the test accuracy, which was evaluated from 1000 tasks randomly drawn from the test set. By default, we used Conv-4 backbone, CNN with four convolutional blocks (Vinyals al., 2016), to not overestimate the performance by using deeper networks. Furthermore, MAMI. and TCMAMI. were trained with a first-order approximation of derivatives for the efficiency (Finn et al., 2017).
Dataset Shift. Additionally, we conducted dataset shift experiments (Chen al., 2019), in which the train set is the entire mini-ImageNet dataset, and validation and test sets are from CUB-200. mini- ImageNet - CUB-200 200 implemented with ResNet-18 18 backbone (He al., 2016) for better convergence. The dataset shift is significantly more challenging than a typical few-shot classification setting because the model has predict the examples from completely different distributions to those of the train set. This is an intriguing situation from a practical perspective since for most real-world problems it cannot be assumed that new tasks would be from among those with which the model is familiar. However, not much research on uncertainty in few-shot learning has been attempted on shifted datasets.
5.2 CALIBRATION RESULTS
Our main contribution is well-calibrated classification. The model is well-calibrated when it makes its decisions based on the predicted uncertainty. The correctness of a perfectly calibrated model will match the confidence level (the largest value of the softmax output). For example, the model miscalibrated or overconfident when the confidence 0.8 on average, but only 70 out of 100 samples correct. The quantified measures for miscalibration include expected calibration error (ECE) and maximum calibration error (MCE),
Bml aacc(B.) (m))) MCE- mE (1,e,,,,,, M) |acc(Bm) conf(Bm))
ECE-
max
where acc(.) and conf(-), respectively, are the empirical accuracy and average confidence for each bin. Here, Bm, referred to as bin, is subset of samples such that the sample's prediction confidence falls into the m-th partition. Refer to Guo et al. (2017) for detailed derivation.
Table I and 2 summarize the error rate and accuracy. MAML+TS denotes temperature scaling after training MAML. Temperature scaling (Guo et al., 2017) is one of the calibration methods widely used in general classification, which learns a temperature parameter control the confidence. works as post-processing on the validation set. However, in few-shot learning setup, train, vali- dation, and test set all have non-overlapped classes, and the unseen classes are given the future tasks. Existing calibration methods such as temperature scaling find the best parameter for the known classes, which is not applicable in few-shot classification. This is why MAML+TS produced somewhat random results; few results are better than MAML but the rest are severely poor.
In most experiments, TCMAML exhibited a significantly decreased error rate than vanilla MAML and ABML, as well as TCProtoNet outperforming ProtoNet. ABML, amortized Bayesian meta- learning (Ravi & Beatson, 2019), is one of the Bayesian methods for few-shot learning and is also based on the MAML algorithm. Notably, while producing calibrated results, TCMAML and TCPro- toNet yielded consistent accuracy without incurring the loss of prediction capability. On the other hand, ABML produced poor accuracy with high ECE/MCE rate except for the 1-shot mini- ImageNet

---

Under review as a conference paper at ICLR 2021
Table I: Calibration results for mini-ImageNet experiments. 95% confidence interval.
The accuracy is reported with
5-Way 1-Shot
5-Way 5-Shot
Methods
Accuracy (%)
ECE (%)
MCE (%)
Accuracy
ECE
MCE
MAML
46,92 + 0.63
4.04
8.77
63.18 1 0.59
1.46
3.98
MAML+TS
46.92 = 0,63
1.39
4.59
63.18 + 0.59
6.01
8.79
ABML
43.47 = 0.58
1,48
4.82
60.06 = 0.53
2.97
4.93
TCMAML
47.20 = 0.63
0.80
5.80
62.96 0.58
1.62
4.09
ProtoNet
48.30 = 0.62
1.26
4.53
67.61 = 0.54
0.70
1.43
TCProtoNet
49.52 = 0.63
0.89
3.00
67.12 0.53
0.55
1.27
Table 2: Calibration results (%) for CUB-200 and mini-ImageNet - CUB-200 experiments. The accuracies are reported in Appendix B.5.
CUB-200
mini-ImageNet CUB-200
5-Way 1-Shot
5-Way 5-Shot
5-Way 1-Shot
5-Way 5-Shot
Methods
ECE
MCE
ECE
MCE
ECE
MCE
ECE
MCE
MAML
2.22
2.94
2.39
3,66
18.02
35.80
18.28
25.27
MAML+TS
8.62
13.47
2.04
3.65
16.71
35.33
20.89
27.99
ABML'
3.34
5,00
2.29
4.59
TCMAMI.
1.40
2.74
2.27
3.36
15.06
27.88
17.53
24.65
ProtoNet 3.88
6.31
0.46
4.44
4.17
6.61
0.50
1.72
TCProtoNet 2.87
5.37
0.34
0.85
3.47
4.64
0.49
1.30
ABML is not implemented dataset shift experiment, due its inefficiency. Refer Appendix B.3, regarding the reimplementation of ABML.
case. Also, TCMAML's improvement was more significant in 1-shot experiments compared to the 5-shot, because 1-shot tasks bring more distributional uncertainty that induces the wrong confidence of a model. For example, in 5-shot mini-ImageNNt, MAML was already well-calibrated and there was no more improvement.
Interestingly, TCProtoNet has a better calibration ability than TCMAML. MAML has explicit inner- level optimization, which adapts rapidly to the task, regardless of uncertainty. ProtoNet, on the other hand, updates the parameters using the distance loss between the support prototypes and query set. Therefore, ProtoNet actually warps the embedding space towards reducing the support-query discrepancy. Thus, can be observed that different learning strategies highly influence the model's calibration abilities.
5.3 TASK UNCERTAINTY REFLECTS PREDICTABILITY
One might be curious whether task uncertainty correlates well the difficulty of a given task. Figure (left) shows that our measure of task uncertainty reflects the model's reliability for making correct predictions. There a clear trend of low accuracy when the task has large discrepancy the support and query distribution. In other words, distributional uncertainty is strongly related to the classification accuracy. However, the epistemic and aleatoric uncertainty are much more exacting to measure and do not intuitively explain the difficulty of a task.
We further studied evaluating only those tasks that were estimated to be well-defined. In Figure (right), excluded the outlier tasks of which uncertainty values exceeded the threshold we set. The baseline accuracy for all 1000 tasks is 76.03%. From removing approximately half of all tasks (when threshold is 3.5), the test accuracy reached 81.01% % (gaining 4.98%).

---

Under review as a conference paper at ICLR 2021
06.0
DUB 200
Task Accuracy (%)
mee tragcles
Test Accuracy (%)
90.
90.0
60
10
Similarity Scores
30 22
Threshold
Figure 3: Left: TCMAML was trained on 5-shot CUB-200 train set, and thereafter evaluated on different test sets. Each represents a test task. 1000 tasks were evaluated for each dataset and each task consisted of 75 query examples, 15 per class be classified. y-axis is task accuracy, the accuracy of 75 queries. Right: The test accuracy (solid line) and the number of tasks (dashed line) that were evaluated while excluding the outlier tasks. The test set was CUB-200.
5.4 META-TRAINTS TRAINING WITH CORRUPTED TASKS
Our task calibration method is robust against data corruption. Meta-training with corrupted data is challenging to the model because it introduces large noise that impedes the accumulation and the transfer of meta-knowledge. Nevertheless, robustness is crucial in that the model should still learn meta-knowledge, even among those tasks, be transferred the target tasks.
To this end, we manipulated the task sampling process in meta-train phase. Specifically, with some fixed probability, we corrupted the tasks to include random Gaussian images. That is, corrupted task consists of 5 support images from mini- Imai--maaeNNet train set and 15 query images that have pixel values from (0,1)) We found out that MAML is already robust against this attack due to the informative feature reuse (Raghu et al., 2019) and overfitting to the uncorrupted images; thus, there was no improvement with TCMAML. On the other hand, metric-based model as ProtoNet is fragile because this type of model learns directly from matching the query examples to the support set. If the support and query set are heterogeneous, ProtoNet is easily damaged by unwanted updates.
Table 3: Test results (%) after meta-training under corrupted tasks condition.
0.25
0.5
Methods
Accuracy
Decrease
Accuracy
Decrease
MAML
62.27 + 0.57
-0.91
61.12 + 0.59
-2.06
TCMAML
61.82 0.58
-1.14
61.03 + 0.57
-1.93
ProtoNet
61.38 0.55
-6.23
43.60 + 0.50
-24.11
TCProtoNet
63.45 + 0.56
-3.67
48.65 = 0.53
-18.47
When 25% of tasks were corrupted (p = 0.25), ProtoNet performance dropped to 61.38%. Not surprisingly, 50% corruption (p 0.5) led to a further decrease, which is 24.01%p lower than without-corruption training. TCProtoNet managed to exceed its counterpart, 63.45% and 48.65% in 25% and 50% corruption, respectively.
6 CONCLUSION
In this study, we asserted that large-scale few-shot classification problems should be examined by carefully estimating uncertainty. We postulate that the few-shot classification contains large discrep- ancy between the support and query set, although typical meta-learning algorithms do not consider this. We demonstrated that our suggested class-wise similarity score is related to distributional un- certainty and reflects the predictability of task. Further, demonstrated that task calibration be extended easily to other methods, such as metric- based learning, via TCProtoNet. The calibra- tion results verified reliable decision-making without the ability to classify being degraded. Lastly,
8

---

Under review as a conference paper at ICLR 2021
found out the task calibration method acquires robustness to the data corruption during meta- training.
REFERENCES
Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. In Advances in Neural Information Processing Systems, pp. 3981-3989, 2016.
Federico Cabitza, Raffaele Rasoini, and Gian Franco Gensini. Unintended consequences of machine learning in medicine. JAMA, 3186)))1 2017.
Anirban Chakraborty, Manaar Alam, Vishal Dey, Anupam Chattopadhyay, and Debdeep Mukhopad- hyay. Adversarial attacks and defences: A survey, arXiv preprint arXiv: 181000009, 2018.
Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. closer look at few-shot classification. In 7th International Conference on Learning Representations, 2019.
Stefan Depeweg, José Miguel Hernández- Finale Doshi- Velez, and Steffen Udluft. Decom- position of uncertainty in Bayesian deep learning for efficient and risk-sensitive learning. In Pro- ceedings of the 35th International Conference on Machine Learning, volume 80, pp. 1192-1201, 2017.
Armen Der Kiureghian and Ove Ditlevsen. Aleatory epistemic? Does it matter? Structural Safery, 31(2):005-112, 2009.
Li Fe-Fei et al. A Bayesian approach to unsupervised one-shot learning of object categories. In Proceedings Ninth IEEE International Conference on Computer Vision, pp. 1134-1141. IEFE, 2003.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. Proceedings of the 34th International Conference on Machine Learning, volume 70, pp. 1126-1135. JMLR. org, 2017.
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-learning. In Ad- vances in Neural Information Processing Systems, pp. 9516-9527, 2018.
Yarin Gal. Uncertainty in deep learning. University of Cambridge, 1:3, 2016.
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee Whye Teh. Neural processes. arXiv preprint arXiv: 2018.
Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard F. Tumne. Meta- learning probabilistic inference for prediction. 7th International Conference on Learning Rep- resentations, 2019.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. Proceedings of the 34th International Conference on Machine Learning, volume 70, pp. 1321-1330. JMLR. org, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 770-778, 2016.
Tove Helldin, Göran Falkman, Maria Riveiro, and Staffan Davidsson. Presenting system uncertainty in automotive UIs for supporting trust calibration in autonomous driving. In Proceedings of the 5th international conference on automotive user interfuces and interactive vehicular applications, pp. 210-217, 2013.
Nathan Hilliard, Lawrence Phillips, Scott Howland, Artëm Yankov, Courtney D Corley, and Nathan O Hodas. Few-shot learning with metric-agmostic conditional embeddings. arXiv preprint arXiv: 1802. 04376. 2018.

---

Under review as a conference paper at ICLR 2021
Heinrich Jiang, Been Kim, Melody Guan, and Maya Gupta. To trust not to trust classifier. In 2018.
Advances in Neural Information Processing Systems, pp.
Taesup Kim, Jaesik Yoon, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian model-agnostic meta-learning. In Advances Neural Information Processing Systems, pp. 7343-7353, 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1412.6980, 2014.
Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. Siamese neural networks for one-shot image recognition. In /CML deep learning workshop, volume 2, 2015.
Balaji Lakshninarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive
uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, pp. 6402-6413, 2017.
Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient-based hyperparameter optimiza- tion through reversible learning. In Proceedings of the 32nd International Conference on Machine Learning, volume 37, pp. 2113-2122 2015.
Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. In Advances in Neural Information Processing Systems, pp- 7047-7058, 2018.
Alexandru Niculescu-Mizil and Rich Caruana. Predicting good probabilities with supervised learn- ing. In Proceedings of the 22nd International Conference on Machine learning, volume 119, pp. 032- ACM, 2005.
Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. TADAM: Task dependent adaptive metric for improved few-shot learning. In Advances in Neural Information Processing Systems, pp. 721-731, 2018.
Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning or feature reuse? Towards understanding the effectiveness of MAML. arXiv preprint arXiv: 199.9977,, 2019.
Sachin Ravi and Alex Beatson. Amortized Bayesian meta-learning In 7th International Conference on Learning Representations, 2019.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In 5th Interna- tional Conference on Learning Representations, 2017.
Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning to quantify classifica- tion uncertainty. In Advances in Neural Information Processing Systems, pp. 3179-3189, 2018.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, pp. 4077-4087, 2017.
Jasper Snoek, Yaniv Ovadia, Emily Fertig, Balaji Lakshminarayanan, Sehastian Nowozin, D) Scul- ley, Joshua Dillon, Jie Ren, and Zachary Nado. Can you trust your model's uncertainty? Eval- uating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems, pp. 13969-13980, 2019.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning compare: Relation network for few-shot learning. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, pp. 1199-1208, 2018.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in Neural Information Processing Systems, pp. 3630-3638, 2016.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The Caltech- USCD birds-200-2011 dataset. 2011.
Zhixiong Xu, Lei Cao, and Xiliang Chen. Meta-learning via weighted gradient update. /EEE Access, 7:1 10111884-11111 10855, 2019.
10

---

Under review as a conference paper at ICLR 2021
A DETAILED DESCRIPTION ABOUT DISTRIBUTIONAL UNCERTAINTY
Here we use the same few-shot classification setup as in the paper. To address the uncertainty in meta-learning, recent works built Bayesian neural networks to formalize posterior distribution over the task-specific parameters Φr, which an adapted result from D(7) and global parameter O. The main idea of the Bayesian approach is using the posterior of Φ, as a prior when computing the predictive distribution. Marginalizing over task-specific parameters would give good estimates of an output, considering model's epistemic uncertainty.
ply|,, Dept. 0) - plul a, d)p(OlDspt 0) d$
(9)
Malinin & Gales (2018) extended above equation explicitly capture distributional uncertainty,
via parameterizing distribution over categorical distribution, p(ula Φb, where μ stands for the
categorical probability; μ [u]..... [NN]" [ply = 1), p(y = N)] Given enough training
data we can assume point estimate for the model parameters, &. Then (9) can be reformulated as:
/ (oll))(() x. ()p(o) ()) 0) du d$
pyy x. Dapt. 0)
(10)
f plulm))( 3) du
(11)
where ob p( spec 0) 8(6 Φ). 011l/ () is a prior for categorical distribution, for example Dirichlet prior, r(E2 a.)) 11 I(a.)
(12)
N
p(u Si a))
I
(13)
where Og is a non-negative concentration parameter for class
In the case when input has high aleatoric uncertainty, p(ular, क) from (13) must be sharp at the center of the (N - 1) simplex, being confident of imposing uncertain predictions. is satisfied with symmetric Dirichlet where a&S are the same and larger than Conversely, when there exists high distributional uncertainty. desired Dirichlet prior should be flat, i.e., a = [I, ... []] This because support and query examples form different distributions, so there no prior belief of mapping query examples to any prediction, x + y.
Although Malinin & Gales (2018) and Sensoy al. (2018) addressed distributional uncertainty by parameterizing Dirichlet prior over categorical distribution, they trained the model with a strong as- sumption that the distributional mismatch of data known in advance. However, in few-shot learn- ing problem setup, it is unknown whether the uncertain prediction is attributed to the distribution, model, or data itself. Moreover, every few-shot learning task could potentially involve distributional uncertainty due to the limited number of support examples. Thus, it is not easy for the model to decide how much of the meta-learned prior experience should make use to solve a given task.
We suppose is more crucial to detect the distributional mismatch in few-shot image classification, with an assumption that the epistemic uncertainty is relatively lower than the aleatoric or distribu- tional uncertainty. Since deep networks such CNN extract highly informative features from the images (Chen et al., 2019; Raghu et al., 2019), uncertainty may not strongly depend on the model's stochasticity. While the Bayesian models can explicitly represent parameter uncertainty in simple problems, it cannot estimate distributional mismatch within complex data. Besides, there is another pitfall. Although few-shot classification constitutes task with only small amount of examples, there are in fact large enough data in total and meta-learner is trained under millions of randomly composed tasks, Enough data typically shrink the model posterior and reduce the episteme.
B EXPERIMENTAL MATERIALS
B. HYPERPARAMETERS
We used a Conv-4 backbone except for dataset shift experiments. The Conv-4 architecture (Vinyals al., 2016) stacks blocks that each comprised of (Convolution + BatchNorm ReL. Max-
11

---

Under review as a conference paper at ICLR 2021
Pool). When training, used data augmentation (RandomResizedCnng) ColorJitter, RandomHori- zontalFlip), and each image is resized to the resolution of 84 (224 in the dataset shift). Meta-training lasts 1600 epochs for 5-shot and 2400 epochs for 1-shot experiments, with 25 episodes per epoch. One episode samples 4 tasks (T = 4) and each task samples 15 queries per class (k' = 15). We used Adam optimizer (Kingma & Ba, 2014) with learning rate Se-4. For MAMI., the learning rate in the inner-loop update is le-2 and the number of inner-loop steps is 5.
In addition, computing weights for each task requires scaling factor, according to (5). a scaling hyperparameter that needs to be tuned, depending on the dataset, type of feature extractor, and the size of matrix (number of classes), etc. Note that T + oo means equally weighted sum, leading to vanilla MAML. In our experiments, we searched T from {1/2, 1.
B.2 LINEAR SCALING TCMAML
In (5), we computed w, with a softmax over the similarity scores. While the softmax function is an exponential scaling (ES), normalizing by linear scaling (LS) is also possible. In this case, we do not need a scaling factor.
1/87
Wy Erar 1/8T
(14)
Table 4 exhibits the results of linear scailng TCMAML. However, most experiments, TCMAML (ES) is comparable with better than TCMAML (LS). Depending on the similarity score values, linear scaling may have small weight variations across tasks, and produce similar results as MAML.
Table 4: ECE (%) and MCE (%) comparisons between MAML, TCMAML with linear scaling, and TCMAML with exponential scaling.
Iing-Nme
CUB-200
mini- Metge-tmageNet CUB-200
1-Shot
5-Shot
1-Shot
5-Shot
1-Shot
5. 5-Shot
Methods
8.77
1,46
3.98
2.94
2.39
3.66
80
18:28
25.27
TCMAML. (1.S)
1.25
4.66
1.86
4.49
2.63
3.34
1.68
2.70
16.46
18:34
24.39
TCMAMI. (FS)
0.80
5.80
1,62
4.09
1.40
2.74
2.27
3.36
15.06
27.
17.53
24.65
B.3
REIMPLEMENTATION OF ABML
ABML (Ravi & Beatson, 2019) is Bayesian approach that estimates uncertainty in meta-learning. We reimplemented the ABML algorithm because 1) they only reported mini- Mini--mmgeNer 5-way 1- shot experiment result and 2) our experimental setup is slightly different from theirs. Therefore, implemented ABML based on our setup.
First, sampled 4 tasks within an episode as did in the TCMAML. experiments. Second, Ravi & Beatson (2019) did not use data augmentation, but we used the same dataloaders as other experiments of ours. This includes the same data augmentation scheme. Third, we used a first-order approximation of derivatives as TCMAML. For the other setups, such as the number of ensemble networks train and validate, learning rates, and KL-divergence reweighting factor, we followed the original paper's default values.
However, we did not implement ABML on dataset shift experiment due to its inefficiency of memory and computation. ABML is Bayesian method, and this implies that ABML is trained by marginal- izing over model parameters. In implementation, this requires significant computation overhead because it should hold several models and receive their gradients in parallel. During the meta- training of ABML, ensemble networks work in parallel, and during the meta-testing, 10 ensemble networks are employed. In Figure 4, TCMAML and ABML show enormous gap of GPU memory usage, even though they are both MAML-based methods. We can also find that ABML takes a lot of time to compute during the meta-train phase. Because of ABML's inefficiency, could not apply it to a ResNet-13 18 model, which is a backbone for dataset shift experiments. It exceeds the capacity of single GPU own, and also it takes excessive amount of time be converged.
12

---

Under review as a conference paper at ICLR 2021
Training Evaluction
Memory Usage (GB)
Time (sec /100 iter)
100
80
69
TCMAML
ABML
TCMAML
ABML
Figure 4: The amount of resource used while training on mini-ImageNet 5-way 5-shot with Conv-4 backbone. Tested on single GeForce RTX 2080Ti GPU. Left: GPU memory usage (GB) during the training (meta-train) phase and the evaluation (meta-test) phase. Right: Time consumption (sec) 100 meta-update iterations in the meta-train phase.
B.4 EXPERIMENTAL SETUP FOR FIGURE 2
We now describe how we generated Figure 2 in the paper. We measured the class-wise similarity scores of 1000 training tasks, where the tasks are sampled from some manipulated distribution. The dataset we used here is minri-mmageet,, and we followed the 5-way 1-shot MAML experiment protocol. We describe way that we manipulated the task sampling distribution.
Standard is general 1-shot experiment. For the support set, image is sampled from each class. For the query set, 15 images are sampled from the corresponding class.
Well-defined tasks are sampled so that there is distributional uncertainty. Although it is hard to perfectly get rid of the uncertainty, it is possible to make the distribution of D( spi and D(F)) similar. To this end, we sample 100 support images per class and corresponding 15 query images. By changing few-shot tasks into data-rich many-shot tasks, we could simulate on the low uncertainty regime. Adapting 100 support images (but we did not change the number of adaptation steps) allows 0 to be near optimal to maximize the likelihood of the query set.
Poorly-defined tasks are constructed to make extremely large support-query discrepancy, large dis- tributional uncertainty. As the Standard setting, Poorly-defined task samples 1 support image and 15 query images, but they are from different classes. For instance, the class labels of support Mlllffish and the class
images can be [hair sl idee carousel, wok, photocopieeeeeccccoooooocco labels of query images be (ant, vase, da Malmatian, school bus, king crab].
B.5 5 TEST ACCURACY FOR TABLE
In Table we display every test accuracy for CUB-200 and dataset shift experiments. Note that ABML poorly performs in CUB-200 5-shot, while it maintained comparable performance in 1-shot. Also, ProtoNet-based methods outstrip MAML-based methods with large margin in dataset shift experiments.
Table Test accuracy (%) with 95% confidence interval.
CUB-200
mini-
CUB-200
Methods
1-Shot
5-Shot
1-Shot
5-Shot
MAML
56.76 1 0.78
75.62 +. 0.58
34.24 + 0.55
49.34 1: 0.57
MAML+TS
56.76 = 0.78
75.62 0.58
34,24 0.55
49.34 0.57
ABML
54.99 = 0.70
69.59 = 0.56
TCMAML
56.55 = 0.75
76.03 = 0.57
33.48 + 0.52
50.44 + 0.56
ProtoNet
57.03 = 0.68
77.79 + 0.49
44.20 = 0.78
64.97 = 0.69
TCProtoNet
56.91 = 0.69
77.23 + 0.50
45.04 + 0.80
64.08 = 0.71
13

---

Under review as a conference paper at ICLR 2021
C
QUALITATIVE EXAMPLE
support set
set
weights
Task
0.2732
T
Task
0.1725
TABAKONI
Task
Task
Task
Tisk
tank, green mamba, cocktail shaker, bolote, tobacco shop, jellyfish. spider web
chime, French huildog,
snorkel, hotdog, robin, Gordon setter,
fire miniature poodlc, tebacco shop, frying pan,
clog
Figure 5: An example of a 4-task batch and their TCMAML weights.
In Figure 5, we depicted an example of weight computation given real images as the input data. This is sampled batch of tasks, where the minibatch size is 4, from mini-Imagell ImageNet train set. As describes, TCMAML gives different weights to each task, regarding the support-query discrepancy. For example, in Task 1, each query image easily matched to their corresponding support images. The computed weight to this well-defined task is 0.2732. However, Task 4 includes somewhat unclear matching between the support and query images. In this kind of task, even the adapted model cannot confidently distinguish the query images, resulting in the lower weight, 0.1725.
D RELIABILITY DIAGRAMS
Reliability diagrams (Niculescu-Mizil & Caruana, 2005) visually present model calibration, plotting empirical accuracy versus average prediction confidence, the largest value of the output distribution. Figure 6 describes reliability diagrams comparing between MAML, ABML and TCMAML. We can find that TCMAML has its bars closer to the y = x line than MAML and ABML, implying more reliable predictions. By the way, CUB experiments show slightly different aspects. MAML is dif- fident on its predictions, with the empirical accuracy higher than the confidence. The model which not sufficiently confident can cause another type of problem. As opposed to the overconfidence,, low confidence often occurs when there are ambiguous images with low accuracy. This situation is vulnerable adversarial attack perturbation of images (Chakraborty al., 2018), readily avail- able to decrease overall accuracy due to the model's low confidence. In this case, the model which can accurately tell easy tasks apart is needed. TCMAML again succeeds in calibration with enough confidence in such tasks.
14

---

Under review as a conference paper at ICLR 2021
МAMΣ.
ABMI
TCМAMI.
0.04037
ECE
0.01476
0.00003
69
8.04815
03
0.05802
0.7
07-
Accuracy
Accurar
5.4
41-
23
62
0.1
85
05
67
28
16
0.1
03
655
60
09
16
01
32
04
86
6.7
36
Confidence
Confidence
Confidence
ECE 0.02215 MCE 0.02943
ECE 0.03343
ECE 0.01398
MCE 0.04995
35-
MCS 0.02735
54.99=0.70%%
22-
0.7
0.-
Accuracy
Accuracy
25
0.5
13
033
03-
62
04
68
55
0.5
611
03
24
0.5
68
1.0
22
0.4
08
Confidence
Confidence
Confidence
ECE 0.110019
ECE = =0.5561
MCE -0.55795
60
MCE -0.27877
Acc 34.2410.555%
Acc -33.4910.52%%
67
97-
Accuracy
Not Implemented
0.4
03
60
22
Confidence
Confidence
Confidence
Figure 6: Reliability diagrams from MAML (left column), ABML (middle), and TCMAML (right) classification results. All experiments are 1-shot, and from the top row datasets are mini- ImageNet, CUB-200, and mini-ImageNNet CUB-200. Red bars indicate the empirical accuracy the samples whose prediction confidence falls into the specific interval. Perfectly calibrated model should have its bars aligned to y x line (dashed). Deviation from the line is highlighted blue average confidence is higher than accuracy; overconfident, and dark if accuracy is higher; diffident. Note that the confidence always above 0.2 because there are classes. Best seen color.
15


Reviews:
[{"review_id": "nQBMoGSLIzD", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presented a task calibration (TC) method, which introduces the notion of \"distributional uncertainty\", for few-shot classification. Two TC extensions of existing methods (MAML and ProtoNet), namely TC-MAML and TC-ProtoNet, have been presented and experimented. The main contributions of this work the authors pointed out are (1) it's in a non-Bayesian fashion thus is computationally efficient and (2) the TC method can be applied to a range of meta leaning models, and (3) the method is more effective for the few-shot meta-learning situation under dataset shift.\n\nI think this paper deals with an interesting idea but should be improved further. Also the contributions are not clearly supported. The authors should do some more work to demonstrate their contributions. My comments are as below.\n\n1) Regarding the contribution (1), the authors only showed that ABML (amortized Bayesian meta-learning) was computationally expensive, while very efficient Bayesian approximation method based on MC-dropout for the implementation in a Bayesian fashion. The authors should evaluate and compare in terms of the computation time to demonstrate the proposed method is really computationally efficient compared to those in a Bayesian fashion. Also, they need to do something to demonstrate that the proposed method can efficiently handle complex and high-dimensional data sufficiently well.\n\n2) Regarding the contribution (2), there are a variety of meta-learning models that have been presented in recent years (e.g., MetaSGD, TAML, and many more). Which of them can be combined with the TC method? It would be great if the authors investigate the applicability of TC to other existing methods and see if TC improves their performance by experiments as well.\n\n3) Regarding the contribution (3), only one dataset shift case (mini-ImageNet -> CUB-200) was experimented. I strongly recommened the authors to evaulate ther method on other dataset shift cases with additional datasets (e.g., Omniglot).\n\n4) In Table 1 and Table 2, I can't find \"significant\" improvement of the proposed methods (TC-) compared to their baselines in terms of \"classification accuracy\" which is the most important metric. In particular in the case of 5-Way 5-shot for mini-ImageNet, TC-MAML was worse than MAML and TCProtoNet was worse than ProtoNet. How, and in which aspect, could you say the proposed methods are \"significantly\" better?\n\n5) For the organization of this paper, things should be more consistent. For example, accuracy was reported in table 1, but wasn't in table 2 (although provided in appendex B.5). In table 3, the experimental results for MAML and its variants are not provided (althouth the authors mentioned that MAML is already robust for the corrupted tasks).\n\n6) Aren't the methods sensitive to the architectural and training configurations of the CNN models? \n\n7) I think in the current manuscript, there are some strong conjectures that need to be supported by theoretical evidences or more experimental results. For example, \"That is, modeling episteme is not much helpful for the calibration in few-shot classification.\" and the sentences that describe the main contributions.\n\n-----\ni've upated my rating after the authors' response.", "extra_scores": {}, "extra_text": {}}, {"review_id": "DaQ3Xy_ZnZ", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a task calibration method to for meta learning, aiming at better task uncertainty estimation. The method modifies the MAML meta-learning approach by weighting the task-specific loss using class-wise similarity, measured by the cosine similarity of the task features. The authors further propose to use this weighting on metric-based models for few-shot classification. \n\nThe change in weighting proposed by the paper, although incremental, provides an appreciable performance improvement of existing meta-learning methods. However, some comparisons to existing uncertainty calibration methods are recommended to show a more comprehensive evaluation. In particular, one claim is its advantages over Bayesian methods. Probabilistic MAML [1] and Bayesian MAML [2] could be readily applied as additional comparisons to the author’s proposed method. For metric-based methods, as the authors indicated, it may worth testing with more recent developments, including TADAM.\n\nAn additional experiment could also be useful to verify that the class similarity weighting is performing its intended purpose. For example, one can construct a task where the labels are shuffled as in Zhang et al. [3]. Because the labels are uninformative, the task should not be useful for meta learning and should provide almost zero weight.\n\nThe description of task corruption experiments can be improved. Were the reported results based on the total four tasks for meta-training only as described in Appendix B.1? Also, it may worth putting the results from MAML variants in addition to ProtoNet. \n\nThe authors motivated the method with the discussion of distributional uncertainty. A more explicit characterization relating the posterior predictive distribution to the proposed method would strengthen the paper.\n\nThere are also several typos in the current submission. The presentation can be improved for better readability in section 3.2. For example subscripts for classes in equation 3 can be changed to be different from the ones for the query inputs and labels to avoid confusion.\n\n\n\n[1] Finn et al., Probabilistic Model-Agnostic Meta-Learning.\n[2] Yoon et al., Bayesian Model-Agnostic Meta-Learning, NuerIPS 2018.\n[3] Zhang et al., Understanding deep learning requires rethinking generalization.\n\n----post rebuttal----\n\nI am downgrading my score as the authors did not address most of my concerns. The paper can be further improved with the suggested experiments as well as discussion on distributional uncertainty but the revised version does not appear to contain any of these suggested changes. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "4aGzQLyJE-", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall, this paper studies uncertainty in few-shot classification. The main challenge of this problem is unknown. The description of the problems in the first page is kind of misleading as authors describe many problems, not sure whether authors intend to resolve them all or just part of them. The main contribution of this work is not well motivated. \n\nMy main concern is that the contribution of this work is not strong. The main contribution of this work is imposing the class-wise similarity on the meta-update rule in Equation (4). The meta-update is part of the model-agnostic meta-learning pipeline as shown in Equation (2) [Finn et al., 2017]. Equation (4) seems reasonable. However, the novelty of this work is not clear. For example, is the class similarity computation first proposed in this work? Is the weighted meta-update never used in existing literature?\n\nMy another concern is that the motivation is not clear. The main problem/problems authors aim to solve are not specified or emphasized. There are many challenges mentioned in this work, such as aleatoric and epistemic uncertainty, few-shot classification, distributional mismatch between support and query data, the trade-off between calibration and the performance of a model. It is not clear what is the main problem that authors aim to solve and how it promotes the contributions of this work.\n\nFigure 1 requires narratives, especially the description about the dash line and solid line with different colors is not clear.", "extra_scores": {}, "extra_text": {}}, {"review_id": "1kxkM9DBopv", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to use class-wise similarity to measure task uncertainty in few-shot learning. In training, if a task has low class-wise similarity, then the task is assigned a higher weight; if a task has high class-wise similarity, then it would be considered ill-defined and assigned a lower weight during training. Experiments on mini-imagenet  and CUB-200 show that it can improve MAML and ProtoNet’s classification performance and uncertainty estimation.\n\nI disagree with the paper’s definition of ‘ill-designed’ tasks. When clases in a training task are very similar, I think it just means that this training task is harder for the model to learn. Therefore, the task at least shouldn’t get a lower weight and somewhat get ignored during training. If at test time, a test task also has very similar classes close to this training task, then this ‘ill-defined’ training task could actually help. With the current method, I think if the data set gets bigger and more complicated, the classification performance will drop significantly. Because the current method would filter out the hard tasks and ignore them during training to some extent. \n\nAs for the experiment part, the paper also only conducted experiments on relatively small and simple data sets, mini-ImageNet and CUB-200. If the authors can add an experiment on Tiny-ImageNet and show some performance improvements there, it can make the experiment part much more solid.\n\nThe paper says ‘A well-calibrated network often exhibits lower accuracy compared to an overconfident deep network’ in the introduction. It is a false claim. There are a lot of  post-training methods to measure uncertainty at test time. They don’t affect the model’s classification performance since they are post-training methods. Moreover, DeepEnsemble improves the classification performance of a classifier at the same time when it improves the model’s uncertainty estimation.\n\n####################Post Rebuttal#######################################\n\nI meant tiered-ImageNet (https://github.com/renmengye/few-shot-ssl-public). I apologize for the mistake. \n\nI think the rebuttal has not addressed my concerns, especially the one related to the 'ill-defined' tasks. The 'ill-defined' tasks detected from the proposed method could be 'hard but good' tasks benefitting the test. The authors didn't deny this. And the proposed method tries to push the model to learn less from such tasks. A natural deduction is that it could affect the classification performance. I think if the authors can take a further step to disentangle uncertainty estimation and classification, this con of the proposed method can be removed in the future. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper tries to address the uncertainty calibration problem in meta-learning by weighting the gradient from different tasks according to class-wise similarity. There have been many concerns raised by the reviewers and most of them either are still not properly addressed after the rebuttal period. 

The main concerns are as follows:
- The problem the paper tries to address is not clear. The use of weighting in meta-update is motivated from distributional uncertainty, but it is not clear how that will improve the task calibration at meta-testing time.
- The proposed update runs into the risk of focusing on simple tasks and down-weighting hard tasks that could be improved with more learning to get more discriminative features. That might hurt the classification performance even though it gets better calibration quality.
- Novelty is limited. The proposed method is a fairly small modification on the original MAML algorithm.
- More comprehensive empirical evaluation is required to support the superiority of the proposed method to other baselines.

I suggest the authors take all the reviewers' comments seriously and improve their work for a better revision.

Decision: Reject