Repurposing Pretrained Models for Robust Out-of-domain Few-Shot Learning
Authors: Namyeong Kwon
Hwidong Na
Gabriel Huang
Simon Lacoste-Julien
Venue: ICLR 2021 Poster (2021)
Keywords: Meta-learning
Few-shot learning
Out-of-domain
Uncertainty
Ensemble
Adversarial training
Stepsize optimization

Abstract: Model-agnostic meta-learning (MAML) is a popular method for few-shot learning but assumes that we have access to the meta-training set. In practice, training on the meta-training set may not always be an option due to data privacy concerns, intellectual property issues, or merely lack of computing resources. In this paper, we consider the novel problem of repurposing pretrained MAML checkpoints to solve new few-shot classification tasks. Because of the potential distribution mismatch, the original MAML steps may no longer be optimal. Therefore we propose an alternative meta-testing procedure and combine MAML gradient steps with adversarial training and uncertainty-based stepsize adaptation. Our method outperforms "vanilla" MAML on same-domain and cross-domains benchmarks using both SGD and Adam optimizers and shows improved robustness to the choice of base stepsize.

Published as a conference paper at ICLR 2021
REPURPOSING PRETRAINED MODELS FOR ROBUST OUT-OF-DOMAN FEW-SHOT LEARNING
Namyeong Kwon, Hwidong Na
Samsung Advanced Institute Technology (SAIT), South Korea {ny [ny.kwon, hwidwwg na l samsung
Gabriel Huang
Simon Lacoste-Julien' Mila, Université de Montréal SAIT AI Lab, Montreal
Mila, Université de Montréal
ABSTRACT
Model-agnostic meta-learning (MAML) is a popular method for few-shot learning but assumes that we have access to the meta-training set. In practice, training the meta-training set may not always be an option due to data privacy concerns, intellectual property issues, merely lack of computing resources. In this paper, we consider the novel problem of repurposing pretrained MAML checkpoints to solve new few-shot classification tasks. Because of the potential distribution mis- match, the original MAML steps may no longer be optimal. Therefore we propose an alternative meta-testing procedure and combine MAML gradient steps with ad- versarial training and uncertainty based stepsize adaptation. Our method outper- forms "vanilla" MAML on same-domain and cross-domains benchmarks using both SGD and Adam optimizers and shows improved robustness to the choice of base stepsize.
INTRODUCTION
Deep learning approaches have shown improvements based on massive datasets and enormous com- puting resources. Despite their success, it is still challenging to apply state-of-the-art methods in the real world. For example, in semiconductor manufacturing (Nishi & Doering, 2000), collecting each new data point is costly and time consuming because requires setting up a new manufacturing pro- cess accordingly. Moreover, in the case of a "destructive inspection" the cost is very high because the wafer must destroyed for measurement. Therefore, learning from small amounts of data important for practical purposes.
Meta-learning Seaaning-to-leamt approaches have been proposed for learning under limited data constraints. A meta-learning model optimizes its parameters for the best performance the distri- bution of tasks. In particular, few-shot learning (FSL) formulates "learning from limited data" as an n-way shot problem, where is the number of classes and A is the number of labeled samples per class. For each task in FSL, a support set is provided for training, while a query set is provided for evaluation. Ideally, meta-learning model trained over set of tasks (meta-training) will generalize well to new tasks (meta-testing).
Model-agnostic meta-learning (MAML) (Finn al., 2017) is a general end-to-end approach for solving few-shot learning tasks. MAML is trained on the meta-training tasks to learn a model initialization (also known checkpoint) such that few gradient steps on the support will yield the best predictions on the query set.
However, in practice may not always be possible retrain or finetune on the meta-training set. This situation may arise when the meta-training data is confidential, subject to restrictive licences, contains private user information, or protected intellectual property such as semiconductor manu-
Work done as visiting researchers at SAIT AI Lab, Montreal.
1 Canada CIFAR AT Chair

---

Published as a conference paper at ICLR 2021
facturing know-how. Another reason is that one may not have the computing resources necessary for running large-scale meta-training.
In this paper, we consider the novel problem of repurposing MAML checkpoints to solve new few- shot classification tasks, without the option of (re))raining on the meta-training set. Since the meta- testing set (new tasks) may differ in distribution from the meta-training set, the implicit assumption -n end-to-end learning- of identically distributed tasks may not hold, so there is no reason why the meta-testing gradient steps should match the meta-training. Therefore, investigate various improvements over the default MAML gradient steps for test time adaptation.
Conceptually, our approach consists of collecting information while training the model on new support set and then proposing ways to use this information to improve the adaptation. In this paper, we consider the variance of model parameters during ensemble training as source of information to use. We propose algorithms that uses this information both to adapt the stepsizes for MAML as well as generate "task-specific" adversarial examples to help robust adaptation to the new task.
Our main contributions are the following:
We motivate the novel problem of repurposing MAML checkpoints to solve cross-domain few- shot classification tasks, in the case where the meta-training set is inaccessible, and propose stepsize adaptation and adversarial data augmentation, which has the particularity that meta-testing differs from meta-training steps.
method based on uncertainty-
Compared to "vanilla" MAML, our method shows improved accuracy and robustness to the choice of base stepsizes on popular cross-domain and same-domain benchmarks, using both the SGD and Adam (Kingma& & Ba, 2014) optimizers. Our results also indicate that adversarial train- ing (AT) is helpful in improving the model performance during meta-testing:
To the best of our knowledge, our work is the first few-shot learning method to combine the use of ensemble methods for stepsize computation and generating adversarial examples from the meta- testing support set for improved robustness. Moreover, our empirical observation of improving over the default meta-testing procedure of MAML motivates further research on alternative ways to leverage published model checkpoints.
2
RELATED WORK
2.1 META-LEARNING AND MODEL-AGNOSTER META-LEARNING
FSL approaches deal with an extremely small amount of training data and can be classified into three categories. First, metric-based approaches solve few-shot tasks by training feature extractor that maximizes inter-class similarity and intra-class dissimilarity (Vinyals et al., 2016; Snell et al., 2017; Sung al., 2018). Second, memory- -bsed approaches utilize previous tasks for new tasks with external memories (Santoro et al., 2016; Mishra et al., 2018). Third, optimization-basee approaches search for good initial parameters during training and adapt the pretrained model for new tasks (Finn et al., 2017; Lee & Choi, 2018; Grant et al., 2018). We focus on the optimization-base approaches and suggest better training especially for the general family of MAML methods.
2.2 UNCERTAINTY FOR MODEL TRAINING
Uncertainty is an important criterion for measuring the robustness of a neural network (NN). Bayesian neural networks (BNN) (Blundell et al., 2015) obtain model uncertainty by placing prior distributions over the weights p(w). This uncertainty has been used to adapt the stepsizes during continual learning in Uncertainty-guided Continual BNNs (UCB) (Ebrahimi et al., 2020). For each parameter, UCB scales stepsize inversely proportional to the uncertainty of the parameter in the BNN to reduce changes in important parameters while allowing less important parameters to be modified faster in favor of learning new tasks. Our approach also decreases the stepsizes for "uncer- tain" parameters, but using different notion of uncertainty, and instead in the context of FSL with a pretrained MAML checkpoint.
Unlike BNNs, deep ensembles (Lakshminarayamanl et al., 2017) estimate uncertainty of NNs ing ensembles over randomly initialized parameters combined with adversarial training (AT). Deep ensembles have been successfully used to boost predictive performance and AT has been used to im-
2

---

Published as a conference paper at ICLR 2021
prove robustness to adversarial examples. Lakshminarayanan et al. (2017) showed that they produce predictive uncertainty estimates comparable in quality to BNNs. Deep ensembles, however, are not directly applicable for FSL tasks, training randomly initialized parameters from scratch with limited amount of training data yields poor performance. Our approach is partly inspired from deep ensembles, adapting it the FSI. setting by using instead parameter perturbations of the MAMI. checkpoint model rather than random initialization. We use in particular multiplicative Gaussian perturbation that rescales the parameters, as the information content of the weights is said to be invariant to their scale (Wen et al., 2018).
2.3 ADVERSARIAL TRAINING FOR ATTACK AND DEFENSE
Deep NNs are sensitive to adversarial attacks (Goodfellow et al., 2015; Madry et al., 2018; Moosavi- Dezfooli et al., 2016). These methods generate an adversarial sample to fool a trained model, where the generated image looks identical to the original one for humans. Goodfellow et al. (2015) pro- posed the fast gradient sign method (FGSM) which generates adversarial example using sign of input gradient. While stronger attacks have been proposed (such as using projected gradient de- scent (Madry et al., 2018)); we will focus on the FGSM approach in this paper for simplicity. In FSL. adversarial attack and defense have also been studied. ADVersarrilMMtta-Lenmm Lear-er (Yin al., 2018) utilized one-step adversarial attack for generating adversarial samples during meta-training. There is little consideration, however, for the degradation of accuracy in the original sample in ad- versarial defense approaches. Xie et al. (2020) reported improvement using generated examples for adversarial attack on the large scale training. Motiian et (2017) propose adversarial domain adaptation for FSL. Despite similar keywords, their work is distinct from ours and relies on using generative adversarial networks discriminators to confuse domains, whereas we rely on adversarial examples for improved robustness. To our best knowledge, there is no prior FSL work which uses adversarial examples to enhance the model at test-time.
2.4 DOMAIN ADAPTATION
Domain adaptation methods (DA) (Ben-David et al., 2010) attempt to alleviate the distribution mismatch between source and target domains. Most of the recently proposed DA approaches based on generative adversarial networks (GANs) (Goodfellow et al., 2014). GAN-based DA ap- proaches require having access to the large amount of unlabeled data from both the source and target datasets (Tzeng et al., 2017; Zhang et al., 2019; Wilson & Cook, 2020). DA methods cannot be di- rectly applied the FSL scenario due to limited number of target domain samples (shots). Some Nomain-adaptive FSL (DA-FSL) methods have been proposed in the case where only a very few samples from the target domain are available (Motiian et al., 2017: Zhao et al., 2020).
DA and DA-FSI. methods cannot be directly applied to our setting because we assume that the meta-training dataset inaccessible - mirroring real world situations in which access to the meta- training set is restricted by privacy and confidentiality concerns. Our approach differs from DA and DA-FSL by not requiring access to the meta-training dataset (source domain).
3 PROPOSED METHOD
At meta-testing time, MAML normally uses the support set to compute fixed gradient steps, which were "calibratedd" using end-to-end learning during meta-training. That is, the learned initialization such that fixed combination of stepsize and loss result in the desired result. However, those stepsizes and losses may be suboptimal on the new task, especially if the new task is out-of-domain with respect to the meta-training tasks.
Our method is based the assumption that the support set can be used to improve the meta-testing procedure itself, beyond merely serving as training examples. We start by leveraging the support set to estimate task-specific uncertainties over the model parameters. Then, we propose two im- provements over the "vanilla" MAML gradient steps: : we scale the gradient steps using layer-wise stepsizes computed from the support set and we train using task-specific adversarial examples.
3

---

Published as a conference paper at ICLR 2021
3.1 MOTIVATING HYPOTHESES
At meta-testing time, we start by assuming that we can estimate task-specific uncertainties over the model parameters. One possibility, which we adopt in Section 3.2, is train deep ensembles (Lak- shminarayanan et al., 2017) on the support set and use the ensemble to estimate variances over model parameters. Each model learns slightly different parameters, and yields slightly different gradients. We regard the variance over the parameters and input gradients as task-specific uncertainties. Given the uncertainty estimates, we propose two modifications to the original MAML gradient steps.
Proposal (Task-specific stepsizes) Use lower stepsizes for model parameters with higher variance.
In our case, the variance could be further amplified if high-variance components were to be moved with large stepsizes. Therefore, we carefully move high-variance components with a lower stepsize with the hope is that we can limit the variance over the model parameters. This approach can be related with the fact that lower stepsizes should be taken for SGD when the gradients are very noisy (Dieuleveut et al., 2020).
Proposal (Task-specitic adversarial examples) Use adversarial examples with higher ad- versarial perturbation on input components with higher variance over the input gradient.
The intuition is that if only slightly perturbed models (from the ensemble) disagree on parts of the input gradient, then it means that they disagree on what to learn and therefore those parts of the input are more vulnerable to attack. Therefore, we propose to use AT with stronger adver- sarial perturbation in the weak parts of the input, with the hope to incur improved robustness on those parts of the input.
We regard adversarial training as form of data augmentation or regularization at meta-testing time, which we hope allows the model overcome the limited size of the support set.
3.2 UNCERTAINTY-BASEE GRADIENT STEPS AT TEST-TIME
We improve over the default MAML gradient steps by implementing the ideas presented in the previ- ous section. The resulting approach is detailed in Algorithm 1 and is a combination of uncertainty- based stepsize adaption (USA, based on Proposal 1) and uncertainty- fast gradient sign method (UFGSM), adversarial training (AT), and generating additional adversarial examples from deep en- sembles (EnAug) which are based on Proposal
Denote L(D.0) 1/ID| E(EyieD lo(x,)) y) the cross-entropy loss for model f over the labeled dataset D {(x,)) ..... Ao(x,y) = I + esi esign((all( y)) the FGSM adversarial example computed from (x., 3) and ( = 1/IDIE lo (o, y), y) the resulting adversarial cross-entropy, which we refer to as AT. For easy reference, all notations are summarized in Ap- pendix A.1.
Starting from the pretrained MAML checkpoint Og. we perturb the model parameters with multi- plicative Gaussian noise to create a deep ensemble (0 )ism<M (lines 2-4). Then, we repeat the following for T steps. At time f, run gradient descent on each model 6ThH of the ensemble (line 7), and AT' loss LAt as (Lakshmii-
where the loss is a combination of the usual cross-entropy narayanan et al., 2017) but on the support set, and the stepsizes o adap are updated using USA (details below). Also, we generate adversarial examples using FGSM and UFGSM (details below) and store them into D (lines 8 and 11). Finally, we run T' gradient steps on the original checkpoint, where the loss combination of cross-entropy on the support set Sptt, AT loss on the support set, and cross-entropy on the aisemble-augmentee
support set Ang (lines 13-16). Note that
recover the original MAML steps for (Axr = Awug 0, A, = 1). while we refer to the case (ΛTT AAug 1, Aa 0) our full method.
4

---

Published as a conference paper at ICLR 2021
Algorithm 1: Uncertainty-basel Gradient Steps at Test-time
Data: New task support set D {(11, v1).... ... (I with ways and & shots. Require: Base stepsize a, pretrained weights 0o. Gaussian variance a, AT coefficient e, size of ensemble M, number of gradient steps T, selection coefficients in (0, 1) for: base stepsize λas adversarial loss ÅAT, and augmented cross-entropy AAug
DAug 0, adap
for 1 to M do 0 <- 00 + (N(),)))
DInitialize deep ensemble
end
for
T do
for = 1 to M do
0. +- 8Th a adap Vor (L( 1) + Cxr(DSM 00
DADE DAUE U {(Aom (z, v). (x, DS0
DEnAug
end
10
a alup < USA (a, 0l:hd) DAug f- D U { UFGSM (8o, i.l ,,,y) (z, y) D
DAlgorithm >Algorithm
11
12 end
13 a f- Ag + (1 da) mddp
for t to T do
A + -1 a @ Ve.,, LCDS,, :) + (D + L(DAE, 0s-1)
15
16 end
USA. We propose uncertainty-basee stepsize adaptation (USA), which assigns lower stepsizes to layers' with higher uncertainty (Algorithm 2) we call this loosely an "inverse-relaionssiip below. Let denote the default (scalar) stepsize and 01:M the parameters of the ensemble models, where M is the size of the ensemble and the number of layers in model is L. To compute the adapted layer-wise stepsizes adup, we compute u, the parameter-wise standard deviation of the model parameters over the ensemble (line 2), apply an inverse-relationship transformation which flips the max with the min (line 3), average over each layer (line 4) and L -n-nomallize the result (line 5). The design choices for USA is explained in Appendix A.2. The resulting stepsizes adap have an inverse-relationship with the variance of each layer. We give an example of applying USA to the 4-ConvNet architecture minilmageNet (Figure 1). On the left, we plot the layer-wise standard deviations of the parameters and on the right the corresponding USA stepsizes, which follow an inverse relationship with the standard deviations.
Algorithm 2: USA
Algorithm 3: UFGSM
Function USA (a, 01:M)
Define: MinMaxNorm (x) Eax (X) -Min
2-Min
u Std (0I:M)
Function UPGSM (0, 0.:M y)
c Max (u) u + Min (u)
u Std ( (V 2o (I, y) 6' E (!
μe <- average of e over each layer
u MinMaxNorm (u)
dlap upp /4'
r' + eu sign (V d (x,
return adup
return x'
end
end
UFGSM. We also propose uncertainty-basee based FGSM (UFGSM) to generate adversarial examples, with higher adversarial perturbation input components with higher variance over the input gradi- ent (Algorithm 3). Starting from an image æ and label y, we compute the input gradient for each model from the ensemble and caleulate the standard deviation we linearly map u between 0 and (line 3), compute the FGSM adversarial example for the pre-
the ensemble (line 2). Then,
In this study, we choose to use layer-wise stepsizes, but it is also possible to use kernel-wise or parameter- wise stepsizes. The motivation for using layer-wise stepsizes is because features from the same layer tend to have the same level of abstraction (Zeiler & Fergus, 2014).
5

---

Published as a conference paper at ICLR 2021
Unertiinyy (Sundad (Sundadd deviation)
0.12
Dompebezzon lave weight Codrocation la bais
Match nomellontoon liger gareza
ccomiliation
leyet weight
0.01
0.08
Sepase
0.000
0.006
0.01
0.004
0.022
Conrl. weight
Conel His
DNI proma
BNVL Jeet
Com wegler
has
1012 fees
Com-, wrjttt
Cond hiar
BNU Jeis
Claos biss
BNd cammu
live Fveta
Limor wegld
Linar Aas
Contt tooight
Court bue
Bill gonms
DN Joya
Conv soight
Conoz bis
DN22 ganmo
Cans? weight
Condt Bim
BNI Jenme
DND bota
Conve anight
Bid groon
Litear woigle
Litrrt his
Conc2
Cand
Cane
Capar
(a) Uncertainty of layers
(b) Uncertainty-based stepsize adaption
Figure I: USA converts uncertainty into layer-wise adapted stepsize. Here a = 0.01. (a) is standard deviation of trained ensemble models' weights. Each layer has difference uncertainty. (b) is adapted stepsize by USA. Each layer has a different stepsize based on the uncertainty.
+ex
sign (2b(
x' Z(UFGSM)
x' (FGSM)
Figure 2: Applying UFGSM to 4- 4-ConvNet on minilmageNet with = 0.05. Starting from the clean image æ, we add the signed gradient sign(Valozz, y)) after rescaling by the uncertainty over the input gradient u, to generate the adversarial example x' (UFGSM). Note how UFGSM generated more natural image than FGSM (rightmost, u ===))
trained model do and rescale it using so that areas of higher variance get more perturbation. We give an example of applying UFGSM to min/ImageNet in Figure
EXPERIMENTAL EVALUATION
We train MAML minilmageNet (Vinyals al., 2016) training split; we then apply our method on the resulting checkpoint. We evaluate our model on the test split of minilmageNet - for the same- domain setting- well as CUB-200-2011 (Welinder al., 2010), Traffic Sign (Houben al., 2013) and VGG Flower (Nilsback & Zisserman, 2008) for the cross-domain setting. These datasets are denoted as Mini. Birds, Signs and Flowers respectively. A desirable feature for an optimizer is to maintain good performance in broad range of stepsizes (Asi & Duchi, 2019). Therefore, we evaluate our approach not only at the optimal stepsize, but also over broad range of base stepsizes? from 10-1 to 1. We evaluate the performance with three metrics: All, Top-1 and Top-40%." If two methods have comparable Top-1 I performance, but one has better Top-40% performance, then that method is more robust to the choice of base stepsize. Detailed experimental setup and pretrained model selection are included in Appendix A.3. Our code is available at https: : /et:://gttttiee rithub... com/ /USA Namyeong USA UFGSM/.
4.1
MAIN RESULTS
Our main results are in Table 1. We compare the default MAML steps, which we take as our base- line (denoted as SGD), with our full method (denoted as SGD+All), which consists of combining
We determine the ranges of stepsize hased training performance min/ImageNet and keep the for other datasets. We selected the minimum and maximum stepsize where performance decreased drastically. 3 All is the average accuracy over all stepsizes. Top-1 is the accuracy of the hest performing stepsize. Top- 40% the average of the top 40% accuracies among all the stepsizes

---

Published as a conference paper at ICLR 2021
Table Main results. We compare the default MAML steps (SGD) with our method (SGD+All) on same-domain and cross-domain benchmarks.
5-way I-shol
S-way S-shol
10-way I-shot
Metric
Dataset
SGD
SGD+AIl
SGD
SGD+AIl
SGD
SGD+AIl
Mini
34.21
36 24
38. 24.3
43.48:444
20331
22.07=04
Birds
30.07
29.83
33.29+020
38.07
17.54
17.81+0.1
All
Flowers
40.54
40.86
44.55
53.10.44s
26. 26
28.07
Signs
14.19
34.76:441
45.39
51.02
23.77
24.88
Avg.
35.50=9.0
35.42+0044
40.34
46,42=
22.35
23.21
Mini
46.71 1::0
46 62
62,46
62.16aam
31.10
31,42:448
Birds
36.03-204
36.43=
50.71 50.71 20.54
$1.82=444
23.16
23.61
Top-1
Flowers
51.97
54.36=114
70.63-44.6
74.52++0.
38.61 =0.40
40.68++.54
Signs
43.44
44.10
a2.52.nsa
34.13
35.10
35.11
Avg.
44.54
45.37
64.08
65.66::472
31,99
32.70
Mini
45.07
45.79
55,38+0.4
60.70
28.65
30.56+a.77
Birds
34.88
35.77
45.50
51.03
21.78aam
22.87
Top-40% Flowers
49.98=0m
53.16=873
65,16
73.55
36,14
39.75
Traffic
+1.81+ue
43,39+1m1
67.34
71.45
33.56=0.1
34.27
Avg.
42.94
44.52
58.34
64.18:4m
30.03
31.8614as
USA, UFSGM, EnAug, and AT (AxT AAay 1,
Algorithm 1). In terms of abso- lute performance (Top-1 1 row), our method outperforms the baseline on cross-domain tasks (Birds, Flowers, Signs), while the performance robustness to the base stepsize (All and Top-40% rows), our method outperforms the baseline over large range of stepsizes for 5-way 5-shot and 10-way I-shot tasks, while for 5-way 1-shot the results either comparable (All) or better (Top-40%) depending the metric considered. More results can be found in Appendix A.6.
comparable on same-domain tasks (Mini). In terms of
4.2 DISCUSSION
Ablation Study. We perform an ablation study for the 5-way 1-shot case in Table and plot the accuracy at different base stepsizes for the Flowers dataset in Figure 3. Comparing SGD to SGD+AT in Table 2, we observe that adversarial training is beneficial over the baseline (Topl and Top-40%), except for large stepsizes with SGD, which is reflected in the All metric and in the dip in performance in Figure 3a. Notice also how the use of AT flattens the accuracy curve near the optimal stepsize. Comparing SGD to SGD+USA, we observe very small but consistent improvement over the baseline in the vicinity of the optimal stepsize (Top-1 and Top-40%). Comparing SGD+US, to SGI+USS+UFGGS
we observe improvement over a wide range of stepsizes, which is reflected ++ UFG-SS
in the curves and in All and Top-40% metrics. Comparing shows that AT+EnAug is beneficial in terms absolute performance (Top-1) well as creating robustness to the choice of stepsize (Top-40%). The drop for the All metric is explained by the dip in the curve for the largest stepsizes. Overall, the best absolute performance (Top-1) is always obtained through some use of adversarial training, while using the full method consistently results in increased robustness to the choice of stepsize (best Top-40% performance). Appendix A.5 shows various AT results.
to
CGD+USA+UFGSM-EDAA
UFGSM vs. FGSM. We compare UFGSM to FGSM (Goodfellow et al., 2015) in Table 3. The results show that our uncertainty- approach consistently outperforms FGSM, which suggests that the uncertainty information extracted from the support set was useful.
SGD vs. Adam. Our method can be used with SGD and Adam, however, we have mainly fo- cused SGD throughout the paper. SGD tends to yield the best results (Top-1 and Top-40%), as shown in Table for the baseline and full method. More results for Adam can found in Sections A.6 and A.8 of the appendix.
Best checkpoint vs. Overfitted checkpoint. We also consider the problem of repurposing an overfitted checkpoint. We find that method improves both absolute performance and robustness

---

Published as a conference paper at ICLR 2021
Table 2: Ablation Study for 5-way 1-shot classification with SGD optimizer. SGD+AT corresponds
to using (AAT = X 1, AAug = 0) Algorithm 1, SGD+USA to (AAT dng = A 0), SGD+USA+UFGSM to (AAT = Aa = 0, Aung (AAT Auug I, Aa = 0).
1) and SGD+USS+UFGGM--AAAA to
5-way 1-shot
Metric
Dataset
SGD+USA
SGD+USA
SGD
SGD+AT
SGD+USA
+UFGSM
+UFGSM
+EnAug+AT
Mini
14.60
37.53=a09
38.71
36.2410a2
Birds
30.07 =a.5
28.81 -0.455
30,39
31.07 =..50
29.83+046
All
Flowers
40.54.1na
38. 80.244
41.09
42.33
40.861045
Signs
34. 19
33.42
35.02
36.101..1
34.761u01
Avg.
35.50+n.9
33.91 =0.41
36.01=0.36
37.05
35.42
Mini
46.71 lins
46.58
46.72...71
46.6610.7
46.62.10.5
Birds
36.03=a44
36.40=z04
36.08=0.4
36.02+0.48
36.43
Top-1
Flowers
51.97 11.79
54.55
52,01
52.30
54.36
Signs
43.43
43.98=com
43.42
43.96
44.10
Avg.
44.5420m
45.38
44.56
44.732041
45.37
Mini
45.07 +n:se
45.53=0.50
45.23 =0.99
45.76
45.79
Birds
34.88
35.57
35.17=mct
35.30
35,77
Top-40%
Flowers 49.98+am
52.92=0.8
50.39=0.1
51,07
53,16:073
Signs
41.83
49.19
42.08 1.00
43.181075
43.39
Avg.
42.94.u70
o4.30.004
43.22
43,83
44.52
0.56
0.57
0.51
0.48
0.68
0.44
0.65
5GD
0.62
0.20
USA
0.28
USA -UFGSM-ENAU
USA -UKGS-------u
0.33
0.3
USA AUPGSM
Adam
USA
0221
ATRGSM #E
HEGAUg
0,001
0.0033
BLDXSS
EL0007
0,0000
DO00
00004
0000
0000
122
120
0.55
09
0.4
09
0D001
0,0033
0
0000
Stepaae
(a) With SGD optimizer
(b) With Adam optimizer
Figure 3: Ablation study for 5-way 1-shot classification on Flowers dataset.
8

---

Published as a conference paper at ICLR 2021
Table 3: Comparing UFGSM against FGSM (Goodfellow et al., 2015) on 5-way 1-shot tasks. For all metrics and datasets, the proposed uncertainty- based method is better.
Table 4: Comparing SGD vs. Adam for default MAML. steps (baseline) and our method (baseline+all) on 5-way I-shot classification.
Baseline
Baseline+Al
SGD+USA
Metric
Datuset
Mctric
Datasct
SGD
Adam
SGD
Adam
+UFGSM
+FGSM
Mini
37.21
31.87
36.24_sz
33.57
Mini
38,71
36.97
Birds
30.07
28.02
29.83 n.
29.5 sus
Birds
31.07=a.9
30,05=a5s
All
Flowers
40.54:14s
40.33
40.86-04s
44.58
All
Flowers
42.33__m
41.17
Signs
34.19.4m
33.98g
34.76_mst
35.10.a4
Signs
36.10_a..
34.782010
Avg.
35.50:
33.55 p:22
35.42
35.69
Avg.
37.05=em7
35.74
Mini
46.71
43.08
46.62 .a.s
44.72,105
Mini
46.66_.677
43.93_uu
Birds
36.03
34.00=axs
36.43-os
35.45
Birds
36.02.....
34,52.071
Top-1
Flowers
54.97
$1.28
$4,36_110
56.04
Top-1
Flowers
$2.30=a77
50.6
50.63ga844
Signs
43.43-a25
42753
44.10
43,01
Signs
43,96=864
42.62=1100
Avg.
44.54:am
42.72
45.37-601
44, 80,04
Avg.
44.73-0.1
42.93-047
Mini
45,07=455
40.18
45.79
42.92e0.
Birds
34.88
35.77_aa5
34.54
Mim
45.76
48.182007
Top-40%
Flowers
49.98
18.18
53.16_us
54.08
Birds
35.30=045
33,88
Signs
41.83-148
40.90eam
43,39
42.26
Top-40%
Flowers
51.07=0.7
49.47-00
Avg.
42.94
40.41 =020
44.52:
43.45
Signs
43.18
41.92:um
Avg.
43.83=0.9
42.11=004
to stepsize. In fact, the gains are more substantial on the overfitted (worse) checkpoint than the best checkpoint (Appendix A.8).
Validating Proposal and 2. Our method is built on the hypotheses that should take small stepsizes on high uncertainty parameters and add more adversarial perturbation on high uncertainty input gradients. To validate those choices, we have tried taking large stepsizes on high uncertainty parameters, and using less adversarial perturbation on high uncertainty input gradients, which re- sulted in lower accuracy and robustness the stepsize (Appendix A.7).
Freezing most uncertain layers. We observed that the batch normalization (BN) layers have the most uncertainty (Figure 1). In Proposal 1, we argued that the higher uncertainty component ampli- fies the error when we use bigger stepsize. As suggested by an anonymous reviewer, this motivates an additional experiment that we perform where we "freeze" the BN layer during meta-testing (i.e. manually set the stepsize for the BN scale and shift parameters to zero). We performed the BN layer freezing experiment on SGD+All (see detailed results in Table 10 and Figure 12 in Appendix A.9). It turns out that SGD+AII (w/ freezing BN) outperforms SGD+AIl (w/o freezing BN) on the All metric, with most improvement in the higher stepsize range (>0.1). This is consistent with our intuition that updating high-uncertainty layers (such BN) with large stepsizes can be harmful.
CONCLUSION
In this paper we considered the novel problem of repurposing pretrained MAML checkpoints for out-of-domain few-shot learning. Our method uses deep ensembles to estimate model parameter and input gradient uncertainties steps through the addition of uncertainty- experiments over popular few-shot benchmarks show that our method yields increased accuracy and robustness to the choice of base stepsize. More generally, our results motivate the use of adversarial learning data augmentation scheme for improving few-shot generalization. In the future, would be interesting to apply our method to related settings such as domain adaption and transfer learning.
the support set, and builds upon the default MAML gradient
adversarial training and adaptive stepsizes. Our
ACKNOWLEDGMENTE
We thank Hugo Larochelle and Reza Babanezhad for insightful discussions, Damien Scieur and Emmanuel Bengio for helpful feedback on the manuscript. We also thank the anonymous reviewers for their comments and suggestions.

---

Published as a conference paper at ICLR 2021
This research partially supported by the Canada CIFAR Al Chair Program, the NSERC Dis- covery Grant RGPIN-200 2019I-2017-06936 and Google Focused Research award. Simon Lacoste-Julien is CIFAR Associate Fellow in the Learning in Machines & Brains program.
REFERENCES
Hilal Asi and John C. Duchi. The importance of better models in stochastic optimization. In National Academy of Sciences, 2019.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wort- man Vaughan. theory of learning from different domains. Machine Learning, 2010.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International Conference on Machine Learning. 2015.
Aymeric Dieuleveut, Alain Durmus, Francis Bach, et al. Bridging the gap between constant step size stochastic gradient descent and markov chains. In Annals of Statistics. Institute of Mathematical Statistics, 2020.
Sayna Ebrahimi, Mohamed Elhoseiny, Trevor Darrell, and Marcus Rohrbach. Uncertainty- continual learning with bayesian neural networks. In International Conference on Learning Rep- resentations, 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, 2017.
lan J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Infor- mation Processing Systems, 2014.
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations, 2015.
Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradient- based meta-learning hierarchical bayes. In International Conference on Learning Representa- tions, 2018.
Sebastian Houben, Johannes Stallkamp, Jan Salmen, Marc Schlipsing, and Christian Igel. Detection of traffic signs real-world images: The german traffic sign detection benchmark. Interna- tional Joint Conference Neural Networks, 2013.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In arXiv preprint 1rXiv:1412.6900 2014.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in Neural Information Processing Systems, 2017.
Yoonho Lee and Seungjin Choi. Gradient-based meta-leaming with learned layerwise metric and subspace. In International Conference on Machine Learning, 2018.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta- learner. In International Conference on Learning Representations, 2018.
Seyed Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. Deepfool: A simple and accurate method fool deep neural networks. Computer Vision and Pattern Recognition, 2016.
Saeid Motiaan, Quinn Jones, Seyed Iranmanesh, and Gianfranco Doretto. Few-shot adversarial domain adaptation. In Advances in Neural Information Processing Systems, 2017.
10

---

Published as a conference paper at ICLR 2021
Maria Elena Nilsback and Andrew Zisserman. Automated flower classification large number of classes. Indian Conference on Computer Vision, Graphics and Image Processing, 2008.
Yoshio Nishi and Robert Doering. Handbook of semiconductor manufacturing technology. CRC press, 2000.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In International Conference on Learning Representations, 2017.
Adam Santoro, Sergey Bartumov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with memory-auggmntte neural networks. In International Conference on Machine Learning. 2016.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, 2017.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip H.S. Torr, and Timothy M. Hospedales. Learning to compare: Relation network for few-shot learning. In Computer Vision and Pattern Recognition, 2018.
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evei, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, and Hugo Larochelle. Meta- dataset: A dataset of datasets for learning to learn from few examples. In International Conference on Learning Representations, 2020.
Hung-Yo Yu Tseng, Hsin-Ying Lee, Jia-Bin Huang, and Ming-Hsuan Yang. Cross-domain few-shot classification via learned feature-wise transformation. International Conference on Learning Representations, 2020.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. Conference on Computer Vision and Pattern Recognition, 2017.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Match- ing networks for one shot learning. In Advances in Neural Information Processing Systems, 2016.
Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pietro Perona. Caltech-uesd birds 200. 2010.
Yeming Wen, Paul Vicol, Jimmy Ba, Dustin Tran, and Roger Grosse Flipout: Efficient pseudo-
independent weight perturbations on mini-batches. In International Conference on Learning Representations, 2018.
Garrett Wilson and Diane J. Cook. A survey of unsupervised deep domain adaptation. In ACM Transactions on Intelligent Systems and Technology. 2020.
Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang, Alan L. Yuille, and Quoc V. Le. Adversarial examples improve image recognition. In Computer Vision and Pattern Recognition, 2020.
Chengxiang Yin, Jian Tang, Zhiyuan Xu, and Yanzhi Wang. Adversarial meta-learning. In arXiv preprint arXiv:1806.03311 2018.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, 2014.
Yabin Zhang, Hui Tang, Kui Jia, and Mingkui Tan. Domain symmetrc networks for adversarial domain adaptation. In Computer Vision and Pattern Recognition, 2019.
An Zhao, Mingyu Ding, Zhiwu Lu, Tao Xiang, Yulei Niu, Jiechao Guan, Ji Rong Wen, and Ping Luo. Domain-adaptive few-shot learning. In arXiv, 2020.
11

---

Published as a conference paper at ICLR 2021
A
APPENDIX
A.
NOTATIONS AND ACRONYMS
Notations and acronyms used repeatedly throughout the paper are summarized below.
Notation
Meaning
USA
Uncertainty- based stepsize adaption
I/USA
Inverse USA
FGSM
Fast gradient sign method optimization (Goodfellow al., 2015)
UFGSM
Uncertainty-based FGSM
/UFGSM
Inverse UFGSM
EnAug
Ensemble augmentation
T
Number of test-time gradient steps
M
Size of deep ensemble
On
Pretrained MAML checkpoint to repurpose
Ensemble element m time
fT
Model parameters after meta-testing
a
Base stepsize
adap
Uncertainty-basee based stepsize (parameter-wise)
Spe
Support set (test-time)
D Aug
Adversarial examples based on Sp
A.2 DESIGN CHOICES FOR USA
Here, we introduce the design choices of the Algorithm 2.
Line 3: We calculate the maximum Max(u) and minimum Min(u) values among the estimated stan- dard deviations u for each parameter. As proposal 1, order to assign to low stepsize to high uncer- tainty, the standard deviation of each parameter is subtracted from the maximum value Max(u) (u) for vevrre-relationssip.. At this time, the largest value among parameter values becomes 0. To pre- vent this, we add the smallest Max(u) (u) also tried alternatives ways like non-inverse- renatere-relationshipp and relative standard deviation, but did not seem to work better. To keep the same scale of the uncertainty before and after the transforma- tion, we chose the inverse-relationship using Max u + Min(u).
+ Min(u). Note that all values of are non-negative. We
Line 4: In this study, each layer, we calculate the average of the parameters of each layer and the values the parameters of the corresponding layer are replaced with the average value. repeats all layers and performs the corresponding operation.
layer-wise adapted stepsize. In order to assign the same stepsize for
Line 5: To more easily compare the effectiveness of USA with the baseline, we the average stepsize equals to the base stepsize, i.c. 1/(p="up) Σ adup rescaling, than the average stepsize can change, which makes difficult distinguish the effect of using an overall different stepsize (for constant SGD e.g.) vs. our adaptation of stepsizes.
If USA is applied without this
A.3 EXPERIMENTAL SETUP AND PRETRAINED MODEL SELECTION
Our baseline model was trained using the same hyperparameters with mintlmageNeet training of MAMI. except the inner loop stepsize. The inner loop stepsize was set to 0.1 for reproducing the 5-way 1 -hoot accuracy reported in the original MAML paper. We trained the model for 150,000 iter- ations. A pretrained model was selected with the validation accuracy among minilmageNet training checkpoints. We used the checkpoint with the highest validation accuracy as the pretrained model 8g. The highest test accuracy we achieved was 49.24% during meta-training. For the checkpoint that we chose with the highest validation performance, the test accuracy 47.58%. Note that the cross-domain performance highly depended on which checkpoint we used. See Fig.4 Appendix A.4.
12

---

Published as a conference paper at ICLR 2021
The size of ensemble is M which is the same the deep ensembles (Lakshminarayanan et al., 2017). The parameter for the FGSM is e = 0.05. The scale value a of Gaussian random perturbation for ensemble model training is 0.05. The gradient step is T 10 which is same with mageee test of MAMI..
We do not split those datasets except minilmageNet, because we do not the datasets in the meta- training. We used the same splits as Ravi & Larochelle (2017) for minilmageNet. Tseng et al. (2020) used randomly split manner for the cross-domain test and Triantafillou et (2020) used all traffic signs dataset for the test.
A.4 CROSS-DOMES ACCURACY WHILE META TRAINING MAML ON MINIIMAGENET
The performance of the cross-domain significantly differed depending the selected checkpoint. Figure 4 shows the performance for every 200 iterations. We selected the checkpoint based on minilmageNet validation performance, which is the highest at the 57,200 iteration.
0.55
minilmageNet (Val)
0.54
minilmageNet (Test)
0.53
Birds
0.52
Flowers
0.51
Signs
0.5
0.48
0.47
0.45
0.44
0.43
0.41
0.4
0.39
0.38
0.37
0.36
0.34
200
4400
8400
12400
16600
20600
24600
28800
32800
36800
40800
45000
49000
53000
57200
61200
65200
69400
73400
77400
81600
85600
89600
93600
97800
101800
105800
110000
114000
118000
122200
126200
130200
134400
138400
142400
Figure 4: Cross-domain performance while meta-training on minilmageNet. X-axis is training iter-
ation meta-training minilmageNet and y-axis is classification accuracy.
A.S ADDITIONAL RESULTS FOR AT-ONLY SETTING.
To evaluate the effect of AT, only applied AT in MAML (A,, >XT 1, 0). Table 5, Figure 5 and 6 show AT results on 5-way 1-shot with SGD and Adam. Adam is worse than SGD in the results. However, found that AT and Adam good combination for meta-test training.
A.6 ADDITIONAL RESULTS FOR 5-WAY 5-SHOT AND 10-WAY 1-SHOT CLASSIFICATION.
We present some additional results for 5-way 5-shot and 10-way 1-shot classification in this section. Table 6 shows the 5-way 5-shot and 10-way 1-shot classification results. Especially in the case of the Top-40% Avg our method significantly increased the performance about 5.84% and 1.83% in 5- way 5-shot and 10-way 1-shot respectively. It means that it can increase the probability select of
13

---

Published as a conference paper at ICLR 2021
Table 5: 5-way 1-shot classification results. In order verify the effectiveness of AT, we applied only AT among the proposed method and the all proposed method.
Baseline SGD
Ours (SGD)
Baseline
Ours (Adam)
Metric
Dataset
AT Only
All
Adam
AT Only
All
Mini
+0.21
34.60=a18
36.24
31.87=0.20
33.40+0.3
33,57+0029
Birds
30.07 +..52
28.81 +0.45
29.83
28.02 0.39
29.37 +0.1
29.51
All
Flowers
40.54..0.4
38.80 24.44
40.86
40.33_0.57
44.53
44.58
Signs
34. 1826
33.42.10.9
34.76:1.61
33.98_044
34.961147
35.10
Avg.
35.50
33.91
35,42
33.55
35.57 ::029
35.69
Mini
46.71 =0.81
46.58=00.5
46.62+a054
43.08=0.10
44.80=20.0
44.72+00.5
Birds
36.03
36.40
36.43
34.00=0.63
35.28
35.45
Top-1
Flowers
51.97
54.55
54.361104
51.28=odd
56.10
56,04-11.4
Signs
43.44:10.9
43.98
44.10
42.53=0.9
43.13
43.01
Avg.
44.54
45.38 +0.7
45.37+0.1
42.72 0.32
44.83 +
44.80+0.54
Mini
45.0
45.53
45.79
40. 18.1
42.59
42.92
Birds
34.88
35.57
35.77
32.38 0.61
34.264073
34.54:10.6
Top-40%
Flowers
49.98
52.92.10.68
53.16:1073 1.73
48. 0.44
53,88
54.08
Signs
41.81
43.19
43.39
40.90
42,06+074
42.26
Avg.
42.94
44.30
44.52
40.41 0.29
43.20
43,45 +..00
0.48
845
044
042
0.8
0.1
0M
0.50
008
0.3
SGD
0.22
033
-ow Cryyy
036
E
ALL
0000
0.00%
0000
0000
0000
0100M
0.004
Oni
02
0.000
0.0000
0000
0004
000
000
0.000
0.000
0000
0004
000
(a) Mini
(b) Birds
(c) Flowers
(d) Signs
Figure 5: 5-way 1-shot classification results with AT SGD.
optimal stepsize for new task. As shown in Figure there are more flatten curve near the highest accuracy in both tasks.
A.7 VALIDATING PROPOSALS AND 2 WITH "INVERSE" COUNTERPARTS.
To validate proposal I, we evaluate the inverse strategy of USA, denoted I/USA, which assigns higher stepsizes to layers with higher uncertainty. Specifically, flip the stepsizes by replacing line 3 of Algorithm 2 with c <- u. To measure the effect of USA, we only applied USA MAML (Aa- = AAT AAug = 0). The results for 5-way 1-shot classification are in Table 7. USA outperforms the baselines over broad ranges of stepsizes compared to Adam and SGD. I/USA significantly decreased the performance of all metrics. Therefore, USA reflected useful knowledge well into the stepsize for a new task. See accuracy by stepsizes in Fig 8. Figure 8 shows the results for verifying of Proposal USA shows more flatten curve in Top-1 accuracy ranges than SGD and I/USA. I/USA degraded the performance through all ranges of stepsizes.
For verifying the proposal 2, we evaluated three methods UFGSM I/FFGSM and FGSM on USA. I/FGGS is inversely implemented method of UFGSM. FGSM uses examples generated by FGSM instead of UFGSM. To measure the effect of UFGSM, we only applied UFGSM in MAML with USA (A, ÀAT 0, AwAS 1). Table 7 shows 5-way 1-shot classification results. UFGSM outperforms all the other methods for every metric. spite of /UFGSM had stronger adversar- ial perturbation than UFGSM, the performance was worse than UFGSM. Note that I/UFGSM and FGSM showed almost similar performances. The reason is that most of the input pixels have an un- certainty close to O; therefore, when scaling after inverse, most pixels have value of 1. Therefore, the generated examples are almost similar to the adversarial example in which FGSM is applied. See accuracy by stepsizes in Fig Figure 9 shows the result for verifying of proposal 2. UFGSM
14

---

Published as a conference paper at ICLR 2021
8665
0.30
0027
0M
0.62
0.4
0.32
o ALL
03
0445
036
0.34
042
0.30
033
0.2
0.3
0.28
0.35
0.3
0.8
0.28
Apppe
022
0,000
0000A
0.9
01
00
QUEEI
0U000
B100
0,000
000000
0.00
0,000
000000
Strpeze
(a) Mini
(b) Birds
(c) Flowers
(d) Signs
Figure 6: 5-way 1-shot classification results with AT on Adam.
Table 6: 5-way S-shot and 10-way 1-shot classification results using SGD. Our method outperformed
both tasks. Especially 5-way 5-shot classification performance significantly improved than baseline.
5-way 5-shot
10-way 1-shot
Metric
Dataset
Baseline
Ours (SGD)
Baseline SGD
Ours (SGD)
SGD
SGD+AT
SGD+AII
SGD+AT
SGD+All
Mini
38.14 14.3.9
41.06
43.48
21.31
20.97
22.07
Birds
33.29
36.04
38.07 10.3
1..5
17.08
17.81
All
Flowers
44.55
48.94
53.10
26.79_o.e
26.501042
28.07:1011
Signs
45.39
47.98
51.02++0.0
23.77=0.13
23.99
24.88++0.1
Avg.
40.34
43.51+0.41
46.42-+0.4
22.35
22.14
23.21++0.6
Mini
62.46
62.07
62.16
31.10 0.66
31,18
31.42-+0..1
Birds
50.71 10.54
51.93
51.82+0..44
23.16 0.21
23.42.10.7
23.61 ::0.9
Top-1
Flowers
70.63
74.14
74.52:10.9
38.61 =0.40
40.84
40.68
Signs
72.52
73.66=409
74.13
35.10_0.21
34.76+t17
35.11
Avg.
64.08
65.45
65.66:4472
31.99_0.99
32.55
32.70
Mini
55.38
59.97 +0,69
60.70
28.65 -064
30.22
30.56++0.7
Birds
45.50+0.4
50.48
51.03
21.78
22.61
22.87++0.0
Top-40%
Flowers
65. 162077
72.61 2078
73.55 1.
36.14 14.53
39.402066
39.75 :
Traffic
67.34
70.562087
71.45
33.56-0.11
34.21 2013
34.27
Avg.
18.34
63.40
64.18 1.52
50.03
31.61 10.44
31.86:0.46
shows flatter curve near the best performing accuracy than SGD, I/UFGSM and FGSM. FGSM and why the two methods gave similar results. As can be seen in Fig 10, almost all of the input gradient uncertainty is near zero. When we inverse the value, almost all the value is 1 (See Fig 11). UFGSM improves performance despite less adversarial perturbation than I/UFGSM and FGSM. Through this, even small change can help model learning if correct (useful) information is reflected.
V/FGGM show very similar trends. We explained the
A.8 5-WAY 5-SHOT CLASSIFICATION RESULTS ON OVERFITTED CHECKPOINT
As can seen in Appendix A.4, overfitted models minilmageNet degraded performance over all datasets. We investigated the 5-way 5-shot classification results on the checkpoint after meta- training with MAML. for 150K iterations. Table 8 and 9 show the results with SGD and Adam. The number in the parentheses is the difference from the baseline for each checkpoint (e.g. SGD or Adam). The results using the (Last) checkpoint are worse than (Validation) checkpoint due the overfitting, However the performance boost from using our method is more substantial on the over- fitted (Last) checkpoint than the best (Validation) checkpoint, both in terms of absolute performance (Top-1) 1) and robustness to choice of base stepsize (other metrics).
A.9 FREEZING MOST UNCERTAIN LAYERS.
Since some of the BatchNorm parameters have the highest uncertainty (see Figure la), we have experimented freezing the BatchNorm parameters during meta-testing (i.e. setting their stepsize to zero). It turns out that SGD+All (w/ freezing BN) outperforms SGD+AIl (wlo freezing BN) on the All metric (Table 10), with most improvement in the higher stepsize range (>0.1) (Figure 12). We
15

---

Published as a conference paper at ICLR 2021
0.66
0.52
0.75
078
0:48
0.7
044
04
055
0.6
0:30
0.5
04
0445
0.36
0.20
04
0.28
AT
ALL
0.2
0000
00000
00000
0.00
0.001
0,007
06
07
0000
00005
0000
0.01
0.00
02
00004
0000
0.08
0.04
0007
00004
00007
0100
0.888
0000
04
07
0033
0.24
0442
0399
0.33
039
0.34
018
0.66
006
013
022
026
320
0.14
COON
BLOCK
0.000
0007
004
0.000
0.00
000
0000
0.008
0.00
0C0
0000
0L008
000
00
(a) Mini
(b) Birds
Flowers
(d) Signs
Figure 7: 5-way 5-shot (Top) and 10-way I-shot (Bottom) classification with MAML(SGD) and our proposed method. In terms of robustness, our method outperforms MAML(SGD) more effectively. In addition, we show that our method outperforms Top-1 accuracy for Birds, Flowers and Signs. Also our method shows flatter curve near highest accuracy sections. This increases the probability of optimal stepsize selection.
0.36
0.52
0.44
OLM
04
033
0.35
032
0.24
023
02
0000
0000
0
0,00
00000
00000
00000
0.034
02
00000
0.
02
00000
01007
0OX
0.000
000
(a) Mini
(b) Birds
(c) Flowers
(d) Signs
Figure 8: 5-way I-shot classification for verifying proposal We compared USA and I/USA.
notice a similar trend for 5-way 5-shot SGD, where SGD (w/ freezing BN) also outperforms SGD (w/o freezing BN) on the All metric, with most improvement in the higher stepsize range.
16

---

Published as a conference paper at ICLR 2021
Table 7: 5-way 1-shot classification results for verifying proposal and 2. I/USA and I/FFGSM degraded performance than USA and UFGSM respectively. Our approach which is applied USA and UFGSM outperformed SGD on almost of the metrics.
Baseline
SGD
SGD+USA
Metric
Dataset
SGD
Adam
+USA
+I/USA
+UFGSM
+I/UFGSM
+FGSM
Mini
37.21
31.85
37.53
28.73
38.71
37.25
36.97
Birds
30.07
28.17
30.39
24.98
31.07
30.27
30.05
All
Flowers
40.54
40.49
41.09
30.52
42.33
41.44
41.17
Signs
34.21
34.28
35.02
27.90
36.10
34.98
34.78
Avg.
35.50
33.68
36.01
28.03
37.05
35.98
35.74
Mini
46.71
43.07
46.72
44.13
46.66
44.31
43,93
Birds
36.03
34.30
36.08
34.20
36.02
34.77
34.52
Top-1
Flowers
51.97
51.38
52.01
49.34
52.30
50,90
50.63
Signs
43.44
43.00
43.42
41.03
43.96
42.82
42.62
Avg.
44.54
42.93
44.56
42.17
44.73
43.20
42.93
Mini
45.07
40.16
45.23
38.33
45.76
43.60
43.18
Birds
34.88
32.64
35.17
30.28
35.30
34.20
33.88
Top-40%
Flowers
49.98
48.28
50.39
42.19
51.07
49.79
49.47
Signs
41.81
41.32
42,08
35.78
43.18
42.17
41.92
Avg.
42.94
40.60
43.22
36.64
43.83
42.44
42.11
044
041
04
032
0.4
0.88
0.36
03
0.88
0.32
028
0.25
-UFFFF
0.32
029
038
006
0.24
02
00001
0000
00000
0000
0000
10000
61
00001
00005
TLCOM
000
0.07
02
0000%
00000
0000
000%
0000
0000%
00000
0000
000%
0000
(a) Mini
(b) Birds
(c) Flowers
(d) Signs
Figure 9: 5-way 1-shot classification results to verify proposal 2.
We compared to UFGSM,
VUFGSM and FGSM.
08
200
500
(a) Uncertainty map (u)
(b) Flatten uncertainty
(c) Histogram uncertainty
Figure 10: (a) shows the re-scaled input gradient uncertainty to generate the UFGSM example. (b)
is a flatten plot of the re-scaled input gradient uncertainty. (c) shows histogram of the uncertainty.
17

---

Published as a conference paper at ICLR 2021
250
20
150
04
02
5000
10000
(a) Uncertainty map (u)
(h) Flatten uncertainty
(c) Histogram of uncertainty
Figure 11: (a) shows the re-scaled inverse input gradient uncertainty to generate the UFGSM exam- ple. (b) is flatten plot of the re-scaled inverse input gradient uncertainty. (c) shows histogram of the uncertainty.
Table 8: 5-way 5-shot classification results between selected checkpoint with SGD
Checkpoint (Validation)
Checkpoint (Last)
Metric
Dataset
Baseline
Ours (SGD)
Baseline
Ours (SGD)
SGD
AT Only
All
SGD
AT Only
All
Mini
38. 14...
41,06
43.48
34.98 0.54
38,664052
41,89+0.1
(-)
(2.92)
(5.34)
(-)
(3.68)
(6.91)
Birds
33. 33.29
36.042o.xx
38.07
31.33_as1
34.3440.p
36,90405
(-)
(2.75)
(4.78)
(-)
(3.01)
(5.57)
Flowers
44.55
48.94
53.10
40.96
40.96-087
45.620.
50,58
All
(-)
(4.39)
(8.55)
(-)
(4.64)
(9,62)
Signs
45.39
47.98+4.15
51.02
42.05 0.35
46,01
49,88 +-0.5
(2.59)
(5.63)
(-)
(3,96)
(7.83)
Avg.
40.34-0.30
43.51
46.42
37.33_a32
41.1520.9
44.81 +3.53
(3.17)
(6.08)
(3.82)
(7.48)
Mini
62.46
62.07
62.16
58.49 0.72
58.44
58.72+1.
(-0.39)
(-0.3)
(-0.05)
(0.23)
Birds
50.71
51.93:10.2
51.821064
46.77=05
48.33
48.58107
(1.22)
(1.11)
(-)
(1.56)
(1.81)
Flowers
70.63
74.14.0.2
74.52
66.31
68.994081
69. 69.731071
Top-1
(-)
(3,51)
(3.89)
(-)
(2.68)
(3.42)
Signs
72.52
73,66
74.13
67.75
70.71
71,08
(-)
(1.14)
(1.61)
(-)
(2.96)
(3.33)
Avg.
64.08
65.45-073
65.66
59.83
61.62+073
62,03
(-)
(1.37)
(1.58)
(-)
(1.79)
(2.2)
Mini
55.38 +0.46
59.97
60.70
50.56
56.44
57.22+1
(-)
(4.59)
(5.32)
(-)
(5.88)
(6.66)
Birds
50.4820.5 20.5020.40
51.03
42. 0.55
47.221073
47.98
(4.98)
(5.53)
(5.09)
(5,85)
Flowers
72.61
73,55 1
59.8 1.00
67.37 4:833
68, 9:.99
Top-40%
(-)
(7.45)
(8.39)
(-)
(7.57)
(9.13)
Signs
67.34+045
70.564087
71.45
61.95_9.6
68.03+064
69.41 +aus
(3.22)
(4.11)
(6.08)
(7.46)
Avg.
58.3410.53
63,40
64.181083
53.61 0074
59.77
60.89
(5.06)
(5.84)
(6.16)
(7.28)
18

---

Published as a conference paper at ICLR 2021
Table 9: 5-way 5-shot classification results of selected checkpoint with Adam
Checkpoint (Validation)
Checkpoint (Last)
Metric
Dataset
Baseline
Ours (Adam)
Baseline
Ours (Adam)
Adam
AT Only
All
Adam
AT Only
All
Mini
40.08 1.33
41.38+1.or
41.64
37.25 0.64
39.47
40.36+0az
(1.3)
(1.56)
(-)
(2.21)
(3.1)
Birds
36. 1..
38.79=1
38.78
35.39_047
37.861072
38.15
(-)
(2.11)
(2.10)
(2.47)
(2.76)
Flowers
18.5
60.92.1.24
60.362114
54. 54.36_0.99
59.092m61
59.32
All
(2.92)
(2.36)
(4.73)
(4.96)
Signs
52.65 0.57
52.37=0.6
52.53
50.48
51.1310.45
51,95
(-0.28)
(-0.12)
(-)
(0.65)
(1.47)
Avg.
46.85
48.37
48.33+091
44.37=067
46.89
47.45+002
(1.52)
(1.48)
(2.52)
(3.08)
Mini
58.89
60.35
60.08
55.25 1.02
57.79
57,86+052
(-)
(1.46)
(1.19)
(-)
(2.54)
(2.61)
Birds
47.92
50.88
50.84
46.02_085
49.89=1
49.78
(-)
(2.96)
(2.92)
(-)
(3.87)
(3.76)
Flowers
73.21
77.51
77.45
69.09 0.77
76.611051
76.51
Top-1
(-)
(4.3)
(4.24)
(-)
(7.52)
(7.42)
Signs
70.98
71.30
71.72++152
67.51
69.75+0x3
70.30
(-)
(0.32)
(0.74)
(2.24)
(2.79)
Avg.
62.75
65.01
65.02.11
59.47=084
63.51
63.61
(-)
(2.26)
(2.27)
(4.04)
(4.14)
Mini
52.17
54.96=1.18
55.75+1.07
46.95_033 (-)
51.81 1071
53.29
(-)
(2.79)
(3.58)
(4.86)
(6.34)
Birds
43.88 1.06
47.931091
48.261007
41.41 (-)
46.71
1
(4.05)
(4.38)
(5.3)
(5.7)
Flowers
68.43
73.86+1.02
74.03
63.44_077
72.41 +0079
72.56+:044
Top-40%
(5.43)
(5.60)
(8,97)
(9.12)
Signs
66.85
67.36=1.
67.89
62.95 081
65.43+008
66.37+071
(0.51)
(1.04)
(2.48)
(3.42)
Avg.
57.83
61.03-1.1
61.48
$3.69
59.09
59.83
(3.2)
(3.65)
(5.4)
(6.14)
055
048
0.80
046
0.42
048
06
026
028
0.36
0120
000%
00007
0,03
2004
100
000
00001
00000
1000
0000
00008
00000
6000
8608
8000
0.08
00004
000P
0008
0.04
000
060
004
0.88
036
044
000
S-Wo
00000)
0.000
0L00X
00008
0,000
0000
1000
R000
000000
0.0%
0000
00004
0,0000
BLE
000
Suppins
(a) Mini
(b) Birds
(c) Flowers
(d) Signs
Figure 12: 5-way I-shot (Top) and 5-way 5-shot (Bottom) classification results freezing BN
layers. We tested freezing BN on SGD+AIL. SGD+All (w/ BN freezing) outperformed SGD+All (w/o BN freezing) in the higher stepsize range (>0.1)..
19

---

Published as a conference paper at ICLR 2021
Table 10: 5-way 1-shot and 5-way 5-shot classification results freezing BN layers. We tested freezing BN SGD+AIL. SGD+AIl (w/ freezing BN) outperformed in All metric 5-way 1-shot and 5-way 5-shot.
5-way 1-shot (SGD+AII)
5-way 5-shot (SGD+AlI)
Metric
Dataset
w/ freezing BN
w/o freezing BN
w/ freezing BN
w/o freezing BN
Mini
37.13
36.24
45.42_mx
43.48190
Birds
30.74:4.0
29.8 23.83
40.29-..m
38.07
Flowers
43.88
40.86.04
59.41.nnx
53.10100s
Signs
36.37
34.76gnm
54.65
$1.0210.9
Avg.
37.03:
35.42:u44
49.94
46.42
Mini
46.48
46.62+05s
62.18 18-
62.
Birds
36.41
36.43
51.90
51.82
Top-1
Flowers
54,16=ss0
54.36=1.54
74.59=000
74.52+a00
Signs
44.18
44.10+1.0
73.98-0os
74.13
Avg.
45.3
45.37
66.606
65.6629772
Mini
45.44
45.792n4
60.70
Birds
35.67
35.77=a555
04.443
51,03
Top-40%
Flowers
52.97
53.1614m
73.56_007
73.551os
Traffic
43.38
43.39210
71.21
71 71,45
Avg.
44.37
44.52,0s
64.1 1
14.1819
20


Reviews:
[{"review_id": "OmTB4AFtCwv", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary \nThis paper considers the problem of adapting a pre-trained model for few-shot learning in case there is a shift of distribution from the meta-training set. If the new tasks significantly differ from the meta-training distribution the model might need to be retrained from scratch but this is not always possible, so the authors propose to \"repurpose\" the model under the assumption that the support set can be used to re-calibrate the pre-trained model to the new shifted distribution.\nFollowing two intuitions/hypotheses: 1) if the model uncertainty of specific parameters is high, then the step size should be small and 2) high uncertainty on input gradients require more adversarial training to improve robustness.\nIn practice, the uncertainty of the parameters is computed by using deep ensembles with perturbed MAML checkpoints rather than random initialization. High variance components are moved with lower step sizes. Moreover, If slightly perturbed models from the deep ensemble disagree on parts of the input gradient, then it means that they might be more prone to adversarial attacks, hence they need to be robustified with stronger adversarial training.\n\n### Questions\n- Q0a: maybe I'm missing the point, but am I wrong or the procedure that you propose can be also applied to any other supervised learning model, not necessarily few-shot? Let's consider you are not only observing new classes, but also new domains. Does the method apply to that situation? \n- Q0b: also the use of deep ensemble and the proposed UFGSM maybe can be useful in the supervised learning scenario to improve robustness. I would suggest to try some experiments in that direction.\n- Q1: from plots in figure 1 it seems that batchnorm parameters are the most uncertain. This is expected since BN capture the statistics of the activations from the meta-distribution and it is the most sensitive to the domain shift. Are you using batch-norm in a transductive setting, i.e. you don't use the running stats, but you always use the batch statistics also at test time? \n    - If no, and you use running stats, then a simple baseline would be to use test time statistics, so I would add this comparison to your experiments.\n    - If yes, I would also try not to update the BN scale and shift parameters in the inner loop since they seem very sensitive. Another alternative would be to have per-step running stats and per-step bn weights as proposed in Antoniou 2019.\n- Q2: the improvement wrt to the SGD baseline can be more appreciated with 5 shots, rather than only 1 sample. I would have expected the opposite because of the limited number of samples. How do you interpret this? \n- Q3: what happens if you try PGD instead of simply using FGSM? \n\n### Considerations\n- I think the paper is well written and motivated. The idea of repurposing a FSL model without retraining from scratch is timely and interesting. The experimental campaign is carefully performed as well as the ablation. I recommend acceptance, but I want to first to hear the author's response to my doubts.\n\n### Bib\nAntoniou et al 2019 How to train your maml\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "HknhhvkTv7-", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper tries to solve a novel practical problem: adapting pretrained meta-learning checkpoints to out-of-domain test tasks. It is an important problem in industry since there are some industrial tasks, especially meta-learning tasks, that have very limited data. Therefore, the meta-training process which requires a relatively large data set is not practical. \n\nTowards this end, the paper proposes to run ensembling fine-tuning with the support set of the test task, and use the weight variances from this process to guide the training.  The paper also proposes to add task adversarial examples to the training set to help the meta fine-tuning process. Experimental results seem promising.\n\nOne main concern I have in mind is, existing domain adaptation or domain transfer methods are trying to solve very similar problems, i.e., the model is pre-trained with large data set from other domains and you need to adapt it to the new domain. What are the differences between the current problem and domain transfer/adaptation problems? Why can’t we apply existing domain adaptation methods to solve the current problem?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ENuFZ4RneTa", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to reutilize pretrained MAML checkpoints for out-of-domain few-shot learning, combining with uncertainty-based adversarial training and deep ensembles.\n\nPros:\n\n1. The idea of combining meta-learning, uncertainty learning and adversarial training is well-structured. In particular, the related work part provides a clear introduction of background work.\n\n2. It is quite novel to leverage adversarial learning as data augmentation for meta-testing in MAML.\n3. The paper provides extensive and convincing experiment results over evaluating the proposed model’s robustness to the choice of base stepsizes.\n\nCons:\n\n1. It would be better if an optimization equation is provided, especially if there are generated adversarial examples.\n\n2. For the ablation study, the authors mention that the best absolute performance (Top-1) is always obtained through some use of adversarial training. Actually, it would be more convincing if they can discuss more choices of \\lambda_{AT} and \\lambda_{AUG} and \\lambda_{a} to present the sensitivity analysis of the hyper-parameters.", "extra_scores": {}, "extra_text": {}}, {"review_id": "8nkIPEcOKyb", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[Summary]\nMAML has two stages: (1) In the meta-training stage, given various tasks (but each say only has a few labeled data), we want to arrive at a representation where it can quickly adapt to any test task later. Let us denote this as $\\theta_0$ (2) In the meta-testing stage, we go from $\\theta_0$, given the few-shot test data, via an SGD process (typical scenario), to get a final model that has good performance on the test data of the test task.\n\nIn MAML, we assume that tasks are (i.i.d.) sampled from a distribution of tasks P(T). This paper considers a setting (as I understand) where the test task is out-of-distribution of P(T). In this setting, the authors argued further that one cannot run the meta training, so the focus of the paper is to improve stage (2) (gradient update process at the test time).\n\nThe proposed method contains two main parts: (A) We use an ensemble method to help understand uncertainty of the network, and finally use different step sizes for different layers for updating the network parameters at test adaptation phase, (B) adversarial training as data augmentation to help the test-time adaptation process. Experiments show that this method can bring some performance improvements\n\n[Assessment]\n1. I am not sure I entirely accept the authors' statement about testing tasks being out-of-distribution compared to the training tasks. In the original MAML setting, the test tasks can be entirely different from the training tasks (for example, we pick test tasks from different classes). Further, it has always been the case in MAML, that one can store a checkpoint $\\theta_0$ trained from some tasks, and  when a new task (maybe entirely different) comes, we start from $\\theta_0$ to adapt. This renders the motivations of this work less meaningful. And indeed, if we look at the performance numbers, the improvement is somewhat marginal.\n\n2. More specifically, at the very least, I think the precision of the paper can be improved. If MAML's purpose is already to be able to adapt to new, unseen, tasks, what is the precise meaning of out-of-distribution tasks here? (for example, one interpretation of this, stated in a formal way, is what I described above as saying that the new task is out-of-distribution of P(T))\n\n3. I do like the use of ensemble and also adversarial examples, and intuitively they can improve the test-time adaptation. However, the arguments for why they are useful are quite ad-hoc (to a degree that I feel superficial, no offense). Can we better justify their usage?\n\n4. What is the computational overhead in order to gain the additional accuracy? The use of ensemble together with adversarial training seems to bring a lot of computation cost, which seems not a good deal in view of the accuracy gain.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper considers a new and practical setting of meta-learning for out-of-domain task adaptation where a pretrained model exists but the original meta-training data is not available. The authors incorporate several ideas including deep ensembles, adversarial training and uncertainty-based step sizes, and achieve competitive performance under this particular setting.

The combination of various methods appears complicated, but the authors provide detailed ablation study to show the effectiveness of each component empirically. During rebuttal and discussion, they addressed many of the concerns from the reviewers. As pointed out by a reviewer, their proposed method would have a value in the domain adaptation area beyond meta-learning.

The remaining concern is on the somewhat ad-hoc combination of multiple methods and lack of a clear single solution for addressing the OOD few-shot learning problem. Nonetheless, the proposed methods show a convincing empirical improvement on the vanilla MAML baseline in the experiments.
Decision: Accept (Poster)