Meta-free few-shot learning via representation learning with weight averaging
Authors: Kuilin Chen
Chi-Guhn Lee
Venue: ICLR 2022 Submitted (2022)
Keywords: few-shot learning
representation learning

Abstract: Recent studies on few-shot classification using transfer learning pose challenges to the effectiveness and efficiency of episodic meta-learning algorithms. Transfer learning approaches are a natural alternative,  but they are restricted to few-shot classification. Moreover, little attention has been on the development of probabilistic models with well-calibrated uncertainty from few-shot samples, except for some Bayesian episodic learning algorithms. To tackle the aforementioned issues, we propose a new transfer learning method to obtain accurate and reliable models for few-shot regression and classification. The resulting method does not require episodic meta-learning and is called meta-free representation learning (MFRL). MFRL first finds low-rank representation generalizing well on meta-test tasks. Given the learned representation, probabilistic linear models are fine-tuned with few-shot samples to obtain models with well-calibrated uncertainty. The proposed method not only achieves the highest accuracy on a wide range of few-shot learning benchmark datasets but also correctly quantifies the prediction uncertainty. In addition, weight averaging and temperature scaling are effective in improving the accuracy and reliability of few-shot learning in existing meta-learning algorithms with a wide range of learning paradigms and model architectures.

Under review as a conference paper at ICLR 2022
META-FREE FEW-SHOT LEARNING VIA REPRESENTA- TION LEARNING WITH WEIGHT AVERAGING
Anonymous authors Paper under double-bling review
ABSTRACT
Recent studies few-shot classification using transfer learning pose challenges to the effectiveness and efficiency of episodic meta-learning algorithms. Transfer learning approaches are natural alternative, but they are restricted to few-shot classification. Moreover, little attention has been on the development of proba- bilistic models with well-calibrated uncertainty from few-shot samples, except for some Bayesian episodic learning algorithms. To tackle the aforementioned issues, we propose new transfer learning method obtain accurate and reliable models for few-shot regression and classification. The resulting method does not require episodic leta-learning and is called meta-free representation learning (MFRL). MFRL first finds low-rank representation generalizing well on meta-test tasks. Given the learned representation, probabilistic linear models are fine-tuned with few-shot samples to obtain models with vell-calibrated uncertainty. The proposed method not only achieves the highest accuracy on wide range of few-shot learn- ing benchmark datasets but also correctly quantifies the prediction uncertainty. In addition, weight averaging and temperature scaling are effective in improving the accuracy and reliability of few-shot learning in existing meta-learning algorithms with a wide range of learning paradigms and model architectures.
INTRODUCTION
Currently, the vast majority of few-shot learning methods are within the general paradigm of meta- learning (a. learning to learn) (Bengio et al. 1991, Schmidhuber] 1987 Thrun & Pratt 1998). which learns multiple tasks in an episodic manner to distill transferrable knowledge (Vinyals et al.) 2016 Finn et al. 2017: Snell al. (017). Although many episodic meta-learning methods report state-of-the-art (SOTA) performance, recent studies show that simple transfer learning methods with fixed embeddings (Chen et al.) 2019 Tian al,, 2020) can achieve similar or better performance in few-shot learning. is found that the effectiveness of optimization baasdd meta-learning algorithms due to reusing high-quality representation, instead of rapid learning of task-specific representation (Raghu et al. 2020). The quality of the presentation is not quantitatively defined, except for some empirical case studies (Gooldblum et al. 2020). Recent machine learning theories (Saunshi et al.) 2021 indicate that low-rank representation leads better sample efficiency learning new task. However, those theoretical studies are within the paradigm of meta-learning and do not reveal how obtain low-rank representation for few-shot learning outside the realm of meta-learning. This motivates us to investigate ways to improve the representation for adapting to new few-shot tasks in meta-free manner by taking the advantage of simplicity and robustness in transfer learning.
In parallel, existing transfer learning methods also have limitations. That is, the existing trans- fer learning methods may not find representation generalizing well to unseen few-shot tasks CCen et al. 2019 Dhillon et al., 2020). compared with -tate-of-the-art meta-learning methods (Ye et al.) 2020 Zhang et al., 2020). Although some transfer learning methods utilize knowledge distillation and self-supervised training to achieve strong performance in few-shot classification, they are re- stricted to few-shot classification problems (Mangla et al.) 2020 Tian et al. 2020). To the best of knowledge, no transfer learning method is developed achieve similar performance to meta- learning in fhw-shot regression. As such, it is desirable to have a transfer learning method that finds high-quality representation generalizing well unseen classification and regression problems.

---

Under review as a conference paper at ICLR 2022
The last limitation of the existing transfer learning methods in few-shot learning is the lack of un- certainty calibration. Uncertainty quantification is concerned with the quantification of how likely certain outcomes are. Despite plethora of few-shot learning methods (in fact, machine learning in general) to improve the point estimation accuracy, few methods are developed to get probabilistic models with improved uncertainty calibration by integrating Bayesian learning into episodic meta- training (Grant et al., 2018; Finn et al., 20188; Yoon et al., 2018 Snell & Zemel (021). Few-shot learning models can used in risk-averse applications such as medical diagnosis (Prabhu et al. 2019). The diagnosis decision is made on not only point estimation but also probabilities associated with the prediction. The risk of making wrong decisions is significant when using uncalibrated mod- els (Begoli et al. 2019). Thus, the development of proper fine-tuning steps to achieve well-calibrate models is the key towards practical applications of transfer learning few-shot learning.
In this paper, we develop simple transfer learning method as our own baseline to allow easy regu- larization towards more generalizable representation and calibration of prediction uncertainty. The regularization in the proposed transfer learning method works for regression and classification prob- lems so that we can handle both problems within a common architecture. The calibration procedure is easily integrated into the developed transfer learning method to obtain few-shot learning models with good uncertainty quantification. Therefore, the resulting method, called Meta-Free Repre- sentation Learning (MFRL), overcomes the aforementioned limitations in existing transfer learning methods for few-shot learning. Our empirical studies demonstrate that the relatively overlooked transfer learning method can achieve high accuracy and well-calibrated uncertainty in few-shot learning when it is combined with the proper regularization and calibration. Those two tools are also portable to meta-learning methods improve accuracy and calibration, but the improvement less significant compared with that of transfer learning.
We use stochastic weight averaging (SWA) (Izmailov al., 2018), which agnostic to loss function types, as implicit regularization to improve the generalization capability of the representation. We also shed light on that the effectiveness of SWA is due to its bias towards low-rank representation. To address the issue of uncertainty quantification, we fine-tune appropriate linear layers during the meta-test phase to get models with well-calibrated uncertainty. In MFRL, hicrarchical Bayesian linear models are used to properly capture the uncertainty from very limited training samples in few-shot regression, whereas the softmax output is scaled with a temperature parameter to make the few-shot classification model well-calibrated. Our method is the first one achieve well-calibrated few-shot models by only fine-tuning probabilistic linear models in the meta-test phase, without any learning mechanisms related to the meta-training or representation learning phase.
Our contributions in this work are summarized as follows:
We propose a transfer learning method that can handle both few-shot regression and clas- sification problems with performance exceeding SOTA.
For the first time, we empirically find the implicit regularization of SWA towards low-rank representation, which is useful property in transferring few-shot tasks.
The proposed method results in well-calibrated uncertainty in few-shot learning models while preserving SOTA accuracy.
The implicit regularization of SWA and temperature scaling factor can be applied to exist- ing meta-learning methods to improve their accuracy and reliability in few-shot learning.
2 RELATED WORK
Episodic meta-learning approaches can be categorized into metric-based and optimization-basee methods. Metric-based methods project input data to feature vectors through nonlinear embeddings and compare their similarity to make the prediction. Examples of similarity metries include the weighted L1 metric (Koch et al. 2015), cosine similarity (Qi et al.) 2018) Vinyals et al., 2016), and Euclidean distance to class-mean representation (Snell et al. (0017). Instead of relying on predefined metrics, learnable similarity metrics are introduced to improve the few-shot classification perfor- mance (Oreshkin et al. 2018 Sung et al., 2018). Recent metric- approaches focus on devel- oping task-adaptive embeddings to improve shot classification accuracy. Those task-adaptive embeddings include attention mechanisms for feature transformation (Fei et al. [022]] Gidaris &
Komodakis 2018
Ye et al. 2020) Zhang et al. 2021), graph neural networks (Garcia & Estrach]
2

---

Under review as a conference paper at ICLR 2022
2018). implicit class representation (Ravichandran et al. 2019), and task-dependent conditioning Oreshkin et al.) 2018 Yoon et al. 2020; 2019). Although metric- aetric-based approaches achieve strong performance in few-shot classification, they cannot be directly applied to regression problems.
Optimization-bai based meta-learning approaches try to find transferrable knowledge and adapt to new tasks quickly. An elegant and powerful meta-learning approach, termed model-agnostic meta- learning (MAML), solves a bi-level optimization problem to find good initialization of model pa- rameters (Finn al. 2017). However, MAML has a variety of issues, such sensitivity to neural network architectures, instability during training, arduous hyperparameter tuning, and high com- putational cost. On this basis, some follow-up methods have been developed to simplify, stabilize and improve the training process of MAML (Antoniou et al. 2018 Flennerha et al. 2020) Lee & Choi 2018 Nichol et al. 2018, Park & Oliva 2019). practice, is very challenging high-dimensional model parameters in low-data regime. Latent embedding optimization (LEO) attempts to learn low-dimensional representation to generate high-dimensional model parameters (Rusu et al. 2019). Mcanwhile, R2-D2 (Bertinetto et al. 2019) and MetaOptNet (Lee et al. 2019) reduce the dimensionality of trainable model parameters by freezing feature extraction layers during inner loop optimization. Note that the proposed method is fundamentally different from R2-D2 and MetaOptNet because our method requires neither episodic meta-leaminng nor bi-level optimization.
learn
Transfer learning approaches first learn feature extractor on all training data through standard supervised learning, and then fine-tune a linear predictor on top of the learned feature extractor in a new task (Chen et al. (011)). However, vanilla transfer learning methods for few-shot learning do not take extra steps to make the learned representation generalizing well to unseen meta-test tasks. Some approaches in this paradigm are developed to improve the quality of representation and boost the accuracy of few-shot classification, including cooperative ensembles (Dvornik et al.) (2019), knowl- edge distillation (Tian et al. 2020), and auxiliary suf--sueereeeed learning (Mangla al. 2020). Nevertheless, those transfer learning methods are restricted to few-shot classification. MFRL aims to find representation generalizing well from the perspective of low-rank representation learning, which is supported by recent theoretical studies (Saunshi et al. (02T). Furthermore, MFLR is the first transfer learning method that can handle both few-shot regression and classification problems and make predictions with well-calibrated uncertainty.
3 BACKGROUND
3.1 EPISODIC META-LEARNING
In episodic meta-learning, the meta-training data contains T episodes tasks, where the 7h episode consists of data D, ably in the rest of the paper. Episodic meta-learning algorithms aim to find common model param- eters which can be quickly adapted to task-specific parameters dr (T = I, D). For example, MAML-type algorithms assume o is one or a few gradient steps away from 0 (Finn et al.) 2017] Grant et al. 2018: Yoon et al. (2018), while other meta-learning approaches assume that é, and share the parameters in the feature extractor and only differ in the top layer (Bertinetto et al.
(r.)) with N, samples. Tasks and episodes are used interchange-
2018
2019
Lee et al. 2019, Snell et al. 2017).
3.2
STOCHASTIC WEIGHT AVERAGING
The idea of stochastic weight averaging (SWA) along the trajectory of SGD goes back to Polyak-Ruppert averaging (Polyak & Juditsky 1992). Theoretically, weight averaging results in faster convergence for linear models in supervised learning and reinforcement learning(Bach & Moulines 2013, Lakshminarayanan & Szepesvari 2018). deep learning, we are more interested in tail stochastic weight averaging (Jain et al., 2018), which averages the weights after T training epochs. The averaged model parameters OSwA can be computed by running s additional training epochs using SGD
T+s
BSWA
1
Σ
8,
(1)
8
i=++1
3

---

Under review as a conference paper at ICLR 2022
where 9g denotes the model parameters the end of the i-th epoch. SWA has been applied to supervised learning of deep neural neural networks to achieve higher test accuracy (lzmallov et al. 2018).
4 METHODOLOGY
The proposed method is two-step learning algorithm: meta-free representation learning followed by fine-tuning. We employ SWA to make the learned representation low-rank and better generalize top layer is fine-tuned with few-shot samples to obtain probabilistic models with well-calibrated uncertainty. Note that MFRL can be used for both regression and classification depending on the loss function. The pseudocode of MFRL is presented in Appendix A.I
to meta-test data. Given meta-test task,
4.1 REPRESENTATION LEARNING
Common representation can be learned via maximization of the likelihood of all training data with respect to 0 rather than following episodic meta-learning. To do so, we group the data D, = {xx Nr,5) ( } N- from all meta-training tasks into a single dataset Dur- Given aggregated
f=1
training data Dtr = {X,Y Y), representation can be learned by maximizing the likelihood (Drr () with respect to Let A = [0y, W]. where f represents parameters in the feature extractor and W denotes the parameters in the top linear layer. The feature extractor h(x) E IP is neural network parameterized by O and outputs a feature vector of dimension p. The specific form of the loss function depends on whether the task is regression classification and can be given as follows:
[MMSE(f), regression
CRP(O) - log (Dir (0)
(C) classification
where
(0)
(Ur - w h (xx=)))
(2)
2N'
1=15=
N'
exp(w. T h(xx))
(C = Vi. log
Σe expl w 1 ((xx))
For regression problems, the model learns T regression tasks (W- [W1, WT] E R(++1)xT) simultaneously using the loss function LMSE given in Eq. 2 whereas the model learns C-class classification model (W- = [WI..., wel E R 1)C+ for classification problems using the loss function Lce Eq. The loss function either Eq. or 3 can be minimized through standard
stochastic gradient descent, where N' = Σ Nr, is the total number of training samples.
Post-processing via SWA Minimizing the loss functions in Eq. 3ndd3by SGD may not necessarily result in representation that generalizes well to few-shot learning tasks the meta-test set. The last hidden layer of modern deep neural network is nigh-dimensional and may contain spurious features that over-fit the meta-training data. Recent meta- theories indicate that better sample complexity in learning new task be achieved via low-rank representation, whose singular values decay faster (Saunshi et al. 2021). We aim to find low-rank representation & h(X) without episodic meta-learning, which is equivalent finding the conjugate kernel R ΦΦ with fast decaying cigenvalues. To link the representation with the parameter space, we can lincarize the neural network by the first-order Taylor expansion or and get the finite width neural tangent kernel (NTK) KNTK (X, X) = J(X) where J(X) Vo)e(X) E N'|0) is the Jacobian matrix, and KNTK is composite kernel containing K" (Fan & Wang 2020). The distributions of eigenvalues for KNTK and KC are empirically similar. Analyzing NTK could shed light on the properties of KC In parallel, KNTK shares the same cigenvalues of the Gauss-Newte matrix G = (X) (X). For linearized networks with squared loss, the Gauss-Newton matrix well
'C is the total number of classes in Dzr- Learning C-class classification model solves all possible tasks in the meta-training dataset because each task D, only contains subset of C classes.
4

---

Under review as a conference paper at ICLR 2022
approximates the Hessian matrix H when y is well-deseribed by fo(x) Marrtens 2020 This is the case when SGD converges to 0r within a local minimum basin. A Hessian matrix with a lot of small cigenvalues corresponds perturbation of model parameters (Keskar et al.) 2017). It is known that averaging the weights after SGD convergence in local minimum basin pushes 0r towards the flat side of the loss valley (He) et al. 2019). As result, SWA could result in a faster decay of eigenvalues in the kernel matrix, and thus low-rank representation. Our conjecture about SWA as implicit regularization towards low-rank representation is empirically verified in Section 5
flat minimum. where the loss function is less sensitive to the
4.2 FINE-TUNING
After representation learning is complete, W is discarded and 0 is frozen a new few-shot task. Given the learned representation, train new probabilistic top layer in meta-test task using few- shot samples. The new top layer will be configured differently depending on whether the few-shot task a regression or a classification problem.
In a few-shot regression task, we learn a new linear regression model y = w h((x) + on fixed feature extractor h(x) TR with few-shot training data D {(x, y: ]=)] |
where denotes the model parameters and is Gaussian noise with zero mean and variance a². To avoid interpolation
on few-shot training data (n < p), a Gaussian prior p(w ^) II."o .V (w;; 0, A) placed over w, where À is the precision in the Gaussian prior. However, it is difficult to obtain an appropriate value for À in few-shot regression task because no validation data is available in D.
Hierarchical Bayesian linear models can be used to obtain optimal regularization strength and grounded uncertainty estimation using few-shot training data only. To complete the specifica- tion of the hierarchical Bayesian model, the hyperpriors on À and a² are defined as p()) = c, d), respectively. The hyper-priors become very flat and non-informative when b, and d are set to very small values. The posterior over all latent variables given the data is (w, A, x, o² y) where X = {x} and y = {v ) However, the posterior distribution (w X, y) is intractable. The iterative optimization based approxi- mate inference (Tipping 2001 is chosen because it is highly efficient. The point estimation for and o² is obtained by maximizing the marginal likelihood function p (y X, x, The posterior of model parameters p (w X,y, X, o²)) is calculated using the estimated À and o². Previous two steps are repeated alternately until convergence.
Gamma (X a, b) and p(a
Gamma(o-2
The predictive distribution for new sample x., is
p (v.1 x. X.y, = (v. x., x,),,o") X,y, dw,
(4)
which can be computed analytically because both distributions on the right hand side of Eq. are Gaussian. Consequently, hierarchical Bayesian linear models avoids over-fitting on few-shot training data and quantifies predictive uncertainty.
In a few-shot classification task, a new logistic regression model is learned with the post-processed representation. typical K-way n-shot classification task D {(x, 34) )=) consists of K classes (different from meta-training classes) and training samples per class. Minimizing an un-regularized cross-entropy loss results in a significantly over-confident classification model be- cause the norm of logistic regression model parameters W E [[++1)xK becomes very large when few-shot training samples can be perfectly separated in the setting nK < A weighted L2 regu- larization term is added to the cross-entropy loss to mitigate the issue
nk K
Mi. log exp(w h(xx)) ))
((W) = Σ
Σ c=1
w We:
(5)
where À is the regularization coefficient, which affects the prediction accuracy and uncertainty. is difficult to select an appropriate value of À in each of the meta-test tasks due to the lack of validation data in D. We instead treat À as a global hyper-parameter so that the value of À should be determined based on the accuracy validation accuracy does not necessarily lead to well calibrated classification models. As such, we introduce the temperature scaling factor (Guo al. 2017) as another global hyper-parameter to
meta-validation data. Note that the selected À with high
5

---

Under review as a conference paper at ICLR 2022
scale the softmax output. Given test sample xx, the predicted probability for class becomes
cxp(w.,(xx.)/ Pe Ee= exp(w
(6)
where 7' is the temperature scaling factor. In practice, we select the 1.2 regularization coefficient X and the temperature scaling factor 7 follows. first, we set 7 1, and do grid search on the mecaavalidattio data to find the À resulting in the highest meta-validation accuracy. However, fine- tuning does not ensure good calibration. It is the temperature scaling factor that ensures the good uncertainty calibration. Similarly, we do grid search of T the meta-validation set, and choose the temperature scaling factor resulting in the lowest expected calibration error (Guo et al. 2017). Note that different values of T do not affect the classification accuracy because temperature scaling is accuracy preserving.
5 EXPERIMENTS
We follow the standard setup in few-shot learning literature. The model trained on a meta-training dataset and The final performance of the model is evaluated on a meta-test dataset. The proposed method is applied to few-shot regression and classification problems and compared against wide range of alternative methods.
selected based the performance meta-validation dataset.
5.1 FEW-SHOT REGRESSION RESULTS
Sine waves (Finn et al. 2017) and head pose estimation (Patacchiola et 2020) datasets used to evaluate the performance of MFRL in few-shot regression. We use the same backbones literature (Patacchiola al. 2020) make fair comparisons. Details of the few-shot regression experiments can be found in Appendix A.2
The results for few-shot regression summarized in Table In the sine few-shot regression, MFRL outperforms all meta-learning methods, demonstrating that high-quality representation can be learned in supervised learning, without episodic meta-learning. Although DKT with spectral mixture (SM) kernel achieves high accuracy, the good performance should be attributed to the strong inductive bias to periodic functions in the SM kernel (Wilson Adams 2013). Additional results for MFRL with different activation functions are reported in Appendix In the head pose estimation experiment, MFRL also achieves the best accuracy. results in improved accuracy, suggesting that SWA can improve the quality of features and facilitate the learning of downstream tasks. In Fig. D uncertainty is correctly estimated by the hicrarchical Bayesian linear model with learned features using just 10 training samples.
both few-shot regression problems, SWA
Sine wave (2-layer MLP)
MSE
MAML (Finn et al.
2017
0.67 0.06
Bayesian MAML Yoon et
2018
0.54 0.05
ALPaCA (Harrison
2018
0.14 0.09
R2D2 (Bertinetto et al. 2019
0.46 + NA
DKT+RBF (Patacchiola et al.
2020
1.38 0.03
DKT+Spectral (Patacchiola et al.
2020
0.08 0.06
MFRL (w:o. SWA)
0.023 0.016
MFRL
0.016 0.008
Head pose (3-layer Cony Net)
MSE
420
440
MAML (Finn et al. 2017]
0.21 0.01
Bayesian MAMI Yoon et al. 2018
0.18 0.01
DKT+Spectral (Paracchiola cf al. 2020
0.10 0.01
MFRL (w.o. SWA) MFRL
0.033 0.006
0.027 0.005
Figure 1: Sine wave regression and uncertainty quantification (10 training samples). The true and the estimated (by MFRL) standard deviation of data generation noise are 0, and 0.093.
Table 1: 10-shot regression on sine waves and head pose estimation.

---

Under review as a conference paper at ICLR 2022
5.2 FEW-SHOT CLASSIFICATION RESULTS
We conduct few-shot classification experiments on four widely used few-shot image recognition benchmarks: minil Net (Ravi Larochelle 2017), mieredI (Ren et al. 2018), CIFAR- FS (Bertinetto et al. (2019), and FCT00 (Oreshkin et al.) 2018). In addition, we test our approach on cross-domain few-shot classification task from the minilmageNett to CUB. The experiment details about the few-shot classification datasets can be found in Appendix A.. The proposed method is applied to three widely used network architectures: ResNet- (Lee al. 2019 Ravichandran al,)
2019), wide ResNet (WRN-28-I1) 10) (Dhillon et al.] 20220 Rusu et 2019). and 4-layer convolu-
tional neural network with 64 channels (Chen et al. 2019 Patacchiola et al. 2020 (in Appendix
A.3
Table 2: Few-shot classification results on minilmageNet and tieredImageNet.
mitillrapeN Swaay
RenollmaagNi Soway
Mechod
Backbose
T-bett
S-shen
S-ahol
Manching Net
Vimals al
2010
ResNer-12 12
033 08
1: 0.880
73.39 1: 0.00
UH 50
0.
80.00 d:: 0.71
Proto
ResNa-12
60.37
0.88
78.02
0.57
83.40
Proto
ResNee-12
63 51
81.08
0.33
84.76
MAML
Finr
2017
ReoNa-12
50.58
1.84
70.85
MAMI.
5WA
ResNer-12
72:47
Munkhalahii cl 2018
ReaNa-12
50.88 0.62
71.94 0.57
TADAM
Orshain a 2015
ResNes 12
58. 08.50000
0.30
76.70 0.30
0.76
Basclinent
5WA
ResNer 12
82.72 0.80
81:25 0.68
70.01 0.82
84.39 0.04
TapNent
Yoop Yoop 2019
ResNee112
00 0.15
76.35 0.10
08 0.15
80.26
McsOpiNa1
Laves
XM19
ResNa-12
62.4
78.63
81.50
Easemble
Durnices
ResNes-1
50.48 0.665
75.62 0.42
Simoo ct
ReaNa-12
62. 03.64422
0.06
78.83 0.45
66. 0.76
Raacehingg
3029
ResNer 12
0.57
76.25 0.51
0.53
FEAT
Se 2000
ResNes-12
66.78 0.20
82.05 0.14
70.80 0.228
84.79 0.
De)
ResNer
0.56
0.87
Ditill
Tan 100
ResNei-13
64.
0.60
82.14 0.43
52
86.03
MIRL
(K.)) SWA)
ResNer 12
612 27
0.86
80.23 0.57
00 0.77
84.
LBU dta 2019
WRN-2--0
0:08
77.09 0.32
33
81.44
Fine tuze
Dhiloo dt 2020
WEN 25 10
57. 07.73-2 0.62
78.17 0.49
66. 58 0.70
$5.55 0.48
0.35
MosFFn Na clal 20100
WRN-25-10
64.13
0.18
80.82 0.17
67.27 0.14
83.28
(won SWA)
WRN-25-10
80.12 0.88
89 0.79
84.42
WRN-25-10
88.34
Table 3: Few-shot classification results on CIFAR-FS and FC100.
Mahsal
Backhone
CHAR--S S-aay
PU100 S-way
Proto Net
Socll otal.
2017
ReiNe:-12 12
72.2 2 0.7
83.5 0.5
41.5 +: 0.7
57.0 4: 0.7
TADAM
Orehhin at
3018
ResNee 12
40.1 1:
56.1 0.4
Baseliaer1
2019
72.2 0.9
43.1
55.7
Mec Ni
Leo all
3019
ResNee112 12
73.8 1: 0.7
85.0 0.5
41.1 11.1100 0.6
55.5 0.6
MTL.
Sin
Ro
45.1
57.6
Shot-mr
Kanchhrddnncc
2019
ResNee-12
696.1 NA
84.7 1
TEAM
Ohinet al, 207V
2017
RoNNs-12
70.4 d: NA
81.3 NA
ResNe:12
70.0 0.5
DSN
Sinae all 2000
Ro
72.3 0.8
8..1
MABAS
Kin 2t b010
RedNer 12
73.5
85.6 0.6
42.3
58.1 0.7
ResNe-12
MERL
(Wo. SWA)
ResNer 12
71.4
86.1
42.0
59.1
RocNee-12
74.0
0.8 0.8
87.4
45.3 0.8
61.1 0.7
08.7
38.2
MERL (Wo. SWAI
28-1
71.7 0.9
86.2
41.
57.3
0.7
MERL
WHN-28-10
76.7 0.9
88.6
45.1 0.8
61.0
The results of the proposed method and previous SOTA methods using similar backbones are re- ported in Table and and The proposed method achieves the best performance most of the exper- iments when compared with previous SOTA methods. Our method is closely related to Baseline++ (Chen et al. 2019) and fine-tuning on logits (Dhilonn et al. 2020). Baseline++ normalizes both classification weights and features, while the proposed method only normalizes features. It allows our method to find a more accurate model in a more flexible hypothesis space, given high-quality representation. Compared with fine-tuning logits, method obtains better results by learning
7

---

Under review as a conference paper at ICLR 2022
new logistic regression model on features, which store richer information about the data. Some approaches pretrain a C-class classification model on all training data and then apply highly sophis- ticated meta- meta-learning techniques to the pretrained model to achieve SOTA performance (Rusu et al. 2019] Sun et al. 2019). Our approach with SWA outperforms those wnerrained-then-meta--ee models, which demonstrates that SWA obtains high-quality representation that generalizes well to unseen tasks, Compared with improving representation quality for few-shot classification via self- distillation (Tian al. 2020), the computational cost of SWA is significantly smaller because does not require training models from scratch multiple times. Moreover, SWA can be applied to find good representation for both few-shot regression and classification, while previous transfer learning
approaches can only handle few-shot classification problems (Mangla et al. 2020: Tian et al.. 2020
MFRL also applied to the cross-domain few-shot classification task as summarized in Table MFRL outperforms other methods in this challenging task, indicating that the learned representation has strong generalization capability. We use the same hyperparameters (training epochs, learning rate, learning rate in SWA, SWA epoch, etc.) as in Table 2. The strong results indicate that MFRL is robust to hyperparameter choice. Surprisingly, meta-learning methods with adaptive embeddings do not outperform simple transfer learning methods like Baseline++ when the domain gap between base classes and novel classes is large. We notice that Tian et al. (2020) also reports similar results that transfer learning methods show superior performance on large-scale cross-domain few-shot classification dataset. We still believe that adaptive embeddings should be helpful when the domain gap between base and novel classes large. Nevertheless, how to properly train model to obtain useful adaptive embeddings in novel tasks is an open question.
Table 4: Cross-domain few-shot classification results on minilmageNet to CUB.
minilmageNNt to CUB 5-way
Method
Backboone
1-shot
5-chor
MAMI
Finn et
2017
WRN-28-10
39.06 : 0.47
55 04 = 0.42
Kusu
WRN-28-10
41.40 +: 0.54
+ 0.48
MTL
WRN
43.15
0.44
Malching
Vinyals et al. 2016
WRN 10
42.04
0.57
0.45
Hu
2020
WRN-28-100
43.27
0.44
Baseline
Chen
2019
WRN-28-10
42.89
0.41
02.12
Baellinc+
Chen al.
2019
WRN 26 10
42.12
0.39
60.21
0.39
MFRL (w.o. SWA)
WRN 28 10
43.68
0.47
63.88
0.42
MFRL
WRN-28 10
46.98
0.51
66.92
0.42
5.3 EFFECTIVE RANK OF THE REPRESENTATION
The rank of representation defines the number of independent bases. For deep learning, noise in gradients and numerical imprecision cause the resulting matrix to be full-rank. therefore,ssmpp counting the number of non-zero singular values may not be an effective way to measure the rank of the representation. To compare the effective ranks, we plot the normalized singular values of the representation of meta-test data in Fig. 2 where the representation with SWA has a faster decay in singular values, thus indicating the lower effective rank of the presentation with SWA. The results empirically verify our conjecture that SWA is an implicit regularizer towards low-rank representation.
SONG PIgANC
GEARIS
-Silta 2049
-20 2043
Normalized singular values for representation with and without SWA. The metric Σo; log o. is used to measure the effective rank of the representation, where o; o,/o max- Faster decay in singular values indicates that fewer dimensions capture the most variation in all dimensions, thus lower effective rank.
Figure 2:
8

---

Under review as a conference paper at ICLR 2022
Figure 3: Study on the temperature scaling factor for 5-way 5hohot classification using ResNet-11 12 for the proposed MFRL and existing episodic meta learning methods. Uncalibrated models are in the first row, and calibrated models with the temperature scaling factor are in the second row.
5.4 FEW-SHOT CLASSIFICATION RELIABILITY
The proposed method not only achieves high accuracy in few-shot classification but also makes the classification uncertainty well-calibrated. A reliability diagram can be used to check model calibra- tion visually, which plots an identity function between prediction accuracy and confidence when the model is perfectly calibrated (DeGroot & Fienberg, 1983). Fig. shows the classification reliability diagrams along with widely used metrics for uncertainty calibration, including expected calibration error (ECE) (Guo et al. (011)). maximum calibration error (MCE) (Naeini et al.] 2015), and Brier score (BRI) (Brier 1950). ECE measures the average binned difference between confidence and accuracy, while MCE measures the maximum difference. BRI is the squared error between the predicted probabilities and one-hot labels. MAML is over-confident because tuning deep neural network on few-shot data is prone to over-fitting. Meanwhile, Proto Net and Matching Net are better calibrated than MAML because they do not fine-tune the entire network during testing. Neverthe- less, they are still slightly over-confident. The results indicate that MFRL with global temperature scaling factor can learn well-calibrated models from very limited training samples.
5.5 APPLICATION IN META-LEARNING
Meanwhile, also apply SWA to episodic meta-learning methods, such Proto Net, MAML and Matching Net, to improve their classification accuracy. The results in Tablet that SWA can improve the few-shot classification accuracy in both transfer learning and episodic meta-leant meta-learning. SWA is orthogonal to the learning paradigm and model architecture. Thus, SWA can be applied to wide range of few-shot learning methods improve accuracy.
Table Application of SWA on meta-leaming methods for the minilmageNet dataset
Proto Net
MAML.
Matching Net
Method
I-shot
5-shot
I-shot
5-shot
I-shot
5-shot
SWA
60.37
78.02
56.58
70.85
63.08
75.99
SWA
63.51
81.98
58.21
72.47
63.76
76.78
Furthermore, the temperature scaling factor can be applied to calibrate meta-learning methods, in- cluding MAML, Proto Net, and Matching Net. The reliability diagrams Fig. B indicate that the temperature scaling factor not only calibrates classification uncertainty of transfer learning ap- proaches, such as the proposed MFRL, but also makes the classification uncertainty well-calibrated in episodic meta-learning methods. Therefore, the temperature scaling factor be applied to wide range of few-shot classification methods to get cell-calibrated uncertainty, while preserving the classification accuracy.

---

Under review as a conference paper at ICLR 2022
6 DISCUSSION
SWA has been applied to supervised learning of deep neural networks (Izmailov et al. 2018 Athi- waratkun al. 2019) and effectiveness was attributed to convergence to a solution on the flat side of asymmetric loss valley (He et al., 2019). However, it does not explain the effectiveness of SWA in few-shot learning because the meta-training and meta-testing losses are not comparable after the top layer is retrained by the few-shot support data in meta-test task. The effectiveness of SWA in few-shot learning must be related to the property of the representation. Although our results empirically demonstrate that SWA results in low-rank representation, further research about their connection is needed.
Explicit regularizers also be used to obtain simple input-output functions in deep neural net- works and low-rank representation, including L1 regularization, nuclear norm, spectral norm, and Frobenius norm (Bartlett al. 2017; Neyshabur et al. 2018 Sanyal et al. 2020). However, some of those explicit regularizers are not compatible with standard SGD training or are computationally expensive. In addition, it is difficult choose the appropriate strength of explicit regularization. Too strong explicit regularization can bias towards simple solutions that do not fit the data. In compar- ison, SWA an implicit regularizer that is completely compatible with the standard SGD training without much extra computational cost. Thus, it can be easily combined with transfer learning and meta-learning to obtain more accurate few-shot learning models. In parallel, SWA is also robust to the choice of the hyperparameters the learning rate and training epochs in the SWA stage (see details in Appendix A.44..
7 CONCLUSIONS
In this article, propose MFRL obtain accurate and reliable few-shot learning models. SWA is an implicit regularizer towards low-rank representation, which generalizes well to unseen meta-test tasks. The proposed method can be applied to both classification and regression tasks. Extensive experiments show that our method not only outperforms other SOTA methods on various datasets but also correctly quantifies the uncertainty in prediction.
REFERENCES
Antreas Antoniou, Harrison Edwards, and Amos Storkey. How to train your maml. In International Conference on Learning Representations, 2018
Ben Athiwaratkun, Mare Finzi, Pavel Izmailov, and Andrew Gordon Wilson. There are many con- sistent explanations of unlabeled data: Why you should average. International Conference on Learning Representations, 2019.
Francis Bach and Eric Moulines. Non-strongly-convee smooth stochastic approximation with con- vergence rate (1/n). In Proceedings of the 26th International Conference on Neural Information Processing Systems- Volume 1, pp- 773-781, 2013.
Peter L Bartlett, Dylan J Foster, and Matus Telgarsky. Spectrally-normalle margin bounds for neural networks. Advances in Neural Information Processing Systems, 30:6240 62499, 2017.
Edmon Begoli, Tanmoy Bhattacharya, and Dimitri Kusnezov. The need for uncertainty quantifi- cation in muchine-assisted medical decision making. Nature Machine Intelligence, 1(1):20-23, 2019.
Y Bengio, Bengio, and Cloutier. Learning synaptic learning rule. In IJCNN-9J-Seatll Inter- national Joint Conference on Neural Networks, volume 2, pp. 969-vol. IEEE, 1991.
Luca Bertinetto, Joao Henriques, Philip Torr, and Andrea Vedaldi. Meta-learning with differen- tiable closed-form solvers. In International Conference Learning Representations, 2019.
Glenn W Brier. Verification of forecasts expressed in terms of probability. Monthly weather review, 78(1),1-3, 1950.
10

---

Under review as a conference paper at ICLR 2022
Wei- Yu Chen, Yen- Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at few-shot classification. In International Conference on Learning Representations, 2019. 2019.
Morris H DeGroot and Stephen E Fienberg. The comparison and evaluation of forecasters. Journal of the Royal Statistical Society: Series D (The Statistician), 32(1-2):12 1983.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255, Icee, 2009.
Guneet Singh Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. A baseline for few-shot image classification. In International Conference on Learning Representations, 2020.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby, An image worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Representations, 2021.
Nikita Dvomikk, Cordelia Schmid, and Julien Mairal. Diversity with cooperation: Ensemble methods for few-shot classification. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3723-3731, 2019.
Zhou Fan and Zhichao Wang. Spectra of the conjugate kernel and neural tangent kernel for linear- width neural networks. Advances in Neural Information Processing Systems, 33, 2020.
Nanyi Fei, Zhiwu Lu, Tao Xiang, and Songfang Huang. (MELR): Meta-learning via modeling episode-level relationships for few-shot learning. In International Conference on Learning Rep- resentations, 2021.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference Machine Learning- Volume 70, pp. 1126-1135. JMLR. org, 2017.
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-learning. In Pro- ceedings of the 32nd International Conference on Neural Information Processing Systems, pp- 9537-9548, 2018
Sebastian Flennerhag, Andrei A Rusu, Razvan Pascanu, Francesco Visin, Hujun Yi,, and Raia Hadsell. Meta-learning with warped gradient descent. In International Conference on Learning Representations, 2020.
Victor Garcia and Joan Bruna Estrach. Few-shot learning with graph neural networks. In 6th International Conference Learning Representations, ICLR 2018, 2018.
Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4367- 4375, 2018
Micah Goldblum, Steven Reich, Liam Fowl, Renkun Ni, Valeria Cherepanova, and Tom Gold- stein. Unraveling meta-learning Understanding feature representations for few-shot tasks. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Ma- chine Learning, volume 1 119 of Proceedings of Machine Learning Research, pp- 3607-3616, Vir- tual, 13-18 Jul 2020. PMLR.
Shaogang Gong, Stephen McKenna, and John Collins. An investigation into face pose distribu- tions. In Proceedings of the Second International Conference on Automatic Face and Gesture Recognition, pp. 265-270, IEEE, 1996.
J Gordon, J Bronskill, M Bauer, S Nowozin, and RE Turner. Meta-learning probabilistic inference for prediction. In International Conference on Learning Representations (ICLR 2019). OpenRe- view. net, 2019.
11

---

Under review as a conference paper at ICLR 2022
Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradient- based mneaallearnnng as hierarchical bayes. In International Conference on Learning Representa- tions, 2018.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, pp. 1321-1330. PMLR, 2017.
James Harrison, Apoorva Sharma, and Marco Pavone. Meta-learning priors for efficient online bayesian regression. International Workshop on the Algorithmic Foundations of Robotics, pp- 318-337, 337. Springer, 2018.
Haowei He, Gao Huang, and Yang Yuan. Asymmetric valleys: Beyond sharp and flat local minima. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 3549-2560) 2019.
Matthew D Hoffman and Andrew Gelman. The no-u-turn sampler: adaptively setting path lengths in hamiltonian monte carlo. J. Mach. Learn. Res., 15(1):1593-1623, 2014.
Shell Xu Hu, Pablo Garcia Moreno, Yang Xiao, Xi Shen, Guillaume Obozinski, Neil Lawrence, and Andreas Damianou. Empirical bayes transductive meta-learning with synthetic gradients. In International Conference on Learning Representations, 2020.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. In 34th Conference on Un- certainty in Artificial Intelligence 2018, UA! 2018, pp. 876-885. Association For Uncertainty in Artificial Intelligence (AUAI), 2018.
Prateek Jain, Sham Kakade, Rahul Kidambi, Praneeth Netrapalli, and Aaron Sidford. Parallelizing stochastic gradient descent for least squares regression: mini-batching, averaging, and model misspecification. Journal of Machine Learning Research, 18, 2018.
Ghassen Jerfel, Erin Grant, Thomas L Griffiths, and Katherine Heller. Reconciling meta-learning and continual learning with online mixtures of tasks. Advances in Neural Information Processing Systems, 32, 2019.
Nitish Shirish Keskar, Jorge Nocedal, Ping Tak Peter Tang, Dheevatsa Mudigere, and Mikhail Smelyanskiy. On large-batch training for deep learning: Generalization gap and sharp minima. In 5th International Conference on Learning Representations, ICLR 2017, 2017.
Jackyeom Kim, Hyoungseok Kim, and Gunhee Kim. Model-agnostic boundary-adversarfe sam- pling for test-time generalization in few-shot learning. In European conference on computer vision. Springer, 2020.
Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. Siamese neural networks for one-shot image recognition. In /CML deep learning workshop, volume 2. Lille, 2015.
Chandrashekar Lakshminarayanan and Csaba Szepesvari. Linear stochastic approximation: How far does constant step-size and iterate averaging go? In International Conference on Artificial Intelligence and Statistics, pp. 1347-1355. PMLR, 2018.
Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 10657-10665, 2019.
Yoonho Lee and Seungjin Choi. Gradient-baan based meta-learning with learned layerwise metric and subspace. International Conference on Machine Learning, pp. 2927-2936, 2018.
Puneet Mangla, Mayank Singh, Abhishek Sinha, Nupur Kumari, Vineeth Balasubramanian. and Balaji Krishnamurthy. Charting the right manifold: Manifold mixup for few-shot learning. In 2020 JEEE Winter Conference on Applications of Computer Vision (WACV), pp. 2207-2216. IEEE, 2020.
12

---

Under review as a conference paper at ICLR 2022
James Martens. New insights and perspectives on the natural gradient method. Journal of Machine Learning Research, 21:1-76, 2020.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta-
learner. In International Conference on Learning Representations, 2018.
Tsendsuren Munkhdalai, Xingdi Yuan, Soroush Mehri, and Adam Trischler. Rapid adaptation with conditionally shifted neurons. In International Conference Machine Learning, pp- 3664-3673 PMLR, 2018.
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated proba- bilities using bayesian binning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
Behnam Neyshabur, Srinadh Bhojanapalli, and Nathan Srebro. A pac-bayesian approach to spectrally-noormalizeed margin bounds for neural networks. In International Conference on Learn- ing Representations, 2018.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv: Tt03:02800099900 2018
Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. In Advances in Neural Information Processing Systems, pp. 721-731, 2018.
Eunbyung Park and Junier B Oliva. Meta-curvature. Advances in Neural Information Processing Systems, 32:3314-3324 2019.
Massimiliano Patacchiola, Jack Turner, Elliot J Crowley Michael O'Boyle, and Amos J Storkey. Bayesian meta-leaming for the few-shot setting via deep kernels. Advances in Neural Information Processing Systems, 33, 2020.
Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM journal on control and optimization, 30(4):838-88) 855,, 1992.
Viraj Prabhu, Anitha Kannan, Murali Ravuri, Manish Chaplain, David Sontag, and Xavier Amatri- ain. Few-shot learning for dermatological disease diagnosis. In Machine Learning for Healthcare Conference, pp. 532-552. PMLR, 2019.
Hang Qi. Matthew Brown, and David G Lowe. Low-shot learning with imprinted weights. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 5822-5830, 2018.
Limeng Qiao, Yemin Shi, Jia Li, Yaowei Wang, Tiejun Huang, and Yonghong Tian. Transductive episodic-wise wise adaptive metric for few-shot learning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3603-3612, 2019.
Siyuan Qiao, Chenxi Liu, Wei Shen, and Alan Yuille. Few-shot image recognition by predicting parameters from activations. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 7229-7238, 2018
Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning or feature reuse? towards understanding the effectiveness of maml. In International Conference on Learning Rep- resentations, 2020.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In 5th Interna- lional Conference on Learning Representations, ICLR 2017, Toulon, Frunce, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL lattps::/ppenreeierr
Fooum?id-x-JY--E Kcll
Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto. Few-shot learning with embedded class models and shot-free meta training. Proceedings of the IEEE International Conference on Computer Vision, pp. 331-339, 2019.
13

---

Under review as a conference paper at ICLR 2022
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B Tenenbaum, Hugo Larochelle, and Richard S Zemel. Meta-leaming for semi-supervised few-shot classifica- tion. In International Conference on Learning Representations, 2018.
Andrei Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. In International Conference on Learning Reprexentations, 2019.
Amartya Sanyal, Philip H Torr, and Puneet K Dokania. Stable rank normalization for improved generalization in neural networks and gans. In International Conference on Learning Represen- tations, 2020.
Nikunj Saunshi, Arushi Gupta, and Wei Hu. A representation learning perspective on the importance of train-validation splitting in meta-learning. In International Conference on Machine Learning, pp. 9333-9343. PMLR, 2021.
Jürgen Schmidhuber. Evolutionary principles in self-referentir learning, or on learning how to learn: the meta-meta-... hook. PhD thesis, Technische Universität München, 1987.
Christian Simon, Piotr Koniusz, Richard Nock, and Mehrtash Harandi. Adaptive subspaces for few-shot learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Patlern Recognition, pp. 4136-4145, 2020.
Jake Snell and Richard Zemel. Bayesian few-shot classification with one-vs-each pólya-gamma augmented gaussian processes. In International Conference on Learning Representations, 2021.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, pp. 2017,
Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele. Meta-transfer learning for few-shot learning. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 403-412, 2019.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning compare: Relation network for few-shot learning. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, pp. 1199-1208, 2018.
Sebastian Thrun and Lorien Pratt. Learning to learn. Springer Science & Business Media, 1998.
Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B Tenenbaum, and Phillip Isola. Rethinking few-shot image classification: good embedding is all you need? In European conference computer vision. Springer, 2020.
Michael E Tipping. Sparse bayesian learning and the relevance vector machine. Journal of machine learning research, (JJun):211 1-424,,, 2001.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in neural information processing systems, pp. 3630-3638, 2016.
Welinder, Branson, Mita, C. Wah, Schroff, Belongie, and P. Perona. Caltech-UCSD Birds 200. Technical Report
California Institute of Technology, 2010.
Florian Wenzel, Kevin Roth, Bastiaan Veeling, Jakub Swiatkowski, Linh Tran, Stephan Mandt, Jasper Snoek, Tim Salimans, Rodolphe Jenatton, and Sebastian Nowozin. How good is the bayes posterior in deep neural networks really? In International Conference on Machine Learning, pp- 10248-10259, PMLR, 2020.
Andrew Wilson and Ryan Adams. Gaussian process kernels for pattern discovery and extrapolation. In International conference on machine learning, pp. 1067-1075, 2013.
Sanghyun Woo, Jongchan Park, Joon- Joon-Young Lee, and In So Kweon. Cbam: Convolutional block attention module. In Proceedings of the European conference on computer vision (ECCV), pp- 3-19, 2018.
14

---

Under review as a conference paper at ICLR 2022
Jin Xu, Jean-Franee Francois Ton, Hyunjik Kim, Adam Kosiorek, and Yee Whye Teh. Metafun: Meta- learning with iterative functional updates. In International Conference on Machine Learning, pp- 10617-10627. PMLR, 2020.
Han-Jia Ye, Hexiang Hu, De-Chuan Zhan, and Fei Sha. Few-shot learning via embedding adaptation with set-to-set functions. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8808-8817, 2020.
Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian model-agnostic meta-learning. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 7343-7353, 2018.
Sung Whan Yoon, Jun Seo, and Jaekyun Moon. Tapnet: Neural network augmented with task- adaptive projection for few-shot learning. In ICML. 2019 (International Conference Machine Learning). ICMI., 2019.
Sung Whan Yoon, Do-Yeon Yeon Kim, Jun Seo, and Jackyun Moon. Xtamet: Learning to extract task- adaptive representation for incremental few-shot learning. In Proceedings of the 37th Interna- tional Conference on Machine Learning, ICMI. 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 10852-10860. PMLR, 2020.
Zhongqi Yue, Hanwang Zhang, Qianru Sun, and Xian-Sheng Hua. Interventional few-shot learning. Advances in Neural Information Processing Systems, 33, 2020.
Chi Zhang, Yujun Cai, Guosheng Lin, and Chunhua Shen. Deepemd: Few-shot image classifica- tion with differentiable earth mover's distance and structured classifiers. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12203-12213, 2020.
Manli Zhang, Jianhong Zhang, Zhiwu Lu, Tao Xiang, Mingyu Ding, and Songfang Huang. (IEPT): Instance-level and episode-level pretext tasks for few-shot learning. In International Conference on Learning Representations, 2021.
A APPENDIX
A.1 PSEUDO CODE FOR MFRL
Algorithm Meta-free representation learning for few-shot learning
Merge all training tasks Dtr IT= Initialize model parameters
10 W
Maximize the likelihood on all training data p (Drr 0) using SGD
Minimize the squared loss for regression problems
Minimize the cross-entropy loss for classification problems
Run SWA to obtain OSWA
Discard W and freeze 0y
Learn new top layer using support data D in a test task:
Learn a hierarchical Bayesian linear model for regression task
Learn a logistic regression model with the temperature scaling factor for classification task
A.2 EXPERIMENT DETAILS
Sine waves are generated by Ain (r (2) -2) +., where amplitude A E [0.1, 5.0), phase & E [0. xt] and is white noise with standard deviation of 0.1 (Finn et al. 2017). Each sine wave contains 200 samples by sampling uniformly from [-5.0, 5.0). We generate 500 waves for training, validation
and testing, respectively. All sine waves are different from each other. We use the same backbone network described in MAML (Finn al. 2017): two-layer MLP with 40 hidden units in each layer. We use the SGD optimizer with a learning rate of 10-3 over x 1 training iterations and run SWA over 2 x 10a training iterations with learning rate of 0.05.
15

---

Under review as a conference paper at ICLR 2022
Head pose regression data is derived from the Queen Mary University of London multi-view face dataset GGogg et al., 1996). It contains images from 37 people and 133 facial images per person. Facial images cover view sphere of 90% in yaw and 120 in tilt. The dataset is divided into 3192 training samples (24 people), 1064 validation samples (8 people), and 665 test samples (5 people). We use the same feature extractor described in literature (Patacchiola convolutional neural network, each with 36 output channels, stride 2, and dilation 2. We train the model on the training people set for 300 epochs using the SGD optimizer with learning rate of 0.01 and run 25 epochs of SWA with learning rate of 0.01.
al. 2020): three-layer
minilmageNet is a 100-class subset of the original ImageNet dataset (Deng et al., 2009) for few- shot learning (Vinyals et al.] (0166). Each class contains 600 images in RGB format of the size 84 x 84. minilmageNNt is split into 64 training classes, 16 validation classes, and 20 testing classes, following the widely used data splitting protocol (Ravi & Larochelle 2017).
tieredlmageNet is another subset of the ImageNet dataset for few-shot learning (Ren et al. 2018 It contains 608 classes grouped into 34 categories, which are split into 20 training categories (351 classes), 6 validation categories (97 classes), and 8 testing categories (160 classes). Compared with minilmageNet, training classes in taered are sufficiently distinct from test classes, making few-shot classification more difficult.
CIFAR-FS is derivative of the original CIFAR-100 dataset by randomly splitting 100 classes into 64, 16, and 20 classes for training, validation, and testing, respectively (Bertinetto et al. 2019).
FC100 is another derivative of CIFAR-100 with minimized overlapped information between train classes and test classes by grouping the 100 classes into 20 superclasses (Oreshkin et al. 20188) They are further split into 60 training classes (12 superclasses), 20 validation classes (4 superclasses), and 20 test classes (4 superclasses).
miniImageNet to CUB is cross-domain few-shot classification task, where the models trained on mageltt and tested on CUB (Welinder et al. 2010). Cross-domain few-shot classification more challenging due the big domain gap between datasets. We can better evaluate the generalization capability in different algorithms. We follow the experiment setup in Yue et al. (2020) and use the WRN2-28-12 10 the backbone.
The backbone model is trained on all training classes using C-class cross-entropy loss by the SGD optimizer (momentum of 0.9 and weight decay of lc-4) with a mini-batch size of 64. The learning rate is initialized as 0.05 and is decayed by 0.1 after 60, 80, and 90 epochs (100 epochs in total). After the SGD training converges, we run 100 epochs of SWA with a learning rate of 0.02. Note that MFRL is not sensitive to training epochs and learning rates in SWA (see Appendix A.4). The training images are augmented with random crop, random horizontal flip, and color jitter.
During testing, we conduct 5 independent runs of 600 randomly sampled few-shot classification tasks from test classes and calculate the average accuracy. Each task contains classes, 1 x 5 or 5 x 5 support samples, and 75 query samples. A logistic regression model is learned using only the support samples. The classification accuracy is evaluated on the query samples.
A.3 ADDITIONAL RESULTS ON FEW-SHOT REGRESSION AND CLASSIFICATION
The additional results on few-shot regression using different activation functions are reported in Table 6 MFRL achieves high accuracy with different activation functions.
Table 6: 10-shot regression on sine waves with different activation functions.
Sine wave
MSE
MFRL (ReLU activation)
0.16 + 0.51
MFRL (tanh activation)
0.018 - 0.011
MFRL (erf activation)
0.016 + 0.008
The few-shot classification results using 4-layer convolutional neural network (or similar archi-
tectures) are reported in Table / and [ Similar to the results using ResNet-12 and WRN-28-100 the proposed method outperforms wide range of meta-learning approaches. Our method is only
16

---

Under review as a conference paper at ICLR 2022
BLE: 0.8515
BOE: 0-0A35
BOE: 8-9236
OKL: 0.3288
0.3230
SKL: 0.3293
DET DeE Eaieet
50ES 8.3827
ECE B-315X
EDES 8-9648
SKL: 0.4853
0.4388
9KE: 0,2791
Figure 4: Reliability diagrams for 5-way 5-shot classification minilmageNet
Figure 5: Reliability diagrams for 5-way few-shot classification using ResNet- backbone
17

---

Under review as a conference paper at ICLR 2022
Table 7: Few-shot classification results on minilmageNet and ticredImageNet..
minilmageNet 5-way
tieredImage 5-way
Method
Backbone
1-shot
5-shot
1-shot
5-shot
Matching Net (Vinyals
2016)
Conv-4
43.56 1 0.84
55.31 1: 0.73
54.48 0.93
71.32 0.78
Proto Net Snell et al. 2017
Conv-4
49.42 + 0.78
68.20 0.66
53.31 0.89
72.69 0.74
MAML Finn et al.
2017
Conv-4
48.70 + 1.75
63.11 = 0.92
SNAIL Míshra et al. 2018
Conv-4
45.10 + NA
55.20 + NA
VERSA (Gordon et al.
2019)
Conv-5
53.40 1 1.82
67.37 1: 0.86
Meta Mixture (Jerfel al.
2019)
Conv-4
49.60 + 1.50
64.60 0.92
RelationNet Sung ct
2018
Conv-4
50.44 + 0.82
65.32 0.70
54.48 0.93
71.32 0.78
FPA Qiao 2018
Conv-4
54.53 + 0.40
67.87 0.20
Shot-free Ravichandran et al.
2019)
Conv-4
49.07 1 0.43
65.73 1 0.36
48.19 0.43
65.50 1 0.39
Baseline++ CChen et al.
2019
Conv-4
47.15 + 0.49
66.18 = 0.18
54.67 + 0.61
72.37 + 0.67
FEAT Ye ct
2020)
Conv-4
55.15 0.20
71.61 + 0.16
DKT Patacchiola et al.
2020
Conv-4
19.73 + 0.07
64.00 + 0.09
MFRL
Conv-4
53.62 + 0.71
71.52 - 0.60
56.24 0.84
72.88 + 0.76
Table 8: Few-shot classification results on CIFAR-FS and FC100.
CIFAR FS 5-way
FC100 5-way
Method
Backhone
1-shot
5-shot
1-shot
5-shot
Proto Net Snell et al.
2017
Conv-4
55.5 + 0.7
72.0 + 0.6
35.3 + 0.6
48.6 0.6
MAMI.
Finn et al.
2017
Conv-4
58.9 + 1.9
71.5 + 1.0
38.1 + 1.7
50.4
RelationNet Sung et al.
2018
Conv-4
55.0 - 1.0
69.3 0.8
Shot-free Ravichandran al.
2019
Conv-4
55.1 0.5
71.7
Baselinet+
Chen al.
2019
Conv-4
55.1 1 0.9
72.3 0.8
35.2 +
49.8 0.7
MFRL
Conv-4
64.3 - 0.9
7..4
40.1 = 0.8
544 0.7
second to few-shot embedding adaptation with transformer (FEAT) Ye al. 2020) minilma- geNet dataset. Recently, meta-lcarned attention modules are built on top of the convolutional neural network to get improved few-shot classification accuracy. Direct comparison those methods with attention modules Ye et al. 2020 Fei et al. 2021 Zhang et al. 2021) may not be fair because recent studies show that transformer itself can achieve better results than convolutional neural networks in image classification (Dosovitskiy et al. 2021). It is difficult to determine whether the performance improvement due the meta-learning algorithm the attention modules. To make fair compar- ison, we add convolutional block attention modules (Woo et al., 2018) on top of ResNet12 features (before global average pooling). As shown in Fig. MFRL with attention modules achieves com- parable results with MELR and ТEPT.
The uncertainty calibration results of MFRL with the temperature scaling factor are presented in Fig. 5] The prediction confidence aligns well with the prediction accuracy. It demonstrates that MFRL with the temperature scaling factor results well calibrated models.
A.4 SENSITIVITY OF MFRL
The performance of MFRL is not sensitive to learning rates SWA. As shown Fig. 6 the representation learned by SWA generalizes better than the one from standard SGD, as long as the learning rate SWA is in a reasonable range. In addition, the prediction accuracy on meta-test tasks keeps stable even after running SWA for many epochs the training data. Therefore, MFRL is not sensitive to training epochs. This desirable property makes the proposed method easy to use when solving few-shot learning problems in practice.
18

---

Under review as a conference paper at ICLR 2022
Table 9: Results of MFRL with attention modules
minilmageNet 5-way
tieredImageNNll 5-way
Method
Backbone
1-shot
5-shot
1-shot
5-shot
FEAT
et al.
2020
ResNett11
66.78 1 0.20
82.05 0.14
70.80 1 0.23
84.79 0.16
MELR
Fei
2021
ResNet-12
67.40 + 0.43
83.40 0.28
72.14 = 0.51
87.01 0.35
IEPT
Zhang ct al.
2021
ResNet-12
67.05 + 0.44
82.90 0.30
72.24 + 0.50
86.73 0.34
MFRL
ResNet- 12
67.18 + 0.79
83.81 + 0.53
71.58 = 0.79
86.87 + 0.62
MFRL Attention
ResNet-11
67.51 + 0.78
83.97 + 0.51
71.97 + 0.80
86.99 + 0.60
mirlimageteet
CIRR FS
18.%%
19.0%
M6N%
18.0%
01.%
83.0%
8%%%
08.0%
4.6%
63.2%
64.7%
SWA spocth
SWA gpacs
(a)
(b)
Figure 6: Evaluation on different learning rates and training epochs in SWA. (a) 5-way 5-shot ac- curacy on minilmageNet with ResNet12 backbone; (b) 5-way 5-shot accuracy on CIFAR-FS with WRN-28-10 backbone.
A.5 COMPARISON WITH EXPONENTIAL MOVING AVERAGING
Exponential moving average (EMA) decays the importance of model weights from early training epochs exponentially. Let Oawg alwvg (1 a) Onew. We try EMA with different values of a. In Table TO EMA improves the performance when a is within a reasonable range. Note that EMA introduces one extra hyperparameter, the forgetting factor. It makes EMA less desirable in practice.
Table 10: Comparison between SWA and EMA
minilmageNeet 5-way
Method
Backbone
1-shot
5-shot
No averaging
ResNet-12
62.27 0.86
80.23 + 0.57
EMA a 0.9
ResNet-12
66.03 = 0.83
82.90 + 0.58
EMA a 0.99
ResNet-12
66.05 = 0.84
82.98 0.57
EMA a = 0.999
ResNet-12
62.49 = 0.81
80.73 + 0.62
SWA
ResNet-12
67.18 + 0.79
83.8 + 0.53
A.6 HIERARCHICAL BAYESIAN LINEAR CLASSIFICATION MODEL
Similar to the hierarchical Bayesian linear regression model, the prior distribution over w is p(w A) Th (w, A,,, where is the precision in the Gaussian prior. The hyperprior on À is defined as p(A) Gamma (A a, b). The posterior over all latent variables given the data is
( X. w. A) (w P(A
(w,, A X,y)
ply X)
19

---

Under review as a conference paper at ICLR 2022
BELLA
Figure 7: Reliability diagrams of hierarchical Bayesian linear classification models with flat and non-informative hyperpriors. The backbone is ResNet-12.
MCMC sampling (Holfiman & Gelman 2014) is used to avoid potential deterioration in predictive performance due to approximated inference. flat and non-informative hyperprior (a 10 ") is used because no prior knowledge is available. In Table [T] the hierarchical Bayesian linear clas- sification model achieves slightly worse performance than the logistic regression model. However, the classification model is not well calibrated, shown in Fig. 7
Table 11: Comparison between logistic regression and hierarchical Bayesian linear models on 5-way 5-shot classification benchmarks using ResNet-12 backbone.
Top layer
minilmageNNe
tieredImageNet
CIFAR-FS
FC-100
Logistic regression
83.81
86.87
87.4
61.1
Hierarchical Bayesian linear classification
81.94
85.32
85.9
59.2
After fine-tuning a and using the meta-validation data, is possible to get better calibrated classi- fication models test tasks. Besides, the classification accuracy is still slightly worse than logistic regression after hyperparameter tuming. Our observations align with a recent study, which shows that the Bayesian classification model cannot achieve similar performance the non-Bayesian counter- part without tempering the posterior (Wenzel et al. (2020). We do not further experiment tempered posterior in the hierarchical Bayesian linear classification model because it introduces an extra tem- perature hyperparameter that requires tuning. The original purpose of introducing the hierarchical Bayesian model is get an accurate and well calibrated classification model without hyperparam- eter tuning. Consequently, the hierarchical Bayesian model is not used in few-shot classification in that hierarchical Bayesian linear classification models cannot achieve high accuracy and good uncertainty calibration from a non -on-informttive hyperprior. If hyperparameter tuning is inevitable, it much easier to tune a logistic regression model with temperature scaling factor, compared with tuning hierarchical Bayesian model. Furthermore, the computational cost of learning hierarchi- cal Bayesian linear classification model via MCMC sampling is much larger than that learning a logistic regression model.
20


Reviews:
[{"review_id": "ZV5zSsLS22S", "reviewer": "Reviewer_useT", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper presents a few-shot learning method based on the reprehensive pre-train learning using stochastic weight averaging (SWA). The merits of the proposed work is that this proposed method can works for both few-shot regression and classification problems and both of them achieves better results compared with the other recent works as reported in the manuscript. And as claimed by the author, this is the first few-shot learning work can works for classification and regression.\n\n1), It is well-known that SWA is an effective tool in weight space to obtain better reprehensive learning as it leads to a more flat minimum. As a result, it is not very supervising that SWA achieves better results in transfer learning for few-learning learning. Actually, SWA improves most applications as the author mentioned for meta-based approaches. Therefore, employing SWA for few-shot learning might not be a contribution that significant enough for an individual work. It will be more interesting to provide more analyse why SWA lead to the better performance compared with other solutions. Moreover, the flat loss surface contribute to the adaption is well discussed in some existing incremental few-shot learning research.\n\n2), It is interesting that the authors use the theory for meta learning to guide the research for transfer learning based few-shot learning, i.e., to make the feature with lower-rank representation. The result shows that the performance is improved. However, the theoretical link is still not clear. Is the low-rank representation criterion general for all reprehensive learning to have better result? Or the SWA is best way to achieve the low-rank representation for few-shot learning? It might be suspected that all those learning methods resulting better performance will contribute to the low-rank representation, e.g., various pre-training losses, network structure, etc.\n\n3), For the transfer learning based few-shot learning method, there will be a problem that by freezing the feature extraction, it is almost impossible to further adapt and generalize to cross-domain test environment if the feature difference between base classes and few-shot classes are significant. As a result, the evaluation performance will be very related to the dataset whether the few-shot/pretrain data are similar or not. In contrast, methods like MAML will adapt and finetune the feature extraction for the novel classes, which leads it may more difficult to train. However, its generalization capability will be stronger when the feature of few-shot classes are different with the pretrain base classes. As a result, to prove the effectiveness of the proposed methods, more ablation should be added with different setting, e.g., the few-shot classes close to the pretrain base classes, few-shot classes far away to the pretrain base classes, etc. In this way, the reader will be able to see the comparative advantages and limitations of the proposed method. Otherwise, it might not be proper to directly conclude the proposed methods is better than the existing methods.\n\n4), Similar like SWA, Exponential Moving Average (EMA) has the similar better results than the directly trained weights, and both are approaches in the weight space. It will be reasonable and interesting to be an ablation study to see if EMA achieves the similar results in few-shot learning or if EMA also contributes to the low-rank representation.\n\n5), Similarly, for other transfer learning based few-shot learning method such as Baseline++, it is straightforward and fully compatible to employ the SWA in its pertrain, the comparison should be analysed if MFRL better than (Baseline++ with SWA) to demonstrate the effectiveness of the MFRL.", "summary_of_the_paper": "This paper presents a few-shot learning method based on the reprehensive pre-train learning using stochastic weight averaging (SWA). The merits of the proposed work is that this proposed method can works for both few-shot regression and classification problems and both of them achieves better results compared with the other recent works as reported in the manuscript. And as claimed by the author, this is the first few-shot learning work can works for classification and regression.\n", "summary_of_the_review": "As mentioned in the Main Review above, the merits and concerns are pointed for reference and discussion. Generally, the main concern is on the originality as the SWA is an existing method, and the detail analysis on the mechanism how SWA specifically contribute for few-shot learning is not adequate.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "UdIIDLqlv4", "reviewer": "Reviewer_jVyb", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "There are several concerns that needs to be addressed, and I believe addressing these can improve the quality of this paper. \n\n \n \n\n- There are some typos in the main text, like Prot Net in 5.5. \n\n \n \n\nI have some major concerns about the opening remarks of this paper listed below: \n\n \n \n\n- In the introduction, the authors have mentioned that : “Despite the success of episodic meta-learning in few-shot learning tasks, they are slow to converge, prone to over-fitting, and tricky to implement (Antoniou et al., 2018 [1]).”  \n\nI think this statement is not generally true, as these are the shortcoming of the MAML algorithm listed in [1], and most of them are addressed in this paper as MAML++. Also, as an example recent metric-based methods (ProtoNet, RelationNet, FT for cross-domain FSL [2]) does not suffer from these issues that much, at least when comparing to the transfer learning methods like Chen et al. [3] and Dhillon et al [4].  \n\nFor example, ProtoNet, has a relatively fast convergence and increasing the ways in meta-training is proved to resolve overfitting. There are also several recent extensions, and reqularizers for few-shot learning that address these problems well. \n\n \n \n\n- In the later part of the introduction, you also mentioned that your algorithm tries to address these shortcomings: “In this paper, we propose a new method that does not rely on meta-training to overcome the limitations of commonly used episodic meta-learning approaches in few-shot learning.”. But, there is no evaluation regarding faster convergence, preventing overfitting, and simpler implementation of your algorithm compared to recent episodic ones. \n\n \n \n\n- I also doubt the validity of this statement in the introduction: “recent studies on transfer learning (Chen et al., 2019; Tian et al., 2020) cast doubt on whether it is the episodic meta-learning algorithm or the learned representation that is responsible for fast adaption to new tasks”.  \n\nSpecifically, I am not so sure about differentiating episodic meta-learning and learning better features. As an example, Goldblum et al. [5] show that meta-learned features have a better generalization compared to conventionally trained networks with exactly the same structure. So, using the meta-learning algorithms leads to the better representation learning and these two are not to independent concepts. \n\n \n \n\n- What do you exactly mean by this?  \n\n“Our method is the first one to achieve well-calibrated few-shot models by only fine-tuning probabilistic linear models in the meta-test phase, without any learning mechanisms related to the meta-training or representation learning phase.”  \n\nThis mechanism is the well-known solution in transfer learning methods which is firstly popularized by [3] for few-shot learning. Am I missing something here? \n\n \n \n\nConsidering all these problems, I think the authors need to rewrite the introduction part and make it more reliable, considering the recent studies in few-shot learning. There are also some concerns for the Methodology part: \n\n \n \n\n- Based on my understanding, your representation learning part is similar to [3], and the only difference is the usage of SWA as post-processinf of the network parameters. Although it is mentioned that the purpose is to find a low-rank approximation, it is not clear that how do you finally deploy it in your algorithm. The algorithm is unclear and seems to be lost in the explanation of various concepts! \n\nAre you using a simple averaging (mentioned in equation (1)) as a regularizer and just describing that this implicitly steers the model towards low-rank representation? \n\n \n \n\n- For the fine-tuning in few-shot classification, it is not clear for me how logistic regression model fits within your framework.  \n\n \n \n\n- I think adapting the temperature scaling for meta-test phase is the only contribution for fine-tuning few-shot classification. Is this new for few-shot learning to be considered as new algorithm? \n\n \n \n\n- Overall, I found it very hard to follow the methodology as in some cases, the main flow of the algorithm is lost due to overexplanation of other concepts. \n\n \n \n\nRegarding experiments: \n\n \n \n\n- The few-shot regression results are interesting, but the few-shot classification results are not fair, because some recent algorithms like MELR [6] are not included in the comparison. \n\n \n \n \n \n \n\nReferences:  \n\n[1] How to train your MAML, ICLR 2018. \n\n[2] Cross-Domain Few-shot Classification via Learned Features-wise Transformation, ICLR 2020. \n\n[3] A closer look at few-shot classification, ICLR 2019. \n\n[4] A Baseline for Few-Shot Image Classification, ICLR 2020. \n\n[5] Unraveling Meta-Learning: Understanding Feature Representations for Few-Shot Tasks, ICML 2020. \n\n[6] MELR: Meta-Learning via Modeling Episode-Level Relationships for Few-shot Learning, ICLR 2021. \n\n \n", "summary_of_the_paper": "This paper proposes a transfer learning method for few-shot regression/classification. In representation learning, it adapts the stochastic weight averaging (SWA) as a regularizer for learning more generalizable features. In fine-tuning phase (adaption to a new task in the meta-test phase), it treats the regression and classification differently. For regression a hierarchical Bayse is used while in classification, a conventional method is merged with temperature scaling. This work achieves comparable results to SOTA, however, they are major concerns that need to be addressed. ", "summary_of_the_review": "In summary, the proposed algorithm seems interesting. However, there are some major problems in the opening remarks of the paper, the explanation of the algorithm is not clear enough, and the few-shot classification performance of the proposed method is not fairly compared with recent methods. These concerns need to be addressed. \n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "SmoE3bG9WJl", "reviewer": "Reviewer_7iTd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The writing is easy to follow. The introduction provides a good motivation for the proposed approach and presents a thorough review of the existing literature. The idea of using SWA for few-shot classification is itself interesting and new as far as I can tell, and the empirical results look promising. The conjecture that it fosters lower-rank representations is supported by empirical observations.\n\nHowever, after reading the paper, I'm left with the question: beyond the application of SWA to the few-shot learning setting, what is MFRL? One of the claimed contributions is that it \"can handle both few-shot regression and classification\", however different adaptation strategies are used for regression (hierarchical Bayesian linear model) and classification (L2-regularized logistic regression with a temperature coefficient). As far as I can tell, the aspects common to both settings are i) learning the representation on the training data using a single objective rather than episodic losses (which has already been explored in previous work), and ii) applying SWA. In fact, Algorithm 1 feels more like two algorithms combined with if-statements than a unique algorithm. I feel that the presentation would have been better and more straightforwardly characterized as an application of SWA to the few-shot learning setting, and I'd be interested in hearing the authors' opinion on this.\n\nAdditional questions/comments:\n\n- Can the authors say a few words on hyperparameter selection using meta-validation data? In the cross-domain setting (e.g. mini-ImageNet -> CUB, or Meta-Dataset), is this a good selection strategy?\n- I am surprised by how well MFRL without SWA performs for regression. Can the authors expand on what aspect of MFRL is responsible for its good performance? As I understand, it consists in training concurrently on all training regression tasks; how is it that it outperforms more sophisticated meta-learning approaches?\n- In the classification setting, MFRL only really stands out when SWA is applied. Are there other aspects to MFRL that positively impact performance?\n- The difference between MFRL and Baseline++ (one normalizes the features, and the other normalizes both the features and the classification weights) is easy to miss. Being such a small implementation difference, it also reinforces the idea that MFRL reduces to SWA, and naming it is not necessary.\n- Presenting results on model calibration is good, but the paper feels lacking in details. Judging by the introduction, using a temperature parameter is sufficient to obtain good model calibration. How come? The submission mentions that \"given a meta-test task, a new  top layer is fine-tuned with few-shot samples to obtain probabilistic models with well-calibrated uncertainty\"; what about the fine-tuning procedure ensures calibration? Is some kind of early-stopping performed by looking at calibration metrics?", "summary_of_the_paper": "The submission introduces a few-shot classification and regression approach called Meta-Free Representation Learning (MFRL). First, a representation is learned on the meta-training data: for regression, the model is trained on all available training regression tasks concurrently; for classification, the model is trained on the full-ways classification problem using all training classes. Then, stochastic weight averaging (SWA) is applied to the model by continuing training for a certain number of epochs and averaging the parameters obtained across those additional epochs.\n\nFor test tasks, the regression or classification layer is discarded, and a new output layer is trained while freezing the network weights. For regression, approximate inference is performed on a hierarchical Bayesian linear model. For classification, a logistic regression model is trained with L2 regularization and a temperature hyperparameter.\n\nExperimental results are presented on the sine wave and head pose problems for regression, and on mini-ImageNet, tiered-ImageNet, CIFAR-FS, and FC100 for classification. The extent to which SWA encourages learning lower-rank representations (as hypothesized) is verified through visualizations of the normalized eigenvalues. Finally, calibration curves are shown for classification to demonstrate how MFRL's temperature scaling factor leads to better calibrated models.", "summary_of_the_review": "The submission applies an existing idea (SWA) to the few-shot learning setting, which is interesting and shows promising results. However, there is a disconnect between the simplicity of the idea and how it's presented by the submission, and overall there aren't a whole lot of new insights to be gained beyond the observation that SWA works well for few-shot learning.\n\n---\n\n**POST-REBUTTAL**: The response addresses most of my concerns. I would like to see those clarifications integrated in the main text.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: By the scores, this submission is quite borderline. This paper introduces stochastic weight averaging into a few-shot learning setting, The reviewers all agreed the work was sound; discussion after the author response focused on the theoretical justifications, degree of novelty and potential impact, and the empirical support. 

The primary concerns were that the work was slightly too incremental to obviously merit publication at this stage: though the empirical results were sound, they mostly follow the existing observation that SWA tends to be beneficial for generalization in other settings; apparently in few-shot learning as well. The positives would be that this is simple enough that it could become a general "best practice" in few-shot learning baselines, and as such communicating this is important.

The other discussion focused around the theoretical justification relating SWA to low-rank solutions. While empirically it does seem that the solutions found by SWA lead to low-rank representations, this is not really adequately explored, and it's not clear enough why this should be expected to happen. I think if this relationship between SWA and low-rank representations were more clearly explored then the paper would be a strong accept.

As it stands, it is quite borderline. Based on the scores (5,5,6), the recommendation is to reject, but it certainly could be included as well, as it has solid execution and is a clear topical fit for ICLR.
Decision: Reject