Year: 2021

Abstract:
Recent studies on few-shot classification using transfer learning pose challenges to the effectiveness and efficiency of episodic meta-learning algorithms. Transfer learning approaches are a natural alternative, but they are restricted to few-shot classification. Moreover, little attention has been on the development of probabilistic models with well-calibrated uncertainty from few-shot samples, except for some Bayesian episodic learning algorithms. To tackle the aforementioned issues, we propose a new transfer learning method to obtain accurate and reliable models for few-shot regression and classification. The resulting method does not require episodic meta-learning and is called meta-free representation learning (MFRL). MFRL first finds low-rank representation generalizing well on meta-test tasks. Given the learned representation, probabilistic linear models are fine-tuned with few-shot samples to obtain models with well-calibrated uncertainty. The proposed method not only achieves the highest accuracy on a wide range of few-shot learning benchmark datasets but also correctly quantifies the prediction uncertainty. In addition, weight averaging and temperature scaling are effective in improving the accuracy and reliability of few-shot learning in existing meta-learning algorithms with a wide range of learning paradigms and model architectures.

text: Recent studies on few-shot classification using transfer learning pose challenges to the effectiveness and efficiency of episodic meta-learning algorithms. Transfer learning approaches are a natural alternative, but they are restricted to few-shot classification. Moreover, little attention has been on the development of probabilistic models with well-calibrated uncertainty from few-shot samples, except for some Bayesian episodic learning algorithms. To tackle the aforementioned issues, we propose a new transfer learning method to obtain accurate and reliable models for few-shot regression and classification. The resulting method does not require episodic meta-learning and is called meta-free representation learning (MFRL). MFRL first finds low-rank representation generalizing well on meta-test tasks. Given the learned representation, probabilistic linear models are fine-tuned with few-shot samples to obtain models with well-calibrated uncertainty. The proposed method not only achieves the highest accuracy on a wide range of few-shot learning benchmark datasets but also correctly quantifies the prediction uncertainty. In addition, weight averaging and temperature scaling are effective in improving the accuracy and reliability of few-shot learning in existing meta-learning algorithms with a wide range of learning paradigms and model architectures.

heading: 1 INTRODUCTION
text: Currently, the vast majority of few-shot learning methods are within the general paradigm of metalearning (a.k.a. learning to learn) (Bengio et al., 1991; Schmidhuber, 1987; Thrun & Pratt, 1998), which learns multiple tasks in an episodic manner to distill transferrable knowledge (Vinyals et al., 2016; Finn et al., 2017; Snell et al., 2017). Although many episodic meta-learning methods report state-of-the-art (SOTA) performance, recent studies show that simple transfer learning methods with fixed embeddings (Chen et al., 2019; Tian et al., 2020) can achieve similar or better performance in few-shot learning. It is found that the effectiveness of optimization-based meta-learning algorithms is due to reusing high-quality representation, instead of rapid learning of task-specific representation (Raghu et al., 2020). The quality of the presentation is not quantitatively defined, except for some empirical case studies (Goldblum et al., 2020). Recent machine learning theories (Saunshi et al., 2021) indicate that low-rank representation leads to better sample efficiency in learning a new task. However, those theoretical studies are within the paradigm of meta-learning and do not reveal how to obtain low-rank representation for few-shot learning outside the realm of meta-learning. This motivates us to investigate ways to improve the representation for adapting to new few-shot tasks in a meta-free manner by taking the advantage of simplicity and robustness in transfer learning.
In parallel, existing transfer learning methods also have limitations. That is, the existing transfer learning methods may not find representation generalizing well to unseen few-shot tasks (Chen et al., 2019; Dhillon et al., 2020) , compared with state-of-the-art meta-learning methods (Ye et al., 2020; Zhang et al., 2020). Although some transfer learning methods utilize knowledge distillation and self-supervised training to achieve strong performance in few-shot classification, they are restricted to few-shot classification problems (Mangla et al., 2020; Tian et al., 2020). To the best of our knowledge, no transfer learning method is developed to achieve similar performance to metalearning in few-shot regression. As such, it is desirable to have a transfer learning method that finds high-quality representation generalizing well to unseen classification and regression problems.
The last limitation of the existing transfer learning methods in few-shot learning is the lack of uncertainty calibration. Uncertainty quantification is concerned with the quantification of how likely certain outcomes are. Despite a plethora of few-shot learning methods (in fact, machine learning in general) to improve the point estimation accuracy, few methods are developed to get probabilistic models with improved uncertainty calibration by integrating Bayesian learning into episodic metatraining (Grant et al., 2018; Finn et al., 2018; Yoon et al., 2018; Snell & Zemel, 2021). Few-shot learning models can be used in risk-averse applications such as medical diagnosis (Prabhu et al., 2019). The diagnosis decision is made on not only point estimation but also probabilities associated with the prediction. The risk of making wrong decisions is significant when using uncalibrated models (Begoli et al., 2019). Thus, the development of proper fine-tuning steps to achieve well-calibrated models is the key towards practical applications of transfer learning in few-shot learning.
In this paper, we develop a simple transfer learning method as our own baseline to allow easy regularization towards more generalizable representation and calibration of prediction uncertainty. The regularization in the proposed transfer learning method works for regression and classification problems so that we can handle both problems within a common architecture. The calibration procedure is easily integrated into the developed transfer learning method to obtain few-shot learning models with good uncertainty quantification. Therefore, the resulting method, called Meta-Free Representation Learning (MFRL), overcomes the aforementioned limitations in existing transfer learning methods for few-shot learning. Our empirical studies demonstrate that the relatively overlooked transfer learning method can achieve high accuracy and well-calibrated uncertainty in few-shot learning when it is combined with the proper regularization and calibration. Those two tools are also portable to meta-learning methods to improve accuracy and calibration, but the improvement is less significant compared with that of transfer learning.
We use stochastic weight averaging (SWA) (Izmailov et al., 2018), which is agnostic to loss function types, as implicit regularization to improve the generalization capability of the representation. We also shed light on that the effectiveness of SWA is due to its bias towards low-rank representation. To address the issue of uncertainty quantification, we fine-tune appropriate linear layers during the meta-test phase to get models with well-calibrated uncertainty. In MFRL, hierarchical Bayesian linear models are used to properly capture the uncertainty from very limited training samples in few-shot regression, whereas the softmax output is scaled with a temperature parameter to make the few-shot classification model well-calibrated. Our method is the first one to achieve well-calibrated few-shot models by only fine-tuning probabilistic linear models in the meta-test phase, without any learning mechanisms related to the meta-training or representation learning phase.
Our contributions in this work are summarized as follows:
• We propose a transfer learning method that can handle both few-shot regression and classification problems with performance exceeding SOTA.
• For the first time, we empirically find the implicit regularization of SWA towards low-rank representation, which is a useful property in transferring to few-shot tasks.
• The proposed method results in well-calibrated uncertainty in few-shot learning models while preserving SOTA accuracy.
• The implicit regularization of SWA and temperature scaling factor can be applied to existing meta-learning methods to improve their accuracy and reliability in few-shot learning.

heading: 2 RELATED WORK
text: Episodic meta-learning approaches can be categorized into metric-based and optimization-based methods. Metric-based methods project input data to feature vectors through nonlinear embeddings and compare their similarity to make the prediction. Examples of similarity metrics include the weighted L1 metric (Koch et al., 2015), cosine similarity (Qi et al., 2018; Vinyals et al., 2016), and Euclidean distance to class-mean representation (Snell et al., 2017). Instead of relying on predefined metrics, learnable similarity metrics are introduced to improve the few-shot classification performance (Oreshkin et al., 2018; Sung et al., 2018). Recent metric-based approaches focus on developing task-adaptive embeddings to improve few-shot classification accuracy. Those task-adaptive embeddings include attention mechanisms for feature transformation (Fei et al., 2021; Gidaris & Komodakis, 2018; Ye et al., 2020; Zhang et al., 2021), graph neural networks (Garcia & Estrach,
2018), implicit class representation (Ravichandran et al., 2019), and task-dependent conditioning (Oreshkin et al., 2018; Yoon et al., 2020; 2019). Although metric-based approaches achieve strong performance in few-shot classification, they cannot be directly applied to regression problems.
Optimization-based meta-learning approaches try to find transferrable knowledge and adapt to new tasks quickly. An elegant and powerful meta-learning approach, termed model-agnostic metalearning (MAML), solves a bi-level optimization problem to find good initialization of model parameters (Finn et al., 2017). However, MAML has a variety of issues, such as sensitivity to neural network architectures, instability during training, arduous hyperparameter tuning, and high computational cost. On this basis, some follow-up methods have been developed to simplify, stabilize and improve the training process of MAML (Antoniou et al., 2018; Flennerhag et al., 2020; Lee & Choi, 2018; Nichol et al., 2018; Park & Oliva, 2019). In practice, it is very challenging to learn high-dimensional model parameters in a low-data regime. Latent embedding optimization (LEO) attempts to learn low-dimensional representation to generate high-dimensional model parameters (Rusu et al., 2019). Meanwhile, R2-D2 (Bertinetto et al., 2019) and MetaOptNet (Lee et al., 2019) reduce the dimensionality of trainable model parameters by freezing feature extraction layers during inner loop optimization. Note that the proposed method is fundamentally different from R2-D2 and MetaOptNet because our method requires neither episodic meta-learning nor bi-level optimization.
Transfer learning approaches first learn a feature extractor on all training data through standard supervised learning, and then fine-tune a linear predictor on top of the learned feature extractor in a new task (Chen et al., 2019). However, vanilla transfer learning methods for few-shot learning do not take extra steps to make the learned representation generalizing well to unseen meta-test tasks. Some approaches in this paradigm are developed to improve the quality of representation and boost the accuracy of few-shot classification, including cooperative ensembles (Dvornik et al., 2019), knowledge distillation (Tian et al., 2020), and auxiliary self-supervised learning (Mangla et al., 2020). Nevertheless, those transfer learning methods are restricted to few-shot classification. MFRL aims to find representation generalizing well from the perspective of low-rank representation learning, which is supported by recent theoretical studies (Saunshi et al., 2021). Furthermore, MFLR is the first transfer learning method that can handle both few-shot regression and classification problems and make predictions with well-calibrated uncertainty.

heading: 3 BACKGROUND

heading: 3.1 EPISODIC META-LEARNING
text: In episodic meta-learning, the meta-training data contains T episodes or tasks, where the τ th episode consists of dataDτ = {(xτ,j ,yτ,j)}Nτj=1 withNτ samples. Tasks and episodes are used interchangeably in the rest of the paper. Episodic meta-learning algorithms aim to find common model parameters θ which can be quickly adapted to task-specific parameters φτ (τ = 1, ..., T ). For example, MAML-type algorithms assume φτ is one or a few gradient steps away from θ (Finn et al., 2017; 2018; Grant et al., 2018; Yoon et al., 2018), while other meta-learning approaches assume that φτ and θ share the parameters in the feature extractor and only differ in the top layer (Bertinetto et al., 2019; Lee et al., 2019; Snell et al., 2017).

heading: 3.2 STOCHASTIC WEIGHT AVERAGING
text: The idea of stochastic weight averaging (SWA) along the trajectory of SGD goes back to Polyak–Ruppert averaging (Polyak & Juditsky, 1992). Theoretically, weight averaging results in faster convergence for linear models in supervised learning and reinforcement learning(Bach & Moulines, 2013; Lakshminarayanan & Szepesvari, 2018). In deep learning, we are more interested in tail stochastic weight averaging (Jain et al., 2018), which averages the weights after T training epochs. The averaged model parameters θSWA can be computed by running s additional training epochs using SGD
θSWA = 1
s T+s∑ i=T+1 θi, (1)
where θi denotes the model parameters at the end of the i-th epoch. SWA has been applied to supervised learning of deep neural neural networks to achieve higher test accuracy (Izmailov et al., 2018).

heading: 4 METHODOLOGY
text: The proposed method is a two-step learning algorithm: meta-free representation learning followed by fine-tuning. We employ SWA to make the learned representation low-rank and better generalize to meta-test data. Given a meta-test task, a new top layer is fine-tuned with few-shot samples to obtain probabilistic models with well-calibrated uncertainty. Note that MFRL can be used for both regression and classification depending on the loss function. The pseudocode of MFRL is presented in Appendix A.1.

heading: 4.1 REPRESENTATION LEARNING
text: Common representation can be learned via maximization of the likelihood of all training data with respect to θ rather than following episodic meta-learning. To do so, we group the data Dτ = {(xτ,j ,yτ,j)}Nτj=1 from all meta-training tasks into a single dataset Dtr. Given aggregated training dataDtr = {X,Y}, representation can be learned by maximizing the likelihood p (Dtr | θ) with respect to θ. Let θ = [θf ,W], where θf represents parameters in the feature extractor and W denotes the parameters in the top linear layer. The feature extractor h(x) ∈ Rp is a neural network parameterized by θf and outputs a feature vector of dimension p. The specific form of the loss function depends on whether the task is regression or classification and can be given as follows:
LRP (θ) = − log p (Dtr | θ) = { LMSE(θ), regression LCE(θ), classification
where
LMSE(θ) = 1
2N ′ T∑ τ=1 Nτ∑ j=1 ( yτ,j −w>τ h (xτ,j) )2 , (2)
LCE(θ) = − N ′∑ j=1 C∑ c=1 yj,c log exp(w>c h(xj))∑C c′=1 exp(w > c′h(xj))
(3)
For regression problems, the model learns T regression tasks (W = [w1, ...,wT ] ∈ R(p+1)×T ) simultaneously using the loss function LMSE given in Eq. 2, whereas the model learns a C-class classification model 1 (W = [w1, ...,wC ] ∈ R(p+1)×C) for classification problems using the loss function LCE in Eq. 3. The loss function - either Eq. 2 or 3 - can be minimized through standard stochastic gradient descent, where N ′ = ∑T τ=1Nτ is the total number of training samples.
Post-processing via SWA Minimizing the loss functions in Eq. 2 and 3 by SGD may not necessarily result in representation that generalizes well to few-shot learning tasks in the meta-test set. The last hidden layer of a modern deep neural network is high-dimensional and may contain spurious features that over-fit the meta-training data. Recent meta-learning theories indicate that better sample complexity in learning a new task can be achieved via low-rank representation, whose singular values decay faster (Saunshi et al., 2021). We aim to find low-rank representation Φ = h(X) without episodic meta-learning, which is equivalent to finding the conjugate kernel KC = ΦΦ> with fast decaying eigenvalues. To link the representation with the parameter space, we can linearize the neural network by the first-order Taylor expansion at θT and get the finite width neural tangent kernel (NTK) KNTK(X,X) = J(X)J(X)>, where J(X) = ∇θfθ(X) ∈ RN
′×|θ| is the Jacobian matrix, and KNTK is a composite kernel containing KC (Fan & Wang, 2020). The distributions of eigenvalues for KNTK and KC are empirically similar. Analyzing KNTK could shed light on the properties of KC. In parallel, KNTK shares the same eigenvalues of the Gauss-Newton matrix G = 1N ′ J(X) >J(X). For linearized networks with squared loss, the Gauss-Newton matrix G well
1C is the total number of classes in Dtr. Learning a C-class classification model solves all possible tasks in the meta-training dataset because each task Dτ only contains a subset of C classes.
approximates the Hessian matrix H when y is well-described by fθ(x) (Martens, 2020). This is the case when SGD converges to θT within a local minimum basin. A Hessian matrix with a lot of small eigenvalues corresponds to a flat minimum, where the loss function is less sensitive to the perturbation of model parameters (Keskar et al., 2017). It is known that averaging the weights after SGD convergence in a local minimum basin pushes θT towards the flat side of the loss valley (He et al., 2019). As a result, SWA could result in a faster decay of eigenvalues in the kernel matrix, and thus low-rank representation. Our conjecture about SWA as implicit regularization towards low-rank representation is empirically verified in Section 5.

heading: 4.2 FINE-TUNING
text: After representation learning is complete, W is discarded and θf is frozen in a new few-shot task. Given the learned representation, we train a new probabilistic top layer in a meta-test task using fewshot samples. The new top layer will be configured differently depending on whether the few-shot task is a regression or a classification problem.
In a few-shot regression task, we learn a new linear regression model y = w>h(x) + on a fixed feature extractor h(x) ∈ Rp with few-shot training data D = {(xi, yi)}ni=1, where w denotes the model parameters and is Gaussian noise with zero mean and variance σ2. To avoid interpolation on few-shot training data (n p), a Gaussian prior p(w | λ) = ∏p i=0N (wi | 0, λ) is placed over w, where λ is the precision in the Gaussian prior. However, it is difficult to obtain an appropriate value for λ in a few-shot regression task because no validation data is available in D. Hierarchical Bayesian linear models can be used to obtain optimal regularization strength and grounded uncertainty estimation using few-shot training data only. To complete the specification of the hierarchical Bayesian model, the hyperpriors on λ and σ2 are defined as p(λ) = Gamma (λ | a, b) and p(σ−2) = Gamma(σ−2 | c, d), respectively. The hyper-priors become very flat and non-informative when a, b, c and d are set to very small values. The posterior over all latent variables given the data is p ( w, λ, σ2 | X,y ) , where X = {x}ni=1 and y = {yi}ni=1. However, the
posterior distribution p ( w, λ, σ2 | X,y ) is intractable. The iterative optimization based approximate inference (Tipping, 2001) is chosen because it is highly efficient. The point estimation for λ and σ2 is obtained by maximizing the marginal likelihood function p ( y | X, λ, σ2 ) . The posterior
of model parameters p ( w | X,y, λ, σ2 ) is calculated using the estimated λ and σ2. Previous two steps are repeated alternately until convergence.
The predictive distribution for a new sample x∗ is
p ( y∗ | x∗,X,y, λ, σ2 ) = ∫ p ( y∗ | x∗,w, σ2 ) p ( w | X,y, λ, σ2 ) dw, (4)
which can be computed analytically because both distributions on the right hand side of Eq. 4 are Gaussian. Consequently, hierarchical Bayesian linear models avoids over-fitting on few-shot training data and quantifies predictive uncertainty.
In a few-shot classification task, a new logistic regression model is learned with the post-processed representation. A typical K-way n-shot classification task D = {(xi, yi)}nKi=1 consists of K classes (different from meta-training classes) and n training samples per class. Minimizing an un-regularized cross-entropy loss results in a significantly over-confident classification model because the norm of logistic regression model parameters W ∈ R(p+1)×K becomes very large when few-shot training samples can be perfectly separated in the setting nK p. A weighted L2 regularization term is added to the cross-entropy loss to mitigate the issue
L(W) = − nK∑ i=1 K∑ c=1 yi,c log exp(w>c h(xi))∑K c′=1 exp(w > c′h(xi)) + λ K∑ c=1 w>c wc, (5)
where λ is the regularization coefficient, which affects the prediction accuracy and uncertainty. It is difficult to select an appropriate value of λ in each of the meta-test tasks due to the lack of validation data in D. We instead treat λ as a global hyper-parameter so that the value of λ should be determined based on the accuracy on meta-validation data. Note that the selected λ with high validation accuracy does not necessarily lead to well calibrated classification models. As such, we introduce the temperature scaling factor (Guo et al., 2017) as another global hyper-parameter to
scale the softmax output. Given a test sample x∗, the predicted probability for class c becomes
pc = exp(w>c h(x∗)/T )∑K c′=1 exp(w > c′h(x∗)/T ) , (6)
where T is the temperature scaling factor. In practice, we select the L2 regularization coefficient λ and the temperature scaling factor T as follows. At first, we set T to 1, and do grid search on the meta-validation data to find the λ resulting in the highest meta-validation accuracy. However, finetuning λ does not ensure good calibration. It is the temperature scaling factor that ensures the good uncertainty calibration. Similarly, we do grid search of T on the meta-validation set, and choose the temperature scaling factor resulting in the lowest expected calibration error (Guo et al., 2017). Note that different values of T do not affect the classification accuracy because temperature scaling is accuracy preserving.

heading: 5 EXPERIMENTS
text: We follow the standard setup in few-shot learning literature. The model is trained on a meta-training dataset and hyper-parameters are selected based on the performance on a meta-validation dataset. The final performance of the model is evaluated on a meta-test dataset. The proposed method is applied to few-shot regression and classification problems and compared against a wide range of alternative methods.

heading: 5.1 FEW-SHOT REGRESSION RESULTS
text: Sine waves (Finn et al., 2017) and head pose estimation (Patacchiola et al., 2020) datasets are used to evaluate the performance of MFRL in few-shot regression. We use the same backbones in literature (Patacchiola et al., 2020) to make fair comparisons. Details of the few-shot regression experiments can be found in Appendix A.2.
The results for few-shot regression are summarized in Table 1. In the sine wave few-shot regression, MFRL outperforms all meta-learning methods, demonstrating that high-quality representation can be learned in supervised learning, without episodic meta-learning. Although DKT with a spectral mixture (SM) kernel achieves high accuracy, the good performance should be attributed to the strong inductive bias to periodic functions in the SM kernel (Wilson & Adams, 2013). Additional results for MFRL with different activation functions are reported in Appendix A.3. In the head pose estimation experiment, MFRL also achieves the best accuracy. In both few-shot regression problems, SWA results in improved accuracy, suggesting that SWA can improve the quality of features and facilitate the learning of downstream tasks. In Fig. 1, uncertainty is correctly estimated by the hierarchical Bayesian linear model with learned features using just 10 training samples.

heading: 5.2 FEW-SHOT CLASSIFICATION RESULTS
text: We conduct few-shot classification experiments on four widely used few-shot image recognition benchmarks: miniImageNet (Ravi & Larochelle, 2017), tieredImageNet (Ren et al., 2018), CIFARFS (Bertinetto et al., 2019), and FC100 (Oreshkin et al., 2018). In addition, we test our approach on a cross-domain few-shot classification task from the miniImageNet to CUB. The experiment details about the few-shot classification datasets can be found in Appendix A.2. The proposed method is applied to three widely used network architectures: ResNet-12 (Lee et al., 2019; Ravichandran et al., 2019), wide ResNet (WRN-28-10) (Dhillon et al., 2020; Rusu et al., 2019), and a 4-layer convolutional neural network with 64 channels (Chen et al., 2019; Patacchiola et al., 2020) (in Appendix A.3).
The results of the proposed method and previous SOTA methods using similar backbones are reported in Table 2 and 3. The proposed method achieves the best performance in most of the experiments when compared with previous SOTA methods. Our method is closely related to Baseline++ (Chen et al., 2019) and fine-tuning on logits (Dhillon et al., 2020). Baseline++ normalizes both classification weights and features, while the proposed method only normalizes features. It allows our method to find a more accurate model in a more flexible hypothesis space, given high-quality representation. Compared with fine-tuning on logits, our method obtains better results by learning
a new logistic regression model on features, which store richer information about the data. Some approaches pretrain a C-class classification model on all training data and then apply highly sophisticated meta-learning techniques to the pretrained model to achieve SOTA performance (Rusu et al., 2019; Sun et al., 2019). Our approach with SWA outperforms those pretrained-then-meta-learned models, which demonstrates that SWA obtains high-quality representation that generalizes well to unseen tasks. Compared with improving representation quality for few-shot classification via selfdistillation (Tian et al., 2020), the computational cost of SWA is significantly smaller because it does not require training models from scratch multiple times. Moreover, SWA can be applied to find good representation for both few-shot regression and classification, while previous transfer learning approaches can only handle few-shot classification problems (Mangla et al., 2020; Tian et al., 2020).
MFRL is also applied to the cross-domain few-shot classification task as summarized in Table 4. MFRL outperforms other methods in this challenging task, indicating that the learned representation has strong generalization capability. We use the same hyperparameters (training epochs, learning rate, learning rate in SWA, SWA epoch, etc.) as in Table 2. The strong results indicate that MFRL is robust to hyperparameter choice. Surprisingly, meta-learning methods with adaptive embeddings do not outperform simple transfer learning methods like Baseline++ when the domain gap between base classes and novel classes is large. We notice that Tian et al. (2020) also reports similar results that transfer learning methods show superior performance on a large-scale cross-domain few-shot classification dataset. We still believe that adaptive embeddings should be helpful when the domain gap between base and novel classes is large. Nevertheless, how to properly train a model to obtain useful adaptive embeddings in novel tasks is an open question.

heading: 5.3 EFFECTIVE RANK OF THE REPRESENTATION
text: The rank of representation defines the number of independent bases. For deep learning, noise in gradients and numerical imprecision can cause the resulting matrix to be full-rank. Therefore, simply counting the number of non-zero singular values may not be an effective way to measure the rank of the representation. To compare the effective ranks, we plot the normalized singular values of the representation of meta-test data in Fig. 2, where the representation with SWA has a faster decay in singular values, thus indicating the lower effective rank of the presentation with SWA. The results empirically verify our conjecture that SWA is an implicit regularizer towards low-rank representation.
0 100 200 300 400 500 600 Singular value index i
0.0
0.2
0.4
0.6
0.8
1.0
No rm
al ize
d sin
gu la
r v al
ue
i/ m
ax
miniImageNet w.o. SWA: ilog i = 68.49 SWA: ilog i = 58.49
0 100 200 300 400 500 600 Singular value index i
0.0
0.2
0.4
0.6
0.8
1.0
No rm
al ize
d sin
gu la
r v al
ue
i/ m
ax
tieredImageNet w.o. SWA: ilog i = 83.98 SWA: ilog i = 80.01
0 100 200 300 400 500 600 Singular value index i
0.0
0.2
0.4
0.6
0.8
1.0
No rm
al ize
d sin
gu la
r v al
ue
i/ m
ax
CIFAR-FS
w.o. SWA: ilog i = 40.94 SWA: ilog i = 33.63
0 100 200 300 400 500 600 Singular value index i
0.0
0.2
0.4
0.6
0.8
1.0
No rm
al ize
d sin
gu la
r v al
ue
i/ m
ax
FC100
w.o. SWA: ilog i = 40.22 SWA: ilog i = 34.34
Figure 2: Normalized singular values for representation with and without SWA. The metric
−
∑
σ̄i log σ̄i is used to measure the effective rank of the representation, where σ̄i = σi/σmax. Faster decay in singular values indicates that fewer dimensions capture the most variation in all dimensions, thus lower effective rank.

heading: 5.4 FEW-SHOT CLASSIFICATION RELIABILITY
text: The proposed method not only achieves high accuracy in few-shot classification but also makes the classification uncertainty well-calibrated. A reliability diagram can be used to check model calibration visually, which plots an identity function between prediction accuracy and confidence when the model is perfectly calibrated (DeGroot & Fienberg, 1983). Fig. 3 shows the classification reliability diagrams along with widely used metrics for uncertainty calibration, including expected calibration error (ECE) (Guo et al., 2017), maximum calibration error (MCE) (Naeini et al., 2015), and Brier score (BRI) (Brier, 1950). ECE measures the average binned difference between confidence and accuracy, while MCE measures the maximum difference. BRI is the squared error between the predicted probabilities and one-hot labels. MAML is over-confident because tuning a deep neural network on few-shot data is prone to over-fitting. Meanwhile, Proto Net and Matching Net are better calibrated than MAML because they do not fine-tune the entire network during testing. Nevertheless, they are still slightly over-confident. The results indicate that MFRL with a global temperature scaling factor can learn well-calibrated models from very limited training samples.

heading: 5.5 APPLICATION IN META-LEARNING
text: Meanwhile, we also apply SWA to episodic meta-learning methods, such Proto Net, MAML and Matching Net, to improve their classification accuracy. The results in Table 5 indicate that SWA can improve the few-shot classification accuracy in both transfer learning and episodic meta-learning. SWA is orthogonal to the learning paradigm and model architecture. Thus, SWA can be applied to a wide range of few-shot learning methods to improve accuracy.
Furthermore, the temperature scaling factor can be applied to calibrate meta-learning methods, including MAML, Proto Net, and Matching Net. The reliability diagrams in Fig. 3 indicate that the temperature scaling factor not only calibrates classification uncertainty of transfer learning approaches, such as the proposed MFRL, but also makes the classification uncertainty well-calibrated in episodic meta-learning methods. Therefore, the temperature scaling factor can be applied to a wide range of few-shot classification methods to get well-calibrated uncertainty, while preserving the classification accuracy.

heading: 6 DISCUSSION
text: SWA has been applied to supervised learning of deep neural networks (Izmailov et al., 2018; Athiwaratkun et al., 2019) and its effectiveness was attributed to convergence to a solution on the flat side of an asymmetric loss valley (He et al., 2019). However, it does not explain the effectiveness of SWA in few-shot learning because the meta-training and meta-testing losses are not comparable after the top layer is retrained by the few-shot support data in a meta-test task. The effectiveness of SWA in few-shot learning must be related to the property of the representation. Although our results empirically demonstrate that SWA results in low-rank representation, further research about their connection is needed.
Explicit regularizers can also be used to obtain simple input-output functions in deep neural networks and low-rank representation, including L1 regularization, nuclear norm, spectral norm, and Frobenius norm (Bartlett et al., 2017; Neyshabur et al., 2018; Sanyal et al., 2020). However, some of those explicit regularizers are not compatible with standard SGD training or are computationally expensive. In addition, it is difficult to choose the appropriate strength of explicit regularization. Too strong explicit regularization can bias towards simple solutions that do not fit the data. In comparison, SWA is an implicit regularizer that is completely compatible with the standard SGD training without much extra computational cost. Thus, it can be easily combined with transfer learning and meta-learning to obtain more accurate few-shot learning models. In parallel, SWA is also robust to the choice of the hyperparameters - the learning rate and training epochs in the SWA stage (see details in Appendix A.4).

heading: 7 CONCLUSIONS
text: In this article, we propose MFRL to obtain accurate and reliable few-shot learning models. SWA is an implicit regularizer towards low-rank representation, which generalizes well to unseen meta-test tasks. The proposed method can be applied to both classification and regression tasks. Extensive experiments show that our method not only outperforms other SOTA methods on various datasets but also correctly quantifies the uncertainty in prediction.

heading: A APPENDIX
text: A.1 PSEUDO CODE FOR MFRL
Algorithm 1 Meta-free representation learning for few-shot learning
Merge all training tasks Dtr = {Dτ}Tτ=1 Initialize model parameters θ = [θf ,W] Maximize the likelihood on all training data p (Dtr | θ) using SGD
Minimize the squared loss for regression problems Minimize the cross-entropy loss for classification problems
Run SWA to obtain θSWA Discard W and freeze θf Learn a new top layer using support data D in a test task:
Learn a hierarchical Bayesian linear model for a regression task Learn a logistic regression model with the temperature scaling factor for a classification task
A.2 EXPERIMENT DETAILS
Sine waves are generated by y = A sin(x−ϕ)+ , where amplitudeA ∈ [0.1, 5.0], phase ϕ ∈ [0, π] and is white noise with standard deviation of 0.1 (Finn et al., 2017). Each sine wave contains 200 samples by sampling x uniformly from [−5.0, 5.0]. We generate 500 waves for training, validation and testing, respectively. All sine waves are different from each other. We use the same backbone network described in MAML (Finn et al., 2017): a two-layer MLP with 40 hidden units in each layer. We use the SGD optimizer with a learning rate of 10−3 over 8 × 104 training iterations and run SWA over 2× 104 training iterations with a learning rate of 0.05.
Head pose regression data is derived from the Queen Mary University of London multi-view face dataset (Gong et al., 1996). It contains images from 37 people and 133 facial images per person. Facial images cover a view sphere of 90◦ in yaw and 120◦ in tilt. The dataset is divided into 3192 training samples (24 people), 1064 validation samples (8 people), and 665 test samples (5 people). We use the same feature extractor described in literature (Patacchiola et al., 2020): a three-layer convolutional neural network, each with 36 output channels, stride 2, and dilation 2. We train the model on the training people set for 300 epochs using the SGD optimizer with a learning rate of 0.01 and run 25 epochs of SWA with a learning rate of 0.01.
miniImageNet is a 100-class subset of the original ImageNet dataset (Deng et al., 2009) for fewshot learning (Vinyals et al., 2016). Each class contains 600 images in RGB format of the size 84 × 84. miniImageNet is split into 64 training classes, 16 validation classes, and 20 testing classes, following the widely used data splitting protocol (Ravi & Larochelle, 2017).
tieredImageNet is another subset of the ImageNet dataset for few-shot learning (Ren et al., 2018). It contains 608 classes grouped into 34 categories, which are split into 20 training categories (351 classes), 6 validation categories (97 classes), and 8 testing categories (160 classes). Compared with miniImageNet, training classes in tieredImageNet are sufficiently distinct from test classes, making few-shot classification more difficult.
CIFAR-FS is a derivative of the original CIFAR-100 dataset by randomly splitting 100 classes into 64, 16, and 20 classes for training, validation, and testing, respectively (Bertinetto et al., 2019).
FC100 is another derivative of CIFAR-100 with minimized overlapped information between train classes and test classes by grouping the 100 classes into 20 superclasses (Oreshkin et al., 2018). They are further split into 60 training classes (12 superclasses), 20 validation classes (4 superclasses), and 20 test classes (4 superclasses).
miniImageNet to CUB is a cross-domain few-shot classification task, where the models are trained on miniImageNet and tested on CUB (Welinder et al., 2010). Cross-domain few-shot classification is more challenging due to the big domain gap between two datasets. We can better evaluate the generalization capability in different algorithms. We follow the experiment setup in Yue et al. (2020) and use the WRN2-28-10 as the backbone.
The backbone model is trained on all training classes using C-class cross-entropy loss by the SGD optimizer (momentum of 0.9 and weight decay of 1e-4) with a mini-batch size of 64. The learning rate is initialized as 0.05 and is decayed by 0.1 after 60, 80, and 90 epochs (100 epochs in total). After the SGD training converges, we run 100 epochs of SWA with a learning rate of 0.02. Note that MFRL is not sensitive to training epochs and learning rates in SWA (see Appendix A.4). The training images are augmented with random crop, random horizontal flip, and color jitter.
During testing, we conduct 5 independent runs of 600 randomly sampled few-shot classification tasks from test classes and calculate the average accuracy. Each task contains 5 classes, 1 × 5 or 5× 5 support samples, and 75 query samples. A logistic regression model is learned using only the support samples. The classification accuracy is evaluated on the query samples.
A.3 ADDITIONAL RESULTS ON FEW-SHOT REGRESSION AND CLASSIFICATION
The additional results on few-shot regression using different activation functions are reported in Table 6. MFRL achieves high accuracy with different activation functions.
The few-shot classification results using a 4-layer convolutional neural network (or similar architectures) are reported in Table 7 and 8. Similar to the results using ResNet-12 and WRN-28-10, the proposed method outperforms a wide range of meta-learning approaches. Our method is only
second to few-shot embedding adaptation with transformer (FEAT) (Ye et al., 2020) on miniImageNet dataset. Recently, meta-learned attention modules are built on top of the convolutional neural network to get improved few-shot classification accuracy. Direct comparison to those methods with attention modules (Ye et al., 2020; Fei et al., 2021; Zhang et al., 2021) may not be fair because recent studies show that transformer itself can achieve better results than convolutional neural networks in image classification (Dosovitskiy et al., 2021). It is difficult to determine whether the performance improvement is due to the meta-learning algorithm or the attention modules. To make a fair comparison, we add convolutional block attention modules (Woo et al., 2018) on top of ResNet12 features (before global average pooling). As shown in Fig. 9, MFRL with attention modules achieves comparable results with MELR and IEPT.
The uncertainty calibration results of MFRL with the temperature scaling factor are presented in Fig. 5. The prediction confidence aligns well with the prediction accuracy. It demonstrates that MFRL with the temperature scaling factor results in well calibrated models.
A.4 SENSITIVITY OF MFRL
The performance of MFRL is not sensitive to learning rates in SWA. As shown in Fig. 6, the representation learned by SWA generalizes better than the one from standard SGD, as long as the learning rate in SWA is in a reasonable range. In addition, the prediction accuracy on meta-test tasks keeps stable even after running SWA for many epochs on the training data. Therefore, MFRL is not sensitive to training epochs. This desirable property makes the proposed method easy to use when solving few-shot learning problems in practice.
A.5 COMPARISON WITH EXPONENTIAL MOVING AVERAGING
Exponential moving average (EMA) decays the importance of model weights from early training epochs exponentially. Let θavg ← aθavg + (1− a) θnew. We try EMA with different values of a. In Table 10, EMA improves the performance when a is within a reasonable range. Note that EMA introduces one extra hyperparameter, the forgetting factor. It makes EMA less desirable in practice.
A.6 HIERARCHICAL BAYESIAN LINEAR CLASSIFICATION MODEL
Similar to the hierarchical Bayesian linear regression model, the prior distribution over w is p(w | λ) = ∏p i=0N (wi | 0, λ), where λ is the precision in the Gaussian prior. The hyperprior on λ is defined as p(λ) = Gamma (λ | a, b). The posterior over all latent variables given the data is
p (w, λ | X,y) = p (y | X,w, λ) p (w | λ) p(λ) p(y | X)
(7)
MCMC sampling (Hoffman & Gelman, 2014) is used to avoid potential deterioration in predictive performance due to approximated inference. A flat and non-informative hyperprior (a = b = 10−6) is used because no prior knowledge is available. In Table 11, the hierarchical Bayesian linear classification model achieves slightly worse performance than the logistic regression model. However, the classification model is not well calibrated, as shown in Fig. 7.
After fine-tuning a and b using the meta-validation data, it is possible to get better calibrated classification models on test tasks. Besides, the classification accuracy is still slightly worse than logistic regression after hyperparameter tuning. Our observations align with a recent study, which shows that the Bayesian classification model cannot achieve similar performance to the non-Bayesian counterpart without tempering the posterior (Wenzel et al., 2020). We do not further experiment tempered posterior in the hierarchical Bayesian linear classification model because it introduces an extra temperature hyperparameter that requires tuning. The original purpose of introducing the hierarchical Bayesian model is to get an accurate and well calibrated classification model without hyperparameter tuning. Consequently, the hierarchical Bayesian model is not used in few-shot classification in that hierarchical Bayesian linear classification models cannot achieve high accuracy and good uncertainty calibration from a non-informative hyperprior. If hyperparameter tuning is inevitable, it is much easier to tune a logistic regression model with a temperature scaling factor, compared with tuning a hierarchical Bayesian model. Furthermore, the computational cost of learning a hierarchical Bayesian linear classification model via MCMC sampling is much larger than that of learning a logistic regression model.

Review summaries:
The submission introduces a few-shot classification and regression approach called Meta-Free Representation Learning (MFRL). First, a representation is learned on the meta-training data: for regression, the model is trained on all available training regression tasks concurrently; for classification, the model is trained on the full-ways classification problem using all training classes. Then, stochastic weight averaging (SWA) is applied to the model by continuing training for a certain number of epochs and averaging the parameters obtained across those additional epochs.  For test tasks, the regression or classification layer is discarded, and a new output layer is trained while freezing the network weights. For regression, approximate inference is performed on a hierarchical Bayesian linear model. For classification, a logistic regression model is trained with L2 regularization and a temperature hyperparameter.  Experimental results are presented on the sine wave and head pose problems for regression, and on mini-ImageNet, tiered-ImageNet, CIFAR-FS, and FC100 for classification. The extent to which SWA encourages learning lower-rank representations (as hypothesized) is verified through visualizations of the normalized eigenvalues. Finally, calibration curves are shown for classification to demonstrate how MFRL's temperature scaling factor leads to better calibrated models.

This paper proposes a transfer learning method for few-shot regression/classification. In representation learning, it adapts the stochastic weight averaging (SWA) as a regularizer for learning more generalizable features. In fine-tuning phase (adaption to a new task in the meta-test phase), it treats the regression and classification differently. For regression a hierarchical Bayse is used while in classification, a conventional method is merged with temperature scaling. This work achieves comparable results to SOTA, however, they are major concerns that need to be addressed. 

This paper presents a few-shot learning method based on the reprehensive pre-train learning using stochastic weight averaging (SWA). The merits of the proposed work is that this proposed method can works for both few-shot regression and classification problems and both of them achieves better results compared with the other recent works as reported in the manuscript. And as claimed by the author, this is the first few-shot learning work can works for classification and regression.