Meta-free few-shot learning via representation learning with weight averaging

Recent studies on few-shot classification using transfer learning pose challenges to the effectiveness and efficiency of episodic meta-learning algorithms . Transfer learning approaches are a natural alternative , but they are restricted to few-shot classification . Moreover , little attention has been on the development of probabilistic models with well-calibrated uncertainty from few-shot samples , except for some Bayesian episodic learning algorithms . To tackle the aforementioned issues , we propose a new transfer learning method to obtain accurate and reliable models for few-shot regression and classification . The resulting method does not require episodic meta-learning and is called meta-free representation learning ( MFRL ) . MFRL first finds low-rank representation generalizing well on meta-test tasks . Given the learned representation , probabilistic linear models are fine-tuned with few-shot samples to obtain models with well-calibrated uncertainty . The proposed method not only achieves the highest accuracy on a wide range of few-shot learning benchmark datasets but also correctly quantifies the prediction uncertainty . In addition , weight averaging and temperature scaling are effective in improving the accuracy and reliability of few-shot learning in existing meta-learning algorithms with a wide range of learning paradigms and model architectures . 1 INTRODUCTION . Currently , the vast majority of few-shot learning methods are within the general paradigm of metalearning ( a.k.a . learning to learn ) ( Bengio et al. , 1991 ; Schmidhuber , 1987 ; Thrun & Pratt , 1998 ) , which learns multiple tasks in an episodic manner to distill transferrable knowledge ( Vinyals et al. , 2016 ; Finn et al. , 2017 ; Snell et al. , 2017 ) . Although many episodic meta-learning methods report state-of-the-art ( SOTA ) performance , recent studies show that simple transfer learning methods with fixed embeddings ( Chen et al. , 2019 ; Tian et al. , 2020 ) can achieve similar or better performance in few-shot learning . It is found that the effectiveness of optimization-based meta-learning algorithms is due to reusing high-quality representation , instead of rapid learning of task-specific representation ( Raghu et al. , 2020 ) . The quality of the presentation is not quantitatively defined , except for some empirical case studies ( Goldblum et al. , 2020 ) . Recent machine learning theories ( Saunshi et al. , 2021 ) indicate that low-rank representation leads to better sample efficiency in learning a new task . However , those theoretical studies are within the paradigm of meta-learning and do not reveal how to obtain low-rank representation for few-shot learning outside the realm of meta-learning . This motivates us to investigate ways to improve the representation for adapting to new few-shot tasks in a meta-free manner by taking the advantage of simplicity and robustness in transfer learning . In parallel , existing transfer learning methods also have limitations . That is , the existing transfer learning methods may not find representation generalizing well to unseen few-shot tasks ( Chen et al. , 2019 ; Dhillon et al. , 2020 ) , compared with state-of-the-art meta-learning methods ( Ye et al. , 2020 ; Zhang et al. , 2020 ) . Although some transfer learning methods utilize knowledge distillation and self-supervised training to achieve strong performance in few-shot classification , they are restricted to few-shot classification problems ( Mangla et al. , 2020 ; Tian et al. , 2020 ) . To the best of our knowledge , no transfer learning method is developed to achieve similar performance to metalearning in few-shot regression . As such , it is desirable to have a transfer learning method that finds high-quality representation generalizing well to unseen classification and regression problems . The last limitation of the existing transfer learning methods in few-shot learning is the lack of uncertainty calibration . Uncertainty quantification is concerned with the quantification of how likely certain outcomes are . Despite a plethora of few-shot learning methods ( in fact , machine learning in general ) to improve the point estimation accuracy , few methods are developed to get probabilistic models with improved uncertainty calibration by integrating Bayesian learning into episodic metatraining ( Grant et al. , 2018 ; Finn et al. , 2018 ; Yoon et al. , 2018 ; Snell & Zemel , 2021 ) . Few-shot learning models can be used in risk-averse applications such as medical diagnosis ( Prabhu et al. , 2019 ) . The diagnosis decision is made on not only point estimation but also probabilities associated with the prediction . The risk of making wrong decisions is significant when using uncalibrated models ( Begoli et al. , 2019 ) . Thus , the development of proper fine-tuning steps to achieve well-calibrated models is the key towards practical applications of transfer learning in few-shot learning . In this paper , we develop a simple transfer learning method as our own baseline to allow easy regularization towards more generalizable representation and calibration of prediction uncertainty . The regularization in the proposed transfer learning method works for regression and classification problems so that we can handle both problems within a common architecture . The calibration procedure is easily integrated into the developed transfer learning method to obtain few-shot learning models with good uncertainty quantification . Therefore , the resulting method , called Meta-Free Representation Learning ( MFRL ) , overcomes the aforementioned limitations in existing transfer learning methods for few-shot learning . Our empirical studies demonstrate that the relatively overlooked transfer learning method can achieve high accuracy and well-calibrated uncertainty in few-shot learning when it is combined with the proper regularization and calibration . Those two tools are also portable to meta-learning methods to improve accuracy and calibration , but the improvement is less significant compared with that of transfer learning . We use stochastic weight averaging ( SWA ) ( Izmailov et al. , 2018 ) , which is agnostic to loss function types , as implicit regularization to improve the generalization capability of the representation . We also shed light on that the effectiveness of SWA is due to its bias towards low-rank representation . To address the issue of uncertainty quantification , we fine-tune appropriate linear layers during the meta-test phase to get models with well-calibrated uncertainty . In MFRL , hierarchical Bayesian linear models are used to properly capture the uncertainty from very limited training samples in few-shot regression , whereas the softmax output is scaled with a temperature parameter to make the few-shot classification model well-calibrated . Our method is the first one to achieve well-calibrated few-shot models by only fine-tuning probabilistic linear models in the meta-test phase , without any learning mechanisms related to the meta-training or representation learning phase . Our contributions in this work are summarized as follows : • We propose a transfer learning method that can handle both few-shot regression and classification problems with performance exceeding SOTA . • For the first time , we empirically find the implicit regularization of SWA towards low-rank representation , which is a useful property in transferring to few-shot tasks . • The proposed method results in well-calibrated uncertainty in few-shot learning models while preserving SOTA accuracy . • The implicit regularization of SWA and temperature scaling factor can be applied to existing meta-learning methods to improve their accuracy and reliability in few-shot learning . 2 RELATED WORK . Episodic meta-learning approaches can be categorized into metric-based and optimization-based methods . Metric-based methods project input data to feature vectors through nonlinear embeddings and compare their similarity to make the prediction . Examples of similarity metrics include the weighted L1 metric ( Koch et al. , 2015 ) , cosine similarity ( Qi et al. , 2018 ; Vinyals et al. , 2016 ) , and Euclidean distance to class-mean representation ( Snell et al. , 2017 ) . Instead of relying on predefined metrics , learnable similarity metrics are introduced to improve the few-shot classification performance ( Oreshkin et al. , 2018 ; Sung et al. , 2018 ) . Recent metric-based approaches focus on developing task-adaptive embeddings to improve few-shot classification accuracy . Those task-adaptive embeddings include attention mechanisms for feature transformation ( Fei et al. , 2021 ; Gidaris & Komodakis , 2018 ; Ye et al. , 2020 ; Zhang et al. , 2021 ) , graph neural networks ( Garcia & Estrach , 2018 ) , implicit class representation ( Ravichandran et al. , 2019 ) , and task-dependent conditioning ( Oreshkin et al. , 2018 ; Yoon et al. , 2020 ; 2019 ) . Although metric-based approaches achieve strong performance in few-shot classification , they can not be directly applied to regression problems . Optimization-based meta-learning approaches try to find transferrable knowledge and adapt to new tasks quickly . An elegant and powerful meta-learning approach , termed model-agnostic metalearning ( MAML ) , solves a bi-level optimization problem to find good initialization of model parameters ( Finn et al. , 2017 ) . However , MAML has a variety of issues , such as sensitivity to neural network architectures , instability during training , arduous hyperparameter tuning , and high computational cost . On this basis , some follow-up methods have been developed to simplify , stabilize and improve the training process of MAML ( Antoniou et al. , 2018 ; Flennerhag et al. , 2020 ; Lee & Choi , 2018 ; Nichol et al. , 2018 ; Park & Oliva , 2019 ) . In practice , it is very challenging to learn high-dimensional model parameters in a low-data regime . Latent embedding optimization ( LEO ) attempts to learn low-dimensional representation to generate high-dimensional model parameters ( Rusu et al. , 2019 ) . Meanwhile , R2-D2 ( Bertinetto et al. , 2019 ) and MetaOptNet ( Lee et al. , 2019 ) reduce the dimensionality of trainable model parameters by freezing feature extraction layers during inner loop optimization . Note that the proposed method is fundamentally different from R2-D2 and MetaOptNet because our method requires neither episodic meta-learning nor bi-level optimization . Transfer learning approaches first learn a feature extractor on all training data through standard supervised learning , and then fine-tune a linear predictor on top of the learned feature extractor in a new task ( Chen et al. , 2019 ) . However , vanilla transfer learning methods for few-shot learning do not take extra steps to make the learned representation generalizing well to unseen meta-test tasks . Some approaches in this paradigm are developed to improve the quality of representation and boost the accuracy of few-shot classification , including cooperative ensembles ( Dvornik et al. , 2019 ) , knowledge distillation ( Tian et al. , 2020 ) , and auxiliary self-supervised learning ( Mangla et al. , 2020 ) . Nevertheless , those transfer learning methods are restricted to few-shot classification . MFRL aims to find representation generalizing well from the perspective of low-rank representation learning , which is supported by recent theoretical studies ( Saunshi et al. , 2021 ) . Furthermore , MFLR is the first transfer learning method that can handle both few-shot regression and classification problems and make predictions with well-calibrated uncertainty . 3 BACKGROUND . 3.1 EPISODIC META-LEARNING . In episodic meta-learning , the meta-training data contains T episodes or tasks , where the τ th episode consists of dataDτ = { ( xτ , j , yτ , j ) } Nτj=1 withNτ samples . Tasks and episodes are used interchangeably in the rest of the paper . Episodic meta-learning algorithms aim to find common model parameters θ which can be quickly adapted to task-specific parameters φτ ( τ = 1 , ... , T ) . For example , MAML-type algorithms assume φτ is one or a few gradient steps away from θ ( Finn et al. , 2017 ; 2018 ; Grant et al. , 2018 ; Yoon et al. , 2018 ) , while other meta-learning approaches assume that φτ and θ share the parameters in the feature extractor and only differ in the top layer ( Bertinetto et al. , 2019 ; Lee et al. , 2019 ; Snell et al. , 2017 ) .

Summary: The submission introduces a few-shot classification and regression approach called Meta-Free Representation Learning (MFRL). First, a representation is learned on the meta-training data: for regression, the model is trained on all available training regression tasks concurrently; for classification, the model is trained on the full-ways classification problem using all training classes. Then, stochastic weight averaging (SWA) is applied to the model by continuing training for a certain number of epochs and averaging the parameters obtained across those additional epochs.  For test tasks, the regression or classification layer is discarded, and a new output layer is trained while freezing the network weights. For regression, approximate inference is performed on a hierarchical Bayesian linear model. For classification, a logistic regression model is trained with L2 regularization and a temperature hyperparameter.  Experimental results are presented on the sine wave and head pose problems for regression, and on mini-ImageNet, tiered-ImageNet, CIFAR-FS, and FC100 for classification. The extent to which SWA encourages learning lower-rank representations (as hypothesized) is verified through visualizations of the normalized eigenvalues. Finally, calibration curves are shown for classification to demonstrate how MFRL's temperature scaling factor leads to better calibrated models.