Meta-free few-shot learning via representation learning with weight averaging

Recent studies on few-shot classification using transfer learning pose challenges to the effectiveness and efficiency of episodic meta-learning algorithms . Transfer learning approaches are a natural alternative , but they are restricted to few-shot classification . Moreover , little attention has been on the development of probabilistic models with well-calibrated uncertainty from few-shot samples , except for some Bayesian episodic learning algorithms . To tackle the aforementioned issues , we propose a new transfer learning method to obtain accurate and reliable models for few-shot regression and classification . The resulting method does not require episodic meta-learning and is called meta-free representation learning ( MFRL ) . MFRL first finds low-rank representation generalizing well on meta-test tasks . Given the learned representation , probabilistic linear models are fine-tuned with few-shot samples to obtain models with well-calibrated uncertainty . The proposed method not only achieves the highest accuracy on a wide range of few-shot learning benchmark datasets but also correctly quantifies the prediction uncertainty . In addition , weight averaging and temperature scaling are effective in improving the accuracy and reliability of few-shot learning in existing meta-learning algorithms with a wide range of learning paradigms and model architectures . 1 INTRODUCTION . Currently , the vast majority of few-shot learning methods are within the general paradigm of metalearning ( a.k.a . learning to learn ) ( Bengio et al. , 1991 ; Schmidhuber , 1987 ; Thrun & Pratt , 1998 ) , which learns multiple tasks in an episodic manner to distill transferrable knowledge ( Vinyals et al. , 2016 ; Finn et al. , 2017 ; Snell et al. , 2017 ) . Although many episodic meta-learning methods report state-of-the-art ( SOTA ) performance , recent studies show that simple transfer learning methods with fixed embeddings ( Chen et al. , 2019 ; Tian et al. , 2020 ) can achieve similar or better performance in few-shot learning . It is found that the effectiveness of optimization-based meta-learning algorithms is due to reusing high-quality representation , instead of rapid learning of task-specific representation ( Raghu et al. , 2020 ) . The quality of the presentation is not quantitatively defined , except for some empirical case studies ( Goldblum et al. , 2020 ) . Recent machine learning theories ( Saunshi et al. , 2021 ) indicate that low-rank representation leads to better sample efficiency in learning a new task . However , those theoretical studies are within the paradigm of meta-learning and do not reveal how to obtain low-rank representation for few-shot learning outside the realm of meta-learning . This motivates us to investigate ways to improve the representation for adapting to new few-shot tasks in a meta-free manner by taking the advantage of simplicity and robustness in transfer learning . In parallel , existing transfer learning methods also have limitations . That is , the existing transfer learning methods may not find representation generalizing well to unseen few-shot tasks ( Chen et al. , 2019 ; Dhillon et al. , 2020 ) , compared with state-of-the-art meta-learning methods ( Ye et al. , 2020 ; Zhang et al. , 2020 ) . Although some transfer learning methods utilize knowledge distillation and self-supervised training to achieve strong performance in few-shot classification , they are restricted to few-shot classification problems ( Mangla et al. , 2020 ; Tian et al. , 2020 ) . To the best of our knowledge , no transfer learning method is developed to achieve similar performance to metalearning in few-shot regression . As such , it is desirable to have a transfer learning method that finds high-quality representation generalizing well to unseen classification and regression problems . The last limitation of the existing transfer learning methods in few-shot learning is the lack of uncertainty calibration . Uncertainty quantification is concerned with the quantification of how likely certain outcomes are . Despite a plethora of few-shot learning methods ( in fact , machine learning in general ) to improve the point estimation accuracy , few methods are developed to get probabilistic models with improved uncertainty calibration by integrating Bayesian learning into episodic metatraining ( Grant et al. , 2018 ; Finn et al. , 2018 ; Yoon et al. , 2018 ; Snell & Zemel , 2021 ) . Few-shot learning models can be used in risk-averse applications such as medical diagnosis ( Prabhu et al. , 2019 ) . The diagnosis decision is made on not only point estimation but also probabilities associated with the prediction . The risk of making wrong decisions is significant when using uncalibrated models ( Begoli et al. , 2019 ) . Thus , the development of proper fine-tuning steps to achieve well-calibrated models is the key towards practical applications of transfer learning in few-shot learning . In this paper , we develop a simple transfer learning method as our own baseline to allow easy regularization towards more generalizable representation and calibration of prediction uncertainty . The regularization in the proposed transfer learning method works for regression and classification problems so that we can handle both problems within a common architecture . The calibration procedure is easily integrated into the developed transfer learning method to obtain few-shot learning models with good uncertainty quantification . Therefore , the resulting method , called Meta-Free Representation Learning ( MFRL ) , overcomes the aforementioned limitations in existing transfer learning methods for few-shot learning . Our empirical studies demonstrate that the relatively overlooked transfer learning method can achieve high accuracy and well-calibrated uncertainty in few-shot learning when it is combined with the proper regularization and calibration . Those two tools are also portable to meta-learning methods to improve accuracy and calibration , but the improvement is less significant compared with that of transfer learning . We use stochastic weight averaging ( SWA ) ( Izmailov et al. , 2018 ) , which is agnostic to loss function types , as implicit regularization to improve the generalization capability of the representation . We also shed light on that the effectiveness of SWA is due to its bias towards low-rank representation . To address the issue of uncertainty quantification , we fine-tune appropriate linear layers during the meta-test phase to get models with well-calibrated uncertainty . In MFRL , hierarchical Bayesian linear models are used to properly capture the uncertainty from very limited training samples in few-shot regression , whereas the softmax output is scaled with a temperature parameter to make the few-shot classification model well-calibrated . Our method is the first one to achieve well-calibrated few-shot models by only fine-tuning probabilistic linear models in the meta-test phase , without any learning mechanisms related to the meta-training or representation learning phase . Our contributions in this work are summarized as follows : • We propose a transfer learning method that can handle both few-shot regression and classification problems with performance exceeding SOTA . • For the first time , we empirically find the implicit regularization of SWA towards low-rank representation , which is a useful property in transferring to few-shot tasks . • The proposed method results in well-calibrated uncertainty in few-shot learning models while preserving SOTA accuracy . • The implicit regularization of SWA and temperature scaling factor can be applied to existing meta-learning methods to improve their accuracy and reliability in few-shot learning . 2 RELATED WORK . Episodic meta-learning approaches can be categorized into metric-based and optimization-based methods . Metric-based methods project input data to feature vectors through nonlinear embeddings and compare their similarity to make the prediction . Examples of similarity metrics include the weighted L1 metric ( Koch et al. , 2015 ) , cosine similarity ( Qi et al. , 2018 ; Vinyals et al. , 2016 ) , and Euclidean distance to class-mean representation ( Snell et al. , 2017 ) . Instead of relying on predefined metrics , learnable similarity metrics are introduced to improve the few-shot classification performance ( Oreshkin et al. , 2018 ; Sung et al. , 2018 ) . Recent metric-based approaches focus on developing task-adaptive embeddings to improve few-shot classification accuracy . Those task-adaptive embeddings include attention mechanisms for feature transformation ( Fei et al. , 2021 ; Gidaris & Komodakis , 2018 ; Ye et al. , 2020 ; Zhang et al. , 2021 ) , graph neural networks ( Garcia & Estrach , 2018 ) , implicit class representation ( Ravichandran et al. , 2019 ) , and task-dependent conditioning ( Oreshkin et al. , 2018 ; Yoon et al. , 2020 ; 2019 ) . Although metric-based approaches achieve strong performance in few-shot classification , they can not be directly applied to regression problems . Optimization-based meta-learning approaches try to find transferrable knowledge and adapt to new tasks quickly . An elegant and powerful meta-learning approach , termed model-agnostic metalearning ( MAML ) , solves a bi-level optimization problem to find good initialization of model parameters ( Finn et al. , 2017 ) . However , MAML has a variety of issues , such as sensitivity to neural network architectures , instability during training , arduous hyperparameter tuning , and high computational cost . On this basis , some follow-up methods have been developed to simplify , stabilize and improve the training process of MAML ( Antoniou et al. , 2018 ; Flennerhag et al. , 2020 ; Lee & Choi , 2018 ; Nichol et al. , 2018 ; Park & Oliva , 2019 ) . In practice , it is very challenging to learn high-dimensional model parameters in a low-data regime . Latent embedding optimization ( LEO ) attempts to learn low-dimensional representation to generate high-dimensional model parameters ( Rusu et al. , 2019 ) . Meanwhile , R2-D2 ( Bertinetto et al. , 2019 ) and MetaOptNet ( Lee et al. , 2019 ) reduce the dimensionality of trainable model parameters by freezing feature extraction layers during inner loop optimization . Note that the proposed method is fundamentally different from R2-D2 and MetaOptNet because our method requires neither episodic meta-learning nor bi-level optimization . Transfer learning approaches first learn a feature extractor on all training data through standard supervised learning , and then fine-tune a linear predictor on top of the learned feature extractor in a new task ( Chen et al. , 2019 ) . However , vanilla transfer learning methods for few-shot learning do not take extra steps to make the learned representation generalizing well to unseen meta-test tasks . Some approaches in this paradigm are developed to improve the quality of representation and boost the accuracy of few-shot classification , including cooperative ensembles ( Dvornik et al. , 2019 ) , knowledge distillation ( Tian et al. , 2020 ) , and auxiliary self-supervised learning ( Mangla et al. , 2020 ) . Nevertheless , those transfer learning methods are restricted to few-shot classification . MFRL aims to find representation generalizing well from the perspective of low-rank representation learning , which is supported by recent theoretical studies ( Saunshi et al. , 2021 ) . Furthermore , MFLR is the first transfer learning method that can handle both few-shot regression and classification problems and make predictions with well-calibrated uncertainty . 3 BACKGROUND . 3.1 EPISODIC META-LEARNING . In episodic meta-learning , the meta-training data contains T episodes or tasks , where the τ th episode consists of dataDτ = { ( xτ , j , yτ , j ) } Nτj=1 withNτ samples . Tasks and episodes are used interchangeably in the rest of the paper . Episodic meta-learning algorithms aim to find common model parameters θ which can be quickly adapted to task-specific parameters φτ ( τ = 1 , ... , T ) . For example , MAML-type algorithms assume φτ is one or a few gradient steps away from θ ( Finn et al. , 2017 ; 2018 ; Grant et al. , 2018 ; Yoon et al. , 2018 ) , while other meta-learning approaches assume that φτ and θ share the parameters in the feature extractor and only differ in the top layer ( Bertinetto et al. , 2019 ; Lee et al. , 2019 ; Snell et al. , 2017 ) .

Summary: This paper proposes a transfer learning method for few-shot regression/classification. In representation learning, it adapts the stochastic weight averaging (SWA) as a regularizer for learning more generalizable features. In fine-tuning phase (adaption to a new task in the meta-test phase), it treats the regression and classification differently. For regression a hierarchical Bayse is used while in classification, a conventional method is merged with temperature scaling. This work achieves comparable results to SOTA, however, they are major concerns that need to be addressed.