Task Affinity with Maximum Bipartite Matching in Few-Shot Learning

1 INTRODUCTION . Leveraging the knowledge of one task in training the other related tasks is an effective approach to training deep neural networks with limited data . In fact , transfer learning , multi-task learning ( Standley et al. , 2020 ) , and meta-learning ( Finn et al. , 2017 ) are examples of training a new task using the knowledge of others . In fact , a strong piece of work ( Standley et al. , 2020 ) has shown that training similar tasks together in multi-task learning often achieves higher accuracy on average . However , characterizing the similarity between tasks remains a challenging problem . In this paper , we present a task similarity measure representing the complexity of utilizing the knowledge of one task for learning another one . Our measure , called Task Affinity Score ( TAS ) , is non-commutative and is defined as a function of the Fisher Information matrix , which is based on the second-derivative of the loss function with respect to the parameters of the model under consideration . By definition , the TAS between two tasks is always greater or equal to 0 , where the equality holds if and only if both tasks are identical . For the classification tasks , the TAS is invariant to the permutation of the data labels . In other words , modifying the numeric order of the data labels does not affect the affinity score between tasks . Additionally , TAS is mathematically well-defined , as we will prove in the sequel . Following the introduction of TAS , we propose a few-shot learning method based on the similarity between tasks . The lack of sufficient data in the few-shot learning problem has motivated us to use the knowledge of similar tasks for our few-shot learning method . In particular , our approach is capable of finding the relevant training labels to the ones in the given few-shot target tasks , and utilizing the corresponding data samples for episodically fine-tuning the few-shot model . Similar to recent few-shot approaches ( Chen et al. , 2021 ; Tian et al. , 2020 ) , we first use the entire training dataset to train a Whole-Classification network . Next , this trained model is used for extraction of the feature vectors for a set of constructed source task ( s ) generated from the training dataset . The purpose of the sources task ( s ) is to establish the most related task ( s ) to the target task defined according to the test data . In our framework , TAS with a graph matching algorithm is applied to find the affinity scores and the identification of the most related source task ( s ) to the target task . Lastly , we follow the standard few-shot meta-learning in which a set of base tasks are first constructed , and a few-shot model is fine-tuned according to the query set of these base tasks . Our approach has a unique distinguishing property from the common meta-learning approaches : our base tasks are constructed only based on the previously discovered related source tasks to episodically finetune the few-shot model . Specifically , the feature vectors of the query data from the base tasks are extracted by the encoder of the Whole-Classification network , and a k-nearest neighbors ( k-NN ) is applied to classify the features into the correct classes by updating the weights in the encoder . Using extensive simulations , we demonstrate that our approach of utilizing only the related training data is an effective method for boosting the performance of the few-shot model with less number of parameters in both 5-way 1-shot and 5-way 5-shot settings for various benchmark datasets . Experimental results on miniImageNet ( Vinyals et al. , 2016 ) , tieredImageNet ( Ren et al. , 2018 ) , CIFARFS ( Bertinetto et al. , 2018 ) , and FC-100 ( Oreshkin et al. , 2018 ) datasets are provided demonstrating the efficacy of the proposed approach compared to other state-of-the-art few-shot learning methods . 2 RELATED WORK . The similarity between tasks has been mainly studied in the transfer learning literature . Many approaches in transfer learning ( Silver & Bennett , 2008 ; Finn et al. , 2016 ; Mihalkova et al. , 2007 ; Niculescu-Mizil & Caruana , 2007 ; Luo et al. , 2017 ; Razavian et al. , 2014 ; Pan & Yang , 2010 ; Mallya & Lazebnik , 2018 ; Fernando et al. , 2017 ; Rusu et al. , 2016 ; Zamir et al. , 2018 ; Kirkpatrick et al. , 2017 ; Chen et al. , 2018 ) are based on the assumption that similar tasks often share similar architectures . However , these works mainly focus on transferring the trained weights from the previous tasks to an incoming task , and do not seek to define the measurement that can identify the related tasks . Though the relationship between visual tasks has been recently investigated by various papers ( Zamir et al. , 2018 ; Pal & Balasubramanian , 2019 ; Dwivedi & Roig , 2019 ; Achille et al. , 2019 ; Wang et al. , 2019 ; Standley et al. , 2020 ) , these works only focus on the weight-transferring and do not use task similarity for discovering the closest tasks for improving the overall performance . Additionally , the measures of task similarity from these papers are often assumed to be symmetric , which is not typically a realistic assumption . For example , it is easier to utilize the knowledge of a comprehensive task for learning a simpler task than the other way around . In the context of the few-shot learning ( FSL ) , the task affinity ( similarity ) has not been explicitly considered . Most of the recent few-shot learning approaches are based on the meta-learning frameworks ( Santoro et al. , 2016 ; Finn et al. , 2017 ; Vinyals et al. , 2016 ; Snell et al. , 2017 ) . In these approaches , episodic learning is often used in the training phase , in which the FSL models are exposed to data episodes . Each episode , consisting of the support and query sets , is characterized by the number of classes , the number of samples per class in the support set , and the number of samples per class in the query set . During the training phase , the loss over these training episodes is minimized . Generally , these episodic learning approaches can be divided into three main categories : metric-based method , optimization-based method , and memory-based method . In metricbased methods ( Vinyals et al. , 2016 ; Snell et al. , 2017 ; Koch et al. , 2015 ; Sung et al. , 2018 ) , a kernel function learns to measure the distance between data samples in the support sets , then classifies the data in the query set according to the closest data samples in the support set . On the other hand , the goal of optimization-based methods ( Finn et al. , 2017 ; Grant et al. , 2018 ; Rusu et al. , 2018 ; Lee et al. , 2019 ; Nichol et al. , 2018 ) is to find the models with faster adaption and convergence . Lastly , the memory-based methods ( Santoro et al. , 2016 ; Ravi & Larochelle , 2017 ; Munkhdalai et al. , 2018 ) use the network architectures with memory as the meta-learner for the few-shot learning . Overall , episodic learning in FSL has achieved great success on various few-shot meta-datasets . Recently , several methods with a pre-trained Whole-Classification network have achieved state-ofthe-art performance on multiple FSL benchmarks ( Chen et al. , 2021 ; Tian et al. , 2020 ; Rodrı́guez et al. , 2020 ; Rizve et al. , 2021 ) . Instead of initializing the FSL model from scratches , these methods focus on leveraging the entire training labels for pre-training a powerful and robust classifier . The pre-trained model , by itself , outperforms several meta-learning approaches in numerous FSL datasets ( e.g. , miniImageNet , tieredImageNet , CIFAR-FS , etc. ) . Next , the Whole-Classification network is used as a feature extractor for a simple base learner ( e.g. , logistic regression , K-nearest neighbor , etc . ) and is often fine-tuned using episodic learning . Various efforts have been investigated to improve the Whole-Classification network for the few-shot learning , including manifold mixup as self-supervised loss ( Mangla et al. , 2020 ) , knowledge distillation on the pre-trained classifier ( Verma et al. , 2019 ; Tian et al. , 2020 ) , and data-augmentation with combined loss functions ( Rodrı́guez et al. , 2020 ; Rizve et al. , 2021 ) . However , none of these approaches consider the task affinity in their training procedure . Here , we propose a task affinity measure that can identify the related tasks to a target task , given only a few data samples . Then we utilize the data samples in the related tasks for episodically fine-tuning the final few-shot classifier . 3 PRELIMINARIES . In this section , we present the definition of the task affinity score . First , we need to define some notations and definitions used throughout this paper . We denote the matrix infinity-norm by ||B||∞ = maxi , j |Bij | . We also denote a task T and its dataset X jointly by a pair ( T , X ) . To be consistent with the few-shot terminologies , a dataset X is shown by the union of the support set , Xsupport and the query set , Xquery , i.e. , X = Xsupport ∪ Xquery . Let PNθ ( T , Xquery ) ∈ [ 0 , 1 ] be a function that measures the performance of a given model Nθ , parameterized by θ ∈ Rd on the query setXquery of the task T . We define an ε-approximation network , representing the task-dataset pair ( T , X ) as follows : Definition 1 ( ε-approximation Network ) . A model Nθ is called an ε-approximation network for a pair task-dataset ( T , X ) if it is trained using the support dataXsupport such thatPNθ ( T , Xquery ) ≥ 1− ε , for a given 0 < ε < 1 . In practice , the architectures for the ε-approximation networks for a given task T are selected from a pool of well-known hand-designed architectures , such as ResNet , VGG , DenseNet , etc . We also need to recall the definition of the Fisher Information matrix for a neural network . Definition 2 ( Fisher Information Matrix ) . For a neural network Nθ with weights θ , data X , and the negative log-likelihood loss function L ( θ ) : = L ( θ , X ) , the Fisher Information matrix is defined as : F ( θ ) = E [ ∇θL ( θ ) ∇θL ( θ ) T ] = −E [ H ( L ( θ ) ) ] , ( 1 ) where H is the Hessian matrix , i.e. , H ( L ( θ ) ) = ∇2θL ( θ ) , and expectation is taken w.r.t the data . In practice , we use the empirical Fisher Information matrix computed as follows : F̂ ( θ ) = 1 |X| ∑ i∈X ∇θLi ( θ ) ∇θLi ( θ ) T , ( 2 ) where Li ( θ ) is the loss value for the ith data point in X . Next , we define the task affinity score , which measures the similarity from a source task , Ta to a target task , Tb . Definition 3 ( Task Affinity Score ( TAS ) ) . Let ( Ta , Xa ) be the source task-dataset pair with Nθa denotes its corresponding ε-approximation network . Let Fa , a be the Fisher Information matrix of Nθa with the query data X query a from the task Ta . For the target task-dataset pair ( Tb , Xb ) , let Fa , b be the Fisher Information matrix of Nθa with the support data X support b from the task Tb . We define the TAS from the source task Ta to the target task Tb based on Fréchet distance as follows : s [ a , b ] : = 1√ 2 Trace ( Fa , a + Fa , b − 2 ( Fa , aFa , b ) 1/2 ) 1/2 . ( 3 ) Here , we use the diagonal approximation of the Fisher Information matrix since computing the full Fisher matrix is prohibitive in the huge space of neural network parameters . We also normalize these matrices to have unit trace . As a result , the TAS in equation ( 3 ) can be simplified by the following formula : s [ a , b ] = 1√ 2 ∥∥∥F 1/2a , a − F 1/2a , b ∥∥∥ F = 1√ 2 [ ∑ i ( ( F iia , a ) 1/2 − ( F iia , b ) 1/2 ) 2 ] 1/2 , ( 4 ) where F ii denotes the ith diagonal entry of the Fisher Information matrix . The TAS ranges from 0 to 1 , with the score s = 0 denotes a perfect similarity and the score s = 1 indicates a perfect dissimilarity . In the next section , we present our few-shot approach based on the above TAS .

Summary: This paper presents a new affinity score based on the Fisher Information matrix from a source to a target task. The authors also develop a few-shot learning procedure based on a pre-trained Whole-Classification network approach. In this procedure, training labels are matched via a maximum matching algorithm, and the target task for training is selected using the proposed affinity score. The effectiveness of the proposed score is demonstrated using benchmark datasets in the problem of few-shot learning.