Task Affinity with Maximum Bipartite Matching in Few-Shot Learning
Authors: Cat Phuoc Le
Juncheng Dong
Mohammadreza Soltani
Vahid Tarokh
Venue: ICLR 2022 Poster (2022)
Keywords: Task Affinity
Transfer Learning
Few-Shot Learning

Abstract: We propose an asymmetric affinity score for representing the complexity of utilizing the knowledge of one task for learning another one. Our method is based on the maximum bipartite matching algorithm and utilizes the Fisher Information matrix. We provide theoretical analyses demonstrating that the proposed score is mathematically well-defined, and subsequently use the affinity score to propose a novel algorithm for the few-shot learning problem. In particular, using this score, we find relevant training data labels to the test data and leverage the discovered relevant data for episodically fine-tuning a few-shot model. Results on various few-shot benchmark datasets demonstrate the efficacy of the proposed approach by improving the classification accuracy over the state-of-the-art methods even when using smaller models.

Published as a conference paper at ICLR 2022
TASK AFFINITY WITH MAXIMUM BIPARTITE MATCH- ING IN FEW-SHOT LEARNING
Cat P. Le, Juncheng Dong, Mohammadreza Soltani, Vahid Tarokh Department of Electrical and Computer Engineering, Duke University
ABSTRACT
We propose an asymmetric affinity score for representing the complexity of uti- lizing the knowledge of one task for learning another one. Our method is based on the maximum bipartite matching algorithm and utilizes the Fisher Information matrix. We provide theoretical analyses demonstrating that the proposed score is mathematically well-defined, and subsequently use the affinity score to propose a novel algorithm for the few-shot learning problem. In particular, using this score, we find relevant training data labels to the test data and leverage the discovered relevant data for episodically fine-tuning few-shot model. Results various few-shot benchmark datasets demonstrate the efficacy of the proposed approach by improving the classification accuracy over the state-of tftte-of-the-eee thet methods even when using smaller models.
INTRODUCTION
Leveraging the knowledge of one task in training the other related tasks is an effective approach to training deep neural networks with limited data. In fact, transfer learning, multi-task learning (Stan- deey et al. 2020), and meta-learning (Finn et al.] 2017) are examples of training new task using the knowledge of others. In fact, strong piece of work (Standley et al., 2020) has shown that training similar tasks together in multi-task learning often achieves higher accuracy on average. However, characterizing the similarity between tasks remains challenging problem. In this paper, we present a task similarity measure representing the complexity of utilizing the knowledge of one task for learning another one. Our measure, called Task Affinity Score (TAS), non-commutative defined as a function of the Fisher Information matrix, which is based on the con-commutative and is of the loss function with respect to the parameters of the model under consideration. By definition, the TAS between two tasks is always greater or equal to O, where the equality holds if and only if both tasks are identical. For the classification tasks, the TAS is invariant to the permutation of the data la- bels. In other words, modifying the numeric order of the data labels does not affect the affinity score between tasks. Additionally, TAS is mathematically well-defined, as we will prove in the sequel.
Following the introduction of TAS, we propose few-shot learning method based on the similarity between tasks. The lack of sufficient data in the few-shot learning problem has motivated us to approach
use the knowledge of similar tasks for our few-shot learning method. In particular, is capable of finding the relevant training labels to the ones in the given few-shot target tasks, and utilizing the corresponding data samples for episodically fine-tuning the few-shot model. Similar to recent few-shot approaches (Chen et al., 2021 Tian et al. 2020), we first use the entire training dataset to train Choll-Classiffcation network. Next, this trained model is used for extraction of the feature vectors for a set of constructed source task(s) generated from the training dataset. The purpose of the sources task(s) is establish the most related task(s) the target task defined according to the test data. In our framework, TAS with a graph matching algorithm is applied to find the affinity scores and the identification of the most related source task(s) to the target task. Lastly, we follow the standard few-shot meta-learning in which set of base tasks are first constructed, and a few-shot model is fine-tuned according to the query set these base tasks. Our approach has unique distinguishing property from the common meta-learning approaches: our base tasks are constructed only based on the previously discovered related source tasks to episodically fine- tune the few-shot model. Specifically, the feature vectors of the query data from the base tasks network, and a k-nearest neighbors (k-NN) is
extracted by the encoder of the applied to classify the features into the correct classes by updating the weights in the encoder.

---

Published as a conference paper at ICLR 2022
Using extensive simulations, we demonstrate that our approach of utilizing only the related training data is an effective method for boosting the performance of the few-shot model with less number of parameters in both 5-way 1-shot and 5-way 5-shot settings for various benchmark datasets. Exper- imental results on minilmageNet (Vinyals et al. (016)), tiered (Ren et al. 2018), CIFAR- FS (Bertinetto al.] (018), and FC-100 (Oreshkin et al.) 2018) datasets are provided demonstrating the efficacy of the proposed approach compared to other state-of-the-an few-shot learning methods.
2 RELATED WORK
The similarity between tasks has been mainly studied in the transfer learning literature. Many ap- proaches in transfer learning (Silver & Benneti 2008) Finn et al., 2016 Mihalkova et al, [2007]
Niculescu-Mizn Mizill & Caruana 2007 Luo et al. [017 2017/ et al. 22014] Pan & Yang] 2010|]]00000a Malla
& [azebnik] [0188] Fernando et al. 2017: Rusu et 2016 Zamir et al. 2018 Kirkpatrick et al.
et al. 2018) are based the assumption that similar tasks often share similar archi- Tectures. However, these works mainly focus on transferring the trained weights from the previous tasks to an incoming task, and do not seek define the measurement that can identify the related tasks. Though the relationship between visual tasks has been recently investigated by various pa- 2019; Dwivedi & Roig 2019 Achille et al.. 2019] Wang et al.J [2019; Standley et al., (020), these works only focus on the eeighttraanferring and do not use task similarity for discovering the closest tasks for improving the overall performance. Additionally, the measures of task similarity from these papers are often assumed to be symmetric, which is not typically a realistic assumption. For example, it is easier to utilize the knowledge of a comprchensive task for learning simpler task than the other way around.
pers (Zamir al. 2018 Pal & Balasubramanian 2019;
In the context of the few-shot learning (FSL), the task affinity (similarity) has not been explicitly considered. Most of the recent few-shot learning approaches are based on the meta- meta-learning frame- works (Santoro et al., 2016, Finn et al.) 2017: Vinyals et al. 2016) [nell et al. 2017). In these approaches, episodic learning is often used in the training phase, in which the FSL models posed to data episodes. Each episode, consisting of the support and query sets, is characterized by the number of classes, the number of samples per class in the support set, and the number of samples per class in the query set. During the training phase, the loss over these training episodes minimized. Generally, these episodic learning approaches can be divided into three main cate- gories: metric-base based method, optimization-bal -based method, and memory- bbase method. In metric- based methods (Vinyals et al., [0166] (nell et al., 2017, Koch et al. 2015 Sung et al. 2018), a kernel function learns to measure the distance between data samples in the support sets, then classifies the data in the query set according to the closest data samples in the support set. On the other hand, the goal of optimization-baser methods (Finn et al.) 2017 Grant et al. 2018 Rusu et al. 2018 Lee jet al. 2019: Nichol et al., 2018) is to find the models with faster adaption and convergence, Lastly, the memory- -aased methods (Santoro et al. 2016:] Ravi & Larochelle] 2017 Munkhdalai et al. 2018)
use the network architectures with memory as the meta-lcarne for the few-shot learning. Overall, episodic learning in FSL has achieved great success various few-shot meta-datasets.
Recently, several methods with pre-trained Chose-Classifiicttinn network have achieved of the-art performance multiple FSL benchmarks (Chen et al. 202T Tian al. 2020 Rodríguez et al. 2020; Rizve et al. 2021). Instead of initializing the FSL model from scratches, these meth- ods focus on leveraging the entire training labels for pre-training a powerful and robust classifier. The pre-trained model, by itself, outperforms several meta-learning approaches in numerous FSL datasets (c.g., minilmageNet, tieredImageNet. CIFAR-FS, etc.). Next, the "hholeCllsssificttin net- work is used as a feature extractor for simple base learner (c.g., logistic regression, K-nearest neighbor, etc.) and is often fine-tuned using episodic learning. Various efforts have been investi- gated to improve the WholeCClassificttion network for the few-shot learning, including manifold mixup as self-supervised loss (Mangla et al. 2020), knowledge distillation on the pre-trained clas- sifier (Verma et al. 2019 Tian et al. 2020 and data-augmentation with combined loss functions [Rodriguez et al. 20200 Rizve et al. 2021). However, none of these approaches consider the task affinity in their training procedure. Here,
propose task affinity measure that can identify the related tasks to a target task, given only a few data samples. Then we utilize the data samples in the related tasks for episodically fine-tuning the final few-shot classifier.
2

---

Published as a conference paper at ICLR 2022
3 PRELIMINARIES
In this section, we present the definition of the task affinity score. First, we need to define some notations and definitions used throughout this paper. We denote the matrix infinity-norm by || = maxi,j Bigl- We also denote a task T and its dataset X jointly by pair X). To be consistent with the few-shot terminologies, a dataset X is shown by the union the support set, Xsupport and the query set, X i.e., X = Xsupport U X Let PN. (T, X E [o, 1] be a function that measures the performance of a given model No. parameterized by 0 E Rdd on the query set Xquery of the task T. We define e-approximation network, representing the task-dataset pair (T, X) as follows:
Definition (s-approximation Network). model Ng called an E-approximation network for a pair task-dataset (T, if is trained using the support data X support such that P Poo (T,X Xueery) 2 1 for given < <
In practice, the architectures for the s-approximation networks for given task T' are selected from a pool of well-known hand-designed architectures, such as ResNet, VGG, DenseNet, etc. We also need recall the definition of the Fisher Information matrix for neural network.
Definition 2 (Fisher Information Matrix). For neural network Na with weights 0, data X, and the negative log-likelihood loss function L(Ø) == L(B, X), the Fisher Information matrix is defined as:
F())= V VoL(O)VoL(()) =-E(H(L(0)]
where is the Hessian matrix, i.e.. H(L())) = V)L(O), and expectation is taken w.nt the data.
In practice, the empirical Fisher Information matrix computed follows:
1 Σ VeX
F())
X
where ('( is the loss value for the 7th data point in X. Next, we define the task affinity score, which measures the similarity from source task, T, target task, Tb.
Definition 3 (Task Affinity Score (TAS)). Let (Ta, Xa) be the source task-dataset pair with No. denotes its corresponding E-approximation network. Let Fa,a be the Fisher Information matrix of No, with the query data gquery from the task Ta. For the larget task-datasel pair (Tè, X1), let Fa,b be the Fisher Information matrix of No, with the support data X xeupport from the task Tb. We define the TAS from the source task Ta to the target task Ty based on Fréchet distance as follows:
s s(a, b) = Trace (Faa + Fo,b ((Fa,Fa.)) 1/2
V2
Here, we use the diagonal approximation of the Fisher Information matrix since computing the full Fisher matrix prohibitive in the huge space of neural network parameters. We also normalize these matrices to have unit trace. As a result, the TAS in equation 3 can be simplified by the following formula:
s(a, b)
p1/2 0.0
F2/2 =
1 V2
1
Σ
(Fit (Pi 1/2 (Frt) )
27 1/2
2
where pir denotes the th diagonal entry of the Fisher Information matrix. The TAS ranges from 0 to 1, with the score = denotes perfect similarity and the = indicates perfect dissimilarity. the next section, we present our few-shot approach based on the above TAS.
4 FEW-SHOT APPROACH
In the few-shot learning problem, few-shot task of M-way K-shot is defined as the classification of M classes, where each class has only K data points for learning. One common approach (some- times called meta-learning learning) to train a model for M M-way K-shot classification is to first construct some training tasks from the training dataset, such that each task has support set with M x K
3

---

Published as a conference paper at ICLR 2022
Phase
fe
source target data
fe
training
C
predictions
meon!
erooder
fully-cornede nedted
emooder
(i) Training Caassiilassiiicationn Network
(ii) Feature Extraction
Phase 2
fe'
F C
fe
aaurno
sourcs
target
controids
centroi cis
Wnrodder
Euly-connrecttd
largal deta
F Faagget
(iii) Matching Labels
(iv) Constructing e-approx. Network
(v) Information Matrix
festire vaciors
encoder
K-shot support set of bese task]
Class
k-nearest neighbors
Phase 3
fe
K-NN
labels
constructed from training set
quary sot
(M--1)
query set base task
updste
entrrpy lces
(vi) Episodic Fine-tuning
The overview of proposed few-shot learning approach. (i) Train the whole- classification network with the entire labels from the training dataset. (ii) Use the encoder fe to extract the feature vectors for each class in the source tasks and the target task, and compute their mean (or centroid). (iii) Maximum matching algorithm is applied to map the source task's s centroids the target task's centroids. (iv) Construct the e-approximation network for the source task(s), and using the modified dataset of the source task(s) to train the e-approximation network. (v) Obtain Fisher Information matrices for the source task(s) and the target task, and computing the TAS between them. The source tasks with the smallest TAS are considered related tasks. (vi) From the related-training set (consists of the data samples from the related tasks), generate few-shot M-way K-shot base tasks. Use the support and query sets of the base tasks to episodically fine-tune fo-
Figure 1:
samples and separate query of data points with the same M labels in the support set. The goal to use the training data (the support set for training and the query set for evaluating the perfor- mance of the training tasks), and the support of the test data train model achieve high performance on the query set of the test data. Recently, there is another few-shot approach called (Chen et al. 202 Tian al. 2020) is proposed use the whole training dataset for training base classifier with high performance. Here the assumption is that the training set is large dataset, which is sufficient for training a high accuracy classifier with all the labels in the training set. On the other hand, the test set, which is used to generate the few-shot test tasks, is not sufficiently large to train a classifier for the entire labels. Here, our proposed few-shot learning approach, whose pseudo-code is shown in Algorithm consists of three phases:
Whhole-Classfffce
Training Whole-Classificca and (ii) in Figure D. we construct Whole- chooll-Clssificationn network to represent the entire training set. To do so, we use the standard ResNet-12 as a common backbone used in many few-shot approaches for constructing the Whole- coole-Classification network. Next, we train this network to classify the whole classes in the training set. Additionally, the knowledge distillation technique can be applied in order to improve the classification accuracy and generalization of the extracted features (Sun et al. 2019). Next, we define a set of few-shot training tasks (i.e.souree tasks) and target task with the number of classes equals to the number of labels the test data. Using the encoder of the trained Whole- Chole-Classificatiie network, extract the embedded features and compute the mean of each label.
Network and Feature Extraction. phase 1 (steps (i)

---

Published as a conference paper at ICLR 2022
Algorithm 1: Few-Shot Learning with Task Affinity Score
Data: (Xtrains Utrain: Aerrin)) ), ( (fXtest U Xeest"), Js Itests Ntest)
Input: The mhooe-Classification network No
Output: Few-shot classifier model fa-
Function mTAS (a,, Ca, Xb, Co No.
Obtain the mapping order: mapping MaximomMatchingg(C..
Fine-tune No. using Xa with mapped labels (building the E-approximation network)
Compute Fa,a: Fa.b using Ne. with Xa. Xt. respectively
return s(a,
12/2 0,
1/2
V2
Function Main:
Train No with the entire Revain labels in Xtroen
D Beginning of phase 1
Construct S source tasks, where each has Thest labels from Xtrain
Construct target task using Reest labels in X Extract set Csouree of class centroids for each source task using the encoder fo of No
10
Extract set Ctarget of class centroids for the target task using the encoder fo of No
p Beginning of phase 2
for i 1,2, do 12
8, mTAS(X source; Ceourcer Xssupoort
Ctorgets Ng)
14
return closest tasks: i" argmin 8t
15
Define set L which consists of labels from closest tasks
Beginning of phase 3
16
Define subset data Xrelated {Xtrain Mtrain E L} Construct M-way K-shot base tasks and fine-tune fo episodically using Xrelated
17
2. Task Affinity. Using the computed mean of the feature vectors from the previous phase, phase 2 (steps (iii), (iv) and (v) in Figure identifies the closest source tasks to the target task by applying the TAS and pre-process step called matching labels.
3. Episodic Fine-tuning. After obtaining the closest source tasks, we construct training dataset called related-trainnng set, consisting of samples only from the labels in the closest source tasks. In phase 3 (step (vi) in Figure [T. follow the standard meta-learning approach by constructing the M-way K-shot base tasks from the related-training set to episodically fine-tune the classifier model. Since the classifier model is fine-tuned only with the labels of the related-training set, the complexity of training of the few-shot model is reduced; hence, the overall performance of the few-shot classification increases.
4.1 PHASE 1 TRAINING WHOLE- CLASSIFICATION NETWORK AND FEATURE EXTRACTION
Recent sho- learning approaches (c.g.. Chen et al. 12021) Tian et al. (2200) have shown advan- tages of a pre-trained "holl-Classification network on the entire training dataset. In this paper, we train a classifier network, No using the standard architecture of 1esNet-12 with the entire training classes. The Chose-Classiifiatiinn network Ng can be described as concatenation of the encoder fo and the last fully-connected layer (illustrated Figure [t)). Since No is trained training set X Xtrain= it can extract meaningful hidden features for the data samples in the training set. Next, we construct target task using Reess classes and its corresponding data points in the support set of the test data (i.c., X support. We also construct set of S few-shot source tasks using the training set, X Xtrain where each task has Reett labels sampled from Nerain classes of the training set. This results for i = 1,2....6 S. Using the encoder part of Ng denoted by fe. we extract the feature centroids from X sourcee and X support Finally, we compute the mean (or centroid) of the extracted features for every class in X source, of the source task T soorree; and similarly for X Xsupport of the target task. That is, we compute [x.1 E.ex. x. [x(x), where X, denotes the set of data samples belonging to class c in either source tasks, or the target task. In phase 2, we use the computed centroids to find the best way match the classes in source tasks to the target task.
the entire
set of source tasks with their corresponding training data, i.e., (T (Tsources X Xsource,)
test
5

---

Published as a conference paper at ICLR 2022
4.2 PHASE 2 TASK AFFINITY
In this phase, our goal is find the most related source task(s) in the training set to the few-shot target task in the test set. To do this end, we apply the proposed task affinity score. However, naively applying the TAS on the extracted centroids may result in finding a non-related source task(s) the target task. This is because neural networks are not invariant to the label's permutation (Zhang et al.) 202T]. For instance, consider task of classifying cat images as and dog images as and consider the E-approximation network No, with zero-error (i.e., E 0). Also, let task J be defined as the classification the dog images encoded as 0 and cat images as 1. Assume that the dataset used for both tasks I and J is the same. If we use No, to predict the data from task J, the prediction accuracy will be zero. For human perception, these tasks are the same. However, the neural network No, can only be good approximation network for task I, and not for task J. A naive approach to overcome this issue is to try all permutations of labels. However, it is not a practical approach, due to the intense computation requirement. As result, propose method of pre-procee neeppocesssig the labels in the source tasks to make the task affinity score invariant to the permutation of labels.
4.2.1 MAXIMUM BIPARTITE MATCHING
We use the Maximum Bipartite Graph Matching algorithm to match each set of Nssss centroids of source tasks to the ones in the target task based the Euclidean distance from each centroid to another. The goal is to find a matching order so that the total distance between all pairs of centroids to be minimized. We apply the Hungarian maximum matching algorithm (Kuhn 1955). as shown in Figure Mii.. The result of this algorithm is the way to match labels in the source tasks to the classes in the target task. After the matching, the matching result and use them to construct an e-approximation network No. for the source task, as marked (iv) in Figure [] Please note that the weights are fine-tuned using the modified matched labels; hence, we use 0' instead of 0 for the parameters of the e-approximation network.
modify the labels of the source tasks according to
4.2.2 TASK AFFINITY SCORE
Now, we can find the closest source tasks to the target task. To do so, for each source task, we compute two Fisher Information matrices (Figure [v)) by passing the source tasks' and target task's datasets through the E-approximation network No., and compute the task aftinity score defined in 4 between two tasks. Since
the matched labels of source tasks w.r.t the target task, the resulted TAS is consistent and invariant to the permutation of labels. The helper function mTAS () in the Algorithm [ implements all the steps in the second phase. Finally, the top-R scores and their corresponding classes are selected. These classes and their corresponding data points are considered as the closest source task(s) the target task. Next, show that our definition of the task affinity score is mathematically well-defined under some assumptions. First, from the equation (4. is clear that the TAS of task from itself always equals zero. Now assume that the objective function corresponding to the e-approximation network is strongly convex. In the Theorem [ we show that the TAS from task A to task B calculated using the Fisher Information matrix of the approximation network of task A on different epochs of the SGD algorithm converges to constant value given by the TAS between the above tasks when the Fisher Information matrix is computed on the global optimum of the approximation network]]
the equation
Theorem Let X A be the dataset for the task Ta with the objective function L,, and B be the dataset for the task TB. Assume XA and XB have the same distribution. Consider an s- approximation network Ng for the pair data-dataset (T),XA) with the objective functions and with the SGD algorithm to result weights O2 at time t. Assume that the function L is strongly con- vex, and its 3rd-order continuous derivative exists and bounded. Let the noisy gradient function in training No network using SGD algorithm be given by: g(Or, c1) L(()) cs, where Os is the estimation of the weights for network N at time I, and V (();) is the true gradient at Or. Assume that e, satisfies
= 0, and satisfies 8 = lim |[ [co,...,(t-]]],, < so almost 1 surely (a.s.). Then the task affinity score between TA and TB computed on the average of estimated
As mentioned in the Theorem 1, we assume that loss function of the approximation network is strongly convex; hence, it has a global optimum.

---

Published as a conference paper at ICLR 2022
weights up the current time t converges constant as t + That is,
81
V2
1
FAi 1/2
FBi"
1/2
F
D
V2
Fx
FF -F 1/2
(5)
where FA, =
FB, = F(Or, X with o, = : Moreover, F² =
F(O", X'A X and F'g F(O", X P ) denote the Fisher Information matrices computed using the optimum approximation network (i.e., a network with the global minimum weights, )*).
To prove Theorem (please see the appendix for the complete proof), we use the Polyak & Juditsky (1992) theorem for the convergence of the SGD algorithm for strongly convex functions. Although the loss function in a deep neural network is not strongly convex, establishing the fact that the TAS is mathematically well-defined for this case is an important step towards the more general deep neural networks and a justification for the success of our empirical observations. Furthermore, some recent works try to generalize Polyyk & Juditsky (1992) theorem for the convex or even some non-convex functions in an (non) -nom)-asymiootic way (Gadat & Panloup 2017). However, we do not pursue these versions here.
4.3 PHASE 3 EPISODIC FINE-TUNING
After obtaining the closest source tasks, we construct a training dataset called related-training set, consisting of samples only from the labels in the closest source tasks. Following the standard meta- learning approach, we consider various M-way K-shot base tasks from the related-training dataset, fine-tune the encoder l' from the original Whole-Classificatee precise, we extract the feature vectors from data samples in the support set and the query set of the base ts) The embedded features from the support set serve as the training data for the k-NN classifier. We utilize the k-NN model to predict the labels for the data samples in the query set of the base task under consideration using their feature vectors. Finally, the cross-entropy loss used to update the encoder fe by comparing the predicting labels and the labels of the query set, and backpropagating the error to the model fa- Then this process can be repeated for several base tasks until fo achieves high accuracy Figure I vi) shows the above process for the jth base task.
network in phase 1. To be more
5 EXPERIMENTAL STUDY
In this section, we present our experimental results in various few-shot learning benchmarks, includ- ing minilmageNet (Vinyals et al. 2016), mageedI (Ren et al., 2018), CIFAR-FS (Bertinetto) eet al. 2018), and FC-100 (Oreshkin et al. 2018) 3 The minilmageNer dataset consists of 100 classes, sampled from ImageNet (Russakovsky et al. 2015), and randomly split into 64, 16, and 20 classes for training, validation, and testing, respectively. Similarly, the tieredImageNet dataset is also a derivative of ImageNet, containing a total of 608 classes from 34 categories. It is split into 20, 6, and 8 categories (or 351, 97, and 160 classes) for training, validation, and testing, respectively. Each class, in both minilmageNet and ticredImageNet, includes 600 colorful images of size 84 x 84.
We the architecture of 1e as the Whole-Classifeee ual blocks, each block has 3 convolutional layers and a max- max-pooling layer. There are different varia- tions of ResNet-12 12 in the literature, each with different size of parameters (c.g.. [64, 128, 256, 512], [64, 160, 320, 640)). In our experiments, we report the number of parameters for all the models we use and for those ones compare with. For training the Whole-Classificatee mizer is applied with the momentum of 0.9, and the learning rate is initialized at 0.05 with the decay factor of 0.1 for all experiments. with batch size of 64, and the learning rate decaying at epoch 90. For tieredImageNet, we train for 120 epochs with a batch size of 64, and the learning rate decaying two times at epochs 40 and 80.
network, which consists of 4 resid-
network, the SGD opti-
minilmageNet experiment, we train the model for 100 epochs
For mine we define 5 = 2000 source tasks by randomly drawing samples from 20 labels (number of labels of the target task) out of 64 training labels from the whole training set. Next, use the chooc-Classification network to extract the feature vectors from S source tasks, and match them with labels of the target task. Next, we construct the e-approximation network using the
Thhe final fine-tuned few-shot model is denoted by for in the Algorithm DDue page limits, we discuss the experiments on CIFAR-FS and FC-100 in the appendix.
7

---

Published as a conference paper at ICLR 2022
Histogram minilmageNet
Histogram Releted Labels minilmageNett
0.08
0.07
0.05
Count
0.04
0.23
0.02
0.1
0.2
0.3
0.4
0.5
0.0
0.7
0.8
1.0
20
30
40
50
60
Task Affinity Score
Training Labet
(a) Task affinity scores in minilmageNe
(b) Related training classes in minilmageNee
Histogram of TAS in tieredimageNet
Histogram Related Labels in tieredImageNet
Densty
0.08
0.02
0.2
0.4
0.6
0.8
50
100
150
200
250
300
350
Task Affinity Score
Training Label
(c) Task affinity scores in ticredImageNet
(d) Related training classes in tieredImageNet
Figure 2: (a) The distribution of TAS found in minilmageNet. (b) The frequency of 64 classes in the top-8 closest source tasks in minilmageNet. (c) The distribution of TAS found in tieredImageNet. (d) The frequency of 351 classes in the top-6 closest source tasks in tieredImageNet.
encoder fe of the ChholeCClassifiattion network, and use it to compute the TAS in the task affinity phase. The distribution of the TAS found from the 2000 generated source tasks in the minilmageNet shown Figure 2a As we can seen, the probability distribution of the task affinity is not non-informative uniform distribution. Only few source tasks, with particular sets of labels, can achieve the small TAS and are considered as the related tasks. Now, we plot the frequency of all 64 labels in the training set which exist in the top-8 closest source tasks in Figure 2b This shows that class with the label 6 has samples in the top-8 closest sources task, and this class the most relevant one to the labels in the target task. Moreover, there are only 49 unique classes that construct the top-8 closest source tasks. This means that there are some classes which do not have any samples among the top-8 closest sources tasks. These labels the ones with the least similarity the labels target task. Similarly, we define 5 = 1000 source tasks tiered by randomly drawing samples from 160 labels of entire 351 training labels. The distribution of the TAS from these 1000 generated source tasks shown in Figure 2c Similar to the minilmageNet case, Figure 2d illustrates the frequency of the 351 training classes the top-6 closest source tasks. Among 35T training classes, there are only 293 unique classes that construct the top-6 closest source tasks.
In the episodic fine-tuning phase, we use the SGD optimizer with momentum 0.9, and the learning rate is set to 0.001. The batch size is set to 4, where each batch of data includes 4 few-shot tasks. The loss defined as the average loss among few-shot tasks. After we trained our few-shot model, we tested it on the query sets of test tasks. Table and Table show the performance of our approach in 5-way 1-shot and 5-way 5-shot on minilmageNet and tieredImageNet datasets, respectively. both cases, approach with the standard ResNet-12 is comparable jet al.) 2021). while outperforming RFS-distill (Tin et al. (222)). EPNet (Rodriguez et al.) 2020) and other methods with significantly smaller classifier model in terms of the number of parameters. Note that, EPNet (Rodriguez et al. 2020) and IE-distill (Rizve et al.] 202T) also utilize the data augmentation during pre-training for better feature extraction.
or better than IE-distill (Rizve)
8

---

Published as a conference paper at ICLR 2022
Table 1: Comparison of the accuracy against state-of-the methods for 5-way 1-shot and 5-way 5-shot classification with 95% confidence interval on minilmageNet dataset.
Model
Backbone
Params
1-shot
5-shot
Matching-Net Vinyals et al. 2016)
ConvNet--4
0.IIM
43.56++0.84 84
00.31
MAML (Finn et al. 2017)
ConvNet-4
0.11M
48.70+1.84
63.11+00.92
Prototypical Net (Snell et al.
2017
ConvNet-4
0.11M
49.42:t0.78
68.20
Simple CNAPS (Bateni et al.
2020
ResNet-18
11M
53. 2.090
70.8+1..0
Activation-Paramt (Qiao
2018
WRN-228-10
37.58M
59.60+40.41
73.74.10.19
LEO (uusu et al.
2018
WRN-28-10
37.58M
61.76:10.08
77.5910011
Baselne+++ (Chen al. 2019)
ResNet-11
11.17M
51.87+0077
75.68+0.66
SNAIL (Mishra et al. (017))
ResNet-12
7.99M
55.71
68.88#00.99
AdaResNet (Munkhdalar
2018
ResNet-12
7.99M
56.88 1.062
71.94.0.57
TADAM (Oreshkin et al.
2018
ResNet-12
7.99M
58. 58.5010.30
76.70:10.30
MTL (Sn et (01,)
ResNet-12
8.29M
61.20+1..0
75.50+0.80
MetaOpiNet (Lee et al. 2019)
ResNet-12
12.42M
62.641
78.63:4..4
SLA-AG (Lee et al. (2200)
ResNet-12
7.99M
62.93++066
79.63+t0.
Constellation (Xu et al.
2020
ResNet-12
7.99M
64.89
79.95+t0.17
RFS-distill (Tian cf al. (0200)
ResNet-12
13.55M
64.82+00.0
82. 14.1440.43
EPNet (Rodriguez et al. 2020
ResNet-12 7.99M
65. 65.66.10.85
81.28110.62
Meta-Baselnne (Chen et al.
2021
ResNet-12
7.99M
63.1710023
79.26.10.17
IE-distill' (Rizve
(021)
ResNet-12
9.13M
65.32:1081
83.6941..2
TAS-simple (ours)
ResNet-12
7.99M
64.71+0.43
82.08+0.445
TAS-distill (ours)
ResNet-12
7.99M
65.13+00.99
82.47+0.52
TAS-distill (ours)
ResNet-12
12.47M
65.68+0.455
83.92+0.55
performs with standard ResNet-12 with Dropblock regularizer, performs with wide-layer ResNet-12
Table 2: Comparison of the accuracy against state-of-the art methods for 5-way 1-shot and 5 5-way 5-shot classification with 95% confidence interval on tieredImageNet dataset.
Model
Backbone
Params
1-shot
5-shot
MAML (Finn et al. 2017)
ConyNet-4
0.IIM
51.67+1.81
70.30=0.08
Prototypical-Net (Snell el 2017)
ConvNet-4
0.11M
53.31+0.89
72.69=0.7
Relation-Net (Sung et al. 2018
ConyNet-4
0.11M
54.4811093
11.72
Simple CNAPS (Bateni al. 2020
R8sNet-18
11M
63.00+11.00
80.0011..00
LEO-trainval Rusu et al. 2018
ResNet-12 7.99M
66.5810.70
85.55
Shot-Free (Ravichandran
2019
ResNet-12 7.99M
63.52an/a
82.59-in/a
Fine-tuning (Dhillon et al. 2019
R2 7.99M
68. 08807740.26
83.7410.18
MetaOptNet (Lee et al.
2019
ResNet-12 12.42M
65.9920.72
81.56+0.58
RFS-distill (Tan et al.
2020)
ResNet-12 13.55M
71.52+0..9
86.03=0.49
EPNet (Rodriguez et al. (2020)
ResNet-12 7.99M
72.60110.9
85.6910.65
Meta- Meta-Baselime (Chen al.
2021
Re 7.99M
68. 10.22
83.7410.11
IE-distill' (Rizve et al. 2021)
ResNet-12
13.55M
72.21
87.08
TAS-simple (ours)
ResNet-12
7.99M
71.98:10.9
86.58-00446
TAS-distill (ours)
ResNet-12
7.99M
72.81+0.48
87.21+0.52
performs with wide-layer ResNet-12 with Dropblock as regularizer
CONCLUSIONS
A task affinity score, which is non-commutative and invariant to the permutation of labels is intro- duced in this paper. The application of this affinity score the few-shot learning investigated. In particular, the task affinity score is applied to identify the related training classes, which are used for episodically fine-tuning a few-shot model. This approach helps to improve the performance of the few-shot classifier various benchmarks. Overall, the task affinity score shows the importance of selecting the relevant data for training the neural networks with limited data and can be applied to other applications in machine learning.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
This work was supported in part by the Army Research Office grant No. W911NF-15-1-0479.
REFERENCES
Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless Fowlkes, Stefano Soatto, and Pietro Perona. Task2Vec: Task Embedding for Meta-Learning. arXiv e-prints, art. 190.:2490,.03545, Feb. 2019.
Peyman Bateni, Raghav Goyal, Vaden Masrani, Frank Wood, and Leonid Sigal. Improved few- shot visual classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14493-14502, 2020.
Luca Bertinetto, Joao F Henriques, Philip HS Torr, and Andrea Vedaldi. Meta-leaming with differ- entiable closed-form solvers. arXiv preprint arXiv:1805.087333 2018.
Shixing Chen, Caojin Zhang, and Ming Dong. Coupled end-to-end transfer learning with general- ized fisher information. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp.
2018.
Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look few-shot classification. arXiv preprint arXiv: 04232, 2019.
Yinbo Chen, Zhuang Liu, Huijuan Xu, Trevor Darrell, and Xiaolong Wang. Meta-baseline: explor- ing simple meta-lcarning for few-shot learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
Guneet S Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. baseline for few-shot image classification. arXiv preprint (rXiv:1909.02729, 2019.
Kshitij Dwivedi and Gemma Roig. Representation similarity analysis for efficient task taxonomy & transfer learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12387-12396, 2019.
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A. Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. CoRR, abs/1701.08734, 2017. URL
08734
Chelsea Finn, Xin Yu Tan, Yan Duan, Trevor Darrell, Sergey Levine, and Pieter Abbeel. Deep spatial autoencoders for visuomotor learning. In Robotics and Automation (ICRA), 2016 IEEE International Conference on, pp. 512-519. IEEE, 2016.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference Machine Learning, pp. 1126-1135. PMLR, 2017.
S. Gadat and Panloup. Optimal non-asymptotic bound of the ruppert-polyak averaging without strong convexity. arXiv preprint
2017.
Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradient- 2018.
based meta-leaming as hierarchical bayes. arXiv preprint
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Ban Barnkkkk-Baiiikka,aaaaa al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-352 2017.
Gregory Koch, Richard Zemel, Ruslan Salakhutdinov, al. Siamese neural networks for one-shot image recognition. In ICML deep learning workshop, volume 2. Lille, 2015.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Citeseer, 2009.
10

---

Published as a conference paper at ICLR 2022
Harold W Kuhn. The hungarian method for the assignment problem. Naval research logisties quarterly, 2(1-2):83-97, 1955.
Hankook Lee, Sung Ju Hwang, and Jinwoo Shin. Self-supervised label augmentation via input trans- formations. In International Conference on Machine Learning, pp. 5714-5724. PMLR, 2020.
Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10657-10665, 2019.
Zelun Luo, Yuliang Zou, Judy Hoffman, and Li F Fei-Fei. Label efficient learning of transferable representations acrosss domains and tasks. In Advances in Neural Information Processing Sys- tems, pp. 164-176, 2017.
Arun Mallya and Svetlana Lazebnik. Piggyback: Adding multiple tasks to single, fixed network by learning mask. CoRR, 06519
2018.
Puneet Mangla, Nupur Kumari, Abhishek Sinha, Mayank Singh, Balaji Krishnamurthy, and Vi- neeth N Balasubramamian. Charting the right manifold: Manifold mixup for few-shot learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp- 2218-2221 2020.
Lilyana Mihalkova, Tuyen Huynh, and Raymond J Mooney. Mapping and revising markov logic networks for transfer learning. In AAAI, volume 7, pp. 608-614, 2007.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta- learner. arXiv preprint urXiv: 2017.
Tsendsuren Munkhdalai, Xingdi Yuan, Soroush Mehri, and Adam Trischler. Rapid adaptation with conditionally shifted neurons. In International Conference on Machine Learning, pp. 3664-3673.
PMLR, 2018
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv:1803.022 020999, 2018.
Alexandru Kiculescu-Mize and Rich Caruana. Inductive transfer for bayesian network structure learning. In Artificial Intelligence and Statistics, pp. 339-346, 2007.
Boris N Oreshkin, Pau Rodriguez, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. arXiv preprint arXiv: 10123, 2018.
Arghya Pal and Vineeth N Balasubramanian. Zero-shot task transfer, 2019.
S. J. Pan and Q. Yang. A survey transfer learning. IEEE Transactions Knowledge and Data Engineering, 22(10):3345-1
Oct 2010. ISSN 1041-4347. doi: 10. 1..1109/KKDE.2000.0
B. Polyak and A. Juditsky. Acceleration of stochastic approximation by averaging. Siam Journal on Control and Optimization, 30:838-852 855, 1992.
Limeng Qiao, Yemin Shi, Jia Li, Yaowei Wang, Tiejun Huang, and Yonghong Tian. Transductive episodic- wise adaptive metric for few-shot learning. In Proceedings of the IEFE/CVF Interna- tional Conference on Computer Vision, pp. 3603-3612, 2019.
Siyuan Qiao, Chenxi Liu, Wei Shen, and Alan Yuille. Few-shot image recognition by predicting parameters from activations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7229-7238. 2018.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In ICLR, 2017.
Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto. Few-shot learning with embedded class models and shot-free meta training. In Proceedings of the IEEE/CVF International Conference Computer Vision pp. 331-339, 2019.
11

---

Published as a conference paper at ICLR 2022
Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. Cnn fea- tures off-the-shelf: An astounding baseline for recognition. In Proceedings of the 2014 IEEE Conference Computer Viston and Pattern Recognition Workshops, CVPRW 14, pp. 512- 519, Washington, DC, USA, 2014. IEEE Computer Society. ISBN 978-1-4799-4300 111109/CVPRW.204..3333 URL http: http:://xx.dii.org.. wtt..o........rd.....o.oo.o ddm. oc ool.ool.oorooo orgg N0.1109/CVPRIC 2ttp://.............. 2014.131
doi:
Mengye Ren, Eleni Triantafilluu, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua Tenenbaum, Hugo Larochelle, and Richard S Zemel. Meta-learning for semi-supervised few-shot classifica- tion. arXiv preprint arXiv: 1803,00676 2018.
Mamshad Nayeem Rizve, Salman Khan, Fahad Shahbaz Khan, and Mubarak Shah. Exploring com- plementary strengths of invariant and equivariant representations for few-shot learning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10836 10846, 2021.
Pau Rodríguez, Issam Laradji, Alexandre Drouin, and Alexandre Lacoste. Embedding propagation: Smoother manifold for fhw-shot classification. In European Conference on Computer Vision, pp. 121-138. Springer, 2020.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenct large scale visual recognition challenge. International journal of computer vision, 1 115(3):211-253 2015.
Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. CoRR, hess/6006.4671,, 2016. URL
org/ abs/abbss/11s 04671
Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osin- dero, and Raia Hadsell. Meta-learning with latent embedding optimization. arXiv preprint arXiv:1807.0596) 2018.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with ing. pp. 1842-1850. PMLR, 2016.
neural networks. In International conference on machine learn-
Daniel L Silver and Kristin P Bennett. Guest editor's s introduction: special issue on inductive transfer learning. Machine Learning, 73(3):215-22) 220, 2008
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. arXiv preprint 1703.05 2017.
Trevor Standley, Amir Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should be learned together in multi-task learning? In Hal Daumé III and Aarti Singh (eds.). Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 9120-91133 9132. PMLR, 13-18 Jul 2020. URL
html
Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele. Meta-transfer learning for few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 403-412, 2019.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning compare: Relation network for few-shot learning. In Proceedings of the IEEE con- ference on computer vision and pattern recognition, pp. 1199-1208, 2018.
Yonglong Tian, Yue Wang. Dilip Krishnan, Joshua B Tenenbaum, and Phillip Isola. Rethinking few-shot image classification: good embedding is all you need? In Computer ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16, pp. 266-282. Springer, 2020.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez- Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In International Conference on Machine Learning, pp. 6447. PMIR, 2019.
12

---

Published as a conference paper at ICLR 2022
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, 29:3630-366 060000363..00 2016.
Aria Y Wang, Leila Wehbe, and Michael Tarr. Neural taskonomy: Inferring the similarity of task-derived representations from brain activity. BioRxiv, pp. 708016, 2019.
Weijian Xu, Huaijin Wang, Zhuowen Tu, al. Attentional constellation nets for few-shot learning. In International Conference on Learning Representations, 2020.
Amir R Zamir, Alexander Sax, William B Shen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In 2018 IEEE Conference on Com-
puter Vision and Pattern Recognition (CVPR). IEEE, 2018.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107- 115, 2021.
A APPENDIX
Here, we fist present the proof of the Theorem 1, and then we provide more experimental results on CIFAR-FS and FC-100 datasets. Additionally, we present the ablation study show the efficacy of our proposed task affinity score. Moreover, we also discuss about the computation complexity of our approach, and the future works.
A.1 PROOF OF THEOREM 1
In the proof of Theorem I, we invoke the Polyak & Juditsky 1992) theorem the convergence of the average sequence of estimation in different epochs from the SGD algorithm. Since the original form of this theorem has focused jective function the training of the approximation network is strongly convex. However, there are some recent works that try to generalize Polyak & Juditsky (1992) theorem for the convex even some non-convex functions in an (non)-asymptotic way Gadat & Panloup (2017). Here, we apply only on the asymptotic version of the theorem proposed originally Polyak Juditsky (1992). We first recall the definition of the strongly convex function.
the strongly convex functions,
have assumed that the ob-
Definition 4 (Strongly Convex Function). A differentiable function f : + R strongly cowve for all x. y E T." and some 1s 0, f satisfies the following inequality:
(6) Theorem 1. Let XA be the dataset for the task TA with the objective function L, and XB be the dataset for the task TB- Assume XA and XB have the same distribution. Consider an e- approximation network No for the pair task-dataset (T Xx) with the objective functions and with the SGD algorithm to result weights Og at time Assume that the functtoon L is strongly con- vex, and its 3rd-order continuous derivative exists and bounded. Let the noisy gradient function in training No network using SGD algorithm be given by: g(0,, = L((0)) + Es, where Og the estimation of the weights for network N at time t, and V ((,)) is the true gradient at lr. Assume that satisfies W[[e) (o. Et-1] = O, and satisfies = lim |[eeer"/eo......
f(y) 2 f(x)+ (()) x+) + +lly 211 x
(e.-..,, < oo almost surely (a.s.). Then the task affinity score between TA and Tn computed on the average of estimated weights up the current time converges constant as 1 -> That
17000
8t =
FA 1/2
FB
1/2
D
F
1/2 1/2
FB
1/2
V
F
V2
F
where FA, T(0,,, FD. F(Or, ((0r,xxaupee Xsupport) with 0. - + Moreover, F
F(O", Xquery) and F F(O", xeupport) denote the Fisher Information matrices computed using the optimum approximation network (i.e. network with the global minimum weights, f*).
Proof of Theorem Consider task TA as the source task and task T3 as the target task. Let 0, be the set of weights at time from the e-approximation network N trained using dataset
13

---

Published as a conference paper at ICLR 2022
XA X with the objective functions L. Since the objective function L is strongly convex, the set of weights 9 will obtain the optimum solutions 8* after training a certain number of epochs with stochastic gradient descend. By the assumption on the conditional mean of the noisy gra- dient function and the assumption on S, the conditional covariance matrix is finite as well, i.e., C = oo E " (c.. .o....... 62-1) < co; hence, we can invoke the following result due Polyak
et al. Polyak & Juditsky (992):
Vi(0. 0*) 2, (O,
as t + oc. Here, H is Hessian matrix, 0* is the global minimum of the loss function, and O, E. 0, In other words, for the network N, vi(0, - 0*) is asymptotically normal random vector:
Vi 07) 2, (V).
where Σ H(L(0))) CH (L(O-)) For a given dataset X, the Fisher Information Fx(0) is a continuous and differentiable function of f, and it also a positive definite matrix; thus, Fx)0)1/2 is well-defined. Now, by applying the Delta method to Equation (9) with dataset XA of task TA,
have:
1 FF 2 N0. 7AA) EA),
(10)
where FAt- = Fx.().). FA = (x) and the covariance matrix is given by ΣA = Jo (ec EJg vee (vec(Fx, (0))/)))) T
Here, vec() is the vectorization operator, 0" is n 1 vector of the optimum parameters, Fx (f") is a Matrix evaluated at the minimum using dataset XA, and Jo(vec(Fx, (0)) is n² x n Jacobian matrix of the Fisher Information Matrix. Likewise, from Equation with dataset XB of task TB, we have:
(F Fn 122 2 N(00 EB).
(11)
where FBL Fxx (01), FB (xx( and the covariance matrix is given by Eg
T
Jo (vec(Fxx (0x)) (f) From Equation (10) and (11 we obtain: -
(F Bt
2 N (H.V..
(12)
where μ (FX FR 1/2 and V = (EA + EB). Since (F1/2 - F1 -
FB 1/2 is
asymptotically normal with the covariance goes to to zero, we conclude that:
approaches infinity, all of the entries go
81 =
F1/2 Al
F1/2
D
F 1/2
F
1/2
(13)
2
F
A.2 EXPERIMENTS ON CIFAR-FS AND FC-100
Here, we conduct the experiments on the CIFAR-FS and the FC-100 datasets. The CIFAR-FS dataset (Bertinetto al. 2018) is derived from the original CIFAR-100 dataset (Krizhevsky al. 2009) by splitting 100 classes into 64 training classes, 16 validation classes, and 20 testing classes. The FC-100 dataset (Oreshkin al., 2018) also subset of the CIFAR-100 and consists of 60 training classes, 20 validation classes, and 20 testing classes. The size of data sample in both of these datasets is x 32.
Similar to previous experiments minilmageNet and tieredImageNet, we train the ResNet- 12 for 100 epochs with batch size of 64, and the learning rate decaying at epoch 60. Next, we compare our proposed fhw-shot approach with other state-of-the-art methods CIFAR-FS and FC-100 datasets. The results in Table 3] and Table Sallefiidiian indicates the competitiveness of our method in both datasets. Our approach with standard ResNet- is comparable to IE-distill (Rizve et al. 2021) and outperforms RFS-distill (Tian et al., 2020) while having a significantly smaller classifier model in term of the number of parameters.
14

---

Published as a conference paper at ICLR 2022
Table 3: Comparison of the accuracy against state-of-the methods for 5-way 1-shot and 5-way 5-shot classification with 95% confidence interval on CIFAR-FS dataset.
CIFAR-FS
Model
Backbone
Params
1-shot
5-shot
MAML (Finn et al. 2017
ConvNet-4
0.11M
58.90
71.50=1.00
Prototypical-Net (Snell et al. 2017
ConvNet-4
0.11M
50.50
72.00=0.60
Relation-Net (Sung et al. 2018
ConvNet-4
0.11M
55.00+1.00
69.30=0..80
Prototypical-Ner (Snell et al.
2017)
ResNet-12
7.99M
72.20
83.5010.50
Shot-Free (Ravichandran et al.
2019
ResNet-12
7.99M
69.20din/a
84.701nm/a
TEWAM (Qiao et
2019)
ResNet-12
7.99M
70. 70.40#n/4a
81.30+tn/a
MetaOptNet (Lee et al.
2019
ResNet-12
12.42M
72.60=0.70
84.30=0.50
RFS- AiS-se Tian et 2020
ResNet-12
13.55M
71.5010.80
86.0010..5
RFS- Than et al. 2020)
ResNet-12
13.55M
73.90:0.80
86.90
IE-distill' (Rizve et al.
2021
ResNet-12
9.13M
75.46-10.86
88.67
TAS-simple (ours)
ResNet-12
7.99M
73.4710042
86.82:10.49
TAS-distill (ours)
ResNet-12
7.99M
74.02+00.5
87.65+t0.58
TAS-distill (ours)
ResNet-12
12.47M
75.56:0.62
88.95-10.65
performs with standard ResNet-12 with Dropblock regularizer. performs with wide-layer ResNet-12
Table 4: Comparison of the accuracy against state-of-the art methods for 5-way 1-shot and 5-way
5-shot classification with 95% confidence interval on FC-100 dataset.
FC-100
Model
Backbone
Params
1-shot
5-shot
Prototypical-Net (Snell
2017)
ConvNet-4
0.IIM
35.30+0.60
48.60=0.00
Prototypical-Ner (Snell el al.
2017
ResNet-12
7.99M
37.50+0.60
52.50=0.60
TADAM Oreshkin al.
2018
ResNet-12
7.99M
40.10=0.40
56.10=0.40
MetaOptNet (Lee et al.
2019
ResNet-12
12.42M
41.10:2..00
35.50=0.60
RFS-simple TTan et
2020
ResNet-12
13.55M
42.60+0.70
59.10+0.60
RFS-distill (Tan et al.
2020)
ResNet-12
13.55M
44.6020.70
60.90
IE-distill (Rizve et al.
2021
ResNet-12
9.13M
44.65+0077
61.2440.75
TAS-simple (ours)
ResNet-12
7.99M
43. 43.10#0.67
60.65+00.2
TAS-distill (ours)
ResNet-12
7.99M
44.62+0.70
61.46:10.65
performs with standard ResNet-12 with Drophlock as regularizer
A.3 ABLATION STUDY
First, show the stability of the TAS by applying distance various classification tasks in CIFAR-10, CIFAR-100, ImageNet datasets using ResNet-18, and VGG-16 as the backhone for the e-approximation networks. For each dataset, we define classification tasks, all of which are variations of the full class classification task. For each task, we consider a balanced training dataset. That is, except for the classification tasks with all the labels, only a subset of the original training dataset is used such that the number of training samples across all the class labels to be equal. Additionally, we use different values for the e-approximation networks, ranging from 0.2 (good) to 0.6 (bad). To make sure that our results are statistically significant, we run our experiments 10 times with each the e-approximation networks being initialized with a different random seed each time and report the mean and the standard deviation of the computed distance.
In CIFAR- 10, we define 4 tasks as follow:
Task is a binary classification of indicating 3 objects: automobile, cat, ship (i.e., the goal is to decide the given input image consists of one of these three objects or not).
Task is a binary classification indicating 3 objects: cat, ship, truck.
Task is a 4-class classification with labels bird, frog, horse, and anything else.
Task 3 is the standard 10 objects classification.
15

---

Published as a conference paper at ICLR 2022
0:332
23445
0.335
645
6.991
0.244
0.4823
22664
9.6777
0.9975
64
6.35
0.3395
22203
Sounce Tor
0.4437
Spuacco Took
02206
0.632
0.9332
625
Target
Tanget
Targel
0322334
033934
0.33844
002755
0.82483
8 8.902
6042588
0.33952
3:65
0003
0.22549
1,0066
square Thi
D.22755
0.C3385
Source Trat
0.04223
0.33755
3.4220
102923
002355
TargetTask
Targut
Target
(a) E 0.2
(b) 0.4
(c) 0.6
Figure 3: Distance from source tasks to the target tasks on CIFAR-IO using ResNet- 18 backbone. The top row shows the mean values and the bottom row denotes the standard deviation of distances between elassification tasks over 10 different trials.
0.7
37
0.8
0.149
0.6416
0.5
1506
Squice
14
0.4623
0.4692
0
0.5114
8.9905
0.5174
0.2
22
0.3646
0.4161
0.3693
0.1
6-4000
8-455
0.4017
0.2588
0.4028
2.4885
Target Task
Target Tasc
Target Task
0.02108
0.C3902
0.04435
0.01798
0.03234
0.03042
Source Teat
Source
0.32112
0.03737
0.04555
3,00055
0.22383
0.02983
D.C2864
8.0005
6000114
0.23422
6.00922
2.00799
0.03115
Target Task
TargetTask
TargetTasc
(a) 0.2
0.4
0.6
Figure 4: Distance from source tasks to the target tasks on CIFAR-10 using VGG-16 backbone. The top row shows the mean values and the bottom row denotes the standard deviation distances between classification tasks over 10 different trials.
In Figure 3l the mean and standard deviation of the TAS between CIFAR- 10 tasks over 10 trial runs, using 3 different values for the ResNet- 18 approximation networks. For 0.2, the approxima- tion network's 's performance on the corresponding task's 's test data is at least 80% and it considered to be good representation. As shown in Figure a)). the standard deviation of TAS is relatively
16

---

Published as a conference paper at ICLR 2022
0.1278
6,2999
0.335
61
6.1301
92426
0.3157
9.457
044244
0:4473
C.1405
3232
02172
0.1382
02453
0.2322
Scurce Traks
0:2371
0.3494
Statce
00b
Tanget
Target
Target
0:31:4
301T5
0.32783
3.01446
2.00074
3.00446
2,00068
0.62947
6,03399
0:02738
0.22325
0.22340
Sounce 1o0k
100303
2.00123
Squrce Thal
0.22323
0.2331
C.0372
1,00112
6,2909
Target
Target
Target
(a) 0.2
(b) 0.4
(c) 0.6
Figure 5: Distance from source tasks the target tasks on CIFAR-I00 using ResNet- backbone. The top row shows the mean values and the bottom row denotes the standard deviation of distances between classification tasks over 10 different trials.
small and the TAS is stable (or consistent) regardless of the network's initialization. Thus. we can easily identify the closest tasks target tasks. The result for 0.4, shown Figure 3bb), the order of tasks remains the same for most cases, however, the standard deviations are much larger than the previous case. Lastly, Figure Ce) shows the results for = 0.6. In this setup. TAS between tasks fluctuates widely and it is considered to be unreliable. Consequently, if the TAS between pair of tasks is computed only once, the order of the closest tasks to the target task would easily be incorrect. Similarly, we conduct the experiment in CIFAR-10 using VGG-16 as the backbone. In Figure 4 we observe similar trend as the approximation network with smaller E value will obtain the more stable and consistent task distances.
In CIFAR-100, which consisting 100 objects equally distributed in 20 sub-classes, each sub-class has 5 object, define tasks follow:
Task 0 is a binary classification of detecting an object that belongs to vehicles and 2 sub- classes or (i.e., the goal is to decide if the given input image consists of one of these 10 vehicles or not).
Task a binary classification of detecting an object that belongs to household furniture and household devices not.
Task is multi-classificatin with 11 labels defined on vehicles 1, vehicles 2, and anything else.
Task 3 multi-classifictiinn with the 21-labels in vehicles vehicles 2, household fur- niture, household devices, and anything else.
In Figure 5) the mean and standard deviation of the TAS between CIFAR-100 tasks over trial runs, using 3 different values for the ResNet-18 18 approximation networks. For æ 0.2, the ap- proximation network's performance on the corresponding task's test data at least 80% and is considered to be good representation. As shown in Figure 5(a), TAS is stable regardless of the network initialization. As E increases from 0.2 to 0.6, we observe the significant rise in standard deviation. For 0.6, the TAS results widely fluctuating and unreliable. In particular, consider Task 0 as the incoming task, and Task 1, 2,3 are the base tasks. From the approximation network 0.2 in Figure 5 a). the closest task to Task 0 is Task 2. However, the approximation network
17

---

Published as a conference paper at ICLR 2022
0.2891
0.5253
0.3819
9.3237
15398
0.515
9.
64237
0.615
05286
M4
Spurca Toak
0.2364
0.4335
0.388
0.2912
2
0.1333
0.2002
2.003
0.382
0.1753
0.3988
3
0.221
62
0:2215
0.428
0.2905
0-4366
12
0.2713
0,4088
0.07923
0.3074
Target Task
Targer
Targel
0,013557
0.04011
0.0240:
D-00035
0.00390
2.09282
b:03947
0.04200
0.01433
0,03952
0.03665
03500
0.23286
0.00906
Spunce Taay
0,0007
0.93866
0.04253
Spunna
0.0128
0.02364
0.01815
602152
C.0353
2,03357
0LENKON
0.01302
0.01144
0.00457
Target
Target
(a) E 0.2
(b) 0.4
(c) E 0.6
Figure 6: Distance from source tasks the target tasks on CIFAR-I00 using VGG-16 backbone. The top row shows the mean values and the bottom row denotes the standard deviation of distances between classification tasks over 10 different trials.
= 0.6 in Figure Fic) identifies Task 1 as the closest task of Task O. Thus, in order to achieve meaningful TAS between tasks, we first need to make sure that we represent the tasks correctly by the approximation networks. the approximation network is not good in term of performance (i.c., e is large), then, that network is not a good representation for the task. Similarly, we conduct the experiment in CIFAR-100 using VGG-16 as the backbone. In Figure O we observe similar trend as the approximation network with smaller value will obtain the more stable and consistent task distances.
In ImtgeNet, we define four 10-class classification tasks in ImageNet dataset. For each class, we consider 800 for training and 200 for the test samples. The list of 10 classes for each task is described below:
Task 0 includes tench, English springer, cassette player, chain saw, church, French horn, garbage truck, gas pump, golf ball, parachute.
Task similar Task O; however, instead of labels of tench, golf ball, and parachute, it has samples from the grey whale, volleyball, umbrella classes.
In Task 2, we also replace 5 labels of grey whale, cassette player, chain saw, volleyball, umbrella Task with another labels given by platypus, laptop, lawnmower, baseball, cowboy hat.
Task includes analog clock, candle, sweatshirt, birdhouse, ping-pong ball, hotdog, pizza, school bus, iPod, beaver.
In Figure and Figure g we demonstrate the mean standard deviation of the TAS between ImageNet tasks over 10 runs using ResNet-18 and VGG- 16 as the backbone. For each backbone, conduct different e values and report results as the above experiments. Similar to the CIFAR-10 and CIFAR-100. observe that the approximation network with better performance (or smaller value) will results more stable TAS. Additionally, regardless of the architecture backbone, the trend of distance between these tasks is consistent across all experiments.
Next, we want to analyze the effectiveness of our approach of choosing the most related data classes by performing several few-shot learning experiments with different number of related classes. Ad-
18

---

Published as a conference paper at ICLR 2022
9.1912
0.3313
0,4728
0.4
0.2522
0.3089
0.4566
145
0.3933
0.5011
65
04
Took
6,2412
0,2921
0392
0.2579
0.1274
Scurce Thaks
0.3228
0.3548
0.46
0.25
Scarce
325
6.3176
2,4472
0,2072
0.4005
0.2844
0.4885
60.35
23
6.4401
0.4525
0.2271
0.05
2.55
TarketTack
Target
Target
0:03240
6.03005
003012
0.68841
1,08533
0.33271
C.ARZEES
U.02213
Source Tack
0.014:3
6.2224
002309
6-02322
1,22529
5.0095
0,00392
0.03613
0.8000
0,0233
0,0124
02044
0.69136
0.83282
0.03214
0.0sat1
0.65533
0.02998
02022
0.02375
Targel
Targut
Target
(a) 0.2
(b) 0.4
(c) E 0.6
Figure 7: Distance from source tasks the target tasks on ImageNet using ResNet-18 18 backbone. The top row shows the mean values and the bottom row denotes the standard deviation of distances between elassification tasks over 10 different trials.
1.43
8.1888
0.35
2.45
3335
0.3748
Souro Toxk
0,2035
0.25
0.3983
0.15
0.7698
0.500
8.3671
0,4102
6.:5
6.3245
6.4225
0 2707
$
0 6.485
0.3108
3.5
Target ask
TorgetT Tast
TargetTas
0.00552
Source Task
8.09894
0.0322
Sourse Tack
8.02255
60652
5,09m11
6/1
0824000
0000
001116
DETEAS
0.01E75
5.0333
602247
0.23555
3 04407
5.08529
0.0005
Target Task
TargetTass Tase
TargetT2ok
(a) 0.2
0.4
0.6
Figure 8: Distance from source tasks to the target tasks on ImageNet using VGG-16 backbone. The top row shows the mean values and the bottom row denotes the standard deviation distances between classification tasks over 10 different trials.
ditionally, we conduct the few-shot experiment with non-related classes and randomized classes to show the efficacy of our proposed method. Table shows the performances of our proposed few- shot method with different settings. We achieve the best performance with 49 related tasks both 5-way 1-shot and 5-way 5-shot classification. We observe drop in performance when the number
19

---

Published as a conference paper at ICLR 2022
Table 5: The performance of TAS with different settings for 5-way 1-shot and 5-way 5-shot classi- fication with 95% confidence interval on minilmageNet dataset.
minilmageNet
Model
Params
1-shot
5-shot
TAS- TAS-simple with top-54 related classes
7.99M
63.54
80.92:t0.55
TAS-simple with top-49 related classes
7.99M
64.71+0.43
82.08+t0.45
TAS-simple with top-38 related classes
7.99M
63.92=0.47
01.299+0.50
TAS-simple with top-32 related classes
7.99M
63.35
80.67:0053
siSpl with top-29 non-related classes
7.99M
61.8210.44
78. 14.10.4..
simple with 32 random classes
7.99M
62.57
78.96.10.
Table 6: The performance TAS with different settings for 5-way 1-shot and 5-way 5-shot classifica- tion with 95% confidence interval on tieredlmageNet dataset.
minilmageNNet
Model
Params
1-shot
5-shot
simps with top-318 related classes
7.99M
71.02=0.42
85.26++0.48
TAS-simple with top-293 related classes
7.99M
71.98-10.39
86.58:10046
TAS-simple with 209-249 related classes
7.99M
70.89
85.65
TAS-simp DAS-simple with top-211 related classes
7.99M
70.11
84.86+0050
TAS TAS-simple with top-223 non-related classes
7.99M
67 67.33+0.50
83.24+0.52
TAS- TAS-simple with 211 random classes
7.99M
68.04+0.16
83.55+00.22
of related classes increases to 54 classes. As we reduce number of related classes from 49 to 32 classes, the overall performance of our model reduces significantly in both 1-shot and 5-shot. As the result, selecting the optimal number relevant classes is crucial to the overall performance of few-shot approach. Additionally,
observe drop in performance when the model utilizing 29 non-related classes as well as 32 random classes. Therefore, we show that using a relevant classes boost the performance of the few-shot model. Moreover, we conduct the similar analysis for the tieredImageNet dataset, in which the results are shown in Table Here, we observe the same trend in previous experiments. Briefly, by selecting the relevant classes, we consistently achieve higher performances when compare with selecting non-relevant or randomized classes.
A.4 COMPUTATION COMPLEXITY
Below are the complexity of 3 phases in our approach:
Phase 1 consists of 2 operations: (i) Train Chole-Classification net. (ii) Feature extraction. These operations is executed only once, and the complexity is constant.
(iii) Matching labels (iv) Constructing approx. net. (v) Fisher Information matrix. The operation (iii) is based on the Hungarian algorithm and the complexity is O(n²), where n is the number of tasks. For each base task, (iv) is done once, and the complexity of finding FIM in (v) is linear with respect to the number of parameters due to the fact that we only consider the diagonal entries of the matrix. Note that all the tasks are few-shot and only consists of few data samples. These operations can effectively implemented in GPU.
Phase consists of 3 operations:
Phase is (vi) Episodic fine-tuning, which often used few-shot approach [R3, R4). Here, use the k-NN as the clustering algorithm. The complexity of k-NN is linear with respect to the number of samples.
A.5 FUTURE WORKS
In this paper, we propose a novel task similarity measure that inherently asymmetric and invariant to label permutation, called Task Affinity Score (TAS). Next, introduce the TAS measure the
20

---

Published as a conference paper at ICLR 2022
few-shot learning approach and achieve significant improvements the current state-of- tr-hh--rtt methods. future work, we would like to extend the applications of TAS to other few-shot learning frameworks, and also other meta-learning areas, e.g., multi-task learning, continual learning. Additionally, we wish to establish theoretical justification for the TAS measure convex or non- convex cases.
21


Reviews:
[{"review_id": "y4IUqc-6_65", "reviewer": "Reviewer_UXcu", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. The idea to exploit Fisher Information Matrix for relevance measurement between different tasks is novel.\n2. The experimental results are promising.\n\nWeakness:\n1. The paper exploits the empirical Fisher Information matrix. The reason for this approximation should be discussed in detail.\n2. The authors did not conduct some necessary experiments to validate the effectiveness of TSA, e.g., the authors should prove that models trained with source tasks selected via TSA are consistently better than those trained with randomly sampled source tasks.\n3. The proof of TSA holds when the loss function is strictly convex, which is a strong assumption.", "summary_of_the_paper": "This paper proposes a few shot learning method, which tries to measure the affinity degree between different tasks. Based on the affinity score, the relevant tasks are exploited for training to boost the performance of target tasks. The Task Affinity Score (TSA) is proposed, which is novel  to measure the dependency between different tasks. The reasonability of TSA is also validated with mathematical proof.", "summary_of_the_review": "This paper exploits the fisher matrix to measure the relevance between different tasks which is an interesting attempt. In addition, the experimental validate the effectiveness of the proposed method.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "i3yqLKCjm2q", "reviewer": "Reviewer_Y7LN", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "pros:\nThe paper is well-written and neatly presented. \nThe idea that leveraging task affinity score to find most relevant tasks for better few-shot learning is impressive, and seems to be technically sound.\n\ncons:\n\tThe contributions are not well highlighted.\n\tThere is no ablation study in this paper, the experimental results are insufficient to validate the idea and many detailed experiments are not provided.\n\tThe main idea and the motivation are easy to follow, however some details of the proposed model are still not well specified.\n\nDetailed comments and questions:\n\t1. In Definition 1, what is the meaning of epsilon-approximate network? How do you determine the value of epsilon, does it have a significant impact on the performance of the model?\n\t2. In Definition 3, why use task Ta's query data to compute the Fisher information matrix instead of using support data?\n\t3. It is mentioned in this paper that the classifier model is fine-tuned only with the labels of the related-training set, the complexity of training of the few-shot model is reduced. However, from Algorithm 1, the relevant task selection seems quite time consuming. Has the author considered the computational complexity of the model?\n\t4. In 1-shot setting, the class centroids of the target task are the sample itself. Will the closest source task calculated in this case be biased due to the randomness of the sample selection?\n\t5. In Section 4.2.2, the author says that top-R scores and their corresponding classes are finally selected. Then Algorithm 1 selects the task with the minimum TAS. Which method does the author use?\n\t6. How to use the source task fine-tuning model, which the authors did not explain? For example, in the miniImageNet, 2000 source tasks are randomly selected, then the model is fine-tuned on each source task to calculate TAS to the target task. Is this process time-consuming? \n\t7. In the tieredImageNet, why only 120 labels are selected for each source task when the test set of this dataset has 160 labels?\n\t8. In Table 1 and Table 2, why use different backbone to compare IE-Distill methods respectively? miniImageNet uses standard resnet-12 while tieredImageNet uses wide-layer resnet-12.\n\t9. What is the baseline of the paper? Is the performance gain from TAS-Simple to TAS-Distill derived from knowledge distillation or the method proposed by the authors?\n", "summary_of_the_paper": "The authors propose a task affinity score based on maximum bipartite matching algorithm and Fisher information matrix. And then utilize this score to find the closest training data labels to the test data and leverage the discovered relevant data for episodically fine-tuning the few-shot model. Experimental results on few-shot learning setting achieve the state-of-the-art performance on four widely-used benchmarks.", "summary_of_the_review": "The overall idea of using task affinity score for few-shot learning is impressive. However, I think this submission is incomplete and lacks a lot of technical details, which makes the paper not easy to be understood. Meanwhile, the paper also has not conducted any ablation study to show the real empirical improvement of the proposed strategies and modules. Moreover, although the method proposed by the author does not introduce additional model parameters, I am quite concerned that the computation cost of the fine-tuning source tasks during the task affinity score calculation.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "KRePbukEMz", "reviewer": "Reviewer_mtpL", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strong points.\n- The authors introduce a new score based on the Fisher Information matrix; its mathematical analysis is also presented.\n- The performance is compared with many meta-learning methods in the empirical study.\n- The paper is well-organized.\n\nWeak points.\n- The proposed few-shot learning scheme seems incremental.\n- The advantage of the proposed score (TAS) is not clear.\n- A more thorough evaluation is needed to show the effectiveness of the proposed method.\n\nComments.\n1. In my understanding, the main contribution of this study is to incorporate a process of constructing a related-training set via Task Affinity Score (TAS) into a few-shot learning procedure. Another approach to consider the task similarity is to embed tasks into hidden space, as in conditional neural processes (NPs) [R1]. The authors should clarify the advantage of the proposed approach compared with such prior works. Also, it would be helpful to add the empirical evaluation with conditional NPs, etc. \n\n2. In Section 3, the authors present TAS as a general one; however, the few-shot learning scheme is constructed using a specific approach, i.e., Whole-Classification. Is it possible to use TAS for few-shot learning based on another approach?\n\n3. In Tables 1 and 2, the authors demonstrate that the proposed method outperforms many previous meta-learning methods in terms of prediction accuracy. However, this is not enough to convince the effectiveness of the proposed method. I would like to see more evidence of why the proposed method works well. For example, it would be helpful to analyze the related source tasks selected by TAS.\n\n4. As another aspect, the authors would add a discussion on the computation complexity of the proposed method.\n\n[R1] M. Garnelo, D. Rosenbaum, C. Maddison, T. Ramalho, D. Saxton, M. Shana- han, Y. W. Teh, D. Rezende, and S. A. Eslami. Conditional neural processes. In International Conference on Machine Learning, pages 1690–1699, 2018. ", "summary_of_the_paper": "This paper presents a new affinity score based on the Fisher Information matrix from a source to a target task. The authors also develop a few-shot learning procedure based on a pre-trained Whole-Classification network approach. In this procedure, training labels are matched via a maximum matching algorithm, and the target task for training is selected using the proposed affinity score. The effectiveness of the proposed score is demonstrated using benchmark datasets in the problem of few-shot learning.", "summary_of_the_review": "This paper proposes a new few-shot learning procedure and shows its effectiveness in terms of prediction accuracy. However, the advantage of the proposal should be discussed compared with previous works; the authors should conduct a more thorough evaluation. Accordingly, I vote for rejecting this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a few-shot learning method that uses Fisher information matrix-based task affinity. The experimental results show that the proposed method achieved better performance than existing methods. This paper is well-written. The newly proposed task affinity score is interesting. The experimental results and theoretical analysis support the effectiveness of the proposed method. The authors are encouraged to address the reviewers' concerns in the paper.  Although the distance between task representations is symmetric in neural processes, they do not use the symmetric distance for meta-learning. They input the task representations into the neural network, so the output can be asymmetric.
Decision: Accept (Poster)