Title: Understanding Few-Shot Learning: Measuring Task Relatedness and Adaptation Difficulty via Attributes
Venue: NeurIPS 2023
Accepted: yes
Keywords: Few-shot Learning

Meta-Learning

Task Relatedness

Task Adaptation Difficulty
Abstract: Few-shot learning (FSL) aims to learn novel tasks with very few labeled samples by leveraging experience from \emph{related} training tasks.
    In this paper, we try to understand FSL by exploring two key questions:
    (1) How to quantify the relationship between \emph{ training} and \emph{novel} tasks?
    (2) How does the relationship affect the \emph{adaptation difficulty} on novel tasks for different models?
    To answer the first question, we propose Task Attribute Distance (TAD) as a metric to quantify the task relatedness via attributes.
    Unlike other metrics, TAD is independent of models, making it applicable to different FSL models.
    To address the second question, we utilize TAD metric to establish a theoretical connection between task relatedness and task adaptation difficulty.
    By deriving the generalization error bound on a novel task, we discover how TAD measures the adaptation difficulty on novel tasks for different models.
    To validate our theoretical results, we conduct experiments on three benchmarks.
    Our experimental results confirm that TAD metric effectively quantifies the task relatedness and reflects the adaptation difficulty on novel tasks for various FSL methods, even if some of them do not learn attributes explicitly or human-annotated attributes are not provided.
    Our code is available at 
    \href{https://github.com/hu-my/TaskAttributeDistance}{https://github.com/hu-my/TaskAttributeDistance}.

Understanding Few-Shot Learning: Measuring Task Relatedness and Adaptation Difficulty via Attributes

Abstract

Few-shot learning (FSL) aims to learn novel tasks with very few labeled samples by leveraging experience from related training tasks. In this paper, we try to understand FSL by exploring two key questions: (1) How to quantify the relationship between training and novel tasks? (2) How does the relationship affect the adaptation difficulty on novel tasks for different models? To answer the first question, we propose Task Attribute Distance (TAD) as a metric to quantify the task relatedness via attributes. Unlike other metrics, TAD is independent of models, making it applicable to different FSL models. To address the second question, we utilize TAD metric to establish a theoretical connection between task relatedness and task adaptation difficulty. By deriving the generalization error bound on a novel task, we discover how TAD measures the adaptation difficulty on novel tasks for different models. To validate our theoretical results, we conduct experiments on three benchmarks. Our experimental results confirm that TAD metric effectively quantifies the task relatedness and reflects the adaptation difficulty on novel tasks for various FSL methods, even if some of them do not learn attributes explicitly or human-annotated attributes are not provided. Our code is available at https://github.com/hu-my/TaskAttributeDistance.

1 Introduction

Learning in human biological system is highly efficient compared to artificial systems. For example, only one example is enough for a child to learn a novel word [9], while thousands of training samples are needed for deep learning models. This learning efficiency comes from the past experiences accumulated by human brain. Inspired by human learning capability, Few-Shot Learning (FSL) aims to learn novel tasks with very few samples by leveraging experience from related training tasks.

Despite the great progress made in FSL during the past decade [48, 42, 17, 45, 22, 10, 16, 50, 27], there is less exploration on what it means for training tasks to be related. Some prior studies [48, 17, 45] focus on constructing related training tasks by sampling categories within a dataset. Recent works [46, 33] consider training tasks sampled from different datasets, known as cross-domain few-shot learning. The latter is considered as a more challenging problem, partly because the category gaps across datasets are usually larger, resulting in training tasks less related to novel tasks. Although this follows the intuition, two natural questions arise: (1) How to quantify the relationship between training and novel tasks? (2) How does the relationship affect the adaptation difficulty on novel tasks for different FSL models?

To quantify the relationship between tasks, one approach is to measure the category relationship in a common representation space. With properly learned representation function, various distribution divergence metrics, like EMD [39] and FID [20], can be used to calculate the distance between feature distributions of different categories, as well as tasks over them. The main obstacle to this

solution is how to learn a good representation function for novel categories with only a few labeled samples. Previous works [32, 33] assume that the representation function learned from training categories can be directly applied to novel categories, but this assumption may not hold due to the large category gaps. Some recent studies [1, 26] utilize the Fisher Information Matrix to quantify the task relatedness without assuming a common representation function. However, this approach has a heavy burden of computing Hessian matrix. Besides, the calculated task distances are highly dependent on the learned model, making them difficult to apply to other FSL models.

To investigate the influence of task relatedness on the difficulty of adapting models to novel tasks, existing studies [37, 40, 32, 33] often make an empirical assumption that a novel task with large distances to the training tasks will be hard to adapt to. With this assumption, some works [40, 32, 33] employ specific distance metrics to quantify the task relatedness, and then construct more challenging benchmarks to explore the generalization ability of different models. Different with the these works, [26] select the most related training data to improve the episodic fine-tune process for FSL models based on an asymmetric task relatedness measure. Despite their empirical success, all the aforementioned works lack theoretical analysis, leaving the connection between task relatedness and task adaptation difficulty not formally clarified.

In this work, we try to overcome the existing obstacles and answer the two questions via attributes. Firstly, we introduce Task Attribute Distance (TAD) as a metric to quantify the task relatedness. Our intuition lies in the attribute compositionality hypothesis (shown in Fig. 1): each category can be represented as a composition of attributes, which are reusable in a huge assortment of meaningful compositions. TAD formulates this intuition by first measuring the total variation distance between attribute distributions of training and novel categories, and then finding the maximum matching with minimum weight of a bipartite graph. Unlike other metrics, TAD only relies on the attribute conditional distributions, making it independent of models and the number of samples in novel tasks. Secondly, we utilize the TAD metric to establish a theoretical connection

grey upperparts

grey upper tail

hooked seabird bill

medium size

long wing

black eyes

black bill

solid tail

black wing

white underparts

white throat

white throat

yellow underparts

yellow throat

rounded wing

all-purpose bill

yellow tail

Laysan

Albatross

Common

Yellowthroat

between task relatedness and task adaptation difficulty in FSL. We provide theoretical proof to verify that TAD contributes to the generalization error bound on a novel task , at different training settings. Importantly, our theoretical results indicate that the adaptation difficulty on a novel task can be efficiently measured by relying solely on the TAD metric, without the need for training any models.

We conduct experiments on three benchmark datasets to validate our theoretical results and explore the generality of TAD metric. To this end, we consider two scenarios that the human-annotated attributes are provided or not available. For the latter case, we propose an efficient way to autoannotate attributes. Experimental results show that as the distance between novel and training tasks increases, the performance of various FSL models on novel tasks decreases. With the TAD metric, we also present two potential applications: intervening a small set of difficult novel tasks before training, and investigating the benefits of heterogeneous training tasks for different models.

Our main contributions can be summarized as follows:

We propose Task Attribute Distance (TAD) metric in FSL to quantify the relationship between training and novel tasks, which is model-agnostic and easy to compute.

We provide a theoretical proof of the generalization error bound on a novel task based on TAD, which connects task relatedness and adaptation difficulty theoretically.

We conduct experiments to show TAD can effectively reflect the adaptation difficulty on novel tasks for various FSL methods with human-annotated and auto-annotated attributes.

2 Related works

Few-shot learning. Most recent few-shot learning (FSL) approaches are based on meta-learning framework. These meta-learning based methods can be roughly categorized into two categories. The

metric-based approach [48, 42, 45, 22, 3, 16, 2, 50] aims to learn a cross-task embedding function and predict the query labels based on the learned distances. The optimization-based approach focuses on learning some optimization state, like model initialization [17, 44] or step sizes [28, 5], to rapidly update models with very few labeled samples. Except for these meta-learning methods, transferbased approaches [10, 11] have recently shown that standard transfer learning procedure of early pre-training and subsequent fine-tuning is a strong baseline for FSL with deep backbone networks.

Meta-learning theory. In parallel to the empirical success of meta-learning methods, a series of theoretical works study how meta-learning utilizes the knowledge obtained from the training task data and generalizes to the novel test tasks. Many works [35, 4, 38, 12, 19] give a generalization error bound on novel tasks from the PAC-Bayesian perspective [30, 31]. These works often assume that each task is sampled from a meta task distribution. Under such assumption, the generalization error bound on novel tasks can be reduced with increasing number of training tasks and training samples [4]. Some recent works replace the meta task distribution assumption with other conditions. For example, [8, 14, 47] assume a common representation function between different tasks, then they give the sample complexity bounds for the expected excess risk on a novel task. However, the above works do not quantify the relationship between training and novel tasks, and seldom explore the adaptation difficulty on novel tasks. In this paper, we propose the TAD as a metric to quantify the task relatedness, and provide a new generalization error bound on a novel task.

Task Difficulty. Exploring the difficulty of few-shot tasks is an important research question in FSL, and has been explored from two aspects: measure the difficulty of (1) training tasks and (2) novel tasks. For training tasks, many previous works [44, 29, 43, 6] have attempted to measure their difficulties based on a model's output, such as negative log-likelihood or accuracy, and use this information to sample different tasks to train the model. The sampling strategy based on training task difficulty is similar to the area of hard example mining [41] or curriculum learning [7] that trains a model according to a specific order of samples to improve its generalization ability. For novel tasks, recent works [1, 37, 32, 40, 26, 33] try to measure the adaptation difficulty of novel tasks based on the relationship between novel and training tasks. Task2vec [1] encodes each novel task into an embedding based on the Fisher Information Matrix, and uses the norm of embedding to reflect task difficulty. Other works utilize specific distribution divergence metrics, such as EMD and FID, to quantify the task relatedness, then select the most related training data to fine-tune models [26] or construct more challenging benchmarks to explore the generalization of models [37, 40, 33]. Closer to our work, [37] introduces a transferability score to quantify the transferability of novel tasks via attributes and explore the generalization of FSL models at different transferability scores. However, different with our work, [37] aims to investigate the benefits of self-supervised pretraining with supervised finetuning in the few-shot attribute classification context.

3 Problem Setting

In few-shot learning problem, a model observes n different training tasks { τ i } n i =1 with each task represented as a pair τ i = ( D i , S i ) , 1 ≤ i ≤ n . D i is an unknown data distribution over the input space X and label space Y i . S i = { ( x k , y k ) } m i k =1 , ( x k , y k ) ∼ D i represents an observed training set drawn from D i . With a model trained on the n training tasks, our target is to adapt and evaluate it on t novel test tasks τ ′ j = ( D ′ j , S ′ j ) , 1 ≤ j ≤ t . D ′ j is an unknown data distribution over X and Y ′ j , and S ′ j is a labeled dataset drawn from D ′ j . Note that, in few-shot learning, the labeled data for each category is very limited, and the categories in training tasks { τ i } n i =1 will not appear in novel tasks { τ ′ j } t j =1 , which means Y i ∩ Y ′ j = ∅ , ∀ i ∈ { 1 , ..., n } , j ∈ { 1 , ..., t } .

4 Generalization Analysis of Attribute-based Few-shot Learning

The key to answering the two questions we raised above lies in a proper metric that quantifies task relatedness and contributes to the generalization error on a novel task. In this section, we begin by introducing the Task Attribute Distance (TAD), which first measures the category relationship through attribute conditional distributions, based on which the relationship between tasks is measured as well. Then, to facilitate our theoretical analysis, we introduce a meta-learning framework with attribute learning. Finally, we provide theoretical proof that establishes a connection between the proposed TAD metric and the generalization error bound on a novel task.

4.1 Model-Agnostic Distance Metric via Attributes

To measure the relationship between two categories y k and y t , a natural idea is to measure the divergence between class-conditional distributions p ( x | y k ) and p ( x | y t ) by a divergence measure, like the L 1 or total variation distance (TV) [18]. Unfortunately, this idea is frequently unfeasible, because the set of measurable subsets under distributions p ( x | y k ) and p ( x | y t ) is often infinite, making it hard to compute the distance from finite samples.

In this paper, we introduce attributes to overcome the above limitation because the values of attributes are usually discrete and finite. The basic idea is based on the attribute compositionality hypothesis presented in the introduction section. Specifically, the distance between two categories can be measured by the difference between the corresponding attribute conditional distributions. Follow the above idea, let A be the attribute space spanned by L attribute variables { a l } L l =1 , we define the distance between two categories y k and y t on L attribute variables as:

where B l is the set of measurable subsets under attribute conditional distributions p ( a l | y k ) and p ( a l | y t ) , and B l is a finite set if the values of variable a l are finite.

Based on the distance between categories, we then define the distance metric between training and novel test tasks. To this end, we represent the set of categories in training task τ i and the set of categories in novel task τ ′ j as two disjoint vertex sets respectively, and construct a weighted bipartite graph G between the two vertex sets, where each node denotes a category and each edge weight represents the distance between two categories in τ i , τ ′ j respectively. Let M = { e kt } denote a maximum matching of G , which contains the largest possible number of edges and any two edges do not connect the same vertex. We choose M with minimum weights and define the task distance as

where | M | is the number of edges in matching M . From this definition, if the two tasks τ i , τ ′ j are identical, the task distance d ( τ i , τ ′ j ) is equal to 0. We call the task distance d ( τ i , τ ′ j ) as the Task Attribute Distance (TAD). Additionally, the TAD is invariant to the permutation of categories in tasks. In other words, modifying the numeric order of categories in τ i or τ ′ j does not affect their distance d ( τ i , τ ′ j ) . To emphasize the attribute conditional distributions with respect to different tasks, we hereinafter add task indexes on the distribution and probability notations, like p i ( a l | y k ) and P i a l | y k [ B ] , although the true distributions are task agnostic.

4.2 A Specific Few-Shot Learning Framework

To facilitate the subsequent theoretical analysis, we consider a meta-learning framework with attribute learning. In this framework, a meta-learner f θ : X → A parameterized by θ learns the mapping from a sample x ∈ X to attributes a ∈ A , and a task-specific base-learner g ϕ i : A → Y i parameterized by ϕ i learns the mapping from attributes a ∈ A to a class label y ∈ Y i for training task τ i . During training, f θ is meta-learned from n different training tasks. To adapt to a novel task τ ′ j , we fix f θ and train a new base-learner g ϕ ′ j parameterized by ϕ ′ j using few labeled samples S ′ j .

In the following theoretical deduction, we will denote p θ ( a, y ) ≜ p ( f θ ( x ) , y ) as a joint distribution over A×Y induced by the meta-learned mapping f θ . Further, with the induced conditional distribution p θ ( a | y ) , we can compute the distance between task τ i and τ ′ j as

Note that d ( τ i , τ ′ j ) (as defined in Eq. (2)) computed based on p ( a | y ) is the model-agnostic distance metric, while d θ ( τ i , τ ′ j ) is a model-related one since it relies on the meta-learned mapping f θ .

4.3 Theoretical Analysis on Generalization

As for the above meta-learning framework, we provide theoretical analysis of the generalization error on a novel task, in terms of the proposed TAD metric. We define the generalization error and empirical error of the meta-learner f θ on novel task τ ′ j as

̸

̸

where ϕ ′ j is the parameters of g ϕ learned from labeled samples S ′ j given f θ , and m ′ j = ∣ ∣ S ′ j ∣ ∣ is the number of labeled samples in S ′ j . I denotes the indicator function. Similarly, we can define the generalization error ϵ ( f θ , τ i ) and empirical error ˆ ϵ ( f θ , τ i ) for training task τ i . With these definitions, we will proceed by introducing an auxiliary lemma, and then stating our main theoretical results. All detailed proofs of the following lemma and theorems are in the Appendix.

Lemma 1. Let A be the attribute space, L be the number of attributes. Assume all attributes are independent of each other given the class label, i.e. p ( a | y ) = ∏ L l =1 p ( a l | y ) . For all a i ∈ A and any two categories y k , y t , the following inequality holds:

where d ( y k , y t ) is the distance as defined in Eq. (1) and ∆ = ∑ a i ∈A 1 2 L ∑ L l =1 ( p ( a l i | y k ) + p ( a l i | y t )) .

Lemma 1 says that under the attribute conditional independence assumption, the distance between two attribute conditional distributions p ( a | y k ) , p ( a | y t ) over attribute space A is bounded by the defined distance d ( y k , y t ) and a non-negative term ∆ . This result enables us to make use of the defined TAD metric in deriving the generalization error bound, leading to the following theoretical results.

Theorem 1. With the same notation and assumptions as in Lemma 1, let H be the hypothesis space with VC-dimension d , f θ and g ϕ be the meta-learner and the base-learner as introduced in Sec. 4.2 respectively. Denote g ϕ ∗ as the best base-learner on some specific tasks given a fixed meta-learner f θ . For any single training task τ i = ( D i , S i ) and a novel task τ ′ j = ( D ′ j , S ′ j ) , suppose the number of categories in the two tasks is the same, then with probability at least 1 -δ , ∀ g ϕ ◦ f θ ∈ H , we have

̸

where λ = λ i + λ ′ j is the generalization error of g ϕ ∗ and f θ on the two tasks, i.e., λ i = E ( x,y ) ∼D i [ I ( g ϕ ∗ i ( f θ ( x )) = y )] , λ ′ j = E ( x,y ) ∼D ′ j [ I ( g ϕ ′∗ j ( f θ ( x )) = y )] . ∆ ′ is a term depending on learned base-learners g ϕ i , g ϕ ′ j and the best base-learners g ϕ ∗ i , g ϕ ′∗ j .

̸

Theoretically, the generalization error on a novel task is bounded by the the training task empirical error ˆ ϵ ( f θ , τ i ) plus four terms: the second term d θ ( τ i , τ ′ j ) is the model-related distance between τ i and τ ′ j , which is derived based on Lemma 1; the third term ∆ ′ reflects the classification ability of g ϕ , which converges to zero if the learned base-learners are equal to the best ones for both tasks; the last term λ is the generalization error of f θ and g ϕ ∗ , which depends on the attribute discrimination ability of f θ and the hypothesis space of base-learner g ϕ . For a reasonable hypothesis space, if f θ has a good attribute discrimination ability on both tasks, the last term usually converges to zero. Next we generalize this bound to the setting of n training tasks.

Corollary 1. With the same notation and assumptions as Theorem 1, for n training tasks { τ i } n i =1 and a novel task τ ′ j , define ˆ ϵ ( f θ , τ n i =1 ) = 1 n ∑ n i =1 ˆ ϵ ( f θ , τ i ) , then with probability at least 1 -δ , ∀ g ϕ ◦ f θ ∈ H , we have

where λ = 1 n ∑ n i =1 λ i + λ ′ j , and ∆ ′ is a term depending on the learned base-learners { g ϕ i } n i =1 , g ϕ ′ j and the best base-learners { g ϕ ∗ i } n i =1 , g ϕ ′∗ j .

Corollary 1 is a straightforward extension of Theorem 1, in which we consider multiple training tasks instead of a single training task. In Corollary 1, the generalization error on a novel task τ ′ j is bounded partially by the average distance between task τ ′ j and n training tasks. Note that the distance d θ ( τ i , τ ′ j ) used in the bound is model-related. Next, we further derive the relationship between model-related distance and model-agnostic distance as follows.

Theorem 2. With the same notation and assumptions as in Corollary 1, assume that the conditional distribution p ( x | a l ) is task agnostic. If the number of labeled samples m i in n training tasks and the number of labeled samples m ′ j in novel task τ ′ j tend to be infinite, the following equality holds:

Theorem 2 shows that when the number of labeled samples in n training task and a novel task tends to be infinite, the model-related average distance can be bounded by the average TAD. Based on the universal approximation theorem [21, 15] and the same assumption as in Theorem 2, if g ϕ ◦ f θ is a complex enough network, the second and fourth terms on the r.h.s. of Eq. (6) in Corollary 1 will converge to 0 , the first term ˆ ϵ ( f θ , τ n i =1 ) and 1 n ∑ n i =1 λ i will converge to Bayes risk [13]. We assume the Bayes risk is small enough, so that it can be ignored. In this case, the generalization error on novel task τ ′ j is only bounded by λ ′ j and the average TAD, i.e. , ϵ ( f θ , τ ′ j ) ≤ λ ′ j + 1 n ∑ n i =1 d ( τ i , τ ′ j ) . Note that the average TAD is independent of models, thus we can rely solely on the TAD metric to measure the adaptation difficulty of each novel task, without the need for training any models.

5 Computing the Model-Agnostic Task Distance

5.1 Distance Approximation

In this section, we discuss practical considerations in computing the TAD on real data, where the attributes often take on discrete and finite values. For continuous and infinite attributes, it is possible to divide their values into many segments or discrete parts. Therefore, we denote V l as a finite set of possible values of attribute a l , and re-express the TAD in Eq. (2) as

Computing the above distance is challenging as it requires finding the maximum matching M with minimum weights, which is a combinatorial optimization problem. The Hungarian algorithm [25] is commonly used to solve the matching problem, but it is computationally expensive. Due to the high computational cost, we do not calculate this distance directly but estimate the approximation instead:

where we assume all tasks have C categories, which holds in most FSL settings. In Eq. (9), we simplify the comparison of attribute conditional distributions between two tasks by replacing the individual differences with the average difference. Through the above approximation, we can avoid seeking the minimum weight perfect matching M and estimate the TAD efficiently.

5.2 Attributes Auto-Annotation

As stated before, our proposed theory and TAD metric rely on category-level attribute annotations that are possibly unavailable for some few-shot benchmarks, like mini ImageNet [48]. To address this issue, we propose a solution to auto-annotate attributes using a pretrained CLIP [36] model, which has demonstrated impressive zero-shot classification ability. We first pre-define 14 attribute labels based on pattern, shape, and texture. Then we create two descriptions for each attribute, such as 'a photo has shape of round' and 'a photo has no shape of round', which are used as text inputs for CLIP. We formulate the annotation problem as 14 attribute binary classification problems for each image using CLIP model. After that, we gather the attribute predictions of all images within the same category, which provides rough category-level attribute information while greatly reduces the cost of attribute annotations. With the above auto-annotation process, we can obtain the category-level attributes for mini ImageNet in just 4 minutes.

6 Experiments

In this section, we conduct experiments to validate our theoretical results and explore the generality of the proposed TAD metric in measuring task adaptation difficulty. Firstly, we evaluate our TAD metric in a scenario where human-annotated attributes are available and the training/novel tasks are sampled from the same dataset but with different categories. Next, we test the TAD in a more general and challenging scenario where human-annotated attributes are not available and the training tasks are constructed by sampling categories from different datasets.

6.1 Setups

Datasets. We choose three widely used benchmarks: (1) CUB-200-2011 ( CUB ) [49]: CUB is a fine-grained dataset of birds, which has 200 bird classes and 11,788 images in total. We follow [23] to split the dataset into 100 training classes, 50 validation classes and 50 test classes. As a fine-grained dataset, CUB provides part-based annotations, such as beak, wing and tail of a bird. Each part is annotated by a bounding box and some attribute labels. Because the provided attribute labels are noisy, we denoise them by majority voting, as in [24]. After the pre-processing, we acquire 109 binary category-level attribute labels. (2) SUN with Attribute ( SUN ) [34]: SUN is a scene classification dataset, which contains 14,340 images for 717 scene classes with 102 scene attributes. In following experiments, We split the dataset into 430/215/72 classes for training/validation/test, respectively. (3) mini ImageNet [48]: mini ImageNet is a subset of ImageNet consisting of 60,000 images uniformly distributed over 100 object classes. Note that, different with CUB and SUN, mini ImageNet does not provide the attribute annotations, which means category-level attribute annotations are not available. Following [10], we consider the cross-dataset scenario from mini ImageNet to CUB, where we use 100 classes of mini ImageNet as training classes, and the 50 validation and 50 test classes from CUB.

Attribute Prototypical Network. Our theoretical analysis is based on a specific meta-learning framework with attribute learning (proposed in Sec. 4.2). Thus, we first instantiate a simple model under that framework as an example to verify our theory. Specifically, we adopt a four-layer convolution network (Conv-4) with an additional MLP as the meta-learner f θ . The convolutional network extracts feature representations from images, then the MLP takes features as input and predicts attribute labels. For base-learner g ϕ i parameterized by ϕ i , we simply choose an nonparametric base-learner like ProtoNet [42], which takes the attributes outputted by f θ as input to calculate cosine distance between test samples and attribute prototypes, then predicts the target label. We call this method as Attribute Prototypical Network (APNet). We train APNet by simultaneously minimizing the attribute classification loss and the few-shot classification loss.

Other FSL Methods. Besides of APNet, we choose five classical FSL methods in the following experiments, since they cover a diverse set of approaches to few-shot learning: (1) Matching Network (MatchingNet) [48], (2) Prototypical Network (ProtoNet) [42], (3) Relation Network (RelationNet) [45], (4) Model Agnostic Meta-Learning (MAML) [17] and (5) Baseline++ [10]. Note that these FSL methods, unlike the above APNet, do not use attribute annotations during training. See more implementation and experimental details in the Appendix.

6.2 Task Attribute Distance with Human-annotated Attributes

Approximate results with limited samples. According to Theorem 2, if the number of labeled samples in tasks tends to be infinite, the TAD serves as a metric to measure the adaptation difficulty on a novel task. However, it is impossible that each task contains infinite labeled samples in real-world applications. Thus, we try to empirically verify the approximate theoretical result with limited labeled samples for each task. We train our APNet on CUB and SUN dataset, then use the provided attribute annotations to calculate the average distance 1 n ∑ n i =1 d ( τ i , τ ′ j ) between each novel task τ ′ j and n training tasks { τ i } n i =1 , respectively. For simplicity, we sample n = 100 , 000 training tasks to estimate the distances to 2 , 400 novel tasks. Following N -way K -shot setting, each task only contains N ∗ K labeled samples for training or fine-tuning models. Fig. 4 shows the task distance and the corresponding accuracy on novel tasks for APNet. We can observe in Fig. 4 that as the distance increases, the accuracy of APNet decreases in both 5-way 1-shot and 5-way 5-shot settings on CUB and SUN. These results verify that TAD can characterize model's generalization error on each novel task and measure the task adaptation difficulty effectively, even with limited labeled samples for

0.14

0.15

0.16

0.17

0.18

0.19

0.20

Distance

55

60

65

70

75

Accuracy (%)

CUB 1-shot

(a)

0.14

0.15

0.16

0.17

0.18

0.19

0.20

Distance

76

78

80

82

84

86

88

CUB 5-shot

(b)

0.16

0.18

0.20

0.22

0.24

Distance

45

50

55

60

65

70

75

SUN 1-shot

(c)

0.16

0.18

0.20

0.22

0.24

Distance

60

65

70

75

80

85

SUN 5-shot

(d)

0.14

0.15

0.16

0.17

0.18

0.19

0.20

Distance

50

55

60

65

Accuracy (%)

CUB 1-shot

0.14

0.15

0.16

0.17

0.18

0.19

0.20

Distance

62.5

65.0

67.5

70.0

72.5

75.0

77.5

80.0

82.5

CUB 5-shot

0.16

0.18

0.20

0.22

Distance

45

50

55

60

65

SUN 1-shot

0.16

0.18

0.20

0.22

Distance

62.5

65.0

67.5

70.0

72.5

75.0

77.5

80.0

82.5

SUN 5-shot

ProtoNet

RelationNet

MAML

Baseline++

MatchingNet

training and fine-tuning models. Note that the confidence interval in Fig. 4 is much larger for the last a few points. We argue this is because these distance intervals contain fewer novel tasks, thus the average accuracy of novel tasks in the interval is more easily affected by random factors.

The number of labeled samples. Additionally, in Fig. 4, when comparing 1-shot and 5-shot results with the same distance interval, we find that the increase of accuracy varies at different distance intervals. For instance, on the CUB dataset (comparing Fig. 2(a) with Fig. 2(b)), when the task distance is 0.14, APNet shows an improvement of approximately 10% in accuracy for the 5-shot setting over the 1-shot setting, whereas it shows an improvement of around 15% for the distance of 0.20. This suggests that increasing the number of labeled samples is more effective for harder tasks. One possible explanation is that as the task distance increases, less knowledge can be transferred from training tasks, making it harder for models to adapt to the novel task. Hence, more information from the novel task is required for adaptation, and the model's performance can get more benefit from additional labeled samples.

Other FSL models. We have shown that the average TAD can effectively reflect the task adaption difficulty for APNet. A natural question is whether the calculated task distance can be directly applied to other FSL methods, even if they do not follow the specific meta-learning framework and do not use the attributes during training. We try to empirically explore this question. With the same distance estimation and experimental setting, we conduct experiments with five different FSL methods on CUB and SUN. Fig. 3 shows the task distance and the corresponding accuracy of 2,400 novel test tasks for them. We observe similar results that with the increase of task distance, the accuracy of all FSL models tends to decrease. This indicates that even though these FSL methods do not use the attribute annotations during training, they implicitly learn mixture of attributes in their representations. Therefore, attribute-based distance can still reflect the task adaptation difficulty for them. These results demonstrate the generality of the proposed TAD metric, and provide some insight into the transferable knowledge that different FSL models have learned.

6.3 Task Attribute Distance with Auto-annotated Attributes

Cross-dataset generalization. Another interesting question is how to utilize the proposed TAD metric in situations where human-annotated attributes are either unavailable or expensive to obtain. One classic example is the cross-domain few-shot learning problem, since it requires the annotation

0.20

0.22

0.24

0.26

0.28

0.30

0.32

0.34

Distance

40.0

42.5

45.0

47.5

50.0

52.5

55.0

57.5

Accuracy (%)

All attributes

ProtoNet

RelationNet

MatchingNet

MAML

Baseline++

(a)

0.22

0.24

0.26

0.28

0.30

0.32

0.34

Distance

42

44

46

48

50

52

54

Accuracy (%)

w/o pattern

ProtoNet

RelationNet

MatchingNet

MAML

Baseline++

(b)

0.20

0.25

0.30

0.35

Distance

40

42

44

46

48

50

52

54

56

Accuracy (%)

w/o shape

ProtoNet

RelationNet

MatchingNet

MAML

Baseline++

(c)

0.26

0.28

0.30

0.32

0.34

0.36

0.38

Distance

42

44

46

48

50

52

54

56

58

Accuracy (%)

w/o texture

ProtoNet

RelationNet

MatchingNet

MAML

Baseline++

(d)

of distinct datasets with a common attribute set. To address this challenge, we use the auto-annotation method described in previous section to annotate the attributes for both mini ImageNet and CUB. We then train the above FSL models on the cross-dataset scenario from mini ImageNet to CUB, and estimate the average task distance with the auto-annotated attributes. Fig. 4(a) illustrates the distance and corresponding accuracy of 2,400 novel test tasks with the auto-annotated attributes. We can find that, with the increase of task distance, the accuracy of different models tends to decrease. This phenomenon is consistent with our previous findings and shows the proposed TAD metric still works with auto-annotated attributes. Besides, we selectively remove some attributes to explore the influence of them in the distance-accuracy curve. Figs. 4(b) to 4(d) show that the exclusion of pattern, shape, and texture attributes exhibits varying degrees of influence on the decreasing tendency. Notably, we discover that texture attributes are more importance than others, as indicated by the more pronounced fluctuations in the curve.

ProtoNet

RelationNet

MatchingNet

MAML

Baseline++

20

25

30

35

40

45

Accuracy (%)

30.38

35.37

32.54

33.20

31.20

40.00

42.70

43.06

38.01

45.96

w/o intervention

w intervention

Test-time Intervention. With the proposed TAD metric, we can quantify the adaptation difficulty of novel test tasks without training and adapting a model. This makes sense in real-world scenarios, as we can identify more challenging tasks before training and take interventions to improve a model's performance on those tasks. In this part, we explore a simple test-time intervention operation that supplies more labeled samples for harder novel tasks. To construct the intervention experiment, we first calculate the distances between each novel task and n training tasks, as done in previous experiments. We manually set a constant threshold value, denoted as r , to identify the harder tasks that exhibit large distances and generally yield low accuracy. Once we identify these challenging tasks, we intervene by providing additional labeled samples, which contain more task-specific information. Specifically, in the 5-way 1-shot setting, we offer 4 extra labeled samples for each intervened task, which can be seen as the additional annotated samples. We set the distance threshold r = 0 . 29 for the cross-dataset scenario. With the above threshold values, we only need to intervene with a small subset of novel tasks (approximately 5%-7%), which greatly reduces the cost of sample annotation. The results of the test-time intervention can be seen in Fig. 5. As illustrated in Fig. 5, the test-time intervention can significantly improve the performance of different FSL models on the challenging novel tasks. Note that we are not proposing a novel intervention method, but demonstrating a practical application of the proposed TAD metric.

Training with partial versus all tasks. Using the proposed TAD, we can also investigate whether training on all available training tasks leads to better generalization compared to training on only the most related ones. While it is generally expected that training with more tasks helps to improve generalization, it remains a question whether this holds for heterogeneous data (e.g. cross-dataset scenario). To explore this, we follow the cross-dataset experiment setup from mini ImageNet to CUB, and select these less related training tasks based on their calculated task distances. We collect the classes in the selected tasks, compute the frequency of each class, and remove the most frequent 36 classes (out of a total of 100 classes in mini ImageNet). We then reconstruct training tasks based on the remaining 64 classes in mini ImageNet and retrain five FSL models. The results are presented in Tab. 1. It can be observed that incorporating more heterogeneous data that is less related to the novel tasks may not lead to much improvement ( for ProtoNet, RelationNet and MAML), but could even result in performance degradation ( for MatchingNet and Baseline++). This result may inspire future research on how to better utilize heterogeneous data to improve the performance of models in the cross-domain few-shot learning problem.

7 Conclusion

We propose a novel distance metric, called Task Attribute Distance (TAD), in FSL to quantify the relationship between training and novel tasks, which only relies on the category-level attribute annotations. We present a theoretical analysis of the generalization error bound on a novel task with TAD, which connects task relatedness and adaptation difficulty theoretically. Our experiments demonstrate TAD can effectively reflect the task adaptation difficulty for various FSL methods, even if some of them do not learn attributes explicitly or human-annotated attributes are not available. We also present two potential applications of our TAD metric: intervening a small set of difficult novel tasks before training, and investigating the benefits of heterogeneous training tasks. We believe our theoretical and empirical analysis can provide more insight into few-shot learning and related areas.

Limitations and broader impact. Our theory and proposed TAD metric assume that attributes are conditionally independent without considering the correlations that may exists between them. Furthermore, identifying which attributes are critical in the distance metric is still an open question. However, we believe that our work lays a solid foundation for measuring task relatedness and adaptation difficulty of novel tasks, which offers a starting point for further research in few-shot learning and related areas.

Acknowledgments

This work is partially supported by National Key R&D Program of China no. 2021ZD0111901, and National Natural Science Foundation of China (NSFC): 61976203 and 62276246.

References

Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless C Fowlkes, Stefano Soatto, and Pietro Perona. Task2vec: Task embedding for metalearning. In ICCV , 2019.

Arman Afrasiyabi, Hugo Larochelle, Jean-François Lalonde, and Christian Gagné. Matching feature sets for few-shot image classification. In CVPR , 2022.

Kelsey Allen, Evan Shelhamer, Hanul Shin, and Joshua Tenenbaum. Infinite mixture prototypes for few-shot learning. In ICML , 2019.

Ron Amit and Ron Meir. Meta-learning by adjusting priors based on extended PAC-Bayes theory. In ICML , 2018.

Antreas Antoniou, Harri Edwards, and Amos Storkey. How to train your MAML. In ICLR , 2019.

Sébastien Arnold, Guneet Dhillon, Avinash Ravichandran, and Stefano Soatto. Uniform sampling over episode difficulty. 2021.

Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In ICML , 2009.

Tianshi Cao, Marc T Law, and Sanja Fidler. A theoretical analysis of the number of shots in few-shot learning. In ICLR , 2020.

Susan Carey and Elsa Bartlett. Acquiring a single new word. Papers and Reports on Child Language Development , 15:17-29, 1978.

Weiyu Chen, Yencheng Liu, Zsolt Kira, Yuchiang Frank Wang, and Jiabin Huang. A closer look at few-shot classification. In ICLR , 2019.

Yinbo Chen, Zhuang Liu, Huijuan Xu, Trevor Darrell, and Xiaolong Wang. Meta-baseline: Exploring simple meta-learning for few-shot learning. In ICCV , 2021.

Nan Ding, Xi Chen, Tomer Levinboim, Sebastian Goodman, and Radu Soricut. Bridging the gap between practice and PAC-Bayes theory in few-shot meta-learning. In NeurIPS , 2021.

Geoff Dougherty. Pattern recognition and classification: an introduction . Springer Science & Business Media, 2012.

Simon Shaolei Du, Wei Hu, Sham M Kakade, Jason D Lee, and Qi Lei. Few-shot learning via learning the representation, provably. In ICLR , 2021.

András Faragó and Gábor Lugosi. Strong universal consistency of neural network classifiers. IEEE Transactions on Information Theory , 39:1146-1151, 1993.

Nanyi Fei, Yizhao Gao, Zhiwu Lu, and Tao Xiang. Z-score normalization, hubness, and few-shot learning. In ICCV , 2021.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML , 2017.

Alison L Gibbs and Francis Edward Su. On choosing and bounding probability metrics. International statistical review , 70:419-435, 2002.

Jiechao Guan and Zhiwu Lu. Fast-rate PAC-Bayesian generalization bounds for meta-learning. In ICML , 2022.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In NeurIPS , 2017.

Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural networks , 2:359-366, 1989.

Ruibing Hou, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen. Cross attention network for few-shot classification. In NeurIPS , 2019.

Dahyun Kang, Heeseung Kwon, Juhong Min, and Minsu Cho. Relational embedding for few-shot classification. In ICCV , 2021.

Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In ICML , 2020.

Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly , 2:83-97, 1955.

Cat Phuoc Le, Juncheng Dong, Mohammadreza Soltani, and Vahid Tarokh. Task affinity with maximum bipartite matching in few-shot learning. In ICLR , 2022.

SuBeen Lee, WonJun Moon, and Jae-Pil Heo. Task discrepancy maximization for fine-grained few-shot classification. In CVPR , 2022.

Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few-shot learning. arXiv , 2017.

Chenghao Liu, Zhihao Wang, Doyen Sahoo, Yuan Fang, Kun Zhang, and Steven CH Hoi. Adaptive task sampling for meta-learning. In ECCV , 2020.

David A McAllester. Some PAC-Bayesian theorems. Machine Learning , 37:355-363, 1998.

David A McAllester. PAC-Bayesian model averaging. In COLT , 1999.

Timo Milbich, Karsten Roth, Samarth Sinha, Ludwig Schmidt, Marzyeh Ghassemi, and Bjorn Ommer. Characterizing generalization under out-of-distribution shifts in deep metric learning. In NeurIPS , 2021.

Jaehoon Oh, Sungnyun Kim, Namgyu Ho, Jin-Hwa Kim, Hwanjun Song, and Se-Young Yun. Understanding cross-domain few-shot learning based on domain similarity and few-shot difficulty. In NeurIPS , 2022.

Genevieve Patterson, Chen Xu, Hang Su, and James Hays. The sun attribute database: Beyond categories for deeper scene understanding. International Journal of Computer Vision , 108:59-81, 2014.

Anastasia Pentina and Christoph Lampert. A PAC-Bayesian bound for lifelong learning. In ICML , 2014.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML , 2021.

Mengye Ren, Eleni Triantafillou, Kuan-Chieh Wang, James Lucas, Jake Snell, Xaq Pitkow, Andreas S Tolias, and Richard Zemel. Probing few-shot generalization with attributes. arXiv , 2020.

Jonas Rothfuss, Vincent Fortuin, Martin Josifoski, and Andreas Krause. PACOH: Bayes-optimal meta-learning with PAC-guarantees. In ICML , 2021.

Yossi Rubner, Carlo Tomasi, and Leonidas J Guibas. A metric for distributions with applications to image databases. In ICCV , 1998.

Mert Bulent Sariyildiz, Yannis Kalantidis, Diane Larlus, and Karteek Alahari. Concept generalization in visual representation learning. In ICCV , 2021.

Abhinav Shrivastava, Abhinav Gupta, and Ross Girshick. Training region-based object detectors with online hard example mining. In CVPR , 2016.

Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In NeurIPS , 2017.

Qianru Sun, Yaoyao Liu, Zhaozheng Chen, Tat-Seng Chua, and Bernt Schiele. Meta-transfer learning through hard tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44:1443-1456, 2020.

Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele. Meta-transfer learning for few-shot learning. In CVPR , 2019.

Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In CVPR , 2018.

Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evci, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, et al. Meta-dataset: A dataset of datasets for learning to learn from few examples. In ICLR , 2020.

Nilesh Tripuraneni, Chi Jin, and Michael Jordan. Provable meta-learning of linear representations. In ICML , 2021.

Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In NeurIPS , 2016.

Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.

Jiangtao Xie, Fei Long, Jiaming Lv, Qilong Wang, and Peihua Li. Joint distribution matters: Deep brownian distance covariance for few-shot classification. In CVPR , 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new metric, the Task Attribute Distance (TAD), to quantify the link between two tasks for FSL. The distance is based on the relationship between some pre-defined attributes and classes in the two tasks. This distance can be computed from a model point of view, considering the distribution of attributes in the features space of a given model, or in a model-agnostic way. The paper presents two main theoretical results about TAD:\n- first, showing the link between TAD from a model computed on training tasks and a given novel task, and the generalization error on this novel task.\n- second, the link between the model-related TAD and the model-agnostic one.\nThen the paper discusses how to empirically estimate the model-agnostic TAD, and how to create attributes automatically for datasets without ones, using a pretrained CLIP model.\nFinally, the paper provides experiments on two datasets with attributes labels (SUN and CUB), and one with attributes from auto-annotation (mini-imagenet). The results show that the task distance is correlated with the performance of the models. A final experiment shows that some models benefits from removing frequent classes in training.\n\nSTRENGTHS:\n- *Originality:* the metric presented is novel and represents an interesting way to measure task difficulty. It uses information about attributes related between the tasks, as opposed to previous works which mainly used model's outputs. \n- *Quality:* The claims are all supported, and the paper presents theoretical results with proofs to link the TAD metric with generalization bounds. The experimental results show the correlation between the distance and the performance. I took a look at the proofs and the results are sounded.\n- *Clarity:* The paper is clearly written and easy to follow.\n\nWEAKNESSES:\nThe main weakness of the paper is the significance and the quality of the analysis regarding the attributes automatically obtained. \n- I found the metric interesting because it can be computed in a model-agnostic way, but its main problem is that it relies on attributes that are not always available on datasets. Even though the paper explains a way to automatically annotate with attributes, an evaluation of the quality of the obtained attributes is missing. We can even see on Figure 4 that the correlation between the obtained attributes and the performance is less clear than on other datasets. It would also be interesting to show more examples of the obtained attributes, for a qualitative analysis. Because it relies too much on attributes, and on their quality, I am afraid this metric will be of limited impact.\n- I think the paper is missing an application of the proposed metric in a setting where it could be useful. Besides the experiments showing the link between the distance and the performance, I'm not really convinced by the last experiment on training with less training tasks. I don't think it is a good representative of potential applications.\n\nQUESTIONS:\n- I would like to see preliminary results for potential applications of this metric to assess the significance of the work. \n- What are the attributes obtained for mini-Imagenet ? Are they sounded ?\n\nLIMITATIONS:\nThe authors have included a discussion about the limitations of their works. I don't think that a discussion on societal impact is needed for this work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work raises two questions regarding the setting of few-shot classifications: (1) How to evaluate the difference between training and novel tasks? (2) How does the difference impact FSL methods? For the first question, the authors proposed Task Attribute Distance (TAD), which uses attributes from CLIP to evaluate task differences. For the second question, the author derives upper bounds for the task and show some empirical evaluation.\n\nSTRENGTHS:\nThe author provides some theoretical insight into the difficulty of few-shot classifications with clear writing.\n\nWEAKNESSES:\n1. It is unclear how the manually selected 25 attribute impact the performance. The author also admits that the 25 pre-defined attributes are not comprehensive enough for the experiment in Sec 6.3. \n2. It is not very intuitive how this work can provide insight into few-shot learning areas. For example, for the experiments conducted in current few-shot papers, what is their average TAD? Can we tell the difficulty of these tasks based on their TAD? If some settings have TAD larger than a threshold, can we tell that these settings are not practical to do few-shot classification at all? It will be much more impactful if the author can point out problems or expectations of the few-shot learning area based on their theorem.\n3. Minor errors: Line 291, ProtoNet calculated Euclidean distance. It is MatchingNet that calculates cosine distance.\n\nQUESTIONS:\n1. It is interesting that Figure 2 shows an almost linear response between distance and accuracy. Is this expected from the theory?\n2. Comparing Figures 2 and 3, it seems that the accuracy of APNet is generally better than other few-shot methods. Does that mean the APNet, which uses additional information from CLIP models, can be considered as an upper-bound method so we know which tasks have reached their best performance and we should not further push it? Can the authors evaluate it?\n\nLIMITATIONS:\nThe author addressed the limitation in the selection of attributes, as I mentioned in the first point in the weakness section. Also, while the author raises several interesting observations and tries to derive the theory behind it, there is still some gap between the theory and the experiments, as I mentioned in the questions sections.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new metric called TAD (Task Attribute Distance) to measure the distance between two tasks which is expected to serve as a proxy for how difficult the target task is in comparison to the source pre-training data and also provide an approximation of the few-shot classification performance of a given task. The proposed method works by calculating a set of attributes for each class per dataset and then by computing an approximation of the maximum bipartite matching score between the classes from the source and target tasks. Authors provide theoretical justification related to the generalization bounds and also provide some empirical results.\n\nSTRENGTHS:\n* The proposed method has a unique viewpoint -- instead of measuring task distance by using gradients or the fisher information matrix like some other methods, using a set of attributes as the anchors and computing distance based on the learned values of those attributes is a novel idea.\n\n* Given that not all datasets come with a set of pre-defined attributes, authors provide a practical backup solution to define a set of prompts per dataset where such attributes are not available and use CLIP to determine to get a score for each attribute -- after that, the original algorithm can be used.\n\n* The paper provides theoretical guarantees regarding the generalization error on a new task in terms of the proposed metric.\n\n* I appreciate the effort from authors to reduce the computational complexity of the bipartite matching by replacing the `hungarian` algorithm with a reasonable approximation.\n\nWEAKNESSES:\n* The practicality of this metric is questionable -- it needs a somewhat large set of attributes per dataset and for most of the datasets, such attributes are not available and it would require users to generate a set of prompts and then feed the data through a CLIP model and perform trial and error before finding the right set of attributes.\n\n* Although in the related works section, the paper mentions a few works from the literature, I do not see any direct comparison between this metric and those in the experimental setup. It is hard to understand how good this metric is compared to other metrics.\n\n* The paper here misses comparing against (both method wise and experiment wise) against a relevant paper called Task2Vec - https://arxiv.org/abs/1902.03545. \n\n* The proposed metric is heavily tied to meta-learning algorithms and also where the number of ways (in K-way N-shot) have to be fixed between meta-train and test - which means to use this metric, one needs to train multiple meta learners with different number of ways. Please let me know if my understanding is incorrect.\n\nQUESTIONS:\n* How will this method work if there is partial overlap in terms of classes between train and test? Is it a strict condition that the classes have to be disjoint between train and test?\n\n* How will the attribute prediction work for domains where CLIP may not work well?\n\n* Why does the attributes have to be dataset specific and defined apriori? Can it be something that is learnable too?\n\nFor other questions, please see my comments under weaknesses section.\n\nLIMITATIONS:\nAuthors mentioned some limitations of their work in the paper and those seem reasonable.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new metric--Task Attribute Distance (TAD)--to measure the distance between training and novel tasks in few-shot learning. TAD, based on pre-defined attributes of images, calculates a minimum matching between classes in training and novel tasks, where each edge in the bipartite graph is represented by the average TV distance between each two categories on all attribute variables. Using a meta-learning framework, the authors give a theoretical guarantee of the positive connection between TAD and adaptation difficulty. Experiments verify that an increased TAD can indeed lead to decreased few-shot learning performance, hence increased adaptation difficulty.\n\nSTRENGTHS:\n-  How to measure task similarity is central to few-shot learning problems. Using attributes to calculate task similarity is intuitive and makes sense to me, and has the advantage of not relying on specific models.\n- It is clever to utilize CLIP to automatically annotate attributes.\n- The theoretical analysis gives a guarantee of the effectiveness of the proposed metric.\n- Experiments on in-domain and cross-domain few-shot learning problems verify that the proposed metric can indeed serve as a surrogate of task adaptation difficulty, independent of the model used.\n- The writing is clear and easy to follow.\n\nWEAKNESSES:\n- The proposed metric only considers the distance between categories of two tasks, but not considers the relationship among categories inside each task. Thus some important characteristics of a task, like finegrainedness, are not taken into account.\n- In Theorem 1, VC dimension can go to a large value for neural networks, so the second term is not negligible, and the whole bound is thus not tight.\n- Theorem 2 and the following analysis depend on the assumption that the number of samples in the support set goes to infinity, thus not applied to few-shot learning, deviating from the main topic of the paper.\n- The theorem can be applied only to the hand-crafted meta-learning algorithm.\n- It seems to differ a lot switching from minimum matching to average difference.\n- Automatic annotation may lead to wrong definitions and assignments of attributes. This problem manifests in Figure 4.\n\nQUESTIONS:\n- Why consider the TV distance?\n- What if decreasing distance further in Figure 2-3, for example, below 0.1?\n\nLIMITATIONS:\nThe authors have adequately stated the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}