Hard-Meta-Dataset++: Towards Understanding Few-Shot Performance on Difficult Tasks
Authors: Samyadeep Basu
Megan Stanley
John F Bronskill
Soheil Feizi
Daniela Massiceti
Venue: ICLR 2023 poster (2023)
Keywords: Few-shot learning
Meta-Dataset
Benchmarks
Evaluation

Abstract: Few-shot classification is the ability to adapt to any new classification task from only a few training examples. The performance of current top-performing few-shot classifiers varies widely across different tasks where they often fail on a subset of `difficult' tasks.
This phenomenon has real-world consequences for deployed few-shot systems where safety and reliability are paramount, yet little has been done to understand these failure cases. In this paper, we study these difficult tasks to gain a more nuanced understanding of the limitations of current methods. To this end, we develop a general and computationally efficient algorithm called FastDiffSel to extract difficult tasks from any large-scale vision dataset. Notably, our algorithm can extract tasks at least 20x faster than existing methods enabling its use on large-scale datasets. We use FastDiffSel to extract difficult tasks from Meta-Datasset, a widely-used few-shot classification benchmark, and other challenging large-scale vision datasets including ORBIT, CURE-OR and ObjectNet. These tasks are curated into Hard-MD++, a new few-shot testing benchmark to promote the development of methods that are robust to even the most difficult tasks. We use Hard-MD++ to stress-test an extensive suite of few-shot classification methods and show that state-of-the-art approaches fail catastrophically on difficult tasks. We believe that our extraction algorithm FastDiffSel and Hard-MD++ will aid researchers in further understanding failure modes of few-shot classification models.

Published as a conference paper at ICLR 2023
HARD-META-DATASEHH TOWARDS UNDERSTAND- ING FEW-SHOT PERFORMANCE ON DIFFICULT TASKS
Samyadeep Basu ; Megan Stanley, John Bronskill, Soheil Feizi, Daniela Massiceti {abasul2, sfeizi]{umd. edu, [j ac uk
(meganstanley, dmassicet htwrrieeeeeemmmmoooosottoeo femicrosoft.....m com
ABSTRACT
Few-shot classification is the ability to adapt any new classification task from only few training examples. The performance of current top-performing few- shot classifiers varies widely across different tasks where they often fail on a sub- set of 'diffiiull" tasks. This phenomenon has weal-world consequences for de- ployed few-shot systems where safety and reliability are paramount, yet little has been done to understand these failure cases. In this paper, we study these difficult tasks to gain a more nuanced understanding of the limitations of current meth- ods. To this end, we develop general and computationally efficient algorithm called FASTDIFFSEL to extract difficult tasks from any large-scale vision dataset. Notably, our algorithm can extract tasks at least 20x faster than existing methods enabling its use on lcaee-scale datasets. We use FASTDIFFSEL to extract difficult tasks from METAADA widely-used few-shot classification benchmark, and other challenging large-scale vision datasets including ORBIT, CURE-OR and OB- JECTNET. These tasks are curated into HARD-META HARD-META-DATAASETT++ shot testing benchmark to promote the development of methods that are robust to even the most difficult tasks. We use WEDD-META-DDTASETT extensive suite of few-shot classification methods and show that state-of-the-art approaches fail catastrophically on difficult tasks. We believe that our extraction algorithm FASTDIFFSEL and further understanding failure modes of few-shot classification models.
new few-
to stress-test an
will aid researchers in
INTRODUCTION
Few-shot classification is the ability to distinguish between a set of novel classes when given only a few labelled training examples of each class (Lake et al.] 201 Fei-Fei et al. 2006). This holds potential across many real- world applications from robots that can identifyy new objects (Ren al. 2020), to drug discovery pipelines that can predict the properties of new molecules (Stanley et al, 2021 A few-shot image classifier is given a few labelled training images of the new object classes, called the support set. Once the classifier has adapted to this support set, it is then evaluated on novel test images of those classes, called the query set. Together, the support and query is called a task.
Recent years have seen rapid progress in few-shot image classification (Snell et al. 2017 Finn et al., 2017b Ye al. 2020 Li et al. 2021 Radffrd et al. 2021 Kolesnikov al.) 2019), however, current top-performing methods display a wide range in performance over different tasks at test time (Fu al. [022 Agarwal al. (02TT. On META-DATASET (MD), widely-used few-shot classification benchmark (Triantafillou et al. 2019), tftt--oftthearr classifiers obtain accuracies as low as 22% on some individual tasks though their average task accuracy is >55% (see Fig []. Few works have undertaken detailed examination of these 'difficult' tasks, yet they remain critical to interrogate for both future algorithmic development and the safety and reliability of deployed systems.
This paper aims to gain more nuanced understanding of these 'difficult" tasks and the limitations of current methods. We define difficult task as one on which a few-shot classifier performs poorly on the task's 's query set, after being adapted to its support set. Current methods for finding supports sets that lead to poor query performance rely on greedy search- based algorithms (Agarwal et al. 202 These approaches, however, incur high computational cost when sampling for a large numbers of
Work done partly during research internship Microsoft Research, Cambridge (UK)

---

Published as a conference paper at ICLR 2023
Hard-MD
MDD
00-
Variable wway. Variable shot Accuracy
60-
40-
20-
WISCOCO
bratic pgg
Hovre 2012
omniglot
aircrutt
ou J.luddd
tumel
Dower
qpickarrat
Dommins
Figure 1: A state-of-the-aat method (Hu et al., 2022) performs consistently worse on dif- ficult tasks in the MD split (HARD-MD) of HARD-MEETA-DATASET+ compared to tasks in META-DATASET (MD) across all 10 MD sub-datasets. The method uses ViT-S initialized with self-supervised DINO weights and is further meta-trained with ProtoNets on MD's ilsvre_2012 split.
tasks, and for tasks with large support sets, as are common (and best) practices in few-shot evaluation protocols. As result, the study of difficult tasks has been limited to small-scale datasets which lacks the challenging examples and the setup of large benchmarks such as NETA-DATASET.
To address this, develop general and computationally efficient algorithm called FASTDIFFSEL to extract difficult tasks from any large-scale dataset. Given (meta- )trained few-shot classifier, query set, and search pool of support images, formulate constrained combinatorial opti- mization problem which learns a selection weight for each image in the pool such that the loss on the query is maximised. The top-k (i.e. most difficult) images per class are then extracted into a support set and paired with the query set to form a difficult task. This optimization can be re- peated to obtain any number of difficult tasks. practice, we find that FASTDIFFSEL is at least 20x faster than existing greedy search-base based algorithms (Agarwal et al. 2021 with greater gains as the support pools and support set sizes increase.
We leverage the scalability of FASTDIFFSEL to extract difficult tasks from a wide range of large- scale vision datasets including DATTAA-DAASEEE (Triantafillou et al. 2019), OBJECTNET (Barbu al.] 2019), CURE-OR (Temel et al., 2018) and ORBIT (Massiceti et al., 2021) and collect these tasks into a new testing set called HARD-MEN -ATTA--AAAST+++ (UARD-MD++1 META-DATASET is motivated by their wollworld nature and that they provide image annotations of quality variations (e.g. object occluded, blurred, poorly framed) to enable future research into why a task is difficult. We provide early insights into this question in analyses.
The addition of datasets beyond
We stress test an extensive suite of state-of-ther the-trterttt shot classification methods on HARD-MD44 cross-vale cols-vvaliatiin the difficulty of our extracted tasks across these pop-performing methods. In Fig we show one such method (Hu et al., (002)) performing consistently worse on the META-DATASET test split in HARD-MD++ than the original MD test split across all 10 sub-datasets. In Section we find that this trend holds true across wide-range of few-shot classification methods.
We release HARD-MD++ along with broad set of baselines drive future research in methods that are robust to even the most difficult tasks. In summary, our contributions are the following:
1. FAST efficient algorithm extract difficult tasks from any large-scale vision dataset.
2. REDD-MEAA-DATASET- a new test-only few-shot classification benchmark composed of difficult tasks extracted from the widely-used few-shot classification benchmark META- DATASET and other large-scale real-world datasets: OBJECTNET, CURE-OR and ORBIT.
3. Extensive stress testing and novel empirical insights for wide range of few-shot classifi- cation methods, including transfer- and meta-learning based approaches on HARD-MD++.
2 FEW-SHOT CLASSIFICATION PRELIMINARIES AND NOTATIONS
A few-shot classification task is typically composed of (i) a support set S which contains a few labelled examples from set of N classes (e.g., ky examples for each class index j [1,N)
2

---

Published as a conference paper at ICLR 2023
Algorithm FASTDIFFSE:: Efficient algorithm for extracting a difficult few-shot task
Require: Q
task query set; N: number of classes (way), P search pool for extracting task support set; fo: (meta-)trainee base model; M: size of search pool: a: learning rate; (Ry) ) set containing number of shots per class, 4- CONCAT(W,)
Vj E [1,
> Concatenate randomly initialized vectors for each class
for j in do
C +
--E 1 w woffoo
> Compute weighted class prototypes for each class
end for
C II...., cyl
i Store the weighted prototypes for cach class b- Compute prototypical loss (Snell et al. 2017] D. Compute gradients with respect to selection weights b Gradient ascent for updating weights b Projection step per class > Extract k2 examples with the highest weights b Obtain the final difficult support set S
L
+ PROTO-LOSSI 2,
B.BCCWAAD(W))
W < w LaWw
W F PROI(W,,ki kj)
Vj E [1 Vj E N] Vj [,N]
EXTRACT(K,, wy
P)
+ CONCAT(S3)
and (ii) a query set Q which contains a disjoint set of test examples for each of those N classes (e.g., 22 examples for each class E NT). Given trained base model ʃe. the goal in few-shot classification is to use S to adapt fo to the task such that the model can then make predictions on the unseen examples in Q. This is typically done for many tasks which are randomly sampled from given dataset D = (z; ys)) consisting of T examples and J classes in total, where x is the input image and y is the corresponding class label. In few-shot sampling, tasks
constructed by first sampling a set of N classes from the larger set of J classes and subsequently sampling the support and query set. Tasks are typically referred to N-way, k-shot tasks if kj- = k, Vj. However, if the number of classes N varies across tasks and the number of shots per class varies across the N classes (ky, Vj [1, NT), they are referred as variable-way, variable-shot tasks. Usually the number of query examples is kept fixed across all the classes (i.e. 9j = Vj E [T)
Base model training. The underlying model fo is typically trained using one of two approaches: (i) meta-learning (Snell et al. [017,] Lee et al.) [2019] Finn et al. 2017a) which involves training the model in an episodic manner on tasks sampled from a base training dataset, or (ii) transfer learning (Tian et al.) 20200 Hu et al.) 2022; Chen et al.] 2019) which involves first pre-training feature extractor on large base dataset in an supervised or self-supervised manner, and then fine- tuning the final classification layer (or the entire model) at test time for each new test task.
3 FASTDIFFSEL: AN EFFICIENT ALGORITHM TO SELECT DIFFICULT SUPPORT SETS
The first step understanding the limitations of current few-shot classification methods is to study the difficult tasks on which they fail or perform poorly. We define a difficult task as one for which a given few-shot classifier, after being adapted to its support set, performs significantly worse on its query set compared to the mean query performance over all tasks. Finding support set from a given search pool that leads to poor performance on a given query set for the purposes of study, however, has combinatorial complexity. Current solutions have therefore turned to greedy search-based al- gorithms (Agarwal et al.] 2021; Fu et al., 2002),, however, the computational cost quickly becomes infeasible for larger search pools and support set sizes, thus limiting study small-scale datasets (e.g. CIFAR-FS (Bertinetto et al. 2019). mini- Imni-lmaeeNet (Vinyals et al. 2016)). We specifically address this limitation by proposing fast and general optimization-based algorithm FASTDIFF- SEL which offers a speedup of at least 20-25x over greedy search-based algorithms, allowing the extraction of difficult few-shot tasks to be scaled to a wide array of large-scale vision datasets.
3.1 PROPOSED METHOD
Overview. We present FASTDIFFSEL, that can sample difficult few-shot classification task in a deterministic way. The key intuition of our approach is use a model's 's loss on the task's 's query set (i.e. after the model has been adapted to the support set) as a proxy for the difficulty of the task. This (olnods/Arnold et al. (202T): Dhillon et al. al.[(2019) (2019a) which show that a task's 's query loss is an effective surrogate for task difficulty and is also simple to compute. Given a model, a fixed query set and a pool of examples, we cast support set extraction as a constrained optimization problem and learn selection weight for each example in the pool such that the loss on the query set is maximized. We can then construct a difficult support set by drawing the examples with the highest selection weights. The extracted support set is then paired with the fixed query set to form the difficult task. We can repeat this optimization for any number of query sets to extract any number of difficult tasks.
3

---

Published as a conference paper at ICLR 2023
Formally, given dataset D, we first sample N unique classes and a query set Q. Here Q {(xr, (r)r,y))) ) where x is the input image and y is the class label. Let D' C 'D denote sub-dataset containing examples from only the N sampled classes and let = DD' - Q denote the set of examples from D' without the query set Q. Allowing P be the search pool from which the difficult support set will be extracted, the goal of the extraction algorithm is to find a support set S C P such that the loss on the query set maximized after the base model fo has been adapted on S. To this end, we assume selection weights w E R M where w is associated with the hth example in P and M = (P] denotes the cardinality of P. The optimization objective is to learn the selection weights w which result in the maximal loss for query set Q with a sparsity constraint on the weights. Formally: Xxg
max W
Σ
(l(rr:B)), fo)
s.t.
w' 10, Iwyllo S kj, Vj e [1.N]
Vi [L
where So is the base model trained with either meta-learning or supervised learning, E is the loss after adapting fo on P where each of its examples are weighted by w, and w, is the selection weight vector corresponding to the jth class. Here w Wi + W2 @ ... ① wx and w/ the selection weight for the ht example in the weight vector w, for the j"h class. Note w are the only learnable parameters. The optimization constraints ensure that each selection weight is either 0 or 1, and that
a maximum of k; examples are selected for each jth class. Different approaches can be used to adapt fe (Requeima al. 2019, Chen al., 2020). In our work, we adopt a ProtoNets adaptation (Snell eet al., (017] as it is highly efficient and has no learnable parameters. This approach computes a mean embedding for each class, with the loss based on the Fuclidean distance between a query image embedding to each of the class prototypes.
We solve Equation in two steps: (i) first, we take gradient ascent step on the selection weights w to obtain w; (ii) second, we project the selection weight vector of each class w, to the to norm ball to obtain the final selection weights w, (V'j E [I, In practice, (ii) is known to be difficult to solve it is harhard and the ly norm constraint is non-convex (Candes al.) 2005 Candes & Tao, 2005 Natarajan 1995 Donoho 2006). We, therefore, relax the Eu norm to an & norm to make the constraint convex following Donoho (2006) which shows that the &, norm relaxation gives effective sparse solutions. The projection step with f relaxation for the jth class can be formalized as follows:
min 1 |w w
w; 2
s.t. |w5ll S kj
(2)
We solve the dual form of the above projection step via Lagrange multipliers (Boyd & Vanden- berghe 2004) to obtain the optimal sparse weight vector Wy:
W arg max min W;
[w
w + (s kj)
2,20
where Xy is the Lagrange multiplier corresponding to the projection step for the 7th class. In practice, we solve the projection step per class to ensure at least a few-exame few-examples per class are selected, and we find that projection step per class is sufficient to learn appropriate selection weights. After gradient ascent and I projection step (i.e. weight vectors have been learned), we select the examples from P to include in the difficult support set. For each jth class, we sort the final selection weight vector w in descending order and extract the ky examples from P which have the highest weights.
The pscudo-code for FAST DIFF-ELL is shown in Algorithm Tand a detailed derivation of the steps for solving the optimization along with the associated hyperparameters can be found in Appendix A.2
Computational complexity. The key advantage of FASTDIFFSEL is that it does not require an iterative exhaustive search through the search pool to select a difficult support set. Consider selecting a task with N classes and & support examples per class from a dataset D', where each class has examples on average. The greedy algorithm in Agarwal al. (202)) runs for iterations and thus
4

---

Published as a conference paper at ICLR 2023
Ours Greedy
llsvrc 2012
Average Task Accuracy (%)
aircraft
atd
vgg_ plower
traffic sign
omniglot
birds
fungi
mscoco
quickdraw
20 Average Times Extract Difficult Support Sat (Minutes)
120
200
Figure 2: FASTDIFFSEL extracts tasks with similar accuracy to those extracted by greedy search algorithms (Agarwal et al. 2021) but at least 20x faster. extract 50 5-way 5-shot tasks per sub-dataset using Hu et al. (2022) as the base model (VTT-S initialized with SSL DINO weights then meta-tran trained with ProtoNets on MD's 2012)) on an A5000 GPU (64GB RAM).
has search time complexity of ((N.k.d.r) along with as many adaptation steps. In comparison, algorithm removes the need for this exhaustive search and large number of adaptation steps, thus offering a significant speedup. practice, we find that our algorithm offers speedups of at least 20x when compared to greedy algorithm (see Fig and Fig 6p.
DIFFICULT SUPPORT SET EXTRACTION ON META -DTTADDTSSTT
Since FIFFSI is general and highly efficient, we can use it to extract difficult tasks from any large-scale vision dataset. MD is one of the most widely-used benchmarks for few-shot classification, thus we primarily use it to validate the effectiveness of our extraction algorithm. We include further analyses on difficult tasks extracted from further datasets in Section 4.2
META-DATASET (MD) (Triantafillou et al. 2019) contains 10 image sub-datasets from a diverse set of domains. All the sub-datasets (except mscoco and traffic_signs) split into disjoint train/val/test classes, whereas mscoco and traffic_sign are test only sub-datasets. During few-shot training, variable-way variable-shot tasks are randomly sampled from the train classes of each sub-dataset. During few-shot testing, 600 variable-way variable-shor tasks are sampled from the test classes of each sub-dataset with the average task classification accuracy reported for each sub-dataset.
4.1 TEST TASK SAMPLERS FOR META-DATASET
We compare tasks sampled from the test split of MD's 's sub- dabbsetsss using 3 methods: (i) MD's default sampler; (ii) the greedy approach Agarral et al. (2021) 1) and (ii) and (iii) can be used to deterministically sample difficult tasks:
Note, only
MD's default sampler. The default task sampler in MD samples imbalanced tasks of variable-way. variable-shots. The shot, way and size the query set depends on the size of the sub-dataset in MD. For more details, refer to Sec 3.2 in Triantafillou al. (2019). We note, however, that the default sampler cannot deterministically sample difficult tasks.
Greedy search-based sampler. We use Agarwal al. 2021 extract difficult tasks from MD. This approach works by iteratively replacing each example in a support set of fixed size with one from given search pool such that the query loss maximized. Because of the large computa- tional cost and time associated with searching the pool each time an example is replaced, greedy approaches do not scale well to MD's sampler where task ways can vary as large as 50 and shots as large as 100. We therefore only consider fixed-way, fixed-shot tasks when sampling via greedy search on MD.
FASTDIFFSEL. We use FASTDIFFSEL to extract difficult tasks from MD. As the trained base model fe, we choose state-of- -ft-----tttttttt method (Hu et al.) 2022) on MD which employs a ViT-S feature extractor (Touvron et al. 2020) initialized with self-supervised DINO weights (Caron al. 2021) pre-trained on 2lsvre.2012. The extractor then further from MD. To compare against the above samplers, consider two configurations:
on the 1isvre_2017 split
Variable-way, variable-shot tasks. Here, we exactly match the variable way and shots per class of each task yielded by MD's default sampler. Specifically, use the default sampler to generate and save the shot and way for a fixed number of tasks, which we then feed into our algorithm to extract difficult tasks of equivalent specifications.
5

---

Published as a conference paper at ICLR 2023
dirn small patch1d
dino base, patchi6
resnotig
Dase
Amail 1k
WE, baso. Jk
bert base, 21k
dlpp Vit base
dipp resset50
dine, resnetto
216
MD 67.1
66.5
61.8
61.3
60.8
61.3
59.6
60.4
65.1
39.8
58.2
53.3
63.7
Increasing Task Accuracy
39.6
41.0
36.9
36.8
37.8
38.5
39.6
39.9
43.2
15.1
42.2
38.2
33.2
(a) Prototypical cal Networks during Adaptation
Hard-MD
47.4
40.1 43.6
44.5
46.9
48.4
51.5
48.9
51.5
18.6
52.4
50.5
36.5
Full Fine tuning during Adaptation
We evaluate wide range of top-performing methods over 200 variable-way, variable-shot difficult tasks per sub-dataset in HARD-MD and find a consistent 20-30% drop in performance compared to MD tasks regardless of feature extractor, pretraining method, and adaptation strategy. We evaluate on all MD sub-datasets except dlsvre_2012. We include per- formance each sub-dataset and their 95% confidence intervals in Appendix H
Figure 3:
Fixed-way, fixed-shot tasks. Here,
the way and shots per class to be fixed for all tasks. While less challenging, it allows us (i) to compare our algorithm to greedily sampled difficult tasks; and (ii) control for task's way being source of difficulty.
4.2 VALIDATION OF DIFFICULT META-DATASET TASKS
Here we compare the efficacy of the above 3 methods and their extracted tasks across the 10 test sub-datasets in MD.
FASTDIFFSEL. vs. default sampler. When compared over 600 variable- way variable-shot tasks sub-dataset, we find that our algorithm extract tasks which are more difficult on average than MD's default sampler, as shown in [ig This is consistent across all 10 sub-datasets. We also show that these tasks are consistently difficult for wide range of top-performing few-shot classification methods, as shown Fig (see Section 5.3. for details). For certain sub-datasets (e.g. quickdraw), the drop in classification accuracy be as large 50% when compared tasks sampled from MD's default sampler.
FASTDIFFSEL vs. greedy sampler. We compare our algorithm to the greedy sampler over 50 5-way 5-shot tasks sampled per sub-dataset. For fairness, we run both methods for the same set of query sets, and report the average task accuracy and extraction time in Fig Here, we see that the average task accuracy is comparably low for each sub-dataset but 1-2 orders of magnitude faster. This speedup is most significant when the search pool is large (e.g. for 2012, quickdraw). Note that the greedy sampler takes between 20-200 minutes per lask, highlighting its impracticality for larger search pools, support set sizes and numbers of tasks.
our algorithm almost
In all, FAS able to consistently extract difficult tasks on which state-of-the-ar methods achieve low classification accuracy, while also offering significant speedup in extraction time com- pared to existing task samplers. It can, therefore, readily leveraged to further the study of failure modes in few-shot classification methods.
STRESS TESTING WITH HARD-MENT WERDD--EA--DATAAEET-
Few-shot classification benchmarks like MD (Triantafillou et al. 2019) and VTAB+MD (Dumoulin jtt al. (021b) are highly challenging but are not specifically geared to driving performance on diffi new
cult tasks. We leverage our extraction algorithm to fill this gap and introduce HARD-MD++4 test-only benchmark of exclusively difficult few-shot classification tasks extracted from 4 large-scale vision datasets. We extensively stress test wide range of top nop-performing methods HARD- HARD-MD++H -DD++ presenting novel insights on the robustness of current methods to difficult tasks.
5.1 TEST DATASETS
HARDD-D++ is composed of difficult tasks extracted from MD (Triantafillou et al. 2019), OR- BIT (Massiceti et al., 2021), CURE-OR (Temel et al. 2018) and OBJECTNET (Barbu et al. 2019). We motivate the choice of MD by the fact that it is of the most widely-used few-shot classifica- tion benchmarks. The remaining datasets are chosen because they specifically curate images with real-world variations and provide corresponding 'quality' annotations. They can, therefore, be lever-

---

Published as a conference paper at ICLR 2023
aged to derive deeper insights into the properties of difficult tasks as explore in Section 5.3.2
Together, these datasets cover a broad range of challenging and real-world tasks.
we FASTDIHFSEL to extract 200 difficult tasks from each dataset. We use a trained base model fo of ViT-S (Dosovitskiy et al. 2020) pre-trained using DINO (Caron et al. 2021) and further meta-trained
Following Section 4
MD's ilsvre 2012 split. For MD, we extract variable-war way variable-shot tasks to align with its existing evaluation protocol. For the remaining datasets, we extract fixed-way fixed-shot tasks to enable controlled analysis on task properties beyond their way and shot. Together, HARDD-DD++ comes to a total of 2400 difficult test tasks extracted across these datasets. We include further implementations details in Appendix B
META-DATASET. We extract 200 variable-way variable-shot difficult tasks from each test sub- datasets in MD. Note, we exclude the ilsvre 2012 subset so as not to prevent the use of feature extractors pre-trained on it.
ORBIT contains 3822 videos of 486 objects captured by people who are blind on their mobile phones. Each frame is annotated with 7 quality issues including blur, framing, lighting and oc- clusion. We extract 200 fixed 5-way, 5-shot difficult tasks from the test split of ORBIT which cover 1198 videos of 158 objects.
CURE-OR contains 1M test images of 100 objects captured under controlled viewpoint variations (e.g., front, back, side etc.) with corresponding annotations. We extract 200 fixed 5-way, 5-shot difficult tasks from CURE-OR.
OBJECTNET contains 50K test images of 313 objects captured under rotation, background, and viewpoint variations. We extract 200 fixed-way fixed shot difficult tasks from OBJECTNET.
5.2 METRICS AND TRAINING
Metries. Model performance on HARDD-DD+ should be reported as the average classification ac- curacy and 95% confidence interval over the difficult tasks per sub-dataset. This should be accom- panied by performance on MD to provide a more complete characterization of model performance.
Training. We primarily advocate for cross-domain or strong generalization (Triantafillou [021), thus following Hu et al.] (2022), any pre-trained checkpoint, algorithm and model architecture can be used when evaluating on HARD-MD+++ This aligns with more recent few-shot learning practices which allow the wide range of publicly-availabe datasets and pre-trained models to be leveraged.
5.3 RESULTS
HARD-MD++ is a challenging benchmark of difficult tasks from a diverse range of datasets. We stress test a wide range of top-performing few-shot classification methods on HARD-MD++ to gauge their robustness to these tasks. We look first at MD and then ORBIT, CURE-OR, and OBJECTNET, primarily focusing on the robustness of different model architectures and pre-training strategies.
State-of-the-art methods. Recent works TTin al. 2020 Hu et al. 2022 Dhillon et al., 2019b 202 show that transfer learning with powerful feature ex- tractor performs extremely well on few-shot classification tasks compared to previous meta-learning methods (Finn et al. 2017a Requeima et al., 2019). We therefore consider wide range of feature extractors and pre- training paradigms: ResNets (He al.] (0155). Vision Transformers (ViTs) (Doso- 2020; Touvron et al.] 2020), self-supervised variants of ResNets and ViTs (Caron Bao et 2021) and the visual encoder of zero-shot models such as CLIP (Radford
Chen et al. 2019 Dumoulin et al.
vitskiy al.
et al.
2021
al.
2021
We investigate two adaptation strategies when adapting to each difficult test task in
HARD-MD++: (i) computing and classifying by mean class prototypes following Prototypical Net- works (Snell al. 2017) and (ii) fine-tuning the entire feature extractor using strong augmentations following (Hu et al. 2022). Further details are provided in Appendix
5.3.1
RESULTS ON DIFFICULT TASKS FROM META-DATASES
Our key finding is that state-of-the-as methods across the board consistently drop at least 20-25% in classification accuracy when adapting to difficult tasks in HARD-MD compared with tasks randomly sampled from MD, shown in Fig Note, HARD-MD refers to the MD split in HARD-MD++.
Detailed findings. We find that there is a consistent drop in classification accuracy on HARD- MD across all oftte-of-the-at methods, regardless of adaptation strategy, feature extractor and pre- training paradigm (see Fig This validates our proposed algorithm in its ability to extract generally challenging tasks, but also highlights the limitations of current approaches. In particular, approaches

---

Published as a conference paper at ICLR 2023
tino, smail paachh16
dmme bese patch16
vit Dass 21k
Mit. base
belt base 21k
clip vit base
dlip_ resnet5o
dino resnet50
reonet18
1osnet34
Senatton
small .ik
Objectliet
13.8
17.3
18.7
20.1
19.3
27.2
16.5
43.6
13.2
CURE-OR 24.7
25.3
27.0
27.0
28.1
26.8
27.8
27.9
27.2
13.2
32.8
26.9
18.7
ORBIT 22.5
25.6
30.2
30.3
32.2
34.7
35.1
39.8
40.4
16.7
38.2
32.3
25.0
Figure 4: stress test a wide range of few-shot classifiers on difficult tasks from OBJECTNET, CURE-OR and ORBIT in HARD-MD++. We find that ViT-B (CLIP) outperforms all the other models by large margin for CURE-OR and OBJECTNET, while being extremely competitive for ORBIT. Evaluation across 200 tasks per domain using Prototypical Networks as the adaptation strategy. More results with fine-tuning in Appendix H
that employ fine-tuning adaptation strategy see a more significant drop accuracy compared to those employing prototypical-styl -tylle adaptation. For example, fine-tuning a ViT-B feature extractor pre-trained on ImageNet-21k ('vit. base 21k' in Fig (Dosovitskiy et al. 2020) on each test task leads to a drop of ~30% in accuracy on HARD- MD while prototypical-ste drop of only ~20% (note, this approach is strongest on original MD our implementation). This suggests that although fully fine-tuning model can significantly increase its accuracy on general tasks, may not hold for specifically difficult tasks. On the other hand, we find the visual encoder of CLIP which also be more robust to difficult tasks. Despite under-performing on MD (ranks 6m),, it displays strong performance HARD-MP MD across both adaptation strategies, achieving the highest accuracy across all methods with fine-tuning. This trend is consistent for fixed-way. fixed-shot tasks on HARD-MD (see Appendix H.. These early results suggest that large-scale vision-language pre-training may offer more robustness when generalising to new difficult tasks.
adaptation leads to
VIT-B architecture ('clip vit base' in Fig 3 (Radford et al. 2021] to
Effect of meta-training on Hlsvr__2012. Hu al. (2022) show that method's performance on MD can be improved by further meta- meta-training its pre-trained feature extractor on MD's 1lsvre_2012 split. We investigate whether this can also improve performance on HARD-MD. We further meta- train subset of the methods on the ilsvre 2012 split using Prototypical Networks (Snell et al.) 2017) and compare the average task classification accuracy Appendix H We find that despite the further meta-training, the tested methods still display a consistent drop of 20-30% on IIARD-MD. This suggests that novel algorithmic contributions, rather than further training, may be required to improve robustness to difficult tasks.
5.3.2 RESULTS ON DIFFICULT TASKS FROM CURE-OR, ORBIT AND OBJECTNET
Similar to Section 5.3.1 our key finding is that ofte-of-theeatt methods consistently achieve low classification accuracy on difficult tasks from CURE-OR, ORBIT and OBJECTNET, shown in Fig 4 In partieular, ViT-B pre-trained with CLIP (Radford et al. 2021) outperforms all other methods by significant margin on both CURE-OR (by >4%) and OBJECTNET (by > 15%). On ORBIT, ViT-B pre-
trained on ilsvre_201 (Deng al. 2009) performs best though ViT-B with CLIP is still competitive.
We can go beyond task classification accuracy with CURE-OR and ORBIT by leveraging their per- image annotations investigate the properties of difficult tasks. In particular, we the object viewpoint annotations in CURE-OR, and all quality issue annotations (except object.not_ppesen in ORBIT. We compare difficult tasks extracted by our algorithm for each dataset with "easy" tasks for a fixed 5-way 5-shot setting. We extract "casy" tasks by instead minimizing the objective in Equa- tion (T) for the same set of query sets. Fig 5 we compare the composition of these difficult and easy tasks by the annotated attributes in their supports sets.
CURE-OR. We randomly sample 200 query sets containing images only showing the object with a front or back viewpoint. The sampled query sets contain 10 examples per class. For each query set, we use our algorithm to extract an easy and difficult support set and visualize the distribution of their annotations in (Leff). Here, we that the easy support sets have higher proportion of images with front or back viewpoints relative to the total size of the support set, while the difficult support sets have significantly higher proportion of images with side viewpoints.
ORBIT. Unlike CURE-OR, ORBIT was not collected in a controlled setting and hence its quality issue annotations have long-tailed distribution. We therefore randomly sample 200 query sets, and for each, we use FASTDIFFSEL to extract an easy and a difficult support set. For each quality issue, we compute the difference (gap) in the proportion of images in the support versus query set with that
8

---

Published as a conference paper at ICLR 2023
CURE-OR
QRBIT
Hard Tasks
Hand losks (Overall Gap) Tesks (Dverall Gap) Handd Tasks
Easy
Capin Proporten of Quality Pequae
Tasks
Prozortor of itrages
am
00
Irnnt
back
000
Quality aalanopenii
Viwwoonts
Figure 5: Difficult few-shot tasks have a larger mismatch between the properties of their support and query sets compared to easy tasks. (Left): Difficult tasks in CURE-OR have support sets with the majority of images showing the object in side view when the query sets contain images of the object only in front/back view: (Right): Difficult tasks in ORBIT have a larger difference in the proportion of quality issues contained in their support versus query sets compared easy tasks.
particular issue. In Fig (Right) we report this gap averaged over 200 tasks. Here, we observe that difficult tasks have larger gap (i.e. difference in support and query set) in quality issues than casier tasks. Together, these results suggest that mismatch between image characteristics within support versus query set (e.g.. viewpoint, occlusion) can be source of task difficulty. While these annotations do not cover all possible image characteristics, they provide a starting point to explore the robustness of few-shot classifiers to more qualitative distribution shifts. In Appendix curate few-shot tasks leveraging image-level annotations from ORBIT, CURE-OR and MS-COCO to show that tasks with distribution shifts with respect natural characteristics have lower accuracy.
6 RELATED WORKS
Difficult tasks. Previous works (Agarwal et al. 2021 Arnold al.) 2021 Dhillon al., 2019a)
have shown that state-of-the-art few-shot classifiers generally display a wide range in performance when adapting different tasks. Agarwal al. (202T) use this observation develop a greedy saurch-based algorithm that can specifically extract difficult tasks for further study. They consider only meta-learning approaches and, due the computational requirements of greedy search, are limited to small-scale datasets including mini-ImageNet and CIFAR-FS. (Amold et al. (021) also study difficult tasks through correlation- based analysis. We extend on both of these works by (i) proposing scalable algorithm FASTDIFF that extract difficult tasks from any large- scale vision dataset, and (ii) conducting deep empirical evaluation on the robustness of a broader range of meta-learning and transfer learning approaches these difficult tasks. Potentially ideas from subset selection (Wei et al. 2015 Killamsetty et al. 2020 can be adapted for few-shot task extraction, but leave it for future work.
Few-shot classification benchmarks. MD (Triantafillou et al. 2019) and VTAB+MD (Dumoulin et al. 2021b) rent literature. They cover a wide range of domains and primarily evaluate the ability of a few-shot classifier to generalise novel object classes, datasets and domains. Other few-shot benchmarks have been introduced to specifically target adaptation to images with high real-world variation, in- cluding ORBIT (Massiceti et al.) 2021) and cross-domain transfer beyond natural images, including BSCD-FSL (Guo et al. (2020). We note, however, that unlike HARD-MD++, none of these bench- marks specifically target difficult tasks for few-shot classification.
of the most challenging few-shot image classification benchmarks in the
7 CONCLUSION
We introduce a general and scalable algorithm FASTDIFFSEL extract difficult tasks for few-shot classification. We apply FASTDI#SSL to 4 large-scale vision datasets: META-DATASET, ORBIT, CURE-OR, and OBJECTNET, and introduce HARD-MD++, a new test-only suite of 2400 difficult tasks from across these datasets. We stress test wide range of top-performing few-shot methods on HARD--D and demonstrate a consistent drop 20-25% in classification accuracy compared to MD. We conduct additional quantitative analyses CURE-OR and ORBIT which show that difficult tasks typically have a distribution shift in the characteristics between a support and query set (e.g- viewpoint, blur). We believe that the efficiency of FASTDIFFSEL along with HARD-MD++ can drive the study of failure modes in few-shot classification methods which is a under explored area of research.

---

Published as a conference paper at ICLR 2023
8 ACKNOWLEDGEMENTTS
This project was supported in part by Meta grant 23010098, NSF CAREER AWARD 1942230, HR001 19S0026 (GARD), ONR YIP award N00014-22-1-2271, Army Grant No. 1911 and the NSF award CCF2212458.
REPRODUCIBILITY STATEMENT
Our work primarily consists of (i) an algorithm FASTDIFFSRI. for extracting difficult few-shot tasks from any vision dataset. (ii) few-shot testing benchmark called HARD-MD+++. For (i), we provide the steps for reproducing the algorithm in Algorithm [ The hyper-parameters for the al- gorithm can be found in Appendix A.2 For (ii), we provide all the relevant details in Appendix B
and Section 5 The details for training and fine-tuning the various models that we stress test on
HARD-M BARD-MD++ can be referred in Appendix F We will publically release the code and HARD-MD++ upon the acceptance of our manuscript.
REFERENCES
Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Char- less C. Fowlkes, Stefano Soatto, and Pietro Perona. Task2vec: Task embedding for meta-learning. CoRR, abs/1902.03545, 2019. /ttpp/tpp///aa..oooo///////////. 01t.//550020005555
Mayank Agarwal, Mikhail Yurochkin, and Yuckai Sun. On sensitivity of meta-lcarning to support data, 2021. hitp:/tces//aaxii..oo//////////p
Sébastien M. R. Arnold, Guneet S. Dhillon, Avinash Ravichandran, and Stefano Soatto. Uniform sampling over episode difficulty, 2021. URL /tRLattte:://arr..orr///b////// 01662
Hangho Bao, Li Dong, and Furu Wei. Beit: BERT pre-training of image transformers. CoRR, abs/2106.08254, 2021. URL htt.:/ppss///ax.v..rr//abb////// /ttp://////b///0.0000./// 08254
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlle dataset for pushing the limits of object recognition models. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, vol- ume 32. Curran Associates, Inc., 2019. URL http::tt:///t https:://proceedli ngs www.riisss..... wwt.rpps.c.c... ca/ pdf
Etienne Bennequin, Myriam Tami, Antoine Toubhans, and Céline Hudelot. Few-shot image classi- fication benchmarks too far from reality: Build back better with semantic task sampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 4767-4776, June 2022.
Luca Bertinetto, Joao F. Henriques, Philip Torr, and Andrea Vedaldi. Meta-learning with differen- tiable closed-form solvers. In International Conference on Learning Representations, 2019. URL
https::/ net http//ooummiii http://www.....oooooooooooo/////
Malik Boudiaf, Imtiaz Masud Ziko, Jérôme Rony, José Dolz, Pablo Piantanida, and Ismail Ben Ayed. Transductive information maximization for few-shot learning. CoRR, foss/2008.11999 2020.
Stephen Boyd and Lieven Vandenberghe. Conver optimization. Cambridge university press, 2004.
John Bronskill, Daniela Massiceti, Massimiliano Patacchiola, Katja Hofmann, Sebastian Nowozin, and Richard Turner. Memory efficient meta-learning with large images, 2021. URL https:
Tarxivv v. org/ wwwp////.......... 01100
E.J. Candes and T. Tao. Decoding by linear programming, IEEE Transactions on Information Theory, 51(12):4203-421 2005. doi: 10.1 109/TTT.2005.88899
10

---

Published as a conference paper at ICLR 2023
Emmanuel Candes, Justin Romberg, and Terence Tao. Stable signal recovery from incomplete and inaccurate measurements, 2005. URL /ttL:ttts///xxiiii /ttpp////////aaaa.ooaa/a//aaaaaa http///ttth/0006666
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Ar- mand Joulin. Emerging properties in sulf-ssee vision transformers. CoRR, abs/2104.14294, 2021. URL/attps://aaxxt http://www..r//////////0//// 14294
Wei- Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. closer look few-shot classification. In International Conference Learning Representations, 2019. URL htt.thppss//ppeeeeiiieeeee net /orprer/onieeeeeeeemuouu weaerummid=Hkxtaae id=HkxLXAACFQ
Yinbo Chen, Xiaolong Wang, Zhuang Liu, Huijuan Xu, and Trevor Darrell. A new meta basalbe for few-shot learning. CoRR, ahs/2003.04390, 2020. URL httts:t//aaxtiiii wttp://wwwr//bb////00000
04390
Arkabandhu Chowdhury, Mingchao Jiang, and Chris Jermaine. Few-shot image classification: Just library of pre-trained feature extractors and simple classifier. CoRR, abs/2101.0055
2021. /tt/:/ttss//arr.....r/////////// 00562 00562
George Corliss. Which root does the bisectioon algorithm find? SIAM Review, 19(2):325-327, 1977. doi: 10.1137/1019044 WRRL/tttts////d...rgr////// 1137 9044
Jia Deng, Wei Dong, Richard Socher, Li-Jia Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009. doi: 10. wwtp://VR...00052000000
Guneet S. Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. A baseline for few-shot image classification. 2019a. doi: 10.48550ARXX.1190000722 URL https://
dexiv.orgg. abs /1909. 02729
Guneet S. Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. A baseline for few-shot image classification. CoRR, abs/1909.02729, 2019b. URL
abs/1909. 02729
D. L. Donoho. Compressed sensing. IEEE Trans. Inf. Theor, 52(4): apr 2006. ISSN 0018-9448. doi: 10. 10.1109/TT.220068711588
URL
2006. 871582]
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. CoRR, abs/2010. 19929, 2020.
http://wwwww...o.///a////aaaa 2200 11929
Vincent Dumoulin, Neil Houlsby, Utku Evei, Xiaohua Zhai, Ross Goroshin, Sylvain Gelly, and Hugo Larochelle. Comparing transfer and meta learning approaches on a unified few-shot classi- fication benchmark, 2021a. URL
org
Vincent Dumoulin, Neil Houlsby, Utku Evei, Xiaohua Zhai, Ross Goroshin, Sylvain Gelly, and Hugo Larochelle. A unified few-shot classification benchmark to compare transfer and meta learn- ing approaches. In Thirty-fifth Conference Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021b.
Li Fei-Fei, Fergus, and P. Perona. One-shot learning of object categories. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(4):594-611, 2006, doi: 10.1109/TPAM..200..9799
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. CoRR, abs/1703.034400, 2017a. URL
03400
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Doina Precup and Yee Whye Teh (eds.). Proceedings of the 34th International Conference Machine Learning. volume 70 of Proceedings Machine Learning Research, pp. 1126-1135, 1135. PMLR, 06-11 Aug 2017b. URL https: http:://pppcceedinngoo mir. wtt...pees//vc finn 1wm.17a... html

---

Published as a conference paper at ICLR 2023
Minghao Fu, Yun-Hao Cao, and Jianxin Wu. Worst case matters for few-shot recognition, 2022. URL
http//aa..//aae/ee/e// 06574
Yunhui Guo, Noel C Codella, Leonid Karlinsky, James V Codella, John R Smith, Kate Saenko, Tajana Rosing, and Rogerio Feris. A broader study of cross-domain few-shot learning. ECCV, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CoRR, abs/1512.03338, 2015.
03385
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. CoRR, abs/2106.09685,
Shell Xu Hu, Da Li, Jan Stühmer, Minyoung Kim, and Timothy M. Hospedales. Pushing the limits of simple pipelines for few-shot learning: External data and fine-tuning make a difference, 2022.
URL [tthhtss::/ //axxiii////xxv../// org/ abs/ 2204. 07305
Yuqing Hu, Vincent Gripon, and Stéphane Pateux. Leveraging the feature distribution in transfer- based few-shot learning. CoRR, abs/2006.03806, 2020. URL https://arxiv. abs/
2006. 03806
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardic, Serge Belongic, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning, 2022. URL http:ppe//aa......rgg/a////////a
12119
Krishnateja Killamsetty, Durga Sivasubramanian, Ganesh Ramakrishnan, and Rishabh Iyer. Glister: Generalization based data subset selection for efficient and robust learning. 2020. doi: 10.48550/ ARXIV.2012.10630 WtLhhttps: /ttp://aaivvvv//aa///aa org ab/////////// 2012. 10630
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Large scale learning of general visual representations for transfer. CoRR, abs/1912. 111370, 2019. URL.ht::://aaxxxi http://awax...oooraa/////////
org 1ttp://ass11911111333
Brenden M. Lake, Ruslan Salakhutdinov, Jason of simple visual concepts. Cognitive Science, 33, 2011.
Jason Gross, and Joshua B. Tenenbaum. One shot learning
Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In CVPR, 2019.
Wei-Hong Li, Xialei Liu, and Hakan Bilen. Cross-domain few-shot learning with task task-specific adapters, 2021. URL http:::ttt/ttttt aawxiv.vvvvv wttpabs///ar..r http://aaaxxi..r//////00000 00358
Tsung- Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. CoRR, fis/140000312, 2014. URL htthhttp//aae htttt:://axxvv.ooog. wttpxiiiorr//a////o
1105.0315
Daniela Massiceti, Lida Theodorou, Luisa Zintgraf, Matthew Tobias Harris, Simone Stumpf, Cecily Morrison, Edward Cutrell, and Katja Hofmann. Orbit: A real-world few-shot dataset for teachable object recognition collected from people who are blind or low vision, 2021. URL
https://ciee fig.y.ffggharrr hwtpsssarrecoo/a///a www./artii........oo.oo........t /ttpc//sseeetttttttTTTTLLLLLLLL -orll world_
Eew-shot Sew-shot_dataset, dataset for teachable object recognition cttpooomiiiiiooooolteeeeeeeeeeee collected
from www.neepppeeeeeee who are blind or low vision/14294557
B. K. Natarajan. Sparse approximate solutions linear systems. SIAM Journal on Computing, 24(2):227-234, 1995. doi: 1011137/5099733999240000 URL https://doi. otg/1.0 1137/50095399922400
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. CoRR, abs/2103.00020, 2021. ARI.https://arxx wwwaaawwww..r..r...aaaeeaaaeeare 22003. 00020
12

---

Published as a conference paper at ICLR 2023
Mengye Ren, Michael L. luzzolino, Michael C. Mozer, and Richard S. Zemel. Wandering within
a world: Online contextualized shotshot learning. CoRR, abs/2007.04546. 2020. URL https:
7 /tw.ii.crg/ab////eae..aa 04546
James Requeima, Jonathan Gordon, John Bronskill, Sebastian Nowozin, and Richard E. Turner. Fast and flexible multi-task classification using conditional neural adaptive processes. 2019. doi:
10.48550ARRXIV11066.000
wwtp://///////////eeee URL http::tttt///ttt httppp///aaaxx..oorr/////////0/ 07697
Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning, 2017. URL http://aaaaaaa..c./a/////.3//.1.
Megan Stanley, John F Bronskill, Krzysztof Maziarz, Hubert Misztela, Jessica Lanini, Marwin Segler, Nadine Schneider, and Mare Brockschmidt. FS-mol: A few-shot learning dataset of molecules. In Thirty-fitth Conference Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL
net forum? did=
701FtuyLlAd]
D. Temel, Lee, and AlRegib. Cure-or: Challenging unreal and real environments for object recognition. In 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA), 2018.
Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B. Tenenbaum, and Phillip Isola. Rethinking few-shot image classification: good embedding is all you need? CoRR, abs/2003.1 11399, 2020.
URL
abw////abb/203333 11539
Hugo Touvron, Matthicu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. CoRR, abs/2012. 12877, 2020. URL
Eleni Triantafillou.
Towards Strong Generalization from Few Examples.
PhD the-
towards strong generalization. few www.://nnnneeeeeeeeeeeeeeeeeeeee htw.a/aa........eeeeeeeeeee///oa thesessss//tttss////
sis. 2021.
URL /ttpp::/www. http://www.gouuett..cooeeeeeeeee
se-2 2 Copyright Database copyright ProQuest LLC; ProQuest does not claim copyright in the individual underlying works; Last updated 2022-01-14.
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Kelvin Xu, Ross Goroshin, Car- les Gelada, Kevin Swersky, Fierre-Antoin Manzagol, and Hugo Larochelle. Meta-dataset: A dataset of datasets for learning learn from few examples. CoRR, abss/193300309 2019. URL
http:://rxi: org/ abs /1903.1903.. 03096
Oriol Vinyals, Charles Blundell, Timothy P. Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. CoRR, abs/1606.04080,, 2016. URL httt/t/////xxx
org/abs/ 1606. 04080
Kai Wei, Rishabh Iyer, and Jeff Bilmes. Submodularity in data subset selection and active learning. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1954-1963, Lille, France, 07-09 Jul 2015. PMLR. URL https: http://ppocccc..ccct//pp/ meil5.html
/ m m.r.press/w3/
Chengming Xu, Siqian Yang. Yabiao Wang, Zhanxiong Wang, Yanwei Fu, and Xiangyang Xue. Ex- ploring efficient few-shot adaptation for vision transformers. Transactions Machine Learning Research, 2022. /tLLhttps:// http://s//ppeeeereeeeeeeeeeoooos
Han-Jia Ye, Hexiang Hu, De-Chuan Zhan, and Fei Sha. Few-shot learning via embedding adaptation with set-to-set functions. In /EEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 8808-8817, 2020.
13

---

Published as a conference paper at ICLR 2023
A FASTDIFFSEL: SUPPORT SET EXTRACTION ALGORITHM
A.1 STEPS FOR SOLVING THE PROJECTION STEP
In this section, provide details how to solve the projection step in Equation (3). The projection step is solved separately for each of the jth class, where j E [1,N] w, is the selection weight
vector for the jth class obtained after a step of gradient ascent on Equation (1. The dual form of
Equation (2) can be expressed via Lagrange multipliers the following:
Wj arg max min [w, - woll2 + As - ký)
(4)
7;20
a(Ag.w,)
We solve Equation 4 in two steps: (i) First, we solve minw, 9( W;) via proximal operators; (ii) Then, obtain the optimal values of the dual parameters Ay.
KKT optimality conditions (due to stationarity) states that Vw,g(A, wy) = However, note that g(Ay, Wy) is combination of smooth function and non-smooth function which can be solved
by proximal operators. Considering Wj E I",, the KKT optimality condition can be stated as the following:
V 2 Iw, w Vw, A5 ky) =
(5)
The value in the th index in wy, which is wÿ can be obtained through:
a0 (w))
20 ')2
X Tlwil
(6)
Ow
Ow
If w/ > O, then the derivative in Equation 6 is w/ wa + Aj. Therefore Equation 6 can be expressed as: wi viy" >j. which holds true for uiy" >y. Similarly, when wz < 0, wy net +
the minimizer. For oif (- Ay], the minimizer is the only point of differentiability for which wi = This operation is called soft-thresholding and can be expressed as:
w Prox (wif) = sign(vif) max (lúi X).0)
Thus, w; = Prox (w,) = [Prox ll- (1i)). ..... Proxx-1-11- (nāf)). The next step is to compute the value of the dual parameter Az. We then compute the derivative g (A₃, wx) the following:
g((2,, w,) |Proxx111111 (w,))l
(8)
-100)- - 21)+ kj
(9)
We solve Equation 9 by the root finding method in (Corliss) 1977). since the optimal X3 E [0, [oo]] The upper bound w,lloo ensures that g' (Ajs w,) does not become negative.
A.2 HYPERPARAMETERS OF FASTDIFFSEL.
Empirically, we perform a grid-search for the learning rate [0.01, 500) with a step size of 10. We specifically find that high learning rate with only one gradient ascent and projection step suffices obtain difficult tasks. In our experiments, we use learning rate a 200, as we find this value to result in the extraction of the most difficult tasks.
A.3 EMPIRICAL RUNNING TIMES
In Fig. (6), we plot the average running time of our framework to extract a single difficult support set across all the domains in META-DATASET. In practice, we find large speedup (at least 20x) using our framework over (Agarwal et al., 2021) for support set extraction. The main advantage of our framework is the elimination of the iterative search in (Agarwal et al. 202I), which is computionally expensive. The speedup is crucial for scalability of difficult task extraction to large-seale vision datasets such MD.
14

---

Published as a conference paper at ICLR 2023
Algonthi
Running Time per Täsk (Minutes)
100
75-
2012_2012
aicraftt
dtd
vau.lewwr
teuffiiaaion
anmiglot
cu.birds
fungi
mococo
quickdhaw
Domalns in Meta-Dataset
Figure 6: Average running times of framework (In practice): For extracting difficult task on an average, we find a speedup of at least 20x (depending on the dataset) when compared to the greedy method in (Agarwal et al.) 202I). Tasks are fixed-way, fixed-shot tasks where way=5, shot 5. On hardware with a5000 GPU with 64GB RAM.
B DESIGN CHOICES FOR DATASETS
Choice for META-DATASET. META-DATASET Triantfill al. 2019) is arguably one of the most challenging few-shot learning benchmarks comprising of 10 different sub-datasets spanning different domains. We primarily build HARD-MD++ top of MD, enable fast adoption of our benchmark by the few-shot learning community. We envision IIARD-M+++ to be an add-on to MD, which the community uses to report performance on difficult tasks.
Choice for CURE-OR and ORBIT. MD though challenging few-shot learning benchmark, does not come with image-level annotations which can be leveraged to understand the properties of difficult tasks. Understanding the properties of difficult tasks is crucial, as they can further guide the devel- opment of future algorithmic methods specifically for tackling difficult few-shot tasks. We ground our choice of annotated datasets in CURE-OR Temel et al., 2018) and ORBIT(Massiceti et al.) (021), as they reflect the variations present in real- world images. While ORBIT an existing few-shot learning benchmmark comprising of real-world videos of everyday objects, CURE-OR is primarily an object recognition dataset. CURE-OR, each image has objects
different viewpoints, which curated in controlled environment. ORBIT consists of 7 unique annotations about various qual- ity issues for each frame in the video. These annotations are blur, viewpoint, framing, occlusion, overexposure, underexposure and object not present. In experiments, specifically leverage all the quality annotations except object_not.present. These issues are reflective of a subset of prop- erties which can arise in real-world images. We use these annotations from ORBIT and CURE-OR to understand the properties of difficult few-shot tasks. In Section 5.3.2] we provide some early analy- in this regard, where we find that difficult tasks have distribution shift between the support and query set, with respect to various image-level properties such as blur, viewpoint etc. Methodological evaluation on CURE-OR and ORBIT, will enable researchers understand the specific dimensions of image characteristics in which in which robustness has improved.
Choice for ORJECTNET. commonly used high- variation test-set in supervised deep learning is OBJECTNET, which is often used for testing the out-of-domanl generalization capabilities of models. In particular OBJECTNET comprises of images of objects various viewpoints, backgrounds and rotations which results in high variation within a class of objects. However, we note that these annotations are not made public by the (Barbu et al. 2019). Considering the ability of our algorithm to extract tasks with distribution shift between support and query with respect to various image characteristics in high variation datasets such as ORBIT and CURE-OR, we extract difficult tasks from ORJECTNET and add them our benchmark HARD-MD+++
15

---

Published as a conference paper at ICLR 2023
As the next step, the difficult tasks from the MD split in HARD-MD++ be annotated similar to CURE-OR or ORBIT to furthen our understanding the properties difficult few-shot tasks.
C WHAT CAN MAKE FEW-SHOT TASK DIFFICULT?
Prior works (Agarwal al. 2021 Massiceti et al. 2021 Dhillon al. 2019a Arnold et al. 2021 have shown that the composition of the support or query set is crucial in determining the difficulty level of task. This difficulty level is usually characterized by the loss accuracy the query set. While an intuitive characterization, it gives limited insights into why the given few-shot task was difficult. In this section, we aim shed light this question by specifically focusing role that natural image characteristics (e.g., quality issues, multiple objects) play in making a task difficult. These natural characteristics can arise the wild, hence we specifically focus on them. We note that there can be other factors controlling task difficulty such as fine- tine-grained classes, however is outside the scope of this paper (see Table for more details). We hypothesize that tasks which are difficult for given few-shot learner have a mismatch (distribution shift) between the support and the query set with respect to certain image-level characteristics. To validate our hypothesis, we curate annotated few-shot tasks where there is mismatch in the characteristics between the support and query set. We leverage three vision datasets which are annotated with various image-level attributes to curate the few-shot tasks: (i) ORBIT (Massiceti et al., (021) which contains annotations designating the frame cluttered or not as well annotations various quality issues such as blur, occlusion, overexposure, underexposure, viewpoint and framing. (ii) CURE-OR (Temel al. 2018) consists of images of various objects different viewpoints. (iii) MSCOCO (Lin et al.) 2014) consists of annotations about the number of objects an image.
the
Mean Accuracy
Tasks with Shift
CURE- OR
140
100-
200
30-
60-
100-
Number Tuaks
50
20
80
100
0
ORBIT-Quailty Issuss
140
108
120
90
75
80
80-
40-
20
40
60
80
100
Quary Accuracy
Figure 7: Curated tasks with a distribution shift between their support and query sets with respect image-level attributes have a lower accuracy average. Across 600 tasks (5-way, 5-shot) per dataset on ViT-S(DINO). Task curation details in Appendix D
Sampling Protocol. For each the annotated datasets, we curate difficult few-shot tasks in two steps: (i) We sample query set Q which contains certain set of attributes; (ii) For curating tasks without shift, we curate a support set containing similar attributes as Q. For tasks with shift, the curated support contains attributes different from those in Q. Further task curation details in Appendix
find that tasks that have distribution shift between the attributes of their support versus query set have lower query accuracy on average compared to tasks that do not have this shift. We also find strong correlation between the query accuracy and the inverse of the query
Results. From Fig
16

---

Published as a conference paper at ICLR 2023
loss with Spearman correlation of 0.90 (See Fig 8 This shows that tasks with high query loss (low accuracy) is representative distribution shifts between the support and query with respect to natural image characteristics.
C.I CORRELATION PLOOTS
2.5
Inverse of Query Loss
1.0
Query Accuracy
Figure 8: Spearman correlation of 0.90 between the query accuracy and inverse of the query loss for tasks sampled from CURE-OR and ORBIT. 600 tasks are sampled from each of the dataset.
D HEURISTIC FEW-SHOT TASK CURATION DETAILS
In Fig 7 we heuristically curate two types of tasks leveraging attribute annotated vision datasets such CURE-OR, ORBIT and MS-Coco: (i) Tasks with shift between the support and query (ii) Tasks without shift between the support and query set. Below we provide details on how these tasks curated:
D.1 MS-COCO
Tasks with Shift. We sample N classes (where N 5) and randomly select a query set Q from the pool of images having only one object. The support set S is randomly selected from the pool of examples containing more than objects. Each task a 5-way, 5-shot task, where there large mismatch between the number of objects present in the support query.
5) and randomly select a query set Q from the pool of images having only object. The support set S is randomly selected from the pool of examples containing only one or two objects. Each task is 5-way, 5-shot task, where there is no significant mismatch between the number of objects present in the support vs. query.
Tasks without Shift. We sample N classes (where
D.2 ORBIT
D.2.1
ORBIT-CLUTTER
Tasks with Shift. We first randomly sample a user and N classes (where N=5). The query set Q then selected from the clean frames. The support set corresponding the the same user and N classes then selected randomly from the cluttered frames. This sampling procedure ensures that there is shift between the support and query with respect to the number of objects in the frame.
Tasks without Shift. We first randomly sample a user and N classes (where N=5). The query set Q is then selected from the clean frames. The support set corresponding the the same user and N classes is then also selected randomly from the clean frames. This sampling procedure ensures that there is shift between the support and query with respect to the number of objects in the frame.
17

---

Published as a conference paper at ICLR 2023
D.2.2 ORBIT-QUALITY ISSUES
Tasks with Shift. We first randomly sample a user and N classes (N = 5) corresponding to the user. We then select query set Q which contains images without any quality issues. To create the support set, we create subset by selecting frames which have one or more quality issues (i.e. blur=-True OR peclusion=TTrue OR wverexposure==Tnut OR underexposure--True OR viewpoint== True OR Traming==True) Then, we randomly sample a support set S from this subset which contains frames with quality issues. This design choice ensures that the tasks contain a significant shift between the support and query with respect to various quality issues.
Tasks without Shift. To create tasks without any shift, use the default sampler from ORBIT, but modify it to select axed-way, fixed-shot tasks with the query set Q and support set S. Because the tasks are sampled randomly, a majority of the tasks will only have minor shifts with respect the various quality issues.
D.3 CURE-OR
Tasks with Shift. We sample N classes (where N 5) and randomly select a query set Q which has objects in either (i) front or back viewpoints; (ii) side-viewpoints and (iii) top viewpoints. If () is selected, then the support set S is composed of images only with side-viewpoints. If (ii) is selected, then the support set is composed of images with only the front or back viewpoints. (iii) is selected, then the support set S is composed of images with front, back or the side viewpoints. This ensures that the tasks have a shift between the support and query with respect to the viewpoint attribute.
Tasks without Shift. We sample N classes (where N = 5) and randomly select a query set Q which has objects in either (i) front or back viewpoints; (ii) side-viewpoints and (iii) top viewpoints.Ir (i) is selected, then the support set S is composed of images only with the front or back viewpoints. If (ii) selected, then the support set S is composed of images only with the side viewpoints. Similarly, if (iii) is selected, the support set is composed of images containing only the top viewpoint of the object. This design decision ensures that tasks do not have any shift between the support and query with respect to the viewpoint attribute.
We note that these rules or heuristics are not exhaustive to create tasks which have shift between the support and query set with respect to various attributes. However shown in Fig 7 these rules are able to sample tasks which have lower accuracy than tasks which do not have any shift between the respective support and query sets. Moreover, designing such heuristic rules is not straightforward and requires a significant amount of human effort. FASTDIFFSEL effectively addresses this problem and provides way to automatically extract difficult tasks, where the support and query set have shifts with respect to various factors.
E MODEL CHOICES
In the past few years, tremendous amount of research has been initiated in developing few-shot learning algorithms mostly spanning meta-learning (Snell et al.] 2017 Lee et al.) 2019 Bertinetto al., 2019:; Finn et al.) 2017a). However recent works (Chen al. 20200 Tian et 2020 have shown that a well-trained feature extracted via supervised learning is competitive and often surpasses state-of-the-art meta-learning algorithms. In (Hu et al.) 2002). the authors show that upgrading the feature extractor and large-scale scale pre-training improves few-shot learning performance on challeng- ing benchmarks such as MD. In our paper, we motivate our model choices from (Hu et al., 2022) and focus on range of pretrained models which are adapted via Prototypical Networks (Snell et al.) 2017) or full-model fine-tuning. Below we provide the architectural details for the models tested on HARD-MD++:
resnet18-100 We use 4 different pre-trained ResNet (He et al. 2015) architectures (resnet-
18,34,50,100 where the pre-training corpus is 2212 (Deng et al. 2009).
small patch16.. We use ViT-S/16 pre-trained with DINO (Caron et al. 202 on
ilsvre-2011 (Deng et al. 2009).
dino base patch16.
1/16 pre-trained with DINO CCaron et al. 2021 on
ilsvre 2012 (Deng al.
2009
18

---

Published as a conference paper at ICLR 2023
Jlaoebase.a 21k. ViT-B/16 pre-trained on ImageNet-21k 21k (Deng al. 2009)
vit.small_1k, VIT-S/16 pre-trained on ilsvre sorc__012@ (Deng et al. 2009).
vit base 1k. ViT-B/16 pre-trained ilsvre 2012 (Deng al. 2009).
beit base 21k. ViT-B/16 pre-trained using BeiT (Bao et al, 2021 on ImageNet-21k (Dng
et al.] 2009).
dino_ resnet50.
ResNet-50
pre-trained with DINO
CCron et al. 2021 on
ilsvre 2012 (Deng al. 2009).
clip clip.resnetSh resmet50. Visual encoder from CLIP (Radford et al.
2021 pretrained on YFCC100M
ResNett550.
.
clip vit base. Visual encoder from CLIP Radford et al. 2021 pretrained on YFCC100M
VVTT-B/16
We believe that these model choices cover a wide-range of pre-training paradigms including both self-supervised and supervised pre-training.
F FINETUNING DETAILS
Fine-tuning during adaptation. We adopt the fine-tuning recipe from (Hu et al. 2022) for fine- tuning the pre-trained checkpoints. We specifically fine-tune the entire backbone for all the models, rather than partial adaptation. The fine-tuning algorithm from (Hu et al.) 2022) has three hyper- parameters: (i) learning rate; (ii) number of fine-tuning steps and (iii) probability of switching on data-augmentattion for the support set. In our experiments, we set the number of fine-tuning steps as 50 and the probability of switching on data-augmentation for the support set as 0.9. We select the optimal learning rate from {0.0001, 0.001, 0.01 with the help of a separate validation set which comprises of tasks per sub-dataset in HARD-MD++.
ProtoNets during adaptation. In this adaptation strategy, the support set is first used to compute class-specific prototypes. Then each query example is assigned the class, based on its closest dis- tance the class- prototypes.. During adaptation, no parameters learnt which makes it extremely fast during adaptation.
For both the types of adaptation MD and HARD-MD++, images are resized to 128x128 following (Hu et al. (022) to ensure fairness in comparison. We note that increasing the resolution to 224x224 can slightly improve performance on MD (Bronskill et al. 202I) with LITE training strategies how- ever we leave it for future exploration.
G META-TRAINING DETAILS
The model used for extracting the difficult support set is meta-trained using Prototypical Networks (Snell et al. 2017) from its pre-trained initialization on the ilsvre 2012 split from MD. We train this model using distributed training 8 a6000 GPUs. We the training for 50 epochs, where in each epoch 2000 episodes from ilsvrc_2012 are sampled. In total, we meta-train on 100k episodes variable-way, variable-shot episodes from flsvre_2012 We also train the models for 100 epochs, but observe that the best model gets selected before the 50th epoch, hence resort to training for 50 epochs across. We train using the SGD optimizer with a momentum of 0.9. We use learning rate of 5e-4 with cosine scheduler for our experiments. During training, following (Hu et al. 2022). we resize the images to 128x128.
H MORE STRESS- STRESS-TESTING TESTING RESULTS
H. FIXED-WAY, FIXED-SHOT RESULTS
Using our algorithm we also extract fixed 5-way, 5-shot tasks from MD. We evaluate all the 13 models these fixed tasks from the original MD as well HIADD-M.. Overall, across both the adaptation strategies we find a drop of ~30% in performance on HARD-MD. Similar to the results for variable-way way, variable-shot sampling, find that ViT-B with CLIP initialization has strong few-shot performance. In particular, ViT-B with CLIP has the best performance on HARD-MA when it is fully finetuned during adaptation, while ranks 2nd when Prototypical Networks is used during
19

---

Published as a conference paper at ICLR 2023
dino small patch1s
dino, base, patch16
resnet2g
mosnet34
resnet50
resoutioz
vi base 21k
small 1k
Bese 1k
beit base
Mi baze
dipp resnetSo
ding resnetso
80.5
Hard-ND
43.7
43.7
Biseeaongg Accuracy
[a] Prototype Networks during Adaptation
dine small patch1s
dioo base petch16
resnatle
resnet34
resnet50
resneti01
WE base,
Daase
21k
base
dipp resnet5o
resnetso
83.5
83.2
81.1
82.1
49.7
81.4
78.6
80.5
Hard-ND
47.5
47.9
26.1
(a) Full Finetuning during Adaptation
Evaluation of fixed 5-way,5-shot tasks on META-DATASET and HARD-META- DATASET): We find a consistent drop in performance across fixed-way, fixed-shot setting with VTT-B(CLIP) performing the best with fine-tuning, while it ranks 2nd with Prototypical Networks.
Figure 9:
dino small ppachh16
dino base, petch16
resneti8
1osnet34
rennetton
base 21k
wit smmmmall 1k
vit. basebaeee .1k
beit baoe 21k
clip vit base
cip resnet50
dino, resnet50
ObjectNet-
15.7
19.1
16.4
18.6
17.6
18.4
25.8
28.6
29.5
17.4
43.4
26.2
13.3
CURE OR 34.7
35.1
33.0
33.9
36.6
36.7
37.3
42.3
14.3
35.5
43.2
26.0
29.0
32.4
32.5
38.5
18.1
32.9
29.3
26.4
Figure 10: Evaluation of fixed 5-way, 5-shot tasks on from OBJECTNET, CURE-OR and ORBIT with full-finetuning: With ORBIT, we find that vision transformers (ViT-S and ViT-B) pre-trained on ilsvre. 2012 to performing the best. For OBJECTNET and CURE-OR, we find that CLIP pre- training to be still performing the best amongst all the other models.
adaptation. However note that ViT-B with CLIP lags behind other models on fixed-way, fixed-shot tasks sampled from original MD.
The combined results with Section imply that vision- vision-language pre- pre-training can offer more robust- ness difficult tasks, than tasks sampled randomly. A thorough and more fine-grained study of vision-language models on HARD-MD can be a direction of future work.
H.2 FINETUNING RESULTS FOR ORBIT, CURE-OR AND OBJECTNET
The full finetuning results for difficult tasks from OBJECTNET, CURE-OR and ORBIT can be referred Fig I0 In particular for OBJECTNET, we find VÎT-B with CLIP to be performing the best, while for CURE-OR, we find ResNet-50 with CLIP initialization to be performing the best. For ORBIT though, we find that vision transformers (VTT-S or ViT-B) pre-trained on ilsvre 2012 to have the strongest performance.
H.3 BREAKDOWN OF RESULTS FOR HARD-META-DATASSS
In this section, provide breakdown of the results and HARD-MD at the sub-dataset level for both the adaptation strategies. In particular we report the average task accuracy along with the 95% confidence interval.
20

---

Published as a conference paper at ICLR 2023
H.3.1
ADAPTATION: PROTOTYPICAL NETWORKS
Hard-MD
Avernge Tack Arcurary
manatoo
menettas
monatoo
Models
Figure II: ilsvrc. 2012
MD Hard-MD
Accuracy
resneddo
menertes
Wi.aase 31x
bare Crosssaats cro
Models
Figure 12: MSCOCO
100
Hard-MD
Average
menetta)
base st sonato
Models
Figure 13: Aircraft 21

---

Published as a conference paper at ICLR 2023
120
Ilard-MD
Averaae
iseneboo monetts un
bett De die
Models
Figure 14: Cu-birds
120
MD
Hard-We DDD
Average
smal pandb1
woneth lecsedo
menettes
de seeaadt
un
balt
Models
Figure 15: Traffic-Sign
MD HarddMMD
Average Taak Accuracy
monettos
2tt
ber
yeus sened
Models
22
Figure 16: Fungi

---

Published as a conference paper at ICLR 2023
120
Iaard-MD
Averaae
iseneboo monettss
un
bet De die
Models
Figure 17: DTD
Hard-MD
rennettr.
sh.seen Jenth 25.
Jonnn 2t
bentaae
de maauea
Figure 18: Quickdraw
MD Hardd-MD
Average Taak Accuracy
monettos
ber
yeus de diaceenn venad
Models
23
Figure 19: Omniglot

---

Published as a conference paper at ICLR 2023
Ilard-MD
Averaae
iseneboo
monetts
un
ah bet
Models
Figure 20: VGG-Flower
H.3.2 ADAPTATION: FULL FINETUNING
Accuracy
Los.
Jeat 2la
Mianna St.
bettbace Be
Models
Figure 21: ilsvre 2012
24

---

Published as a conference paper at ICLR 2023
122
Ilard-MD
Average
iseneboo
monettos
un
hett
Models
Figure 22: MSCOCO
300
Hard-MD
Average Accuracy
ww1a pubete
basa
Models
Figure 23: Aircraft
Average
bethace
Models
Figure 24 Cu-birds 25

---

Published as a conference paper at ICLR 2023
122
Ilard-MD
Averaae
iseneboo
monettos
un
bett
Models
Figure 25: Traffic-Signs
MD Hard-MD
Accuracy
puidile
monat3s manatoo
remnetiss
So
St
base
Models
Figure 26: Fungi
MD HarddMD
Average Taak Accuracy
meneets
monettos
yeus
La her
Modela
26 Figure 27: DTD

---

Published as a conference paper at ICLR 2023
Iaadd-MD
Averaae Accuracy
iseneboo
monettos un
bett
Models
Figure 28: Quickdraw
Average
Weaane 3iee
base
Models
Figure 29: Omniglot
Average Accuracy
Los.
Models
Figure 30: VGG-Flower
27

---

Published as a conference paper at ICLR 2023
H.4 IMPACT OF META- TRAINING ON LLSVRC22012
We meta-train a subset of the 13 models on the ilsvre flsvrc_2012 split from META-DATASET, In particular, meta-train siinosmmallppa patcch1 dino_base patch 16, @ino_resnet50 and vit_base_ 11k from their pre- trained initialization. Prior work (Hu et al., [022 Chowdhury et al.) 202T) has shown that fine-tuning from strong pre-trained checkpoint during adaptation more beneficial than meta-trainiin on ilsvre 2012, therefore we primarily focus on the full fine-tuning experiments (see Section 4 in our paper. However, to verify if meta-training on ilsvre 2012 leads any significant improvements, we evaluate 4 meta-trained models on HARD-MD, From Fig 31 we find that the ilsvrc_2012 meta- trained models still suffer on MARD-MD and incur a ~20-30% performance drop.
dino small patchis
dine, base patthhi
dino resnetSo
We Sase, Jk
ND
68.8
64.0
67.7
Hard-MD
39.8
41.6
34.0
44.1
Figure 31:
Effectiveness of Meta-Training on the ilsvre. 2012 split from META-DATASET. We observe a 20--30% drop in performance the MD split from HARD-MD.
I VISUALIZATION DIFFICULT TASKS
In this section, provide visualizations of some of the difficult tasks which extracted by algorithm. We provide only a few visualizations for representative purposes and believe a large-scale qualitative study of the difficult tasks in
is future course of study.
Extracted Support Set
Query Set
1
Class
Class
Class
Class 3
Class
Class 4
70
Class
Class 5
Class 5
Observation: Change in lighting condition between support and query set for certain classes
Figure 32: Difficult Task from Traffic-Sign: This extracted task from traffic_sign has accuracy of 15%. We find that this task has distribution shift between the support and query with respect to various lighting conditions. For e.g., Class 5 in the extracted support set has almost no light when compared to the images from Class the query set.
28

---

Published as a conference paper at ICLR 2023
Extracted Support Set
Query Set
Class
Class 2
Class
Class
Class
Figure 33: Difficult Task from aircraft: This extracted task from aircraft has accuracy of 21%. We find that this task has distribution shift between the support and query with respect to background. For e.g., Class class in the query set has all the airplanes on the ground. Similar for Class I and Class 2, where there is distribution shift with respect background characteristics.
the extracted support set has all the airplanes in the sky, whereas
Extracted Support Set
Query Set
A
Class 1
Class 2
Class 3
Class 4
Class 5
Figure 34: Difficult Task from fungi: This extracted task from fungi has an accuracy of 23%. For Class 4, our algorithm extracts images which consistently have the fungi background of vegetation, whereas in the query set only one image in Class has a background of vegetation.
29

---

Published as a conference paper at ICLR 2023
Extracted Support Set
Query Set
Class
Class
Class 4
Class 5
Figure 35: Difficult Task from 2012: Although do not include ilsvre_2012 in HARD- MD++, we present difficult task which has an accuracy of 20%. Class 5 has clear distribution shift with respect to viewpoint and background. There are multiple objects in the extracted support set of Class 1, whereas Class in the query set only contains images of single objects.
These are randomly picked tasks from the pool of difficult tasks extracted by our algorithm, where we find that difficult tasks have a distribution shift between the support and query with respect to various characteristics such lighting, background, viewpoint. Annotation of the difficult tasks in HARD-MD++ is the next step to understand failure modes of few-shot learning methods on deeper level.
INTEGRATING OUR FRAMEWORK WITH FINE-GRAINED CLASS SAMPLERS
Our framework does not put any explicit constraints on the class-sampling procedure. As shown in Equation TT, our framework extracts difficult tasks where there is weak relationship between the support and query. In all our experiments, we follow the same class sampling protocol from MD [Tn- antafillou et al. (019). Note that the ilsvre 2012 split from MD uses a fine- ane-grained class-sampling protocol our framework extracts more difficult tasks than the ones sampled using fine-grained class samplers for this setting (see Fig 330). Therefore, our framework is complementary to any fine-grained class samplers. Recently (Bennequin et al. 2022) introduce a new fine-grained class- sampler for tieredImageNet consisting of tasks with classes which are semantically closer. The sampler can select tasks with different levels of coarsity, thereby providing control over the diffi- culty of the meta-test set. To understand the complementary nature of our framework, integrate the class- sanssssn protocol from (Bennequin et al.) 2022) with our optimization steps. We pre-train ResNet-12 on the training split from tiered Net consisting of 351 classes following the perised pre-training protocol in (Bennequin et al.) 2022). We then use this base model along with our framework extract difficult tasks from the test-set tiered-Image ImageNet..
Ist. Qui
2nd. Qui
3rd. Qul
4th.
Qil 62.7
Fine fine-grained protocol
42.01
51.2
56.7
Ours + Fine-grained protocol
24.3
29.3
31.4
33.1
Table 1: Our framework is complementary to the fine-grained sampling protocol from (Ben- nequin et al., al.,|2022) 2022). We find that our framework when used with the fine-grained sampling protocol extracts more difficult tasks, than fine-grained tasks itself, highlighting its complementary nature. Across 5000 tasks (5-way, 5-shot). Different quartiles represent different levels of task coarsity (smaller the coarsity, higher is the difficulty).
30

---

Published as a conference paper at ICLR 2023
K EFFECT OF THE NUMBER OF OPTIMIZATION STEPS
In our framework, take step of gradient ascent and subsequent step of the projection step per class. Empirically, we find that only one step is sufficient to extract difficult support sets, if the learning rate (a) for gradient ascent is set high 200. In practice, in our experiment test-bed, we observe that the difficulty of the task decreases if the number of iterations increases to greater than 1. This could be attributed the fact that the selection weight vector becomes sparse after one iteration, which might not be a good starting point for further steps of the optimization. Below we plot the effect of the number of iterations on the weighted prototype loss for a subset of the domains from HARD HARD-MD++, which we use as measure of the stopping criterion for our algorithm.
Query Accuracy with Weighted Proto-L Loss
70
60
40
30-
Number iterations
Figure 36: Number of Optimization Steps (Omniglot).
Query Accuracy with Weighted Proto-LosS
60
50
30
10
Number Iterations
Figure 37: Number of Optimization Steps (Aircraft).
31

---

Published as a conference paper at ICLR 2023
Query Accuracy with Weighted Proto-Loss
95
90
80
70
60
Number Iterations
Figure 38: Number of Optimization Steps (cu-birds).
Query Accuracy with Weighted Proto-Loss
100
90
80
70
60
50
40
Number of Iterations
Figure 39; Number of Optimization Steps (ygg-flower).
Note that post-processing step from the learnt selection weights extract tasks with given value of shot per class (i.e. ky samples for each the jth class). Empirically, we track the query accuracy with the extracted support and always find the lowest accuracy on the query set to be after one iteration of the algorithm.
L RESULTS ON MS-COCO (FULL)
The MS-COCO split in META-DATASET consists of object-cropped images based on the bounding box annotations. However, the original MS-COCO (Lin et al. (2014) (hereby denoted as MS-COCO (Full)) with bounding box and segmentation annotations can be a measurement for the number of objects present in the image. To understand, if distribution shift between the support and query with respect to the number of objects is a factor in increasing task difficulty, we perform a fine- grained study using
framework and MS-COCO (Full). In particular we fix the query set to have images with objects in the set: {1,2, 4, 6, 10). For these fixed query sets, we use our framework to extract easy and difficult support sets.
32

---

Published as a conference paper at ICLR 2023
Hard Tasks Easy Tasks
Number Objects Support
12
10
Number Objects Query
Figure 40: Large distribution shift with respect to number of objects between support and query for difficult tasks from MS-COCO (Full).
From Fig we find that our framework extracts difficult tasks where there is large distribution shift between the support and query with respect to the number of objects. Along with the results for ORBIT and CURE-OR (Section this validates that our framework extracts tasks which have weak relationship between the natural characteristics of the support and query set (e.g., occlusion, viewpoint, number of objects).
M RESULTS WITH 600 TASKS PER DATASET
dino, small patch16
dino, base, bast__aacchee
patch16
resnet18
reanet34
resnetso
reanet10)
base,
Mic. smaalmlll. 1k
Vit base,
beit base, 21k
clpp baze
eip. resnet50
dmo resnet5o
200 tasks-
47.6
48.2
43.7
44.5
47.0
48.6
51.5
49.0
51.3
18.5
52.2
50.6
36.7
600 tasks-
48.1
43.6
46.9
48.4
51.5
18.6
52.4
50.5
36.5
Figure 41: Results with 600 tasks on HARD-MD.
From Fig 41 we find that the ordering of the different models in terms of performance remain the same even when evaluated 600 tasks per dataset. For e.g., VIT-B with CLIP initialization still has the best performance on the HARD-MD split from HARD-MD+++
N FORMULATION AND RESULTS WITH TASK2VEC
Vee (Achille et 2019) is task embedding method which can provide vectorial representa-
tions for tasks. The core idea in task2vec is to compute an embedding based on the diagonal of the Fisher information matrix of probe network. Given a probe network Pw: parameterized by weights w, the task embedding is the diagonal of the Fisher Information Matrix (FIM) F:
F (Bn.y--Ma)pp V.. log(p (y [)) log(p)
(10)
In practice, the diagonal of the Fisher Information Matrix, which is a vector (where / = Diag F))
used the task embedding. principle task2vec can be used with our optimization framework to extract difficult tasks. Given fixed query set Q and search pool P, where each example in P has selection weight w, associated to
max ((L(Q),(P.
(11)
33

---

Published as a conference paper at ICLR 2023
inseuco
Traflic sign
llvrre 2012
omniggol
aiccraft
cu birds
did
quickdraw
lungi
veg Dower
DINO Loss DINO
43.8
29.7
Task2Ves
43.1
39.2
51.3
49.2
432
78.1
76.4
32.1
39.2
89.3
Table 2: Accuracy of extracted tasks using Proto-Loss vs. Tesk2Vec across different domains from META-DATASET: Using task2vec within our framework results in easier tasks than using Proto-Loss (current loss in our framework). Across 600 variable-way, variable-shot tasks per dataset. (Lower is better)
8.L. w² E [0,1];
[ E
w|w/lo S kj,
where t is the task embedding from task2vec and ((.) is the mean-squared emror. We solve eq. 11 using the same projection algorithm used in eq. (3) In this framework, we use ViT-S pre-trained with DINO (Caron al. 2021) as the probe network.
Findings. We find that using the task2vec formulation results extracted tasks (Table. 2 which easier than the Proto-Loss formulation used in framework eq. (T). Second, using task2vec is more computationally expensive than our method, as it involves computing the diagonal of the FIM which involves computing the gradients, whereas our current formulation only needs compute the Proto-Los which requires one forward pass.
O ABLATIONS WITH DIFFERENT BASE MODELS
miscoco
trafiic.sign
10s2
omnnggltt
aircraft
cu.birds
did
quickdraw
fungi
vgg flw-elower
VIT (S)+ DINO
21.06
11.4
30.5
43.8
29.7
67.1
66.3
8.2
25.7
83.3
(IT + DINO
21.4
11.9
31.3
43.2
28.8
67.9
67.
9.1
25.3
81.9
(B)+ CLIP
224
35.4
70.1
27.1
VIT (S)+ Deil
22.
142
34.1
45.1
30.1
69.3
70.1
113
26.3
814
(B)
33.2
31.3
71.1
68.1
Table 3: Effectiveness of our framework in extracting difficult support sets using different base models. Across 600 variable-way, variable-shot tasks per domain from META-DATASET,
In Table. 33.
we find that our framework flexible enough to be used with different base models for extracting difficult tasks. While in principle any base model can be used with our optimization framework, we choose ViT(S)+DINO as its the few-shot pipeline used by (Hu et al. 2022) which results in strong cross-domain few-shot performances on MD.
P INTEGRATION WITH DIFFERENT ADAPTATION STRATEGIES
In paper, choose the loss with prototypical networks (nelll al. 2017) the adaptation strategy due to it's fast adaptation. In principle, other relatively fast adaptation strategies such as R2D2 (Bertinetto et al.) 2019) be used. However, there an inherent problem using adaptation strategies such as R2D2 with MD. The adaptation step requires using the entire search pool P with the adaptation strategy. With META-DATASET'S variable-way, variable-shot sampling protocol, the search pool P can be extremely large due to large number of ways in certain tasks. For e.g., R2D2 requires solving a ridge-regressor in the adaptation step, where the optimal ridge regressor weights are denoted by:
W =
+X
(12)
where X contains the embeddings of the support set and X E g|'lxe, where [P| is the size of the search pool and e is the embedding size of the features from the base model. XX" E |PIX and grows quadratically with the size of the search pool size, which can be as large as 50k for certain datasets in MD such quickdraw 12sv2e_2012. Given the high dimensionality of XXx", its inversion will be extremely expensive making it infeasible to use it for fast difficult task extraction from large-scale vision datasets.
34

---

Published as a conference paper at ICLR 2023
Q
COMPARISON WITH FILTERED OUT DIFFICULT TASKS
Filtered Difficult Our Framework
Average Number of Ways
10-
dtd
vgg_ flowr
2012
omniglot
sign
quickdraw
macoco
Domains Meta-Dataset
Figure 42: Our extracted difficult tasks have lower number of ways than filtered out difficult tasks from META-DATASET'S random sampler.
We use our framework to extract 200 difficult tasks and run MD's random sampler for 10000 itera- tions and filter 200 tasks with the lowest accuracies. We then compare the number of ways in both the set of difficult tasks. Previously (Triantafillou et al., 2019) has shown that large number of ways in given task can be source of task difficulty. From Fig [2] we observe that across all the domains in MD, our framework is able to extract tasks with lower number of ways than the filtered out difficult tasks from MD's random sampler. This shows that a large number of ways not the primary source of difficulty in the tasks extracted by our framework. Furthermore in Sec- tion 5.3.2 through the lens of annotated datasets, we validate that the difficult tasks extracted by our framework have distribution shift between the support and query with respect to various natural characteristics.
R RESULTS ON ADVERSARIAL TRAINING WITH DIFFICULT TASKS
We note that adversarial training with difficult tasks have been previously investigated in (Agarwal al., 202T). Similar the observations in (Agarwal et al., 202T), we find that adversarial training does not have intended improvements during meta-testing. In particular, trained with DINO) with prototypical networks on 10000 difficult training tasks extracted using framework and 10000 randomly sampled tasks. We find that adversarial meta-training using difficult tasks does not lead to improvements on HARD-META MADDMEEA--DAASEETT This shows that one needs carefully design new training paradigms for improving the difficult tasks from HARD-META- DATASET.
meta-train VIT-S (pre-
traffic sign
ilsvre
omniglot
aircraft
birds
quickdraw
fungi
vgg flower
VTT + DINO
21.06
11.4
30.3
43.8
29.7
67.1
66.5
8.2
25.7
83.5
(S)+ Adversarial Training
21.4
12.3
30.1
42.3
29.5
66.5
67.1
9.1
26.
83.2
Table 4: Adversarial training does not improve performance the HARD-MD split from HARD-MD++. Across 600 variable-way. variable-shot tasks from each domain.
We note that the results in Table 4 are initial results adversarial meta-training and more carefully designed adversarial meta-training methods might be required to improve performance on HARD- MD++.
35

---

Published as a conference paper at ICLR 2023
S INITIAL RESULTS WITH TRANSDUCTIVE APPROACHES
We use the transductive fine-tuning approach introduced in (Dhillon et al. 2019a) and evaluate on the tasks from HARD-MD+++ We note that in our paper, we primarily evaluate inductive approaches in line with recently developed methods for MD (Triantafillou et al. 2019 Hu et al. 2022 Li et al., 2021I. Evaluating and understanding the true effectiveness of transductive approaches will require separate line of work. However, in this section we provide some initial results on how baseline transductive fine-tuning approaches fare the HARD-MD split from HARD-MD++, Essentiallyy we add regularizer to the fine-tuning cross-entropy loss to seek outputs from the query examples with a peaked posterior (i.e. low Shannon Entropy) similar (Dhillon al. 2019a).
dine small patchce patch16
dino, base patch16
resnett8
resnet34
resmetso
resnat101
wit. been, 21k
vit small Ik
Wit base, .1k
beit, base, 21k
clip Base
resnet30
dine, resnet50
FT
47.4
48.1
43.6
44.5
46.9
48.4
51.5
48.9
51.5
18.6
52.4
50.5
36.5
Transductive
52.1
Figure 43: Transductive Fine-tuning vs. Normal Fine-tuning (FT) on HARD-MD.
From Fig 43
find no clear benefit with transductive fine-tuning for the HARD-MD split from + However, investigation of more transductive approaches such as (Boudiaf et al.,
2020
Hu al. 2020) is warranted to fully validate the effectiveness of transductive approaches and can be good direction for future work.
T RESULTS WITH ADAPTERS
We leverage recently developed parameter efficient fine-tuning adapter based methods such as (Xu al. 2022) and compare with full network fine-tuning on the HARD-MD split from HARD-MD++.
tino small patchie
doc petchle
bari hare 21k
base 21k
small 14
We 28
vi base
Adepter 48.3
48.9
52.9
49.6
52.3
19.1
$3.9
Figure 44: Adapter Fine-tuning vs. Normal Fine-tuning (FT) on the HARD-MD split from HARD-MD++.
Note that the adapter design in (Xu al. 2022) is developed for VIT backbones and find the effectiveness of their method on HARD-MD with 7 different vision transformer pre-trained back- bones. general from Fig 44 we find that carefully designed adapter based methods outperform full-network fine-tume fine-tuning (though by small margin). A thorough evaluation of parameter efficient fine-tuning techniques (Jia al. 2022
Hu et al. 2021) on MARD-MD direction for future work.
36


Reviews:
[{"review_id": "w2_RRFB20k3", "reviewer": "Reviewer_tAM3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new algorithm to build efficiently a few-shot classification benchmark, named HARD-META-DATASET++ tailored to difficult tasks. In particular, the proposed algorithm is able to sample difficult tasks from various large-scale vision datasets in a deterministic way using a constrained optimization problem. The principle is to learn, through this constrained optimization problem, selection weights for each instance in a search pool in order to build a support set associated with a given query set. Compared to previous related approaches that use greedy search, the proposed algorithm is faster and thus enables to sample tasks in large-scale vision datasets. Using this algorithm, they build HARD-META-DATASET++ by sampling difficult tasks on META-DATASET, ORBIT, CURE-OR and OBJECT-NET. They then proposed to test various few-shot classification state-of-the-art approaches on HARD-META-DATASET++ and they show that, on average, these approaches fail on these hard tasks. ", "summary_of_the_review": "The paper proposes a constrained optimization algorithm for few-shot classification hard task sampling.  As such, I believe that the contributions of this paper are not significant enough to be published at the ICLR. In particular, as such the paper just brings a new benchmark on which state-of-the art approaches are weaker but it does not address at all the problem of better understanding and definition of why a task is difficult in few-shot, in particular in real-world settings. More insights are needed on this particular point and in relation with the images and labels of the targeted task.\n\n\n", "strength_and_weaknesses": "**Strengths**\n\n+ This paper targets an important issue in the few-shot classification community: the lack of benchmarks that are enough representative of real-world few-shot tasks. This problem is currently understudied. In particular, the proposed approach focuses on hard tasks defined as tasks for which a given few-shot classifier performs worse than for all other tasks (on average).\n+ The paper is easy to follow and well-structured.\n+ The resulting HARD-META-DATASET++ if it is publically released.\n\n**Weaknesses**\n+ My main concern is related to the proposed definition of a difficult task. The proposed definition is relative to other tasks but does not take into account the content of the task itself, i.e. the labels and the image. Indeed, in real-world scenarios, hard tasks are for instance fine-grained classification ones or tasks with shifts between the support sets and the query set.  \n+ Moreover, the proposed approach to sample hard tasks depends on the chosen loss, here proto-loss but also on a meta-trained based model. This point should be more discussed and in particular what is the impact of the choice of these two ingredients on the findings of the paper.\n+ The paper title mentioned understanding few-shot performance but the \"understanding part\" is not enough detailed and developed in the paper.\n+ Some important references are missing such that the paper of [(Benequin el al, 2022)](https://openaccess.thecvf.com/content/CVPR2022W/VDU/html/Bennequin_Few-Shot_Image_Classification_Benchmarks_Are_Too_Far_From_Reality_Build_CVPRW_2022_paper.html) (Few-Shot Image Classification Benchmarks are Too Far From Reality: Build Back Better with Semantic Task Sampling, that also tackles the weaknesses of current few shot benchmarks for real-world applications. Moreover, various approaches have been proposed to define a task such as for instance [Task2vec](https://arxiv.org/abs/1902.03545) (Achille et al, 2019) and the paper should mention them and position compared to them.\n+  Transductive inference is widely used in few-shot learning. What about transductive few-shot learning in the proposed approach?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "svUyE9evadm", "reviewer": "Reviewer_yM4m", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors proposed a novel efficient algorithm to extract hard tasks from existing large-scale datasets.\n\nFinding the difficult episodes, (support set, query set) pairs, is a combinatorial problem, and the authors propose to use a model’s loss on the task’s query set, as a proxy for the difficulty of the task, and cast the problem as an optimization problem.\n\nUsing the proposed method, the authors extracted a dataset from (MetaDataset, ORBIT, CURE-OR and OBJECTNET). \n\nThe resulting dataset is used as a test-only few-shot classification benchmark, and is named HARD-META-DATASET++. The authors evaluate a few popular few-shot classification methods on this benchmark.", "summary_of_the_review": "The paper presents an interesting way to construct the difficult subset from large scale datasets. The prospered methods shows promising effectiveness in construction such difficult set. The authors successfully constructed the HARD-META-DATASET++, and conducted the evaluation of various feature extractors (fig 4). However, the paper lacks comparison between the proposed approach and existing approaches. The proposed method also has some constraints: hand-picked f (ProtoNet), and the authors used 1 gradient step to solve the optimization wo/ justification.", "strength_and_weaknesses": "Strength\n- casted the difficult support set extraction as a constrained optimization problem, to optimize the \"selection weights\" (eq 1.)\n- solved the optimization problem using:\n   1) ProtoNet as feature extractor\n   2) applies 1-step gradient descent w.r.t the \"selection weights\"\n   3) projection w and uses l1 relaxation, and solved the relaxed version using its dual form\n- The difficult episodes are obtained by drawing the examples with the highest \"selection weights\"\n- The algorithm can speedups of difficult episode search by 20x\n- The constructed hard datasets is difficult for FSL models (fig 3)\n\n\nWeakness\n- Does number of steps have any effect on the final set? E.g., the current optimization of the \"selection weights\" uses 1 projection step.\n- The optimization involves using f, and its set to a ProtoNet. Does the selection of f affect the accuracy of the difficult set selection?\n- There are other methods to extract hard episodes, have the author compared with existing approaches?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "VVpqG16A5E", "reviewer": "Reviewer_Q1sQ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the failure cases in FSL and proposes an efficient algorithm to extract the difficult tasks from large-scale datasets. Based on the proposed algorithm, the paper builds a new test-only few-shot classification benchmark named HARD-META-DATASET++.", "summary_of_the_review": "This paper builds a new test-only few-shot classification benchmark named HARD-META-DATASET++. However, I have some concerns that whether the HARD-META-DATASET++ can be a reliable estimation (Please refer to 1) in Weaknesses). \n", "strength_and_weaknesses": "Strength:\n1) The paper is writing clearly and easy to read.\n2) The paper builds a new test-only few-shot classification benchmark named HARD-META-DATASET++. \n\nWeaknesses:\n\n1) The support samples of \"difficult task\" is selected by fixing the query samples which means the difficulty of each task is highly correlated with the fixed query data. The hypothesis is somewhat unreasonable, since the FSL aims to adapt to the whole new class not the fixed query samples. The reviewer supposes that the tasks in the HARD-META-DATASET++ may not be a reliable estimator in the test-time in FSL. \n\n2) The proposed method can be seen as an extension of the paper[1], since the paper[1] have already proposed that FSL are extremely sensitive to the data used for adaptation and used a greedy algorithm to find those difficult tasks. Although the proposed method is more efficient, the novelty is somewhat limited.\n\n3)The evaluation process consists 200 tasks using Prototypical Network, which is not enough(suffering from high randomness). In recent literature, the number of evaluation tasks is usually more than 2000.\n\n\nAs for questions, I would like to ask:\n\n1 In FSL, we usually report the mean and the variance of the accuracy over 2000 tasks. The variance of the accuracy also denotes the model's performance of the challenging tasks(The higher variance, the lower accuracy on more difficult tasks). Besides, the average accuracy of several worst cases can also be a good estimator. Why is the accuracy on HARD-MD  a better criterion for evaluation?\n\n2 How to use the \"difficult task\" in the training phase in FSL? Can the \"difficult task\" in base classes help the model improve the generality to novel classes?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "GBkKVLP1FH", "reviewer": "Reviewer_trsF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper mainly focuses on few-shot learning. Different from the previous methods pursuing good overall accuracy, the authors study the performance on hard few-shot tasks. Specifically they propose a novel algorithm based on constrained optimization to find difficult support samples given query samples. Such a method is much efficient than greedy search. Based on the proposed method, the authors conduct extensive experiments on a new version of Meta-Dataset using existing methods and provide many interesting results.", "summary_of_the_review": "This paper presents a new Few-shot learning setting, in order to better understand the difficulty of FSL. \nMy personal suggestion is that some insightful analysis may be appreciated, and importantly some recent adaptation methods may be compared in this setting. This can show that this setting is really meaningful.", "strength_and_weaknesses": "Strength:\n1.\tThe idea of analyze the performance on hard tasks is practical.\n2.\tThe proposed algorithm is simple yes effective.\n\n\nWeaknesses:\n1.\tWhile the authors provide abundant results, the corresponding analysis is not so insightful. It is common sense that there are many potential reasons that can lead to low accuracy on query samples. For example, the support samples can either be hard to fit or easy to overfit, which are totally different when considering the properties of these hard tasks. It is hard to understand why these previous methods have large performance gap between the original MD and Hard-MD++ solely based on the average accuracy. It would be better if other metrics can be provided, e.g. the training loss curves on each dataset, the relation between the performance and shot/way number, etc.\n\n2.\tThe paper would be more comprehensive if other adaptation methods like DCM [1], TSA [2] and eTT [3] can be used in the experiments.\n\n3.\tI wonder if it is possible to measure the average ‘hardness’ of the original experiment setting on MD, i.e. 600 episodes for each dataset. This is helpful for indicating the chance of handling such hard tasks in real-life scenarios where hard and easy tasks are mixed.\n\n[1] Powering Finetuning for Few-shot Learning: Domain-Agnostic Bias Reduction with Selected Sampling. AAAI 2022\n[2] Cross-domain few-shot learning with task-specific adapters. CVPR 2022\n[3] Exploring Efficient Few-shot Adaptation for Vision Transformers. TMLR 2022\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster