MetaCoCo: A New Few-Shot Classification Benchmark with Spurious Correlation
Authors: Min Zhang
Haoxuan Li
Fei Wu
Kun Kuang
Venue: ICLR 2024 poster (2024)
Keywords: Benchmark; Few-shot classification; Spurious-correlation shifts
Primary area: datasets and benchmarks
TL;DR: We present MetaCoCo, a benchmark with spurious-correlation shifts collected from real-world scenarios for few-shot classification.

Abstract: Out-of-distribution (OOD) problems in few-shot classification (FSC) occur when novel classes sampled from testing distributions differ from base classes drawn from training distributions, which considerably degrades the performance of deep learning models deployed in real-world applications. Recent studies suggest that the OOD problems in FSC mainly including: (a) cross-domain few-shot classification (CD-FSC) and (b) spurious-correlation few-shot classification (SC-FSC). Specifically, CD-FSC occurs when a classifier learns transferring knowledge from base classes drawn from \underline{seen} training distributions but recognizes novel classes sampled from unseen testing distributions. In contrast, SC-FSC arises when a classifier relies on non-causal features (or contexts) that happen to be correlated with the labels (or concepts) in base classes but such relationships no longer hold during the model deployment. Despite CD-FSC has been extensively studied, SC-FSC remains understudied due to lack of the corresponding evaluation benchmarks. To this end, we present Meta Concept Context (MetaCoCo), a benchmark with spurious-correlation shifts collected from real-world scenarios. Moreover, to quantify the extent of spurious-correlation shifts of the presented MetaCoCo, we further propose a metric by using CLIP as a pre-trained vision-language model. Extensive experiments on the proposed benchmark are performed to evaluate the state-of-the-art methods in FSC, cross-domain shifts, and self-supervised learning. The experimental results show that the performance of the existing methods degrades significantly in the presence of spurious-correlation shifts. We open-source all codes of our benchmark and hope that the proposed MetaCoCo can facilitate future research on spurious-correlation shifts problems in FSC.

Published as a conference paper at ICLR 2024
METACOCo: A NEW FEW-SHOT CLASSIFICATION BENCHMARK WITH SPURIOUS CORRELATION
Min Zhang'
Haoxuan Li
Fei Wu'
Kun Kuang"
Zhejiang University "Peking University [zhangmin, mi lab, wufei, kunkuang]e e edu. en, bxli8stu.pku.ee
ABSTRACT
Out-of-distribution (OOD) problems in few-shot classification (FSC) occur when novel classes sampled from testing distributions differ from base classes drawn from training distributions, which considerably degrades the performance of deep learning models deployed in real-world applications. Recent studies suggest that the OOD problems in FSC mainly including: (a) cross-domain few-shot classifi- cation (CD-FSC) and (b) ppurious-correlation few-shot classification (SC-FSC). Specifically, CD-FSC occurs when a classifier learns transferring knowledge from base classes drawn from seen training distributions but recognizes novel classes sampled from unseen testing distributions. In contrast, SC-FSC arises when a classifier relies on non-causal features (or contexts) that happen be correlated with the labels (or concepts) in base classes but such relationships no longer hold during the model deployment. Despite CD-FSC has been extensively studied, SC-FSC remains understudied due to lack of the corresponding evaluation bench- marks. To this end, we present Meta Concept Context (MetaCoCo). a benchmark with
shifts collected from real-world scenarios. Moreover, to quantify the extent of spurious-correlation shifts of the presented MetaCoCo, we further propose metric by using CLIP pre-trained vision-language model. Extensive experiments on the proposed benchmark are performed to evaluate the state-of-the-ane methods in FSC, cross-domain shifts, and self-supervised learning. The experimental results show that the performance of the existing methods de- grades significantly in the presence of spurious-correlation shifts. We open-source all codes of our benchmark and hope that the proposed MetaCoCo can facilitate future research on spuriousscoorelation shifts problems in FSC. The code is avail- able at: https//gghh..ccc//ccmm////ncccco
INTRODUCTION
Few-shot classification (FSC) aims to recognize unlabeled images (or query sets) from novel classes with only few labeled images (or support sets) by transferring knowledge learned from base classes. Despite the impressive advances the FSC, in real-world applications, outto--distribution (OOD) problems in FSC occur when the novel classes sampled from testing distributions differ from the base classes drawn from training distributions, which significantly degrades the performance and robustness of deep learning models, and has gained increasing attention in recent years (Song al., 2022 Li et al. (202d).. As shown in Figure [ the OOD problems in FSC can be broadly categorized into two categories with different forms of distribution shifts: (a) cross-domain few-shot classifi- cation (CD-FSC) and (b) spurious-correlation few-shot classification (SC-FSC), as established by previous works (Triantafillou et al. 2020; Yue al. 2020 Luo et al. 2021 Li al. 2022).
Cross-domain few-shot classification (CD-FSC). Cross-domain shifts occur when a classifier learns transferring knowledge from base classes drawn from seen training distributions but rec- ognizes novel classes sampled from unseen testing distributions. For example, in COVID-19 pre- dictions, we may want to train a model on patients from a few sampled countries and then deploy the trained model to a broader set of countries. Existing OOD methods in FSC have shown con- siderable progress solving the cross-domain shifts problem (Hou et al. 2019 Doersch et al.)
"Corresponding author.

---

Published as a conference paper at ICLR 2024
Base Classes
Novel Classes
mini
VGG
CUB
Flower
Aircraft
(a) Cross-domain few-shot classification (CD-FSC)
Dog
Cat
Bicycle
Bear
Train
Horse
Autumn
Snow
Rock
Water
Grass
Dim
Autumn
Snow
Rock
Water
Grass
Dim
(b) Spurious correlation few-shot classification (SC-FSC)
Figure Example of cross-domain shifts and spurious-correlatinn shifts in FSC. (a) In Meta- dataset with cross-domain shifts (Triantafillou et al. (020), the model is trained on base classes sampled from three datasets including minilmageNet, CUB-200-2011 and Aircraft, then tested on novel classes drawn from VGG Flower. (b) In our proposed MetaCoCo with spurious-correlation shifts, each class (or concept, e.g., dog) consists of different backgrounds (or context, e,g., autumn).
2020
2021 Liang et al. 2021 Wang & Deng Guo et al. 2020 Wang & Deng 2021; Sun et al.
2021 Li et al. 2023a Oh et al. 2022 Zhang et al. 2020 2022b). Mcanwhile, two standard cross-domain benchmarks have been proposed to evaluate the effectiveness of these methods, iee. Meta-dataset (Triantafillou al. 2020) consisting of 10 existing datasets, and BSCD-FSL (Guo ctt al. 2020) consisting of 4 existing datasets. Figure aa) shows the example of cross-domain shifts on Meta-dataset, where mini (miniImageNet), CUB (CUB-200-201)) 1) and Aircraft are used as the base classes with VGG Flower as the novel classes, with each dataset exhibits a distinct distribution.
Spurious-correlation few-shot classification (SC-FSC). Spurious-correlatton shifts arise when classifier relies on spurious, non-causal context features that are not essential to the true label or concept, which can significantly reduce the robustness and generalization ability of the model. In the COVID-19 example, a recent nationwide cross-sectional study found spurious correlations be- tween long-term PM2.5 exposure and COVID-19 deaths in the United States due to county-level socioeconomic and demographic variables as confounders (Wu et al. 2020). To this end, models trained on base classes with spurious features and evaluated on novel classes without the relation- ship suffer substantial drops in performance. As shown in Figure bb). show the example of spurious-coris correlation shifts in our proposed benchmark, where each class presents a range of non- causal contexts, such autumn snow. Meanwhile, the concepts of the base classes and the novel classes would be distinct in the FSC problem, e.g., e.g.,"dop "dog in the autumn" in the base class and "cat in the autumn" in the novel class, which emphasizes the impact of spurious correlation between con- cepts and contexts in the proposed benchmark. Despite the widespread of spurious-correlation shifts in the real-world FSC problems (Wang al. 2017a| Yue al. 2020 Luo al. 202 I Zhang al.
2023b), SC-FSC remains understudied due to lack of the corresponding evaluation benchmarks.
Shortcomings of spurious- shifts benchmarks in traditional machine learning. Re- cently, ppurious-correlation shifts in traditional machine learning (TML) have been investigated ex- tensively (Arjovsky al. 2019; Sagawa et al. 2019 Rosenteld et al., 2020 Ahmed et al., 2020) Bae et al. 2021; (Shen et al. 2021). and various benchmarks have been created, including toy datasets, e.g., ColoredMNIST (Arjovsky et al. 2019), and real-world datasets, e.g., NICO (He et al. 2021) These TML benchmarks cannot be used directly to evaluate the performance in FSC problems with spurious-correlatin shifts, following the reasons below: (1) The number of classes. Most TML benchmarks the binary classification problem, but for FSC problems, need enough classes to split base and novel classes. (2) The number of samples. FSC needs adequate samples from base classes to learn the transferring knowledge novel classes with few labeled images. (3) The num-
2

---

Published as a conference paper at ICLR 2024
ber of contexts. Contexts in TML benchmarks are commonly limited, but FSC with many classes requires more contexts to build stronger spurious-corvelation correlation shifts. To the best of our knowledge, there does not exist unified study and the benchmark of spurious-correlate
shifts for FSC.
In this paper, we present Meta Concept Context (MetaCoCo), a large-scale benchmark with a total of 175,637 images, 155 contexts and 100 classes, with spurious- correlaioo shifts arising from var- ious contexts the real-world scenarios. The basic idea of constructing spurious-correlation shifts label the images with the main concepts and contexts. For example, in the category with "dog" as the main concept, the images are categorized into different contexts such as "autumn", "snow", and "rock", which denotes that the "dog" is in the autumn, in the snow, or on the rock, respectively. With the help of these contexts, one can easily design a the model in some contexts and testing the model in other unseen contexts for studying spurious- correlation shifts as well as the unseen concepts for studying few-shot classification problems.
wor.elation-shheeeeeeeeee -shif setting by training
Furthermore, we propose metric by using CLIP as a pre-trained langua-e model to quantify and compare the extent of spurious correlations on MetaCoCo and other FSC benchmarks. We conduct extensive experiments on MetaCoCo to evaluate the state-of-the-art methods in FSC, cross- domain shifts, and self-supervised learning. We open-source all codes for our benchmark and hope the proposed MetaCoCo will facilitate the development of ppurious-correkann robust models.
2 COMPARISON WITH EXISTING BENCHMARKS
MetaCoCo provides unified framework to facilitate the development of models robust to spurious- correlation shifts in FSC. We next discuss how MetaCoCo is related to existing benchmarks.
Relation to few-shot classification benchmarks. Few-shot classification (FSC) has attracted atten- tion for its ability to recognize novel classes using few labeled images. Many methods have been proposed solve the FSC problems, including (1) Fine-tuning based methods (Chen et al. 2019
Tian et al., [0000a] Chen et al. (021). which address the problem by learn to transfer. (2) Metric- based methods ( (Vinyals et al. 2016 Snell et al., [017 Li al. [019aa Zhang et al. (022aa) which solve the problem by learn to compare.
(3) Meta-based methods (Finn et al. [017 Rusu et al.,
2019
Bae al. 2021 Zhang et al.
2020 which tackle the problem by learn to learn.
Many FSC benchmarks have been proposed to evaluate the effectiveness of these methods, including minilmageNet (Vinyals al.]|2016 (016)), Places (Zhou et al., 2017). CIFAR-FS (Bertinetto al.) [2019) Plantae (Van Horn et al. 2018). CUB-200-2011 (Wah et al. 2011), Stanford Dogs (Khosla et al., 201T]. Stanford Cars (Krause et al. 2013), etc. These datasets are generally divided into training, validation and testing sets with non-overlap classes. While these datasets are useful testbeds for verifying FSC methods, they follow the independent and identically distributed (IID) assumption.
Relation to cross-domain shifts FSC benchmarks. Cross-domain shifts have been widely studied in the FSC community, which aims to learn the transferring knowledge from seen training distri- butions to recognize unseen testing distributions. Many CD-FSC methods have been proposed to address the cross domain problem (Tseng et al. 2020; Sun et al. 2021] [Ling et al., 202 Wang & Zhang al. 2022b), which can be mainly divided into bi-level opti-
Deng 2021 Li al. mization sssegg et al. 2020 Triantafillou et al. 2021 Li et al., [2236] /Zhan et al. 2023c). domain
2022
adversarial learning (Motian et al. 2017] Zhao et al. 2021 adversarial data augmentation (Wang & Deng 2021; Sun et al.) 2021), and module modulation (Liu et al. 2021 Li et al. 2022). Some benchmarks have been proposed
evaluate the effectiveness of these SCC methods, including Meta-dataset (Trintafillou et al. 2020) consisting of 10 existing datasets, and BSCD-FSL (Guo al.] 2020) consisting of 4 existing datasets. They usually use the save--on--ommin- as the testing domain and the others as training domains. However, these benchmarks use different datasets as domains to construct cross-domain distribution shifts, causing them to fail to reflect spu- rious correlation shifts that occur in real- world applications (see more discussion in Appendix A)
setting
Relation to spurious-corree correlation shifts TML benchmarks. Spurious-correlatee studied recently in traditional machine learning (TML) (Sagawa et al. 2019 Krueger et al. 2021
shifts have been
Yao et al. 2022 Bai et al. 2024] Tang et al. 2024). Many methods mainly focus on causal
learn
ing (Peters al. 2015 K Kuang et al. [2018] Kamath et al. 202.: Wu et al.) 2022 Wang et al.
2024
Li et al.] 2024:
Zhu et al. all]||202 (2024), invariant learning (Arjovsky et al. 2019 Chang et al. [020
Rosen-
feld et al. 2020
Huang et al. 2023), and distributionally robust optimization (Arjovsky al.
2019)
3

---

Published as a conference paper at ICLR 2024
Table 1: A summary of the existing benchmarks and our proposed spurious-correlattoon benchmark, Le., MetaCoCo. C and N are the number of classes and samples, respectively. The subscripts "all",
"train", "val" and "test" mean the all dataset, training set, validation, and testing set, respectively.
Dataset
Cut
Crein
Cust
Clest
Nai
Nreus
Neai
Niess
Conilxxi
Summlarit
midrllageNNe (Vinvals ot al.
2016
100
64
16
20
60,000
38,400
9,600
12,000
0
0.211
CIFAR-FS (Knznewssy et
100
20
60,000
38,400
9,600
12,000
0.181
Stanford Dogs (Kllosla et al.
2011
120
70
20
30
20,580
12.165
3.312
5.103
0
0.244
Stanford Cars (Kranse o1
196
130
16,185
10,766
1.394
4,025
0.164
Aircraft (Wah 2r 2011
100
70
15
15
10,000
5,000
2,500
2,500
0
0.228
CUB-200=2011 [Wal
2011
11,788
7,648
1,182
2,958
0.266
Describable Textures (C Cippot
2014
47
33
7
5,640
3,960
840
840
0.194
Truffic Sigus (Houben
2013
50,000
0.193
Omniglot (Lake et cl. 2013
50
25
20
32,460
17,660
1,620
13,180
0.212
Fungi Schinoleer &
1394
200
89,760
64,449
12,195
13,116
0.191
VGG Nlwer Nilsback & Zusiermnn 2008
102
71
15
16
189
5,685
1,109
1,425
0.177
MSCOCO el 2011
860,001
513,021
346,980
Quick Draw Jongejann el ar.
2016
145
241
52
52
50,426,266
34,776,331
7,939,640
7,710,295
0.168
CrupDisguess (Mohanily el 2016
43,456
ChesiX Wang ar 2017B
8
25,848
25,848
0.183
EuroSAl (Helher ml 2019
10
27,000
27,000
0.173
ISIC2018 [ Codelta m1 2019
10,015
10,015
0.186
MetaCoCu (Our)
100
20
175,637
136,666
5,839
12,268
155
0.142
etc. Some toy benchmarks, e.g.. ColOTedMNIST (Arjovsky et al. 2019) and real-world benchmarks, e.g., NICO (He et al., 2021) and MetaShift (Liang & Zou 2022), have been proposed to evaluate the performance of these methods. These TML. benchmarks do not be used directly in the FSC setting, due to lack of sufficient classes, number of samples, and number of contexts. Although IFSL (Yue et al.] (020) and COSOC (Luo et al.] (02T)) have experimentally proved the importance of spurious-correlation shifts, there is still a lack of a benchmark for evaluation. Therefore, we pro- pose MetaCoCo in this paper to reflect spurious pporeaus-oorellation shifts arising in real-world scenarios.
3 PROBLEM AND EVALUATION SETTINGS
FSC aims to recognize unlabeled images (or query sets) from novel classes with only few labeled images (or support sets). Following the previous studies (Vinyals et al., 2016 Tian et al. 2020b), we adopt episodic paradigm to train and evaluate the few-shot models. Specifically, each N-way K-shot episode Te has a support set S. = {(1,, 3/1) : ...... I,] and query set Q. = {(x,, y:)
i= Ig 1, ................................ Lg + 1g). where x; E X is the image and y, E y is the label from set of N classes C.. with I = N K and Lg be the image numbers in the support and query set, respectively.
Let SxX) and (x') be the image spaces of Se and Qe and S.) and (3) be the corre- sponding label spaces, respectively. The label space of S. and Qe is same but the image space is different, i..., (X) * (X') and (D)) Qe ())). During the training phase, for mesa-based and metric-based methods, episodes are randomly sampled from the base classes set Dj to train the model. Instead, for fine-tuning based methods, a mini-batch images is randomly sampled from D, to train the model. During the testing phase, the trained model is fine tuned with S. and evaluated with Q. in novel episodes sampled from the novel classes set D,, Note that Dj contains more im- ages and classes compared with D,, but label spaces are disjoint, i.e., Dz()²) * D, (3 The model architectures have a feature encoder fo and classifier co parameterized by and Φ. The fo aims to extract features, fo X + 2, and the co predicts the class of extracted features, Co Z - y
3.1 CROSS-DOMAN SHIFTS AND SPURIOUS-CORRELATEI SHIFTS
In Table I we summarize the statistics of the existing benchmarks and our proposed spurious- correlation benchmark, i.e., MetaCoCo. Specifically, Meta-dataset (Triantafillou et al. 2020) and BSCD-FSL (Guo et al.) 2020) are two commonly used cross cross- domain benchmarks, where Meta- dataset has 10 existing datasets, including ILSVRC-2012 (Deng al. 2009). Omniglot (Lake al.) 2013), Aircraft (Wah et 201T). CUB-200-201 1 (Wah et al. 2 2011).
et 20144) Quick Draw (Jongejan et al. al, Fungi (Schroeder & Describable Textures (Cimpoi
VGG Flower (Nils- back & Zisserman 2008). Traffic Signs (Houben et [d.33) and MSCOCO (Lin et al. 2014). BSCD-FSL (Guo et al. 2020) has 4 existing datasets, including CropDiseases (Mohanty et al. 2016), EuroSAT (Helber et al. 2019), ISIC2018 (Codella et al. 2019 Tschandl et al.) 2018), and
'D") and (2)) can he defined similarly, meaning the label spaces of D3 and D,,, respectively.
4

---

Published as a conference paper at ICLR 2024
0.25
Compa srelare
0.70
0.15
0.10
190 -BAE Dhg
Ohe Fulp Mao M4
Deer Dop Cheak hare
11.10
0.34
0.22
02s
(a) Mac the existing benchmarks and MetaCoCo
(b) Mer. and Mre MetaCoCo
Figure 2: (a) The sample-averaged similarity Mee between concepts and images the existing FSC benchmarks and the proposed MetaCoCo, where MetaCoCo has significantly lower similarity between contexts and images. (b) The context-iman similarities Mre (horizontal axis) versus the concept-image similarities Moe (vertical axis) the sample points in the MetaCoCo.
ChestX (Wang et al. 2017b). The main differences between cross-domain benchmarks and our pro- posed MetaCoCo benchmark are as follows: (1) The cause of shifts. The shifts in cross-domain benchmarks caused by varying distributions between various datasets. Instead, the shifts in MetaCoCo are caused by varying both concepts and contexts. For example, for cross-domain shifts, the FSL model is trained on minilmageNet and tested ATroSAT. Whereas for spurious-correlation shifts, the FSL model is trained and tested on images that have distinct associations with the con- texts. (2) The use of contexts. In contrast the existing few-shot classification benchmarks, as shown in Table [ the proposed MetaCoCo benchmark further uses context information collected from real-world scenarios to reflect the spurious-correlation shifts.
3.2 SIMILARITY BETWEEN THE CONCEPT AND CONTEXT INFORMATION
For images containing both conceptual and contextual information, a greater similarity between im- age and context implies that the benchmark has more spurious-con correlas-oo shifts. intuitively show that MetaCoCo has considerably more opurious-correlation shifts than the existing FSC benchmarks including cross-dee domain-shiff benchmarks, we introduce a novel metric that uses CLIP (Radford) eet al. 2021 l) pre-trained dision-language model. By calculating the cosine distance of text and image features extracted by pre- pre-trained text and image encoder from CLIP, the similarity Mee between conceptual language information and image visual knowledge, and the similarity Mre be- tween contextual language expression and image visual knowledge are calculated as follows:
Mce d(Ex, of"), Mre d(zus
where -(.,-) the cosine distance measurement, 2g the image features extracted by pre- pre-trained image encoder by CLIP, and 2 and 21 represent the text features of concept and context extracted by pre-trained text encoder by CL.P, respectively. Figure shows the sample-averaged similar- ity Mae between concepts and images on the existing FSC benchmarks as well as the proposed MetaCoCo. can be seen that MetaCoCo has significantly lower similarity between concepts and images. This is because the added context information in the image introduces correlatoosss with the concepts, e.g., "grass" and "dog", thus weakening the direct correlation between the images and the concepts labels, and presenting more challenging evaluating benchmark for the FSC. (eguree])) further shows the context-image similarities Mie (horizontal axis) versus the concept- image similarities Mar (vertical axis) of the sample points in the MetaCoCo. We find that the overall context-image similarities are slightly higher than the concept-image similarities, suggesting that couritos shifts are substantial in the proposed benchmark.
3.3 EVALUATION STRATEGIES
Before presenting the datasets, we first discuss the evaluation strategies in MetaCoCo, including:
(1) Fine-tuning based methods. Fine-tuning based methods follow the transfer learning procedure, including two phases: pre-training with base classes and test-tuning with novel classes. In the pre-
SSince the existing FSC benchmarks lack context information as shown in Table we not able to compute their sample-averaged similarity Mee hetween contexts and images.
5

---

Published as a conference paper at ICLR 2024
training with base classes phase, the base classes Db is used train Chase-Class classifier below:
r = arg min
Σ LCE (oo((c(((())))))))
0.0
where T is the sample number of Do, and L is the cross-entropy loss, In the test-tuning with novel classes phase, each episode Te (Se, Qe) is sampled from novel classes D, and new Ce-class classifier is re-learned based on few labeled images S. and tested on Qe.
(2) Metric-based methods. Metric-based methods directly compare the similarities (or distance) between query images and support classes, i..., learning to compare, through the episodic training mechanism. Taking Prototypical Network (ProtoNet) (Snell et al. (017) as an example, it takes the mean vector of each support class as its corresponding prototype representation, and then compares the relationships between query images and prototypes. The prototype pn. of each class in the support set S. can be formulated as pn K Σ (oi fo(1;) I[y: = n), where I(.) is the indicator function, then the the metric loss on Qe can be computed as:
((0) = Ig I Σ log P()))..), where P(v) exp( exp(-DD(s(zx))pp))) (E.
(3)
Ea exp(-D((o(tt..
and D): denotes distance measurement, egg, the squared euclidean distance in the ProtoNet.
(3) Meta-based methods. Meta-based methods aim to make the trained model able to quickly adapt unseen novel tasks by a few gradient steps in the testing phase. Specifically, the learning paradigm of meta-based methods has two levels, i.e. inner-level and outer-level, to update the base and meta learner, respectively. Model-agnostic meta-learning (MAML) (Finn al., (017) one representative method, whose core idea is to train a model's 's initial parameters by using the two levels. Specifically, the base learner is optimized on the support set S. that
(0 (0 Yout V (8,0) Lea (fo (oa). where (0', (0, ()) Vin V 10,,0) Lee (cp (co
(4)
and the Tiin and nout are the learning rates of the inner level and the outer level, respectively,
4 METACOCo: A NEW FEW-SHOT CLASSIFICATION BENCHMARK WITH SPURIOUS CORRELATION
MetaCoCo aims to present an environment for evaluating the fine control of spurious-correlation shifts in the FSC problems. Specifically, approach consists of (1) dataset generating, and (2) episode sampling, whose operational procedures are detailed below.
Dataset generating. Compared with the existing benchmarks, the samples in MetaCoCo consist of both conceptual and contextual information, and many of these images exhibit a strong correlation with the context, which inereases the impact of spurious-correlation shifts between the training data and the testing data on the prediction performance. Specifically, we first select 100 categories of common objects following DomainNet (Peng al., (2019). These categories include 155 contexts, which are collected from the adjectives or nouns appeared more frequently with these categories from WordNet (Miller 1995). Then the images are collected by searching a category name com- bined with a context name (e.g., "dog on grass") ") in various image search engines. One of the main challenges is that the downloaded data contains large portion of outliers. To clean the dataset, we manually filter out the outliers, which takes around 2,500 hours in total. To control the annotation quality, we assign two annotators to each image and only take the images agreed by both annotators. After the filtering process, we kept 17.6k images from the 1.0 million images crawled from the web. The dataset has an average of around 1,000 images per category (see Appendix B for more details).
Episode sampling. MetaCoCo has 100 categories, and the number of matching contexts for each category is inconsistent, resulting in an inconsistent number of samples for each category. We sort the samples from most to least. The first 64 categories with the largest number of samples are used as training data, then 20 categories are selected as testing data, and the last 16 categories are used as validation data. FSC adopts episodic paradigm to train and test the model. Each N-way K-shot

---

Published as a conference paper at ICLR 2024
Table 2: Experiments in state-of-the-ard few-shot classification and self-supervised learning meth-
ods. "rot." and "jig." mean using the Rotation and Jigsaw self-supervised pretext tasks, respectively.
Method
Conference
Backbone
Type
GL LLL 1T
1-shoo
5-shol
Baseline (Chen et al.
2019)
[CT. 2119
ReeNar12
Fing-luning
46.78
60.78
Baselin--
(Chen el
2019
ICLR 2019
ResNet12
Fine tuning
46.95
58.50
RU-S-imple
20206
BOCV 2020
ResNet12
Fing-tuning
47.02
56.71
Neg Cosine Lau al.
2020
ECCV 2020
ResNet12
Fine tuning
50.78
62.34
SKD.GENO (Kanaseaaaa
et
2020
BMVC
ResNet12
Fine-tuming
51.34
63.21
FRN Wonthedmer ot 2021
CVPR 2021
ResNet12
Fine-tuning
50.23
60.56
Yang at Yane ai
2022
BCCV 2022
ResNet12
Fine-tuning
58.01
69.32
LP-FT-FB
Wang Ct sl: 2022
ICLR 2023
ResNet12
Fine-tuning
56.21
70.21
MAML
(Fim
2017
ICML 2017
ResNet12
Meta
45.01
54.21
Versa
Garoks al 2018
NeurIPS 2018
ResNet12
Meta
39.64
53.06
R2D2 Berlaneto el 2019
ICLR 2019
ResNet12
Meta
/
45.25
60.14
MTL SS0 41 2019
CVPR 2019
ResNet12
Meta
44.23
58.04
ANIL Raghll
2020
ICLR 2020
ResNet12
Meta
36.58
50.54
BOIL (Ol
XUAD
ICLR 2021
ResNet12
Meta
44.00
55.61
CDKT*MI 1F 2025
NewrIPS 2023
ResNet18
Meta
44.
61.42
CDKT+PI.
ResNe188
Meta
CovaMiNer porraae
20196
XXXT 2019
RcNNcil2"
Matric
"7.81
588.43
DN4 Li dl. 2019/
CVPR 2019
ResNet12
Metric
45.04
68
CAN
Houb drale 2019
NewrIPS 2019
ResNet12
Metric
48.95
62.36
Deep
ResNe112
Metric
FGFL
(Cheng Bi Pizz
ICCV 2023
ResNet12
Metric
46.78
64.32
PUTM TTan clide
ICCV
ResNe118
Metric
72.36
TSA-DETA Zhaang al
202.5g
ICCV 2023
ResNet18
Metric
51.42
61.58
MoCo (Ho at 2020
CVPR 2020
ResNet50
Self-supervised learning
56.90
70.08
SimCI Chnen
20200
2020
ResNe150
Self-supervised learning
ProtoNer [Snell of 2017
NeurlPS 2017
ResNet18
Metric
14.
57.84"
rot. + SSFSL Ciunci 2020
BCCV 2020
ResNet18
Self-supervised Jearning
/
40.65
54.31
rot. HTS Zhinls A0004
ECCV 2022
ResNet18
Self-supervised learning
42.06
55.13
jig. SSPSI.
3r (0X
RCCV
ResNe118
Self-supervised learning
not. + jiy. SSTSL (Sure
2020
BCCV 2020
ResNe118
Self supervisd learning
44.46
59.01
ProtoNet (Sncil ot al. 2017
NourlPS
ResNeil2
Metric
442.99
39.30
rot. +SLA
Lo 2020
ICML 2020
ResNet12
Self-supervised learning
40.29
58.09 09
not. HTS Zauns eiiil 20122
RCCV
ResNet12
Self-supervised
43.19
ProtoNet (Sncll o a. 2017
NeurlPS 2017
WRN-28-10
Metric
"43.67
60.78"
rot. + BF33 (iddans clal 2019
ICCV 2019
WRN-28-10
Self-supervised Jearning
43.78 57.64
rot. HTS Zhiing A07681
ECCV 2022
WRN-28-10
Self-supervised learning
45,31
62,31
episode Te has a support set S, and a query set Qe, where Se and Qe share the same categories but different images. Therefore, we have two sample episodic strategies: independent and identically distributed (IID) episode, i.e., the support and query images with the same contexts, and out-of- distribution (OOD) episode, i.e., the support and query images with the different contexts.
EXPERIMENTS
In this section, we evaluate the spurious-correlation performance of the state-of-the-arr methods optimized with different learning strategies. These experiments further demonstrate that SC-FSC is still a major challenge. (see Appendix C and D for more experimental details and results).
5.1 EXPERIMENTAL. SETUP
Few-shot classification methods. We evaluate the performance with a large number of algorithms
that span different learning strategies, including: (1) Five five.fiietuunng based methods: Baseline (Chen
ct al.
2019), Baseline++ (Chen et al., (019)), RFS-simple (Tian et al. (022a), Neg-Cosine (Liu et al.
2020
and SKD-GENO (Rajasegaran et al. 2020). (2) Six metric-based methods: ProtoNet (Snell
et al.
2017). RelationNet (Sung et al. 2018). CovaMNet (Li et al. 2019b). DN4 (Li et al. 2019a),
CAN
(Hou al. 2019) and RENet (Kang et al.) (022). D. (3) Six meta-based methods: MAML (Finn
et al.
2017). Versa (Gordon et al.) 2018). R2D2 (Bertinetto et al. 2019), MTL (Sun et al.) 2019).
ANIL.
(Raghu et al.) 2020) and BOII. {Oh et al. (222). (4) Six self-supervised learning methods:
MoCo
(He al. (2020). SimCLR (Chen et al. 2020). SSFSL (Su et al. 2020). HTS (Zhang et al.)
2022a),
SLA (Lee et al.
2020) and BF3S (Gidaris et al. 2019). (5) Seven cross-domain methods:
Linear Yue et al.] 2020). Cosine (Yue et al. (020), K-NN (uue al. 2020). ATA (Wang & Deng
2021.
FT (Tseng et al.; 2020), LRP (Sun et al. 2021) and IFSL Yuue et al. 2020).
Backbone architectures. Following prior literatures (Li et al. 2023d), all ine-tuning based meth- ods, metric-based methods and meta-based methods adopt three different embedding backbones from shallow deep, i.e., Conv64F, ResNet12 and ResNet18. For other learning strategy methods,
7

---

Published as a conference paper at ICLR 2024
Table 3: Experiments of cross-domain and spurious-corrclatioo few-shot classification methods
Method
Conference
Type
GL
LL
TT
5-way
L-shot
5-way 5-shot
RelationNet Suung
2018
CVVRR 2018
Metric
45.32 0.48
57.73
+ATA (Wang & Deng
2021
IJCAI 2021
CD-FSC
V
43.24 0.47
56.94 0A7
+FT (Tseng et al. 2020
ICLR 2020
CD-FSC
45.37 0.50
58.
0.48
"GNNN (Satorras & Estrach
2018
ICLR 2018
Metric
7
48.14
61.34 0.56
+ATA Wang Deng
UCAI 2021
CD-FSC
/
46.78
61,78
+ FT seenng CT a K120
ICLR 2020
CD-FSC
47.30 0.56
65.90 0.56
TPN
[Lu ct m.. 2018
ICLR 2019
Metric
49.65 0,51
60.62 0,47
+ATA (Wang Deng
2021
IJCAI 2021
CD-FSC
47.15 0.53
60.33 0.31
+FT
2020
ICLR 2020
CD-FSC
45.62 www.
55.78
Lanear Semonttion 20220)
NeurlPS 2020
Fine-tuning
43.31
S7.87
Cosine Yue ct al.
2020
NeurIPS 2020
Fine-tuning
42.81 0:42
56.33
0.11
A-NN Yuc ei
20.0
NeurPPS 2020
Fine-tuning
42.22 tus
57.93
0.42
"MAMI.
2017
2017
Meta
7
44.09 0.0
53398 0.48
+IFSL Yue
2020
NeurIPS 2020
SC-FSC
43,42
55.00
MTL (unn all 2019
CVPR 2019
Meta
/
43, 0.43
57.18 0.48
+IFSI, Yue et al. 2020
NeurlPS 2020
SC-FSC
/
43.42 0.48
56,90
0.48
Matching Nel Vinyais
2016
NeurIPS 2016
Metric
43.72 0
0.49
+IFSI Yuc 2t 2020
NeurIPS 2020
SC-FSC
44.11
0.49
55.86
SIB (Hu el ai. 2020
ICI R 2020
Meta
/
48 43
057
58.53
0.51
+LL Yue et al. 2020
NeurlPS 2020
SC-ESC
47.97
054
58.4
0.50
5-way-I-shot
5-way-5-shot
40
OOD episode
OOD opisode ITD episnde
ITD opisade
30
Busehne DMi
DNMiaselliee
Figure 3: Experiments of the test-tuning phase with different sampling episodes, i.e., IID and OOD.
adopt different feature backbones based the corresponding original papers, egg., ResNet10 for cross-domain few-shot classification methods, WRN-28-10 for self-supervised learning methods.
Evaluation protocols. Following the prior work (Li al.) 2023d). in this paper, control the evaluation for all methods. evaluate them on 600 sampled tasks and repeat this process five times, i.e.. total of 3,000 tasks. The top-1 mean accuracy will be reported. All images are resized into 84 x 84 by using the single center crop (Li et al. 2019b). Three common tricks are used: (1) Global- label (GL) indicates that the global labels of the training set are used for pre-traine pre-training during the training phase. (2) Local-label (LL) means that only the specific local labels are used in the episodic training phase. (3) Test-tune (TT) means test-tuning of using the support set at the testing stage.
5.2
MAIN RESULTS
In this section, we conduct extensive experiments various methods with six learning strategies.
Experiments in fine-tuning, metric- and meta-based methods and self-supervised methods. We evaluate the performance of 17 competing few-shot methods and six self-supervised methods in MetaCoCo. The results of the 5-way 1- or 5-shot setting are shown in Table From Table we have the following findings: (1) We find that the performance of all methods decreases compared with existing FSC benchmarks (Li et al. 2023d), which demonstrates that these methods are insufficient problem. (2) Previous works introduced self-supervised
in solving the cuurious-coorelationn. leamiing to improve the generalization of FSC models, but experiments have shown that this is not suitable for the SC-FSC problem. In some cases, using self-supervised learning even damages the performance, ie., ProtoNet has 43. 14% in 1-shot, but the accuracy by using rotation is 40.65%.
Experiments in CD-FSC and SC-FSC methods. Table T displays the accuracy of seven CD-FSC methods. These methods have significant performance on solving the cross-domain-shinn problem on the Meta-dataset (Triantafillou et al. 2020) and BSCD-FSL (Guo et al.) 2020). However, in MetaCoCo, the advantages of these methods disappear, resulting in weaker performance, even worse than
FSC methods. It is worth noting that the main motivation of IFSL YYue et al. 2020) is to the idea of causality to solve the impact of spurious correlation between contextual information and images on the model training phase. However, we observe a substantial decrease of the performance the wallworldd spurious-correlation benchmark, i.e., MetaCoCo.
8

---

Published as a conference paper at ICLR 2024
5-way-1-shot
10-way-1-shol
40
20
30
Conv64F
Conv64F
20
ResNet12
10-
ResNetl
ResNett8
ReaNett 8
10
0
Baselino
Daseline
DNI
Figure 4: Experiments of different backbone architectures under 5-way and 10-way 1-shot settings.
LibII
60-
+ 5-way I I-shot 5-way 5-shot
40
40-
5-way I-shot
30
10-way I-shot
Versa
R2D2 Boseline
CAN ProtoNet DN4
ANTlRasclnnt++
Versa R202 Baseline CAN ProcoNat DN4
ANiLBeselnett
Figure 5: Experimental results of different ways (left) and shots (right) on testing performance.
To this end, according to these experimental results, we observe that most methods are insufficient to solve the nourious-correlation-- FSC problem. We hope the proposed MetaCoCo can facilitate future research on the important and real-world problem for few-shot classification.
5.3 IN-DEPTH STUDY
To further analyze the influence of spurious shifts in MetaCoCo, we conduct in-depth experiments.
Effeet of the IID and OOD episodes. Figure 3 shows the results of FSC methods under 5- 5-way 1- and 5-shot settings. The IID and OOD episodes represent same and different contexts of the sup- port and query sets during the test-tuning phase, respectively (see Section 4). These results clearly denote that the learning process the IID episode is better than the optimization process of the OOD episode. This further demonstrates that the model tends to utilize contextual information during the learning process. Once images do not match the contexts, the performance will deteriorate.
Effect of different backbone architectures. In Chen et al. (2019), they change the depth of the feature backbone to reduce intra-class variation for all methods. Following this paper, we start from Conv64F and gradually increase the backbone ResNet12 and 18. The experiments under 5-way and 10-way I-shot settings are shown Figure It is arguably common sense that the stronger backbone used, the performance is best. However, we surprisingly find that this may not be always in the SC-FSC problem. Figure shows the performance degradation in some settings.
Ways and shots analysis. We further study the performance of "ways" (Figure Ieft) and "shots" (Figure 5 right). As expected, we found that the difficulty increases as the way increases, and performance degrades. More examples per class, on the other hand, indeed make it easier to correctly classify that class. Interestingly, Versa presents poor performance with increasing the way but improves at a high rate when shot increases, which further represents that the contextual effects become larger when the task becomes difficult. CAN has the best accuracy under all settings because uses transduction strategy to introduce query samples in the training phase, which destroys the strong spurious correlations between contexts and images.
CONCLUSION
In this paper, we present Meta Concept Context (MetaCoCo), large-scale, diverse and realistic environment benchmark for spurious-correlation few-shot classification. We believe that our ex- ploration various modes on MetaCoCo has uncovered interesting directions for future works: remains unclear what is the best learning strategy for avoiding the effect of spurious-correlation con- texts and the most appropriate episodic sample. Current models even including these cross- coos--dmmann FSC models don't work when trained mismatching contexts. Current models also not robust to the amount of data in testing episodes, each excelling in a different part of the spectrum. We believe that addressing these shortcomings constitutes important research goal moving forward.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported in part by National Natural Science Foundation of China (No. U20A20387, 62376243, 62037001, 623B2002), the StarryNight Science Fund of Zhejiang University Shang- hai Institute for Advanced Study (SN-ZJU-SIAS-OOIE (P22KS00111). All opinions in this paper are those of the authors and donot necessarily reflect the views of the funding agencies.
and Project by Shanghai AI Laboratory
REFERENCES
Faruk Ahmed, Yoshua Bengio, Harm van Seijen, and Aaron Courville. Systematic generalisation with group invariant predictions. In International Conference on Learning Representations, ICLR, 2020.
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXi:: 2019.
Jun-Hyun Bae, Inchul Choi, and Minho Lee. Meta-learned invariant risk minimization. arXiv preprint 1..iv:2103.1944: 2021.
Shuanghao Bai, Min Zhang, Wanqi Zhou, Siteng Huang, Zhirong Luan, Donglin Wang, and Badong Chen. Prompt-based distribution alignment for unsupervised domain adaptation. In Proceedings of the AAAI conference on artificial intelligence, AAAI, 2024.
Luca Bertinetto, Joao F Henriques, Philip HS Torr, and Andrea Vedaldi. Meta-learning with dif- ferentiable closed-form solvers. In Proceedings of the International Conference on Learning Representations, ICLR, 2019.
Shiyu Chang, Yang Zhang, Mo Yu, and Tommi Jaakkola. Invariant rationalization. In International Conference on Machine Learning. ICML, pp. 1448-1458, PMLR, 2020.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference Machine Learning, ICML, pp. 1597-1607. PMLR, 2020.
Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look few-shot classification. In International Conference on Learning Representations, ICLR, 2019.
Yinbo Chen, Zhuang Liu, Huijuan Xu, Trevor Darrell, and Xiaolong Wang. Meta-baseline, explor- ing simple meta-learning for few-shot learning. In Proceedings of the IEEE/CVF International Conference Computer Vision, ICCV, 2021.
Hao Cheng, Siyuan Yang, Joey Tianyi Zhou, Lanqing Guo, and Bihan Wen. Frequency guidance the IEEE/CVF International Conference on Com-
matters in few-shot learning. In Proceedings the puter Vision, ICCV, pp. 11814-11888 11824, 2023.
Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 3606-3613, 2014.
Noel Codella, Veronica Rotemberg, Philipp Tschandl, M Emre Celebi, Stephen Dusza, David Gut- man, Brian Helba, Aadi Kalloo. Konstantinos Liopyris, Michael Marchetti, et al. Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic). arXiv preprint arXiv: 1902.09902.033666 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. 2009 IEEE Conference on Computer Vision and Puttern Recognition, CVPR. pp. 248-255 Iece, 2009.
Carl Doersch, Ankush Gupta, and Andrew Zisserman. Crosstransfomeers: spatially-awan aware few-shot transfer. In Advances in neural information processing systems, NeurIPS, volume 33, pp. 21981- 21993, 2020.
10

---

Published as a conference paper at ICLR 2024
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adaptation of deep networks. In Proceedings of the 34th Interaational Conference on Machine Learning, ICML, 2017.
Spyros Gidaris, Andrei Bursuc, Nikos Komodakis, Patrick Pérez, and Matthieu Cord. Boosting few-shot visual learning with self-supervision. In Proceedings of the IEEE/CVF international conference on computer vision, ICCV. pp. 8059-8068, 2019.
Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard Turner. Versa: Versatile and efficient few-shot learning. In Third workshop on Bayesian Deep Learning, 2018.
Yunhui Guo, Noel Codella, Leonid Karlinsky. James V. Codella, John R. Smith, Kate Saenko, Tajana Rosing, and Rogério Feris. broader study of cross-domain few-shot learning. European Conference on Computer Vision, ECCV, 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, pp. 9729-9738, 2020.
Yue He, Zheyan Shen, and Peng Cui. Towards non-iid image classification: A dataset and baselines. Pattern Recognition, 110:107388 2021.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benehmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226, 2019.
Ruibing Hou, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen. Cross attention net- work for few-shot classification. In Advances in neural information processing systems, NeurIPS, volume 32, 2019.
Sebastian Houben, Johannes Stallkamp, Jan Salmen, Marc Schlipsing, and Christian Igel. Detection of traffic signs real-world images: The german traffic sign detection benchmark. In The 2013 international joint conference on neural networks (IJCNN), pp. 1-8. Ieee, 2013.
Shell Xu Hu, Pablo G Moreno, Yang Xiao, Xi Shen, Guillaume Obozinski, Neil D) Lawrence, and Andreas Damianou. Empirical bayes transductive meta-learning with synthetic gradients. Inter- national Conference Learning Representations, ICLR, 2020.
Shanshan Huang. Haoxuan Li, Qingsong Li, Chunyuan Zheng, and Li Liu. Pareto invariant repre- sentation learning for multimedia recommendation. In ACM International Conference on Multi- media, 2023.
Jonas Jongejan, Henry Rowley, Takashi Kawashima, Jongmin Kim, and Nick Fox-Gieg, The quick, draw!-ai experiment. Mount View, CA, accessed Feb, 17(2018):4 2016.
Pritish Kamath, Akilesh Tangella, Danica Sutherland, and Nathan Srebro. Does invariant risk mini- mization capture invariance? In International Conference on Artificial Intelligence and Statistics, AISTATS, pp. 4069-4077. PMLR, 2021.
Dahyun Kang, Heeseung Kwon, Juhong Min, and Minsu Cho. Relational embedding for few-shot classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, ICCV, pp. 2833, 2021.
Tianjun Ke, Haoqun Cao, Zenan Ling, and Feng Zhou. Revisiting logistic-softmax likelihood in bayesian meta-learning for few-shot classification. In Advances neural information processing systems, NeurIPS, 2023.
Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Fei-Fei Li. Novel dataset for fine- grained image categorization: Stanford dogs. In Proc. CVPR Workshop on Fine-Gre Grained Visual Categorization, FGVC, 2.11.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- iie-grained categorization. In Proceedings of the IEEE international conference on computer vision work- shops, pp. 554-561, 2013.
11

---

Published as a conference paper at ICLR 2024
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distriuttion generalization via risk extrapo- lation (rex). In International Conference Machine Learning, ICML, 2021.
Kun Kuang, Peng Cui, Susan Athey, Ruoxuan Xiong, and Bo Li. Stable prediction across unknown environments, In proceedings of the 24th ACM SIGKDD international conference knowledge discovery & data mining, KDD. pp. 1617-1626, 2018.
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 133066666: 2015.
Hankook Lee, Sung Ju Hwang, and Jinwoo Shin. Self-supervised label augmentation via input trans- formations. In International Conference on Machine Learning, ICML, pp. 5714-5724. PMLR, 2020.
Haoxuan Li, Yan Lyu, Chunyuan Zheng, and Peng Wu. TDR-CL: Targeted doubly robust collabo- rative learning for debiased recommendations, In International Conference Learning Repre- sentations, ICLR, 2023a.
Haoxuan Li, Yanghao Xiao, Chunyuan Zheng, and Peng Wu. Balancing unobserved confounding with a few unbiased ratings in debiased recommendations. In Proceedings of the ACM Web Conference, WWW, pp. 1305-1313, 2023b.
Haoxuan Li, Chumyuan Zheng, and Peng Wu. StableDR: Stabilized doubly robust learning for recommendation on data missing not at random. In International Conference on Learning Repre- sentations, ICLR, 2023c.
Haoxuan Li, Chunyuan Zheng, Yanghao Xiao, Peng Wu, Zhi Geng, Xu Chen, and Peng Cui. De- biased collaborative filtering with kernel-based causal balancing. In International Conference on Learning Representations, ICLR, 2024.
Wei-Hong Li, Xialci Liu, and Hakan Bilen. Cross-domain few-shot learning with task ssecispciific adapters. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, CVPR, pp. 7161-7170, 2022.
Wenbin Li, Lei Wang. Jinglin Xu, Jing Huo, Yang Gao, and Jiebo Luo. Revisiting local descriptor based image-to-class measure for few-shot learning. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR, 2019a.
Wenbin Li, Jinglin Xu, Jing Huo, Lei Wang. Yang Gao, and Jiebo Luo. Distribution consistency based covariance metric networks for few-shot learning. In Proceedings of the AAAI conference 2019b.
on artificial intelligence, AAAI, pp.
Wenbin Li, Chuanqi Dong, Pinzhuo Tian, Tiexin Qin, Xuesong Yang. Ziyi Wang, Jing Huo, Yinghuan Shi, Lei Wang, Yang Gao, et al. Libfewshot: A comprehensive library for few-shot learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, TPAMI, 2023d.
Hanwen Liang Qiong Zhang, Peng Dai, and Juwei Lu. Boosting the generalization capability in cross- few-shot learning via noise-enhanced supervised autoencoder. In Proceedings of the JEEE/CVF International Conference on Computer Vision, ICCV, pp.
2021.
Weixin Liang and James Zou. Metashift: A dataset of datasets for evaluating contextual distribution shifts and training conflicts. In International Conference on Learning Representations, ICLR, 2022.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In 13th European Conference, Zurich, Switzerland, September 12, 2014, Proceedings, Part V 13, ECCV, pp. 740- 755. Springer, 2014.
12

---

Published as a conference paper at ICLR 2024
Bin Liu, Yue Cao, Yutong Lin, Li, Zheng Zhang, Mingsheng Long, and Han Hu. Negative margin matters: Understanding margin in few-shot classification. In Computer Ecion-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IV 16, pp. 438-455. Springer, 2020.
Yanbin Liu, Juho Lee, Minseop Park. Saehoon Kim, Eunho Yang. Sung Ju Hwang, and Yi Yang. Learning to propagate labels: Transductive propagation network for few-shot learning. arXiv preprint arXiv: /805. 10002, 2018.
Yanbin Liu, Juho Lec, Linchao Zhu, Ling Chen, Humphrey Shi, and Yi Yang. A multi-mode mod- ulator for multi-domain few-shot classification. In Proceedings of the IEEE/CVF International Conference Computer Vision, ICCV, pp. 8453-8463 2021.
Xu Luo, Longhui Wei, Liangjian Wen, Jinrong Yang, Lingxi Xie, Zenglin Xu, and Qi Tian. Rectify- ing the shorteut learning of background for few-shot learning. Advances in neural information processing systems, NeurlPS, 2021.
George A Miller. Wordnet: a lexical database for english. Communications of the ACM, 38(11): 39-41, 1995.
Sharada P Mohanty, David P Hughes, and Marcel Salathé. Using deep learning for image-based plant disease detection. Frontiers plant science, 7:1419, 2016.
Saeid Motiian, Quinn Jones, Seyed Iranmanesh, and Gianfranco Doretto. Few-shot adversarial domain adaptation. In Advances neural information processing systems, NewrIPS, volume 30, 2017.
Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics Image Processing, pp. 722-729. IEEE, 2008
Jaehoon Oh, Hyungjun Yoo, ChangHwan Kim, and Se- Young Yun. Boil: Towards representation change for few-shot learning. In International Conference Learning Representations, ICLR, 2020.
Jaehoon Oh, Sungnyun Kim, Namgyu Ho, Jin-Hwa Kim, Hwanjun Song, and Ye-Yo Yun. Un- derstanding cross-domain few-shot learning based on domain similarity and few-shot difficulty. In Advances Neural Information Processing Systems, NeurIPS, 2022.
Xingchao Peng. Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEF/CVF international conference computer vision, pp. 1406-1415, 2019.
J Peters, Peter Buhlmann, and N Meinshausen. Causal inference using invariant prediction: identi- fication and confidence intervals. arxiv. Methodology, 2015.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, ICML, pp. 8748-8763. PMLR, 2021.
Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning feature reuse? towards understanding the effectiveness of MAML. In International Conference on Learning Representations, ICLR, 2020.
Jathushan Rajasegaran, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Mubarak Shah. Self-supervised knowledge distillation for few-shot learning. arXiv preprint arXiv:2006.0978 097855 2020.
Elan Rosenfeld. Pradeep Ravikumar, and Andrej Risteski. The risks of invariant risk minimization. arXiv preprint arXiv:2010.05761, 2020.
13

---

Published as a conference paper at ICLR 2024
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, IJCV, 2015.
Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-leariing with latent embedding optimization. In International Conference on Learning Representations, ICLR, 2019.
Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generaliza- tion. International Conference Learning Representations, ICI.R, 2019.
Victor Garcia Satorras and Joan Bruna Estrach. Few-shot learning with graph neural networks. In International Conference Learning Representations, ICLR, 2018.
Brigit Schroeder and Yin Cui. Egvex fungi classification challenge 2018. Available online: github. (accessed on 14 July 2021), 2018.
Zheyan Shen, Jiashuo Liu, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. Towards out-of-distribution generalization: survey, arXiv preprint arXiv:2108. 136244 2021.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in neural information processing systems, NeurlPS, 2017.
Yisheng Song, Ting Wang, Puyu Cai, Subrota K Mondal, and Jyoti Prakash Sahoo. A comprehen- sive survey of few-shot learning: Evolution, applications, challenges, and opportunities. ACM Computing Surveys, 2022.
Jong-Chy: Su, Subhransu Maji. and Bharath Hariharan. When does self-supervision improve few- shot learning? In European Conference on Computer Vision, ECCV, pp. 645-666. Springer, 2020.
Jiamei Sun, Sebastian Lapuschkin. Wojciech Samek, Yunqing Zhao, Ngai-Man Cheung, and Alexander Binder. Explanation-guiiee training for cross-domain few-shot classification. 25th International Conference Pattern Recognition, ICPR, 2021.
Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele. Meta-transfer learning for few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 403 412.412,77 2019.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: relation network for few-shot learning. IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2018.
Zihao Tang, Zheqi Lv, Shengyu Zhang, Fei Wu, and Kun Kuang. Modelgpt: capabilities for tailored model generation, 2024.
Unleashing I1m's
Long Tian, Jingyi Feng, Xiaoqiang Chai, Wenchao Chen, Liming Wang, Xiyang Liu, and Bo Chen. Prototypes-oriented transductive few-shot learning with conditional transport. In Proceedings of the IEEE/CVF International Conference on Computer Vision, ICCV, pp. 16317-16326. 2023.
Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B Tenenbaum, and Phillip Isola. Rethinking few-shot image classification: good embedding is all you need? European Conference on Computer Vision, ECCV, 2020a.
Zhuotao Tian, Hengshuang Zhao, Michelle Shu, Zhicheng Yang, Ruiyu Li, and Jiaya Jia. Prior guided feature enrichment network for few-shot segmentation. IEEE Transactions Pattern Analysis and Machine Intelligence, TPAMI, 14(2):0050-1055, 2020b.
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evei, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, et al. Meta-Dan Dataset: A dataset of datasets for learning to learn from few examples. 2020.
14

---

Published as a conference paper at ICLR 2024
Eleni Triantafillou, Hugo Larochelle, Richard Zemel, and Vincent Dumoulin. Learning universal template for few-shot dataset generalization. In International Conference on Machine Learning, ICML, pp. 10433. PMLR, 2021.
Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The ham10000 dataset, large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data, 5(1):1-9, 2018.
Hung- Yu Tseng, Hsin- Ying Lee, Jia-Bin Huang, and Ming- Hsuan Yang. Cross-domain few-shot classification via learned feature-wise transformation. In International Conference on Learning Representations, ICLR, 2020.
Grant Van Hom, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 8769-8778, 2018.
Oriol Vinyals, Charles Blundell, Tim Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In Advances in neural information processing systems, NeurIPS, 2016.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
Hao Wang, Jiajun Fan, Zhichao Chen, Haoxuan Li, Weiming Liu, Tianqiao Liu, Quanyu Dai, Yichao Wang, Zhenhua Dong, and Ruiming Tang. Optimal transport for treatment effect estimation. In Advances in neural information processing systems, NeurIPS, volume 36, 2024.
Haoqing Wang and Zhi-Hong Deng. Cross-domain few-shot classification via adversarial task aug- mentation. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelli- gence, IJCAL, 2021.
Heng Wang, Tan Yue, Xiang Ye, Zihang He, Bohan Li. and Yong Li. Revisit finetuning strategy for few-shot learning to transfer the emdeddings. In International Conference on Learning Repre- sentations, ICLR, 2022.
Xiao Wang, Peng Cui, Jing Wang, Jian Pei, Wenwu Zhu, and Shiqiang Yang. Community preserving network embedding. In Proceedings of the AAAI conference on artificial intelligence, AAAI, volume 31, 2017a.
Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Sum- mers. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on seakly-supervised classification and localization common thorax diseases. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, CVPR, pp. 2097-2106, 2017b.
Davis Wertheimer, Luming Tang, and Bharath Hariharan. Few-shot classification with feature map reconstruction networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, pp. 8012-8021, 2021.
Peng Wu, Haoxuan Li, Yuhao Deng, Wenjie Hu, Quanyu Dai, Zhenhua Dong, Jie Sun, Rui Zhang, and Xiao-Hua Zhou. On the opportunity causal learning in recommendation systems: Foun- dation, estimation, prediction and challenges. In International Joint Conference on Artificial Intelligence, IJCAI, 2022.
Xiao Wu, Rachel C Nethery, M Benjamin Sabath, Danielle Braun, and Francesca Dominici. Ex- posure air pollution and covid-19 mortality in the united states: A nationwide cross-sectional study. MedRxiv, pp. 2020-04 04, 2020.
Jiangtao Xie, Fei Long. Jiaming Lv, Qilong Wang, and Peihua Li. Joint distribution matters: Deep brownian distance covariance for few-shot classification. In Proceedings of the IEEE/CV Con- ference on Computer Vision and Pattern Recognition, CVPR, pp. 7972-7981, 2022.
Zhanyuan Yang, Jinghua Wang, and Yingying Zhu. Few-shot classification with contrastive learning. In European Conference Computer Vision, ECCV, pp. 293-309. Springer, 2022.
15

---

Published as a conference paper at ICLR 2024
Huaxiu Yao, Yu Wang, Sai Li, Linjun Zhang, Weixin Liang, James Zou, and Chelsea Finn. Im- proving disttiddittiittiion robustness via selective augmentation. In International Conference Machine Learning, ICML, pp. 25407-25437. PMLR, 2022.
Zhongqi Yue, Hanwang Zhang, Qianru Sun, and Xian-Sheng Hua. Interventional few-shot learning. 2020.
Ji Zhang, Lianli Gao, Xu Luo, Hengtao Shen, and Jingkuan Song. Deta: Denoised task adaptation for fho- learning. Proceedings of the IEEE/CVF International Conference on Computer Vision, ICCV, 2023a.
Min Zhang, Donglin Wang, and Sibo Gai. Knowledge distillation for model-ggnostic meta-learning. In European conference on artificial intelligence, ECAI, pp. 1355-1362. 2020.
Min Zhang, Siteng Huang, Wenbin Li, and Donglin Wang. Tree structure-aware few-shot image classification via hierarchical aggregation. In European Conference on Computer Vision, ECCV, pp. 453-470. Springer, 2022a.
Min Zhang, Siteng Huang, and Donglin Wang. Domain generalized few-shot image classification via meta regularization network. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP, pp. 3748-34888111 3152. IEEE, 2022b.
Min Zhang, Junkun Yuan, Yue He, Wenbin Li, Zhengyu Chen, and Kun Kuang. MAP: Towards balanced generalization of iid and ood through model-agmostic adapters. In Proceedings of the IEEE/CVF International Conference Computer Vision, ICCV, pp. 11921-11931, 2023b.
Min Zhang, Zifeng Zhuang, Zhitao Wang, Donglin Wang, and Wenbin Li. RotoGBML: Towards nut-of-distribution generalization for gradient-basee meta-learning. In JEEE International Con- ference on Multimedia and Expo, ICME, 2023c.
An Zhao, Mingyu Ding, Zhiwu Lu, Tao Xiang, Yulei Niu, Jiechao Guan, and Ji-Rong Wen. Domain- adaptive few-shot learning. In Proceedings the IEEE/CVF Winter Conference Applications of Computer Vision, WACV, pp. 1330-1399 2021.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 mil- lion image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, TPAMI, 40(6)):452-1444 2017.
Minqin Zhu, Anpeng Wu, Haoxuan Li, Ruoxuan Xiong, Bo Li, Xiaoqing Yang, Xuan Qin, Peng Zhen, Jiecheng Guo, Fei Wu, et al. Contrastive balancing representation learning for hetero- geneous dose-response curves estimation. In Proceedings of the AAAI conference artificial intelligence, AAAI, volume 38, pp. 17175-17183, 2024.
16

---

Published as a conference paper at ICLR 2024
A MORE DISCUSSION ON THE EXISTING BENCHMARKS
In Table 1 we have summarized statistics of existing benchmarks. A brief introduction of bench- marks mentioned in this paper is the following. For more details, please refer to the original paper.
minilmageNeet Vinyals et al., 2016). minilmageNet is the subsets of the ILSVRC-12 dataset (uus- sakovsky et al. 2015).
Fine-grained benchmarks. CUB-200-2011 (Wah et al. 2011 and Stanford Cars (Krause et al. 2013) are initially designed for fine-grained classification.
Stanford Dogs Khosla et al. 201
Meta-dataset (Triantafillou et al.,
2020). Meta-dataset is a cross-domain FSC benchmark and has
10 existing datasets, including ILSVRC-2012 (Deng et al.) 2009), Omniglot (Lake al.] 2015)
Aircraft (Wah et al.] 2011), CUB-200-2011 (Wah et al. 201 1). Describable Textures (Cimpoi et al.
2014), Quick Draw (Jongejan al.
20T6), Fungi (Schroeder & Cuil 2018), VGG Flower (Nilsback & Zisserman 2008). Traffic Signs Houben et al. 2013) and MSCOCO (Lin et al. 2014).
BSCD-FSL (Guo al. 2020) BSCD-FSL also cross-domain FSC benchmark and has ex- isting datasets, including CropDiseases (Mohanty et al. (016), EuroSAT (Helber et al. 2019). ISIC2018 (Codella et al.
2019 Tschandl et al.] and ChestX (Wang et al. 2017b).
B DETAILED DATASET STATISTICS
Tables 4 and 5 show the number of samples for concepts and detailed statisties of the MetaCoCo benchmark, respectively. In particular, our benchmark contains 100 concepts (or categories), 155 contexts and 17.6k images. These concepts are from common objects following DomainNet (Peng itt al. 2019). The 155 contexts are collected from the adjectives or nouns appeared more frequently with these concepts from WordNet (Miller 1995). In addition, we show the statistics of samples in each concept in Table 4
C EXPERIMENTAL DETAILS
In this paper, many feature backbones are used to fair evaluate the performance of few-shot classi- fication methods. Specifically, Conv64F contains four convolutional blocks, each of which consists of a convolutional (Conv) layer, a batch-normalization (BN) layer, a ReLU/LeakyReLU layer and a max-pooling (MP) layer, where the numbers of filters of these blocks are [64, 64, 64, 64]. ResNet12 consists of four residual blocks, each of which further contains three convolutional blocks (each is built as Conv-BN- BN-ReLU-MP) along with a skip connection layer, where the numbers of filters of these blocks are {64, 160, 320, 640). ResNet18 8 is the standard architecture used in previous works. One important difference between ResNet12 and R8sNer18 is that ResNet12 uses Dropblock in each residual block. while ResNet18 does not. In addition, the number of filters of these blocks in ResNet18 is {64, 128, 256, 512]. ResNet10 is a common backbone architecture in cross-domain few-shot classification methods. It has the same number of filters of blocks ResNet18, but the number of layers is in each stage. WRN-28-10 is frequently used in self-supervised learning methods, where 28 means the number of layers and 10 the number of the width.
D MORE EXPERIMENTS
In Tables 6 and and Figures and we show the additional experiments to supplement the results in our main paper. From these additional experiments, we find that most of the existing few-shot classification methods are not robust in the apurious-correlation problem. We hope that these studies and the proposed MetaCOCo can facilitate future research on real-world problems.
Table 4: Number of samples for concepts in the MetaCoCo benchmark.
Tiaa Samala
90.0
4215
0/7
2053
2872
3297
2012
2018
2011
2H16
111
7144
3114
1914
2017
7011
1955
101
1977
1915
1981
1176
(191
1230
02
big
1494
1904
146
(40
Aroples
841
301
774
711
157
581
190
106
81
87
17

---

Published as a conference paper at ICLR 2024
Table 5: Detailed statistics of the MetaCoCo benchmark.
Concedts
Contents
dog
bic
1ible
beir
bocse
Tesce
ox
bayycle
pomncyylee
Irann
col
elepliar
tus
char
Indk:
orplase
pants
sheep
belioogeer doar monkey
553
559
440
575
naldoor
456
131
dim
322
241
414
166
318
390
521
489
235
268
338
202
145
485
209
0
308
477
194
white
218
153
63
426
156
110
256
209
0
136
53
252
163
238
263
505
60
557
rook
195
327
335
browen
357
145
65
967
246
363
407
214
D
25
106
55
204
0
211
238
wood
0
1,583
0
T69
446
0
365
bluc
groce
gray
275
26
446
os soow
250
141
L12
140
O
L8B
360
136
IIS
136
71
128
118
87
I8L
J02
large
87
57
58
91
63
109
109
159
62
217
116
993
lying
234
145
220
0
173
73
D
116
with people
0
126
173
173
129
50
58
beach
265
199
46
ST
50
fareat
246
95
roud
57
179
O
140
241
358
91
149
sihee
0
108
111
G7
D
24
130
apen
yellow:
28
20
98
34
SS
standing
ITS
DI
city
0
1.49
201
75
in river
130
176
water
140
174
11
O
71
117
dark
164
walking
glass
211
mrange
107
16
2%
71
hasd
plattic
long
130
195
79
clood
176
pink
nound
187
hase
320
bably
in garage
143
yashi
II
mack
station
brandh
2,00
leady
256
shared
velodrone
or wharf
doublc Aodker
1I
I
0
helipert
bridge
S6
44
burnos
patt
at aispoct
156
155
sterping
110
caling graw
133
in nace
150
takin off
mountan
shout
18

---

Published as a conference paper at ICLR 2024
coneetts
bone deace car Diyycle Eooorcycle
train cow elephant
chair nirplace
parss shepp Telicopter
open mouds
0
0
0
0
0
0
5
5
booth
0
0
112
0
0
0
0
howlling
Khuki
O
D
0
0
D
Q
71
D
or dock
U
D
0
0
0
O
9
head
acound
climbing
steaz
web
keather
curting
concrese
on izeberg
shoultee
80
right
with flower
grazing
paper
ypotied
at
with boz
hanging
clon
upaa
aubwaay
jumping
Rower
badling
pooocian
de surset
dokk
fihiin
puppl
off
zold
diny
noaing
beine
rectangutar
19
thit
trick
in tuuker
city
bind foeder
shelven
1
wall
post
shury
sinking
cargo
hright
shell
muido trviiic light
16
emppyy
cress tumnel
36
full
11
plasing
light
starin
solorfal
dark broon
young
19

---

Published as a conference paper at ICLR 2024
Caooggte
Conrets
unbeella lion
squinel
borr
well
Brand
liger
groite
1011 bet Bulloon
seal
tong
Juiket
poose
Vongano
Bower
skip
COCTES
bor
460
417
0
1358
171
201
444
185
17h
152
7401
151
457
191
1.66
.11
300
13.
339
376
236
315
oakdoor
255
324
358
197
-002
259
133
378
434
267
dim
294
107
0
179
56
166
257
250
227
253
251
115
139
0
131
193
257
221
302
310
147
133
94
Whate
136
135
76
0
65
53
74
107
D
U5
0
146
131
94
37
20
0
52
91
147
nock
253
241
0
265
364
201
140
250
254
123
236
272
258
122
86
110
322
203
205
73
191
84
272
234
265
32
205
210
252
115
brows
229
U
135
0
45
22
seod
1
DI
II
II
D
D
0
0
0
0
76
blue
0
9G
BS
36
D
D
104
348
53
23
100
gray
136
24
12S
bang
O
S8
D
S8
D
D
O
D
16
IOL
culing
95
127
112
70
45
73
93
76
LIO
O
116
fying
wilh people
40
25
1.8
34
20
ED
69
R2
beach
168
G7
50
furzat
road
stlver
running
SU
14
69
744
11A
39
D
66
76
D
lag
16
59
24
yellow
standing
in chy
200
D
siling
56
S1
B4
103
water
dark
70
home
walking
glas
oragge
tee
132
in hand
96
DI
D
70
D
lax
dreadd
greand
D
DI
lokie mounten
hac
un
bubyy
124
76
BIG
yadhi
mack
at station
O
wooden
250
11
D
D
firm
Ieafy
stured
wehdfmm::
D
si wharf
226
double decker
cross bridge
19%
in suraci
197
ut helipoct
O
bridge
D
D
to
D
99
D
sleeping
in desert
6B
cating grass
off
power
in circus
mosmaan
11
113
short
20

---

Published as a conference paper at ICLR 2024
Conaeata
unteclla
Eus
squazeel
bual
walf
lannl
liner
pindic
bullir
oxt
sailbeu
Erng
jackei
nabii
posse
kasaae
liseer
dip
cistia
but
open mouth
0
boodh
0
howiling
110
brick
or dock
O
0
75
U
28
0
0
head
O
0
0
O
D
DI
II
0
O
O
0
cround cloud
O
C
O
0
clinting
web
sidk
leather
kobbrg
in shinlder
sight
Dower
grazing
paper
spotee
hassing
clean
square
subway
jumping
lundun
porcelaia
at umor
desk
65
fighring
pepple
gold
diry
nostng
restangelar
thin
thick
bockest
biid Foeder
4.5
shelives
will
shiny
sinking
with cago
zokde meffiic light
crppty
cross nunnet
full
playing
light beows
staving
dak browT
21

---

Published as a conference paper at ICLR 2024
Careva
takatlly
saakows DSCOAI lost
lockea
ost
preapi: Jasta
1ng 1SB capei
eap
relogeodee Brus adti
201
255
137
585
255
327
255
307
114
55
243
eater
246
251
292
111
211
197
330
175
350
240
189
50
Rood
901
46
IOI
20
BS
94
25
23 cww
50
kepz
53
275
22
123
809
72
47
exti
LAE
brigg
alerr
11.
asin
6
127
petioe
151
staniing
drskg
gluc
in kind
132
platic
156
king
ground
pucki
an mack
msodai
in fatta
104
72
Franch
laaty
an hvigg
in burmer
76
slidi me
in aao
Asspin
22

---

Published as a conference paper at ICLR 2024
5-way-5-shot
10-way-5-hhot
60
40
Conv64F
Conv64F
ResNet12
ResNet12
ResNet18
ResNet18
40
K202 Baseline CAN ProtoNer DN4
ANIlBaseline
Versa
X202 Baseline CAN ProtoNet DN4 ANllBaelineet
Figure 6: Experiments different backbone architectures under 5-way and 10-way shot settings.
20
S-way S-shol
10-way 1-shol 10-way S-shot
30
10-way S-ahot
10
20
Versa
Baseline CAN ProcoNet DN4
ANTlBaseline++
Versa
R2D2 Baseline CAN ProteNet DN4
ANTLBaseline++
Figure 7: Experimental results of different ways (left) and shots (right) on testing performance.
Cenoaph
Cancosa
Dlck ypaRt oitics
Vebaac bracily pasppn
wadhover assodic lsadi realbo Betea
Lngdie css wisder paassk
des big 1&B eape: USS INfRINTINN lonic VCTO
KNN monds
113
heeds
uncrdd
clinbing
skin
28
07
22 sick
KS
centing
kecberg
spttal
a yar
13
daen
iabway
jessing
probbh
Eghting
papke
of
pill
in porcc
nn Kecler
dullvos
bright
37
aepor
avo tasad
plasing
23

---

Published as a conference paper at ICLR 2024
spoo Talverr skatdbard
Nockqokk Nelge oI
laptop
Traee
steopp
gias
5
192
waze
260
mundor
01
0
0
0
0
0
0
0
Q
0
46
dire
0
0
0
0
0
0
0
0
0
0
wabio
22
25
27
49
35
163
125
18
0
64
413
167
288
421
151
150
156
neck
47
Kack
wood
15
245
Noz
204
green
20
24
13
IS
Q
27
456
22
pray
21
1h
14
26
IW
100
25
5D
O
lagzz
25
10
51
16
120
34
17
byln
with people
ou leuch
Q
sruil
DG
24
DM
DO
Ir fivest
rea
0
Q
ciiver
EST
10
178
41
muriigg
epan
yellow
17
sonding
is city
<ating
Iall
In water
dask
likking
glam
LS
13
12
in tm
Ir band
fiiiz
ground
pesie moumaaee
pock
noaad
hom.
D
bubry
It garage
ststaton
woodon
is fasts
brunch
leafs
abarad
veleddmme
what
rmec heidgg
it sussec
al Ielipoet
in bridge
is burnos
ansic
It
sippet
sheppig
is dovert
D
rating grase
D
it rave
takizz off
short
24

---

Published as a conference paper at ICLR 2024
speaet
vacspack
besdge
Lewi
open roooth
booth
lanelita lixx
0
0
0
D
9
0
beic
0
0
0
0
0
D1
9
Mhkii
0
0
0
0
0
D
D
0
at dook
0
0
0
O
D
D
0
head
0
arouad clood
stick
leather
callery
concrvae
al park
forhirg
shculler
with
grating
poper
spatted
at yord
with twe
langing
dlan
sabwway
jumping
laeling
poseclaia
at eanoor
dest
Rebinng
pill
dity
ressint
beigs
rectangola:
this
thick
in bucker
chelvee
en wall
on pool
in bio
sliney
with
auide traliic light
erepey
cros tannel
D
full
0
16
playing
0
0
D
light boows
0
0
D
colorial
25

---

Published as a conference paper at ICLR 2024
10
Table 6: Experiments of -tate-of-the-art few-shot classification methods under 5-way 1- and 5-shot setting. Three common backbones are used and the 95% confidence intervals are displayed.
Method
Backbone
Type
5-way 1-shot
5-way 5-shot
Versa Gordon el al.
2018)
Conv64F
Meta
20.00 = 0.00
51.95 + 0.35
R2D2 Bertinetto et
2019
Conv64F
Meta
39.56 + 0.36
51.29
+: 0.36
ANIL
Raghu al.
2020)
Conv64F
Meta
37.24 +:2 0.36
48.04
0.39
CAN Hou ct al. 2019
Conv64F
Metric
43.02 + 0.42
55.48 0.40
ProtoNet ( cl al.
2017
Conv64F
Metric
37.92 + 0.35
51.39 0.38
DN4 (Li et al. 2019a
Conv64F
Metric
38.87 +
52.72
Baseline CChnn et al.
2019
Conv64F
Fine-tuning
37.97 + 0.32
52.08 0.35
Baseline++ (Chen et al. 2019
Conv64F
Fine-tuning
40.45 t 0.36
53.42 + 0.37
Versa Gordon al.
2018)
ResNet12
Meta
39.64 + 0.39
53.06 + 0.40
R2D2 Bertinetto et
2019
ResNet12
Meta
45.25 +
60.14
ANIL
Raghu et al. 2020)
ResNet12
Meta
36.58 + 0.38
50.54 0.40
CAN Hou et al. 19
ResNet12
Metric
48,93 + 0.42
62.36 0.40
ProtoNet Snell al.
2017
ResNet12
Metric
42.69 + 0.39
59.50 0.40
DN4 (Li et al. 2019a
ResNet12
Metric
45.04 + 0.39
57.68 0.39
Baseline (Chen el al. 2019
ResNet12
Fine-tuning
46.78 + 0.41
60.78 + 0.40
Baseline++ (Chen el al.
22019]
ResNet12
Fine-tuning
46,95 + 0.42
58.50
Versa Gordon et al.
2018)
ResNetT8
Meta
38.88 0.36
53.18
R2D2 Bertinetoo cI al. 2019
ResNet18
Meta
45.26 + 0.38
58.53 + 0.38
ANIL
Raghu el al. 2020)
ResNet18
Meta
37.26 + 0.36
49.42 + 0.39
CAN
Hou al. 2019
18
Metric
+ 48.18
62.38
ProtoNet Snell et al.
2017
ResNet18
Metric
43.14 + 0.37
57.84 0.39
DN4 Li ct al.
2019a
ResNet18
Metric
43.73 + 0.38
56.56 4: 0.39
Baseline (Chen al.) 2019
ResNet18
Fine-tuning
46,49 + 0.41
60.22 0.39
Baseline++ (Chen et al.
2019
ResNet18
Fine-tuning
47.26 0.40
60.10 0.38
26

---

Published as a conference paper at ICLR 2024
Table Experiments of state-of-the-ard fho-shot classification methods under 10-way 1- and 5-shot setting. Three common backbones are used and the 95% confidence intervals are displayed.
Method
Backbone
Type
10-way 1-shot
10-way 5-shot
Versa (Gordon et al.
2018
Conv64F
Meta
10.00
36.88 =
R2D2 [Brrtnetto el al. 2019
Conv64F
Meta
24.80 0.20
35.98 = 0.21
ANIL Raghu et
2020)
Conv64F
Meta
22.39 + 0.19
33.05 = 0.22
CAN Hou et al. 2013
Conv64F
Metric
28.19 0.23
39.06 0.23
ProtoNet Snell ct al.
2017
Conv64F
Metric
23.72 + 0.19
35.49 = 0.22
DN4 (Li et al.
2019a
Conv64F
Metric
26.26
37.54 =
Baseline (Chen et al.
2019
Conv64F
Fine-tuning
23.71 = 0.18
35.70 = 0.21
Baseline++ (Chnn al. 2019
Conv64F
Fine-tuning
25.55 +: 0.20
36.34 = 021
Versa Gordon et al. 2018
ResNet12
Meta
10.00 0.00
40.21 =
R2D2 Berttnetto et al. 2019
ResNet12
Meta
31.16 + 0.23
42.10 = 0.22
ANIL (Raghu et al.
2020)
ResNet12
Meta
22.94 0.20
33.77 = 0.22
CAN (ouu et al. T9
ResNet12
Metric
31.92 0.24
44.78 0.23
ProtoNet Snell al. 2017
ResNet12
Metric
29.59 0.22
42.38 = 0.24
DN4 (Li et al.
2019a
ResNet12
Metric
30.69
40.31 1
Baseline (Chen et al.
2019
ResNet12
Fine-tuning
31,88 0.24
43,40 = 0.23
Baseline++ (Chen al.
2019
ResNet12
Fine-tuning
31.01
39.89 122
Versa Gordon et al. 2018
ResNet18
Meta
10.00 0.000
36.51 = 021
R2D2 Bertinetto et al.
2019
ResNet18
Meta
29.73 + 0.22
39.74 = 021
ANIL
Raghu et al.
2020)
ResNet18
Meta
10.47 0.10
33.83 =
CAN Hou et al. 2009
ResNet18
Metric
33.16 0.25
43.47 = 0.23
ProtoNet Snell al.
2017
ResNet18
Metric
29.34 0.22
41.34 1 0.22
DN4 (Li et al. 2019a
ResNet18
Metric
29.05 #021
30.12 0.78
Baseline (Chen ei al.
2019
ResNet18
Fine-tuning
33.12 4 0.25
44.63 = 0.24
Baseline++ (Chen al.
2019
ResNet18
Fine-tuning
32.19 0.23
43.32 +
27


Reviews:
[{"review_id": "vB6RfKZEW7", "reviewer": "Reviewer_F7nA", "summary": "Out-of-distribution problems in few-shot classification are important and challenging, which considerably degrades the performance of deep learning models deployed in real-world applications. However, although cross-domain shifts and spurious-correlation shifts are two important real-world distributions, the lack of spurious dataset has left a gap in this important research area. In this paper, the authors present a new benchmark to accelerate the development of the spurious-correlation few-shot classification problem. Overall, I think it is important to study the spurious-correlation few-shot classification and the authors have done a great job to this area.", "questions": "See Weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1: The paper is well-motivated to propose a benchmark with spurious-correlation shifts collected from real-world scenarios. The presentation of this paper is fairly clear, and the details of the dataset construction are also explained thoroughly.\n\nS2: The authors further proposed to use of a pre-training vision-language model CLIP for measuring the quantifying and comparing the extent of spurious correlations on MetaCoCo and other FSC benchmarks, which is quite impressive.\n\nS3: Extensive experiments are conducted to investigate the ability of current methods for addressing real-world spurious correlations, with the baselines including a wide range of existing representative approaches.", "weaknesses": "W1: Currently, many vision-language pre-training models, such as CLIP [1] and CoOp [2], evaluate the performance in few-shot setting. I hope to see the performance of pre-trained large models like CLIP on the MetaCoCo with spurious-correlation shifts.\n\n[1] Radford, Alec et al. Learning transferable visual models from natural language supervision, ICML.\n\n[2] Zhou, Kaiyang and Yang, Jingkang and Loy, Chen Change and Liu, Ziwei, Learning to prompt for vision-language models, IJCV.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EGNELG5Cnp", "reviewer": "Reviewer_QLBK", "summary": "This paper presents a novel dataset, called METACoCo. The dataset targets at one research void that existing public datasets cannot evaluate – the spurious-correlation problem in few-shot classification. One metric is proposed to evaluate the spurious-correlation shifts. Experiments are conducted to evaluate the performance of existing few-shot classification models etc. The results shows that the evaluated models degrade significantly in the presence of spurious-correlation shifts.", "questions": "What is the difference between spurious-correlation studied in this paper and overfitting in few shot learning? Can we view spurious-correlation as one type of model overfitting? If this is true, it would be interesting to have a comprehensive study of overfitting problem in few-shot learning, in addition to the spurious correlation studied in this paper.", "limitations": "", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The main contribution of this paper is the created dataset. The dataset enables researchers to study spurious-correlation problem in few shot learning. The conducted experimental analysis is persuasive.", "weaknesses": "One of my concerns is the technical contribution of this work. Although this paper points out spurious-correlation that have not been well addressed by existing methods and a dataset is proposed, the technical contribution of this work would be more significant if a model can be provided to address the problem.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PO6kZicdgD", "reviewer": "Reviewer_xD6Q", "summary": "The authors present a large-scale, diverse, and realistic environment benchmark for spurious-correlation few-shot classification. Extensive experiments on the proposed benchmark are performed to evaluate the state-of-the-art methods in FSC, cross-domain shifts, and self-supervised learning. The experimental results show that the performance of the existing methods degrades significantly in the presence of spurious correlation shifts.", "questions": "1. In the experiment, why not use the most recent methods for comparison? I have only listed some references for 2021-2022, and the authors can use other recent methods rather than the enumerated ones.  I'm just curious about how recent methods perform on the MetaCoCo dataset.\n2. How to understand the conceptual and contextual information mentioned in the manuscript?\n3. The network structure of ResNet-50 is not explained in EXPERIMENTAL SETUP, it is suggested that the author explain it in backbone architectures.\n\n[1]  Yang, Zhanyuan, Jinghua Wang, and Yingying Zhu. \"Few-shot classification with contrastive learning.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.\n[2]  Wertheimer, Davis, Luming Tang, and Bharath Hariharan. \"Few-shot classification with feature map reconstruction networks.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.\n[3]  Wang, Heng, et al. \"Revisit Finetuning strategy for Few-Shot Learning to Transfer the Emdeddings.\" The Eleventh International Conference on Learning Representations. 2022.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and neatly presented. \n2. Exploring the spurious-correlation few-shot classification is a very interesting task.\n3. Sufficient experimental results show that spurious correlation shifts damage the model performance.", "weaknesses": "1. Contributions are not clearly and accurately stated.\n2. The explanation for the spurious-correlation few-shot classification is not easy to understand.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)