Understanding Cross-Domain Few-Shot Learning Based on Domain Similarity and Few-Shot Difficulty
Authors: Jaehoon Oh
Sungnyun Kim
Namgyu Ho
Jin-Hwa Kim
Hwanjun Song
Se-Young Yun
Venue: NeurIPS 2022 Accept (2022)
Keywords: Cross-domain Few-shot Learning
Pre-training
Domain Similarity
Few-Shot Difficulty

Abstract: Cross-domain few-shot learning (CD-FSL) has drawn increasing attention for handling large differences between the source and target domains--an important concern in real-world scenarios. To overcome these large differences, recent works have considered exploiting small-scale unlabeled data from the target domain during the pre-training stage. This data enables self-supervised pre-training on the target domain, in addition to supervised pre-training on the source domain. In this paper, we empirically investigate which pre-training is preferred based on domain similarity and few-shot difficulty of the target domain. We discover that the performance gain of self-supervised pre-training over supervised pre-training becomes large when the target domain is dissimilar to the source domain, or the target domain itself has low few-shot difficulty. We further design two pre-training schemes, mixed-supervised and two-stage learning, that improve performance. In this light, we present six findings for CD-FSL, which are supported by extensive experiments and analyses on three source and eight target benchmark datasets with varying levels of domain similarity and few-shot difficulty. Our code is available at https://github.com/sungnyun/understanding-cdfsl.

Understanding Cross-Domain Few-Shot Learning Based on Domain Similarity and Few-Shot Difficulty
Jaehoon Oh" KAIST DS Daejeon, South Korea jhoon. oh@kaist.ac ac. kr
Sungnyun Kim" KAIST AI Seoul, South Korea
Namgyu Ho* KAIST AI Scoul, South Korea itsnamgyu@kaist. ac.kr
1an43970kaist ac kr
Jin Kim NAVER AI Lab, SNU AIIS Seongnam, South Korea 1inha. kim/navercorp.com com
Hwanjun Song! NAVER AI Lab
Se- Se-Young Yun KAIST AI Scoul, South Korea
Seongnam, South Korea
hwanjun ju wongdnavercorp..com
yunseyoungokaist.cm ac kr
Abstract
Cross-domait few-shot learning (CD-FSL) has drawn increasing attention for handling large differences between the source and target domains-an important concern in real-world scenarios. To overcome these large differences, recent works have considered exploiting small-scale unlabeled data from the target domain during the pre-training stage. This data enables self-supervised pre-training on the target domain, in addition to supervised pre-training the source domain. In this paper, we empirically investigate which pre-training is preferred based on domain similarity and few-shol difficulty of the target domain. We discover that the performance gain of self-supervised pre-training over supervised pre-training becomes large when the target domain dissimilar the source domain, the target domain itself has low few-shot difficulty. We further design two pre-training schemes, mixed-so -upeevs-eeeeieee and two-stage learning, that improve performance. this light, we present six findings for CD-FSL, which are supported by extensive experiments and analyses on three source and eight target benchmark datasets with varying levels of domain similarity and shot difficulty. Our code is available at cdffl
httpp:://githut: com
Introduction
Few-shot learning (FSL) is a machine learning paradigm to learn novel classes from few examples Unlike standard supervised learning, model is pre-trained on the source dataset consisting of base classes and then transferred into the target dataset consisting of novel classes with few examples, where base and novel classes are disjoint but share similar data domains. However, this underlying assumption is not applicable to real-world scenarios because source (base classes) and target (novel classes) domains are different in general. This leads poor generalization performance because of the change in feature and label distributions, posing new challenge in FSL.
with supervised information 666 by
[24 64
In this regard, cross-domain few-shot fewssffew-shoot learning (CD-FSL) is gaining immense attention with the BSCD- FSL (Broader Study of CD-FSLL) benchmark 24] which enables us evaluate real-world few-shot learning tasks. The BSCD-FSL benchmark a collection of four different datasets with varying
"Equal contribution. Corresponding authors.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

levels of domain similarity to large-scale natural image collections, such as ImageNet M Although
there are two possible directions for FSL, meta-learning [17 351 641 and transfer learning 62). transfer learning has been reported to have higher performance than meta-learning approaches in cross-domain scenarios. Therefore, following the transfer learning pipeline, recent studies for CD- have mainly focused on improving the pre-training phase before fine-tuning on the target labeled data with novel classes.
FSL 47 30
To address the challenge of different domains, there have been recent efforts to leverage unlabeled examples from the target domain as auxiliary data for pre-training, in addition to labeled examples from the source domain. For example, along with the supervised cross-entropy loss, STARTUP 4 and Dynamic Distillation 300 incorporate distillation loss and FixMatch-lik, loss for self-superrviion, respectively. In other words, they develop sophisticated pre- tre-training approaches that can leverage source and target data together. However, the basic pre-training schemes, supervised learning (SI.) on the source domain and self-supervised learning (SSL) on the target domain, have not been thoroughly studied with respect to their pros and cons in CD-FSL.
In this paper, we establish an empirical under- standing of the effectiveness of SL and SSL for a better pre-training process in CD-FSL. To
High
B
ChestX
this end, we begin by scrutinizing an opposing
Few- Shot Difficulty
finding of the previous works 47 50] We dis-
cover that readily available SSL methods, e.g., SimCLR [] can outperform the standard SI.
1SIC
CUB
Cars
method for pre-training, even when the amount
of unlabeled target data for SSL is much smaller
Plantae
than that of labeled source data for SL (see Sec-
CropDisease
ERTSAT
Phaces
Low
tion
Small
Large
Next, we investigate why the CD-FSL per- formance depends on different mre-training schemes using the two properties: domain sim- ilarity and few-shor difficulty. Domain Similar- ity the similarity between the source and target domains, which is known to affect the transfer- ability of the source domain features into the target domain I0O (6). However, we find it in- sufficient to identify the effectiveness of SL and SSL based on domain similarity alone. To solve this conundrum, we propose Few-Shot Difficulty as a measure of the inherent hardness of dataset, based the upper bound of empirical FSL performance. By grounding analysis these two metrics, we discover coherent insights on CD-FSL pre-training schemes, depicted in Figure [] Our analyses point to two conclusions: (A) When domain similarity is small, SSL is preferred due to the limited transferability of source information. On the other hand, (B) SL is preferred when domain similarity is large and few-shot difficulty is high, because supervision from the source dataset achieves stronger performance compared to self-supervision on difficult target data (see Section 5
Domain Similarity to ImageNet
Figure 1: Our insights on the pre-training ap- proaches. (A) SSL preferred for all datasets with small domain similarity. (B) SL is preferred for high-difficulty datasets with large domain similar- ity. The formal definitions of similarity and diffi- culty are explained in Section 3.2
Finally, to investigate whether SL and SSL can synergize, we design a joint learning scheme using both SL and SSL, coined as mixed--supervised learning (MSL). is observed that SL and SSL can synergize when they have similar performances. Furthermore, we extend our analysis to two-stage pre-training scheme, motivated by recent works on CD-FSL 47 BD] We observe that this generally improves performance because the SL pre-trained model provides good initialization for the second phase of pre-traane pre-training (see Section 5
Related Work
2.1 Few-Shot Learning (FSL.
FSL has been mainly studied the literature based on two approaches, meta-learning and transfer learning. In the meta-learning approach, model is trained on the meta-train set (i.e., source data) in an episodic manner, mimicking the evaluation procedure, such that fast adaptation possible on the meta-test set (i.e., sho-shot target data). This family of approaches include learning good

---

and learning an update rule model is pre-trained on the source dataset following the general supervised learning procedure in a mini-batch manner, and subsequently fine-tuned on the target dataset for evaluation.
initialization E8
18 48 E31, learning a metric space 666
54
58
[50
19]. By contrast, the transfer-based approach A 12
2.2 Cross-Domain Few-Shot Learning (CD-FSL)
CD-FSL has addressed a more challenging and realistic scenario where the source and target domains Such cross-domain setting makes it difficult to transfer source information
are dissimilar 24] 64
In general, the most recent methods have been developed on top of the fine-tuming paradigm because this paradigm outperforms the traditional meta- leatallerniig approach such as FWT 24. STARTUP 47 and Dynamic Distillation
into the target domain owing to large domain differences A44 36
40
73
30 the two representative algorithms, and they suggested using small-scale unlabeled data from the target domain in pre-training such that a pre-trained model can be well-adaptable for the target domain. Specifically, both algorithms first train a teacher network with cross-entropy loss labeled source data. Then, STARTUP trains the student network with cross-entropy loss on the source data together with two unsupervised losses on the target data: distillation loss [2S] and self-supervised loss (i.e., SimCLR (.). Dynamic Distillation trains the student network with cross-entropy loss on labeled source data and KL loss based on FixMatch 55 on unlabeled target data.
2.3
Self-Supervised Learning (SSL)
SSL has attracted attention as method of learning useful representations from unlabeled data 14
13
When this field first emerged, hand-crafted pretext tasks, such as solving jigsaw puzzles 41 and predicting rotations (2), were designed and utilized for training. In recent times,
72
46
there has been an effort to use contrastive loss, which enhances representation learning based on
augmentation and negative samples B 61 26 2 This contrastive loss encourages the alignment of positive pairs and uniformity of data distribution on the hypersphere (67] This improves the transferability model by encouraging it to contain lower-level semantics compared to supervised approaches [31]. However, this advantage is conditional on the availability of numerous negative samples. To alleviate such constraint, non-contrastive approaches that do not use negative samples have been proposed [ 5
7 63) In our empirical study, we two contrastive approaches, SimCLR [3] and MoCo 26 and two non-contrastive approaches, BYOL and SimSiam [5] The
details of each algorithm are described in Appendix A
For the completeness of our survey, we include prior works that address SSL for cross- and/or few-shot learning. Kim et al. [22] addressed self-supervised pre-training under label-shared cross-domain, while our setting does not share the label space between domains. Ericsson et al. [16] observed that SSL on the source data improves performance on the BSCD-FSL. dataset. However, domain-specific SSL (i.e., SSL on target data) was not addressed. Cole et al. B showed that adding data from different domains can lead to performance degradation when data is numerous. Phoo and Hariharan 47 and Islam ct al. [30 argued that plain SSL methods struggle to outperform SL for CD-FSL. We investigate domain-specifer SSL and demonstrate its superiority, which opposes the finding from previous studies.
3 Overview
Table 1: Summary of the notations.
We clarify the scope of our empirical study, propose formal definitions of do- main similarity and few-shot difficulty. and describe experimental configura- tions. Table I summarizes the nota- tions used in this paper.
Notation
Description
DB, DN Co, Cn
Source and target datasets, Dp n DN = Base classes for DD and novel classes for Dx Unlabeled target data for SSL
DU (xDx Dr. (CDN)
Labeled target data for evaluation, Du Dr, # classes and examples for n-way k-shot
n Ds (CDL)
A support set with size nk for fine-tuning
3.1 Scope of the Empirical Study
Da (CDL)
A query set for evaluation, Ds n Do =
Our objective is to learn a feature ex- tractor I on base classes CB in source data DB, which can extract informa- tive representations for novel classes
A feature extractor (backbone network)
hgg
A classification head for SI. during pre-training
hasl
A projection head for SSL during pre-training
A classification head during fine-tuning
3

---

CN in target data Dv. Typically, classifier is fine-tuned and the model ff is evaluated using labeled target examples D (c. Dvy after pre-training f on the source data DB under the condition that the base classes are largely different from the novel classes.
further assume that additional unlabeled data Dy (c D is available in the pre-training phase. We follow the split strategy used in Phoo and Hariharan 47 where 20% of the target data DN is used as the unlabeled data Dy for pre-training. Note that the size of the unlabeled portion is very small (e.g., only a few thousand examples) compared to large-scale datasets typically considered for self-supervised learning. In this problem setup, the pre-training phase of CD-FSL can be carried out based on three learning strategies:
Following the recent literature 47
50
Supervised Learning: Let f and hyt be the feature extractor and linear classifier for the base classes Cg, respectively. Then, model hyo f is pre-trained only for the labeled source data DB by minimizing the standard cross-entropy loss fee in a mini-batch manner Bl
(al [DBl
Σ leelty o [()),))-
(a.y)EDs
Self-Supervised Learning: Let hast be the projection head. Then, model hest is pre-trained only for the unlabeled target data Dr. which is much smaller than the labeled source data, by minimizing contrasvee self-supervised loss leef (e.g., NT-Xent),"
(f hest: Du) =
Σ Je DE
Bselt (21:22; {2 + leelf 22:
(2)
Du
where 2, = has! o f(A(()))
and (x is the i-th augmentation of the same input æ. This training loss forces 21 to be similar 22 and dissimilar the set of negative features {2-}. In addition, there are non-contrastive SSL methods that do not rely on negative examples, i.e., {= }= =0. We provide a more detailed explanation of SSL losses, including multiple (non-)constrastive approaches in Appendix A
Mixed-Superviae Learning: MSL exploits labeled as well unlabeled data from different domains simultaneously. MSL. can be intuitively formulated by minimizing the interpolation of their losses in Eqs. 1 and 2
(3) 1 and the feature extractor f is hard-shared and trained through SL and SSL losses with a balancing hyperparameter 7, This can generalization of STARTUP and Dynamic Distillation, which use Eq. 3 in the second pre-training phase with moderate modification after the typical pre-trate neeitraiiing phase using SL.
(fal(f (f, har, hesl: DB, Du) (1 (1 ?) Lalf. hat DB) + Lasl(f, hest Du),
where 0
Our analysis focuses on pre- pre-trrreeiii training and fine-tuning schemes due to the superiority of transfer- based methods over typical FSL algorithms such as MAML [17]. which is shown in (24] Based on the three learning strategies above, we conduct an empirical study to gain an in-depth understanding of their effectiveness in the pre-training phase, providing deep insight into the following questions:
1. Which is more effective for pre-training, using only SL SSL?
Section
2. How apply domain similarity and few-shot difficulty to identify the effective pre-training scheme between SL. and SSL, for CD-FSL?
D SSeciin
Can MSL, combination of SL and SSL, as well as a two-stage scheme improve performance?
>> Section 0
3.2 Domain Similarity and Few-Shot Difficulty
We present a procedure for estimating the two metrics on datasets, which are used to analyze the domain similarity introduced in [TO. which is based on Earth
pre-training schemes. First, because the distance between the two domains can be considered as Mover's Distance (EMD 52
The batch loss the entire data is used for ease of exposition.

---

the cost of moving images from one domain to the other in the transfer learning context [T0 36 Further details on this metric, e.g., advantages of EMD, are explained in Appendix D
We can easily compute EMD using the retrieved sample representations We create the prototype vector Pa. which is an averaged representation for all examples belonging to class i. Next, let i CB and j E Cy be a class in base (source) and novel (target) classes, respectively. Then, the domain similarity between the source and target data is formulated as
where EMD(DB, = EiecenJEcN Sis dis
Sim (DB, DN) = exp(-a EMD(DB,DN) DN))
Dieco.JECx fis 1px [Dxl
|Palil [DBl
subject to fig 20,
fis E. jECx
E
(4)
iECD
iECg
where di.j = I|p Pyll2; Sug is the optimal flow between p; and PA subject to the constraints for EMD; D(i returns all examples of the specified class i in D; and a typically set 0.01 ITO. Namely, EMD can be interpreted as the weighted distance of all combinations between the base and novel classes. The larger similarity indicates that source and target data share similar domains.
Next, we propose few-shot difficulty, which quantifies the difficulty of a dataset based on the empirical upper bound of few-shot performance in our problem setup, regardless of its relationship the source dataset. To capture the upper bound of FSL performance, we use 20% of the target dataset as labeled data to pre-train the model in a supervised manner. Then, the pre- trained model is evaluated on the remaining unseen target data for the 5-way k-shot classification task 5 As the generalization capability indicates the hardness (56]. the classification accuracy for unseen data is used and converted into the few-shot difficulty using an exponential function with a hyperparameter 33 (the default value is 0.01),
(5) where Acc(D, k) returns the average of 5-way k-shot classification accuracy over 600 episodes for the given data D. Note that in our paper, k is set to 5, but the order of difficulty is the same regardless of High few-shot difficulty implies that the achievable accuracy is low even when there is no domain difference between pre-training and evaluation.
Dif k) = exp(-3 3 Acc(D,
3.3 Experimental Configurations
Cross-Domain Datasets. We use ImageNet, tieredImageNet, and minilmageNet as source datasets for generality. Regarding the target domain, we prepare eight datasets with varying domain similarity and few-shot difficulty; domain similarity is computed based both the source and target datasets, while few-shot difficulty is computed based on the target dataset. To summarize their order in Figure 1 domain similarity to ImageNet: Places > CUB > Cars > Plantae > EuroSAT CropDisease ISIC > ChestX, and few-shot difficulty: ChestX > ISIC > CUB > Cars > Plantae > Places > EuroSAT> CropDisease. For instance, Places data has the largest domain similarity to ImageNet. while ChestX has the highest few-shot difficulty. Appendix B provides the details of each dataset. The detailed values for domain similarity and few-shot difficulty are reported in Appendix and respectively. These are visualized in Appendix F We also provide the results on the case when source and target domains are the same, i.e., the standard FSL setting, in Appendix O
Evaluation Pipeline. We follow the standard evaluation pipeline of CD-FSL 24). The evaluation
process is performed in an episodic manner, where each episode represents a distinct few-shot task. Each episode is comprised of a support set D3 and a query set Do. which are sampled from the entire labeled target data D,. The support set Dz and query set D2 consist of classes that randomly selected among the entire set of novel classes CN. For the n-way k-shot setting, & examples are randomly drawn from each class for the support set Ds. while ki (typically 15) examples for the query set DQ. Thus, the support and query set are defined as,
Dg = {(7] vrxk and DQ = [xx v VT
(6)
"To extract the representation of images, we follow Li et al. 36 by using a large model trained on a large-scale dataset, ResNet101 pre-trained on ImageNet. Note that Cui et al. I0 used JFT dataset [57]. which is not released for public use. Furthermore, described Tff Appendix Our analysis consistent regardless of the feature extractor used.
measure domain similarity using different feature extractors,
'ee use few-Shot learning task instead of classification on the entire data, preventing the performance from being distorted by other factors, such data imbalance and the number of classes.
5

---

Table 2: 5-way k-shot CD-FSL performance (%) of the models pre-trained by SL and SSL.. We report the average accuracy and its 95% confidence interval over 600 few-shot episodes. B and S indicate base and strong augmentation, respectively. The best accuracy is marked in bold for each backbone.
Source Data
Pre-train Scheme
EuT
CropDiscase
ISIC
ChestX
Method
Aug.
I=1
k=5
k=1
k=5
k=
R=
k=1
k=5
ImageNet
SL
Default
66.14 83
84.73+.51
84.18
92.81+:45
31.11+555
44.10=.55
22.48
25.51
tiered ImageNet
SL
Default
68
79.87:..67
66.82.1.0
87.19..59
30.35:1
41.67:.55
22.34
25.08.1.45
S
60.07..88
19.951.66
65.70...4
86.341.60
29.75.1.56
40.601.588
22
25.20...11
SimCLR
B
x0
87. .7
90.94
97.44:.29
34.13:t.09
44.37
21,41=
25.05
S
84,30
94.1211.32
91,0011.76
36.39 97,461.34
47
21.55
25.26.1.44
B
51.21 01
68. 74
70.22::05
67.11=
27.79 53
36.60=259 59
21.44 43
24.28+t.3
MoCo
69.11
81.01
80.08
92.48+.52
29.54+:59
39.28
21.74=
24.58+.44
SSL
B
60,98=
84, 86
81.588.75
96.82+
35.31
49.26 64
22.65=
28.80+
BYOL.
S
66. 86
8..83
85.77
96.93..3u
34,53+.62
47.59 63
22.75
28.36-4.46
SimSiam
44.06
61.031.72
75.36.1.82
92.311.44
26.991.52
35.681.52
22.02.A1
26.06.1.46
S
70.80=
85.10#:57
84.72
96.05
30.174:56
39.51=.55 55
22.17=
26.56+446
(a)
ResNer18 is used as a backbone.
mini ImageNet
B
64.031.91
82.72.1.59
73.381.87
91.531.49
30.68.1.58
41.77.1.59
22.64
26.26.1.45
SL.
Default
8
65.03 88
84.00# 56
72.82:..7
91.32 49
29.91 1911:55 54
40.84 56
22.88 42
27.01
SimCLR
B
66.77=
46
89.33
96.82
33.32+
44.50= 64
22.26=
44
S
79.50
92.36...7
89.49...4
97.24
34,901.64
46.761.61
21.97
25.624.44
48.70 48.70::90
66.85 82
68.77:1.92
87.67.1.57
27.7611:44
38.033.57
21.55=
24.48.1.44
McCo
S
76.20=
80.19 49
93.41
30.20+.55
41.14=.57
21.64=40
24.49+
SSL
B
61.18=
83.11
80.50:
94.85
33.02
46.72+.65
22.90=
27.40+.47
BYOL
66.45 so
86.55
80. 10.100
94.53+.41
33.50+:59
45.99
23.11=
27.71
SimSiam
44.57
63.671.67
82.831.73
95.37
30,74...0
41.28.162
22.76
27.501.47
S
71.66 88
85.21
81.25
95.13
31 80.800.5 59
41.44..59
23.22 A1
27.83
(b) ResNet1O used a backbone.
For evaluation, a classifier is fine-tuned on the support set Ds, using features extracted from the fixed pre-trained backbone f. Note that g is for the evaluation purpose different from hyt and hgal for pre-training. The fine-tuned model f is then tested on the query set Dq. We set n = 5 and k = (1,5,, and the accuracy averaged over 600 episodes following convention 24 97
Implementation. We use different backbone networks depending on the source data. For ImageNet and tieredImageNet, R8 is used as the backbone, while ResNet10 is used for minilmageNet. For R pre-trained on ImageNet with SL, we use the model provided by PyTorch 45 repository. This setup is exactly the same for all pre-training schemes. Additional details on the training setup are provided in Appendix C
4 Supervised Learning on Source vs. Self-Supervised Learning on Target
We begin by investigating the superiority of SSL on the target dataset over SL on the source dataset for pre-training. We compare the CD-FSL performance of pre-trained models using four representative
(widely cited) SSL methods (SimCLR B. MoCo 26. BYO (23] and SimSiam (5) with that of an SL method (Default) in Table 2 Four different domain datasets from the BSCD-FSL benchmarks (EuroSAT, CropDisease, ISIC, and ChestX) are used as target data. Table 2 provides empirical evidence of the findings in this section. Recent literature has reported that SSL pre-training does not work better than SL for the CD-FSL task because of insufficient unlabeled examples in the target However, our observation contradicts this previous finding.
domain 77 30
OBSERVATION 4.1. SSL on the target domain can achieve remarkably higher performance over SL on the labeled source domain, even with small-scale (i.e., a few thousand) unlabeled target data.
EVIDENCE, SSL methods are observed outperform SL in most cases, even though SSL does not leverage source data for pre-training In particular, SSL methods show much higher performance compared to the model pre-trained on the entire ImageNet dataset, which has more than 1.2M training examples. This leads to the conclusion that SSL on the target domain can be better than SL on the source domain for CD-FSL pre- tre-training. In other words, unlabeled target data available at the pre-training phase is worth more than labeled source data, even if the unlabeled target data is much smaller (e.g.. 8k examples for CropDisease) than the labeled source data. In Appendix G we show that SSL can outperform SL using even smaller portions of unlabeled target data.

---

100
100
SL
SC
Acarney
Neamaey
75
Placze (0.934)
CUB
Can (0.818)
Martac EundAT Cuntaraae iSIC
Chal
Plccs (0.334)
CUB
Chask (8.110) (0.505) (0.833) (8.000) (0.000)
Ponter E isic
(0.225)
(8516) (0,05) (0,83)) (0.00)) (0.000)
(0,255)
(0,818)
(a) 5- way I-shot
(b) 5-y 5-shot
Figure 2: 5-w k-shot CD-FSL performance (%) of SL and SSL according domain similar- ity (values in x-axis), with ImageNet source data. The red box shows that SL outperforms SSL in the second largest domain similarity, while SSL outperforms SL in the largest domain similarity.
OBSERVATION 4.2. SSL achieves significant performance gains with strong data augmentation.
EVIDENCE. In addition, the results in Table provide the performance sensitivity to data augmen- tation. For this study, two types of augmentation are used: (I) base augmentation from 3 which consists of random resized crop, color jitter, horizontal flip, and normalization, and (2) strong aug- mentation from [30]. which adds Gaussian blur and random gray scale to the base (see the detail of the augmentations in Appendix [. With strong augmentation, SSL methods exhibit significant performance gains of up to 27.50%p compared to base augmentation, i.e., MoCo on EuroSAT in Table@i), However, SI. does not benefit from strong augmentation as SSL does. This has also been observed the literature 3 Therefore, the performance of SSL can be further improved for CD-FSL. if more suitable augmentation is applied. Based on this observation, we use strong augmentation for SSL as the default setup the rest of our paper.
Meanwhile, the superiority among SSL algorithms varies with target dataset. In Table 2 we observe that SimCLR performs best in EuroSAT and CropDisease, while ISIC, SimCLR and BYOL both perform well. For ChestX, BYOL and SimSiam show good performance. The SSL methods can be categorized into two groups: contrastive (SimCLR and MoCo) and non-contrastive (BYOL and SimSiam). For the rest of our paper, we focus our analysis on SimCLR and B BOL, which are representative methods from each group with robust performance. The results for other target datasets are presented in Appendix
5 Closer Look at Domain Similarity and Few-Shot Difficulty
We investigate why the CD-FSL performance depends on different pre- pre-training schemes, i.e., SL SSL, based on the two metrics: domain similarity and few-shot difficulty in Eqs. 4 and ( We analyze the relationship between few-shot performance and the two metrics on various target datasets and provide insights for developing
effective pre-training approach.
Including BSCD-FSL, we consider four additional datasets from different domains: Places, CUB, Cars, and Plantae. Note that these additional datasets are known to be more similar to ImageNet than the BSCD-F'L datasets are [66. and our estimated similarity shows the same trend. We mainly use ImageNet as the source dataset to make analysis more reliable. We analyze their domain similarity and few-shot difficulty and display them in Figure T where ImageNet used as source data for domain similarity. In this section, to select the SSL method for each dataset, use SimCLR for all datasets except ChestX, where BYOL used, based on the performance observed in Section
5.1 Domain Similarity
shows the CD-FSL performance of the pre-trained models using SL and SSL for eight target datasets with varying domain similarity, where all the datasets are sorted by domain similarity. common belief about domain similarity is that, as domain similarity increases, it is more beneficial Our analysis shows that this
Figure 2
for pre-training to use large amount of labeled source data TC belief is partially true.
OBSERVATION 5.1. SL does not consistently benefit from large domain similarity.
EVIDENCE. For the aforementioned belief to true, the performance gain of SI. over SSL should be greater domain similarity increases. However, although SL outperforms SSL in the CUB dataset
7

---

Places
Performane Com
+0.5
Planta
11.0
Performane
Cam
+0.0
Coptinenss
+0.0
ISIC
-10
-20
0.55
0.52
Fex-Shot Difficully
Few-Shol Difficull
(a) Small Similarity (ImageNet)
(b) Large Similarity (ImageNet)
Performae Gai
Performane Gain
10.0
CrepDr
10.0
Places
BuroSKT
Plantoc
IS0C!
Clas
.0.-
ChestX
CUB
0.40
0.500
Fow Sbot Disficalyy
Few Sbot Diffiouly
(c) Small Similarity (minilmageNet)
(d) Large Similarity (minilmageNet)
Figure 3: 5-way k-shot performance gain of SSL over SL for the two dataset groups according to the few-shot difficulty (small markers: k=1, large markers: k=5). Results are shown for two source datasets: ImageNet and minilmageNet, each with their corresponding backbones.
with the second largest domain similarity, in the Places dataset with the largest domain similarity, SSL rather exhibits higher CD-FSL accuracy than SL (see the red box in Figure 2 Furthermore, in the ChestX dataset with the smallest domain similarity, SI. and SSL have similar performances. These results demonstrate that unlike prior belief, large domain similarity does not always guarantee the superiority of SL. In other words, there an inconsistency that cannot be explained solely by domain similarity, and we explore why this inconsistency occurs by taking few-shot difficulty into account.
5.2 Few-Shot Difficulty
In this sense, we study the impact of few-shot difficulty by categorizing the eight datasets into two groups: with small domain similarity (i.e., BSCD-FSL) and another with large domain similarity (iee,, other datasets). Figure shows the performance gain of SSL over SL for datasets with varying few-shot difficulty for each group. The performance gain of SSL over SL. is defined as (Errorg- Error Frrorg, which indicates the relative improvement the classification error.
OBSERvATION 5.2. Performance gain of SSL over SL becomes greater smaller domain similarity or lower few-shot difficulty.
EVIDENCE. For both groups, the performance gain of SSL over SL becomes greater few-shot difficulty decreases. In particular, the performance gain is the greatest on the CropDisease and Places datasets with the lowest few-shot difficulty in each group, while the performance gain is the least the ChestX and CUB datasets with the highest few-shot difficulty in each group. For the target data with higher few-shot difficulty, it may not be easy to learn discriminative representations by solely using SSL without label supervision.
Meanwhile, comparing the two groups (BSCD-FSL vs. other datasets), it is observed that the performance gain SSL over SL is significantly worse for the group with large domain similarity. Namely, the performance gain is near or less than zero when domain similarity is large because features learned from SL with label supervision can be better transferred. Note that the negative value of performance gain means that SL outperforms SSL. Furthermore, the performance gain is closely related to the source dataset size for the datasets with large similarity (see Figures b) and 3 d)). For instance, the CUB dataset, the performance gain (A=5) is 2.276 and -0.249 for ImageNet and minilmageNet, respectively. However, when domain similarity is small (see Figures 3 () and 3(c)), the source dataset size does not significantly affect the performance gain of SSL over SL.
In summary, we first conclude that SSL. advantageous to SL. when the target domain extremely dissimilar to the source domain (i.e., the performance gain is greater than 0), which is in line with

---

Table 3: 5-way 5-shot CD-FSL performance (%) of the models pre- pre-trained by SL, SSL, and MSL including their two-stage versions. ResNet18 is used as the backbone model, and ImageNet is used as the source data for SL and MSL. The balancing coefficient y in Eq. B of MSL is set be 0.875. Datasets are grouped by domain similarity and sorted by few-shot difficulty in ascending order in each group (CropDisease < ChestX
Places CUB). The best results are marked in bold.
Pre-train
Small Similarity
Large Similarity
Method
Scheme
CropDisease
EuroSAT
ISIC
ChestX
Places
Plantae
Cars
CUB
ST.
Default
S4 92
10
25.51
79.22
63.21
66.38
83.93
Single-Stan Stage
SimCLR
97.462.34
94.12++.22
47.85
25.26::44
80.43:t.61
60.07
44.55
47.36:1.79
SSL
BYOL
96.932.30
87.83
47.59
46.36
72.47=.63
61.02
48.564.76
SimCLR
96.50
90. 40
45.38:.61
24.05
82.56
64.762.x7 x3
51.844.79
64.53:
MSL
BYOL
96.74
90.824.40
49.14
29.58
81..77
67.39
46.764.73
69.67
(a) Performance comparison for single-stage schemes.
SimCLR
97.88 30
95,28::.27
48.38:..0
25.25
84.40.t.33
66.35
51.31
57.11
SL-+SLL
Two- Stage
BYOL
97.58:1.6
91.82:1.39 39 49.32:.63
28.27
78.87
67.83
54.70+
60.60
SimCLR
97.49:.30
91.70::335
47.43
26.24
85.76:1.52
69.24
58.97
81,51
SL-+ML
BYOL
97.09+.31
90.89=.40
50.72:.67
30.20
83.29
74.16
68.87
84,34
STARTUP
96.06+.33
89.70=:41
46.02:.59
27 46
85.00
85.00::552
69.40
68.43
89.60
MSL
DynDistill 97.60+ 92.28:: 46 50.06
29.65
82.224
71.492106
69.45=1112 86.54
(b) Performance comparison for two-stage schemes.
Observation This implies supervision with huge amount of source data cannot overcome domain differences. However, when domain similarity is large, the few-shot difficulty must be considered to determine better strategy between SSL and SL. Namely, SL becomes more preferable as few-shot difficulty increases due to the benefits from supervision on the source dataset. The same trend is observed when tieredImageNet used as the source dataset (Appendix M.
Advanced Scheme: MSL and Two-Stage
In this section, we further study SL and SSL in a more advanced scheme from the domain similarity and few-shot difficulty perspective, in line with previous observations. We first investigate whether SL and SSL can synergize by studying MSL. Next, we analyze the two-stage pre- pre-training scheme used in recent works 47 30).
6.1 Can SL and SSL Synergize?
To identify whether SL and SSL can complement each other, we first consider a mixed-loss pre- training scheme, MSL, described in Eq. 3.. We define that synergy between SL and SSL occurs when MSL is superior to both SL and SSL. Table summarizes the performance of the models under each pre-training scheme on eight target datasets, grouped by their domain similarity (BSCD- FSL vs. other datasets) and then sorted by the few-shot difficulty in ascending order. In MSL, the hyperparameter 7 is set be 0.875 found by grid search, detailed in Appendix [
OBSERVATION 6.1. SL and SSL can synergize when SL. and SSL have similar performances.
EVIDENCE. Table 3ja,, observed that SL and SSL can synergize (i.e., MSL > SL, SSL) on four datasets: ISIC, ChestX, Places, and Plantae. SL and SSL have similar performances on these datasets, as shown by the large markers (I==) in Figures 3[a) and B(b). MSL can learn diverse features, owing to differences in training domains (i.e. source target) and learning frameworks (i.c., supervised vs. unsupervised), which allows for synergy [6
22 21 However, when either SI. or SSL significantly outperforms the other, MSL does not perform best. In addition, MSL. performance can be improved further in the large similarity group by emphasizing the SL. component through a larger batch size (Appendix K
6.2 Extension to Two-Stage Approach
We extend the single-stage to two-stage approaches, extracting more sophisticated target representa- tions. In two-stage pre-training, model is pre-trained in prior with labeled source data in the first

---

phase and further trained through SSL or MSL in the second phase, i.e., - SSL or SL MSL. This pipeline has been adopted by recent algorithms, such as STARTUP 47 and DynDistill BQ but they additionally maintain an extra network or incorporate the knowledge distillation in the second phase, i.e., SI. - MSL. Table $ b) summarizes the CD-FSI. performance of two-stage schemes.
OBSERVATION 6.2. Two-stage pre-training schemes are better than their single-stage counterparts.
EVIDENCE. sto-stage pre-training approaches generally achieve much higher performance than their single-stage counterparts, i.., SI. SSL. outperforms SSI., and SI. MSI. outperforms MSL.. When SL is used separately in the first phase, it appears to provide good initialization for the second phase because a converged extractor on the source data is better than a random extractor [40] Also, the benefit of the two-stage pre-training is significant when domain similarity is large. This observation is promising for practitioners because pre-trained models on ImageNet or bigger datasets readily accessible. In addition, our simple two-stage methods, without any additional techniques, are shown to achieve comparable performance to the meticulously designed two-stage approaches such STARTUP, even though our main goal is analysis of basic pre-training methods. Appendix summarizes the full results including meta-learning based algorithms.
7 Conclusion
We established a thorough empirical understanding of CD-FSL. Our work is a pioneering study that unveils hidden findings in the empirical use of CD-FSL, We believe it can inspire subsequent studies like theoretical analysis, which our paper did not cover. particular, we focused on the effectiveness of SL,, SSL, and MSL, which can be realized with single- and two-stage pre-training schemes. We (1) observed that their performances are closely related to domain similarity between the source and target datasets and few-shot difficulty of the target dataset, and (2) proposed how they can be effectively combined for pre-training. Through our empirical study, we presented six findings that have been either misunderstood unexplored. To justify all the findings, extensive experiments were conducted on benchmarks with varying degrees of domain similarity and few-shot difficulty.
Acknowledgements
This work was supported by Institute of Information & communications Technology Planning & Evaluation (IIT)) grant funded by the Korea government(MSI1 (No.2019-0-00075, Artificial Intelligence Graduate School Program(KAIST), 10%) and Institute of Information & communications Technology Planning & (valuationn(ITPP) grant funded by the Korea governmen((MSIT) (No. 2022- 0-00871, Development of AI Autonomy and Knowledge Enhancement for AI Agent Collaboration, 90%)
References
[1] M. Andrychowicz, M. Denil, S. Gomez, M. W. Hoffman, D. Pfau, T. Schaul, B. Shillingford, and N. De Freitas. Learning to learn by gradient descent by gradient descent. In NeurIPS, 2016.
[2] M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006.0988 2020.
[3] T. Chen, S. Korbblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In ICMI., 2020.
[4] W.-Y. Chen, Y.-C. Liu, Z. Kira, Y.-. F. Wang, and J.-B. Huang. A closer look at few-shot classification. ICLR, 2019.
[5] X. Chen and K. He. Exploring simple siamese representation leariin.. In CVPR, 2021.
[6] X. Chen, H. Fan, R. Girshick, and K. He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003. Rtt.:20033.02997 2020.
[7] N. Codella, V. Rotemberg, Tschandl, M. E. Celebi, S. Dusza, D. Gutman, B. Helba, A. Kalloo,
K. Liopyris, M. Marchetti, et al. Skin lesion analysis toward melanoma detection 2018:
10

---

A challenge hosted by the international skin imaging collaboration (isic). arXiv preprint arXiv:1902.03368, 2019.
[8] E. Cole, X. Yang, K. Wilber, O. Mac Aodha, and S. Belongie. When does contrastive visual representation learning work? arXiv preprint 2xXi:2205.00887,, 2021.
[9] M. Cover. Elements of information theory. John Wiley Sons, 1999.
[10] Y. Cui, Y. Song, C. Sun, A. Howard, and S. Belongie. Large scale fine-grained categorization and domain-specitn transfer learning. In CVPR, 2018.
[11] Deng, W. Dong. R. Socher, L.-J. Li, Li, and Fei-Fei, ImageNet: large-scale hierarchical image database. In CVPR, 2009.
[12] G. S. Dhillon, P. Chaudhari, A. Ravichandran, and S. Soatto. A baseline for few-shot image classification. In ICLR, 2020.
[13] C. Doersch, A. Gupta, and A. A. Efros. Unsupervised visual representation learning by context prediction. In ICCV, 2015.
[14] A. Dosovitskiy, P. Fischer, J. T. Springenberg, M. Riedmiller, and Brox. Discriminative unsupervised feature learning with exemplar convolutional neural networks. IEEE Transactions on Puttern Analysis and Machine Intelligence, 3809) 2015.
[15] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, Uszkoreit, and N. Houlsby. An image worth 16x 16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL http:::/
[16] L. Ericsson, H. Gouk, and T. M. Hospedales. How well do self-supervised models transfer? In CVPR, 2021.
[17] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, 2017.
[18] C. Finn, K. Xu, and S. Levine. Probabilistic model-agnostic meta-lcarning, arXiv preprint arXiv:1806.00 02817, 2018.
[19] S. Flennerhag, A. A. Rusu, R. Pascanu, F. Visin, H. Yin, and R. Hadsell. Meta-learning with warped gradient descent. In ICLR, 2020.
[20] Gidaris, P. Singh, and N. Komodakis. Unsupervised representation learning by predicting image rotations. In ICLR, 2018.
[21] R. Gontijo-Lopes, Y. Dauphin, and E. D. Cubuk. No one representation rule them all: Over- lapping features of training methods. In International Conference on Learning Representations, 2022. URL wtt.t:ttt/////. htttps://openreviiee http:///pwaarrreeeeeeeeeorrraraa
seet/forum?id=BKKK -4gbar//------- 4qbGgIE3
[22] T. G. Grigg, D. Busbridge, Ramapuram, and R. Webb. Do self-supervised and supervised methods learn similar visual representations? arXiv preprint arXiv:21 2021.
[23] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. H. Richemond, E. Buchatskaya, C. Doersch, B. A. Pires, Z. D. Guo, M. G. Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning, arXiv preprint
Ttt::2006.07733. 2020.
[24] Y. Guo, N. C. Codella, L. Karlinsky, V. Codella, R. Smith, K. Saenko, T. Rosing, and R. Feris. A broader study of cross-domain few-shot learning. In ECCV, 2020.
[25] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778. 2016.
[26] K. He, Fan, Y. Wu, S. Xie, and Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.

---

[27] P. Helber, B. Bischke, A. Dengel, and D. Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):217-2226, 2019.
[28] G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in neural network. In NeurlPSW. 2015.
[29] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 4700-4708, 2017.
[30] A. Islam, C.-F. Chen, R. Panda, L. Karlinsky, R. Feris, and R. J. Radke. Dynamic distil- lation network for cross-domain few-shot recognition with unlabeled data. arXiv preprint 0rXiv:2106.07807, 2021.
[31] A. Islam, C.-F. Chen, R. Panda, L. Karlinsky, R. Radke, and R. Feris. A broad study on the trans- ferability of visual representations with contrastive learning. arXiv preprint arXiv:2 135117 2021.
[32] D. Kim, Saito, T.-H. Oh, B. A. Plummer, S. Sclaroff, and K. Saenko. Cds: Cross-domain self-supervised pre-training. In ICCV, 2021.
[33] D. P. Kingma and Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6998 69800, 2014.
[34] J. Krause, M. Stark, J. Deng, and L. Fei-Fei. 3d object representations for fine fineinrinedd catego- rization. In ICCVW, 2013.
[35] K. Lee, S. Maji, A. Ravichandran, and S. Soatto. Meta-learning with differentiable convex optimization. In CVPR, 2019.
[36] H. Li, P. Chaudhari, H. Yang, M. Lam, A. Ravichandran, R. Bhotika, and S. Soatto. Rethinking the hyperparameters for fine-tuning. arXiv preprint 1rXiv:2002.11770, 2020.
[37] H. Liu, J. Z. HaoChen, A. Gaidon, and T. Ma. Self-supervised learning is more robust to dataset imbalance. arXiv preprint arXiv:21 2021.
[38] D. Maclaurin, D. Duvenaud, and R. Adams. Gradient- based hyperparameter optimization through reversible learning. In ICML, 2015.
[39] S. P. Mohanty, D. P. Hughes, and M. Salathé. Using deep learning for image-based plant disease detection. Frontiers in Plant Science, 7:1419, 2016.
[40] B. Neyshabur, H. Sedghi, and C. Zhang. What being transferred in transfer learning? arXiv preprint arXiv: 1687, 11687 2020.
[41] M. Noroozi and P. Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In ECCV, pages 69-84. Springer, 2016.
[42] M. Noroozi, H. Pirsiavash, and P. Favaro. Representation learning by learning count. In ICCV, pages 5898-5906 2017.
[43] J. Oh, H. Yoo, C. Kim, and S.-Y. Yun. BOIL: Towards representation change for few-shot learning. In ICLR, 2021.
[44] J. Pan and Q. Yang. A survey on transfer learning. IEEE Transactions Knowledge and Data Engineering. 22(10):1345-1359 2009.
[45] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Te- jani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning library. H. Wallach, H. Larochelle, A. Beygelzimer, F. d' A'Alché-Buee E. Fox, and R. Garnett, editors, NeurIPS. 2019.
12

---

[46] D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A. Efros. Context encoders: Feature learning by inpainting. In CVPR, pages 2536-2561 2544, 2016.
[47] C. P. Phoo and B. Hariharan. Self-training for few-shot transfer across extreme task differences. In ICLR, 2021.
[48] A. Raghu, M. Raghu, S. Bengio, and O. Vinyals. Rapid learning or feature reuse? towards understanding the effectiveness of maml. arXiv preprint 190011799 2019.
[49] A. Ramdas, S. J. Reddi, B. Póczos, A. Singh, and L. Wasserman. On the decreasing power of kernel and distance based nonparametric hypothesis tests in high dimensions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
[50] S. Ravi and H. Larochelle. Optimization as a model for few-shot learning, ICLR, 2016.
[51] M. Ren, E. Triantafillou, S. Ravi, Snell, K. Swersky, J. B. Tenenbaum, H. Larochelle, and R. S. Zemel. Meta-learning for semi-supervised few-shot classification. arXiv preprint 2018.
arXivv
[52] Y. Rubner, C. Tomasi, and L. J. Guibas. A metric for distributions with applications to image databases. In Sixth International Conference on Computer Vision (IEEE Cat. No. 98CH36271). pages 59-66. IEEE, 1998.
[53] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattem recognition, pages 4510-4520, 2018.
[54] Snell, K. Swersky, and R. S. Zemel. Prototypical networks for few-shot learning. arXiv preprint
2017.
[55] K. Sohn, D. Berthelot, N. Carlini, Z. Zhang, H. Zhang, C. A. Raffel, E. D. Cubuk, A. Kurakin, and C.-L. Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In NeurlPS, 2020.
[56] H. Song, M. Kim, S. Kim, and J.-G. Lee. Carpe diem, seize the samples uncertain" at the moment" for adaptive batch selection. In CIKM, 2020.
[57] C. Sun, A. Shrivastava, S. Singh, and A. Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEFE international conference on computer vision, pages 843 852, 2017.
[58] Sung, Y. Yang, L. Zhang. T. Xiang, P. Torr, and M. Hospedales. Learning to compare: Relation network for few-shot learning. In CVPR, 2018.
[59] M. Tan and Q Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105-6114, 14. PMLR, 2019.
[60] M. Tan and Q. Le. Efficientnetv2: Smaller models and faster training. In International Conference on Machine Learning, pages 10096-10106. PMLR. 2021.
[61] Y. Tian, D. Krishnan, and P. Isola. Contrastive multiview coding. CECCV, 2020.
[62] Y. Tian, Y. Wang. D. Krishnan, J. B. Tenenbaum, and P. Isola. Rethinking few-shot image classification: good embedding all you need? In ECCV, 2020.
[63] Y. Tian, X. Chen, and S. Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. In International Conference Machine Learning, pages 10268-10278. PMLR, 2021.
[64] H.-Y. Tseng, H.-Y. Lee, J.-B. Huang, and M.-H. Yang. Cross-domain few-shot classification via learned feature-wise transformation. In ICLR, 2020.
[65] G. Van Horn, O. Mac Aodha, Song, Cui, C. Sun, A. Shepard, H. Adam, P. Perona, and
S. Belongie. The inaturalist species classification and detection dataset. In CVPR, 2018.
13

---

[66] O. Vinyals, C. Blundell, T. Lillicrap, D. Wierstra, et al. Matching networks for one shot learning. In NeurIPS, 2016.
[67] Wang and Isola. Understanding contrastive representation learning through alignment and uniformity the hypersphere. In ICML, 2020.
[68] X. Wang, Y. Peng, L. Lu, Z. Lu, M. Bagheri, and R. M. Summers. ChestX-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In CVPR, 2017.
[69] Y. Wang, Q. Yao, J. T. Kwok, and L. M. Ni. Generalizing from few examples: survey on few-shot learning. ACM Computing Surveys (CSUR), 53(3):1-34, 2020.
[70] P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. Caltech-UCSI Birds 200. Technical Report CNS-TR-20111 2010-00,, California Institute of Technology. 2010.
[71] Zbontar, L. Jing, I. Misra, Y. LeCun, and S. Deny. Barlow twins: Self-supervised learning redundancy reduction. In International Conference on Machine Leariing pages 02310-12320 PMLR, 2021.
[72] R. Zhang, P. Isola, and A. A. Efros. Colorful image colorization. In ECCV, 2016.
[73] W. Zhang, L. Deng, L. Zhang, and D. Wu. Overcoming negative transfer: A survey. arXiv preprint arXiv:2009. 2009,00909, 2020.
[74] B. Zhou, A. Lapedriza, A. Khosla, A. Oliva, and A. Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(6):1452-111) 1464, 2017.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] In the abstract and introduction, we established the contributions and scope of our paper the six findings for CD-FSL, which are also described in Figure ! in a compact manner.
(b) Did you describe the limitations of your work? [Yes] Refer to Conclusion. As our work is pioneering study, we exhaustively analyzed CD-FSL and provided several insightful observations; however, there was lack of theoretical analysis.
(c) Did you discuss any potential negative societal impacts of your work? [No] We have checked the ethics guidelines and think no corresponding aspects were found.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We read and ensure it. 2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or URL)? [Yes] Refer to Abstract. We provided the code URI., where we also described how to set up the data. (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? (Yes] Refer to Appendix B and C for dataset and implementation details.
(c) Did you report error bars (c.g.. with respect to the random seed after running experi- ments multiple times)? [Yes]] We added the 95% confidence interval of 600 episodes, following other few-shot learning works.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] Refer Appendix C.T for the training details, including the amount of resources used.
4. If you are using existing assets (e.g., code, data, models) or curating/releusing new assets...
(a) If your work uses existing assets, did you cite the creators? Yes]] We provided the full details on the existing algorithms and datasets in Appendix A and
(b) Did you mention the license of the assets? [Yes] We included the license information of datasets and our own code assets in the code URL attached in Abstract.
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] Refer Abstract. We attached our modified code asset as a form of URL, and this will
be open via GitHub.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "ohI7eubE483", "reviewer": "Reviewer_jPm7", "summary": "In this paper, the authors propose to use Domain Similarity and Few-Shot Difficulty to understand cross-domain few-shot learning. In particular, based on the Domain Similarity and Few-Shot Difficulty, the authors have several observations regarding the performance of self-supervised pre-training and supervised pre-training for cross-domain few-shot learning. Moreover, the authors also propose two additional pre-training strategies to improve the performance of cross-domain few-shot learning. ", "questions": "1. Another experiment that is worth consideration is supervised pre-training on the labeled target domain data. This can further confirm the usefulness of self-supervised pre-training.\n\n2. Would the observations hold even in the cases of larger shots, such as k=20 and k=50？\n\n3. The analysis on the data augmentation is too coarse-grained. What's the individual impact of Gaussian blur and random gray scale on the performance of SSL?\n\n", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Overall, the paper is well-written and structured. The considered problem, i.e., cross-domain few-shot learning, is also important in the area of few-shot learning.\n2. The paper also has several observations regarding cross-domain few-shot learning. \n3. The experimental designs are interesting and shed light upon several things about cross-domain few-shot learning. \n\nWeaknesses:\n\n1. Most of the observations are not surprising. Due to the domain difference, it is expected that self-supervised pre-training on the target can achieve better results than supervised pre-training on the source. \n\n2. The domain similarity in the paper is computed based on EMD. It is not clear if the computed domain similarity reflects the actual similarity between the datasets.  \n\n3. The lack of visualization on the trained representation makes it hard to understand what causes the difference between self-supervised pre-training and supervised pre-training for cross-domain few-shot learning. \n\n4. The proposed improved schemes achieve marginal performance improvement compared with the baselines. \n\n######################\nPost-rebuttal: the rebuttal provided by the authors addresses my concerns. In the revised version, the authors are expected to add the additional results which can greatly improve the paper. I increase my score from 4 to 5. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "e5yP7Ev1Sh_", "reviewer": "Reviewer_Ger5", "summary": "This paper studies the problem of cross-domain few-shot learning in images, and focuses on pre-training and fine-tuning schemes. Through empirical study, the paper shows the findings about the effectiveness of supervised pre-training on base classes and self-supervised pre-training on few-shot data. The paper also incorporates two metrics, domain similarity and few-shot difficulty, to help diagnose the model performance. Finally, a two-stage pre-training scheme with a combination of supervised learning and self-supervised supervised learning is proposed to improve performance in cross-domain few-shot learning.\n\n", "questions": "Please refer to the above \"Weaknesses\".", "limitations": "No limitation and potential negative societal impact section is included in the main paper.\n", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Strengths:\nThis is one experimental analysis paper, and the paper is organized in a systematic way in both writing and experimental presentation.\n\n- Weaknesses:\n1. To quantify the difficulty of a dataset, the paper defines the metric of few-shot difficulty, based on the empirical upper bound of few-shot performance. Specifically, it is shown in L188 that “we use 20% of the target dataset as labeled data to pre-train the model in a supervised manner. Then, the pre-trained model is evaluated on the remaining unseen target data for the 5-way k-shot classification task.” One question arises for the selection of the 20% of the target dataset as labeled data, is there a class split control? the class split control will affect the baseline results significantly which is already shown in some few-shot related papers. Also, the random selection for constructing the “5-way k-shot classification task” will also affect the baseline results. How are the class split control and the construction of “5-way k-shot classification task” realized?\n\n2. L218: In implementation, though different backbone networks (e.g. ResNet18 and ResNet10) are used depending on the source data, they are still shallow backbones. Will these conclusions still hold for deeper backbones?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "XZFl_nlZH0", "reviewer": "Reviewer_eHxj", "summary": "This paper investigates using (extra) unlabeled target domain data for cross-domain few-shot learning (CD-FSL), as well as the impact of source-target domain similarity and the inherent few-shot difficulty. Specifically, given the unlabeled target domain data, the authors set up two learning schemes, i.e., the self-supervised learning and the mixed-supervised learning. Through extensive experiments, they reveal some observations w.r.t. the domain similarity and the few-shot difficulty.", "questions": "\n- Do the unlabeled data and the labeled data have overlapping classes? How do you ensure this?\n- In practice, how do you identify the target domain in advance and prepare the unlabeled images?\n- Why do you use Eq.5 to measure the difficulty of few-shot learning? You should explain its rationality. If you choose different settings of $\\beta$ and $k$, does it change your observations?", "limitations": "The authors have pointed out some limitations in the conclusion part. ", "rating": 4, "confidence": 3, "soundness": 1, "presentation": 3, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- Extensive experiments.\n- The paper is easy to follow.\n\n### Weakness\n\n- The assumption that there are extra unlabeled samples for the target domain is controversial. In realistic few-shot learning, the testing set (consisted of the support and query images) may be collected on-the-fly. How can you identify the target domain in advance and collect the unlabeled images?\n- In Section 4, some important descriptions are misleading as if SSL is superior than SL for base-training (e.g., L224 \"investigating the superiority of SSL over SL for pre-training\"). The real reason is that they use different domains for base-training: the SSL uses the target-domain and the SL uses the source domain. \n- It is not clear that on the target domain, whether the unlabeled data (for base-training) and the labeled data (for support and query) have overlapping classes. \n- In overall, the observations and discoveries provide limited knowledge for understanding the cross-domain few-shot learning. Adding target domain data improves target-domain accuracy is natural, even if the target domain samples are unlabeled. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept