ConFeSS: A Framework for Single Source Cross-Domain Few-Shot Learning
Authors: Debasmit Das
Sungrack Yun
Fatih Porikli
Venue: ICLR 2022 Poster (2022)

Abstract: Most current few-shot learning methods train a model from abundantly labeled base category data and then transfer and adapt the model to sparsely labeled novel category data. These methods mostly generalize well on novel categories from the same domain as the base categories but perform poorly for distant domain categories. In this paper, we propose a framework for few-shot learning coined as ConFeSS (Contrastive Learning and Feature Selection System) that tackles large domain shift between base and novel categories. The first step of our framework trains a feature extracting backbone with the contrastive loss on the base category data. Since the contrastive loss does not use supervision, the features can generalize better to distant target domains. For the second step, we train a masking module to select relevant features that are more suited to target domain classification. Finally, a classifier is fine-tuned along with the backbone such that the backbone produces features similar to the relevant ones. To evaluate our framework, we tested it on a recently introduced cross-domain few-shot learning benchmark. Experimental results demonstrate that our framework outperforms all meta-learning approaches and produces competitive results against recent cross-domain methods. Additional analyses are also performed to better understand our framework.

Published as a conference paper at ICLR 2022
CONFESS: A FRAMEWORK FOR SINGLE SOURCE CROSS-DOMAIN FEW-SHOT LEARNING
Debasmit Das, Sungrack Yun & Fatih Porikli Qualcomm Al Research." {debadas, sungrack, fporiklii@qgtss qualcomm. com
ABSTRACT
Most current few-shot learning methods train a model from abundantly labeled base category data and then transfer and adapt the model sparsely labeled novel category data. These methods mostly generalize well on novel categories from the same domain the base categories but perform poorly for distant domain categories. In this paper, we propose a framework for few-shot learning coined as ConFeSS (Contrastive Learning and Feature Selection System) that tackles large domain shift between base and novel categories. The first step of our framework trains feature extracting backbone with the contrastive loss on the base category data. Since the contrastive loss does not use supervision, the features can generalize better to distant target domains. For the second step, we train a masking module select relevant features that are more suited to target domain classification. Finally, a classifier fine-tuned along with the backhone such that the backbone produces tested it on
features similar to the relevant ones. To evaluate our framework, a recently introduced cross-domain few-shot learning benchmark. Experimental results demonstrate that framework outperforms all meta- meta-learning approaches and produces competitive results against recent cross cross-domain methods. Additional analyses also performed to better understand our framework.
INTRODUCTION
Recently, there has been an expansion the quality and quantity of datasets (Zhang et al., 2018 Sun et al. 2017), computing resources (Jeon al. 2019), and deep neural architectures (Dhillon & Verma 2020). When trained with vast amounts of data, these deep neural network models deliver improved
performance applications like image recognition, action localization, speaker verification, text
analysis, and gene sequence prediction (Nguyen et al. 2018 Yun et al., 2019: Yao et al., 2019 Zhou al. 2018). However, data collection and annotation and cost, which are particularly difficult for specialized domains such as medical imaging and satellite imagery, where domain expertise is needed. Moreover, most neural networks fail to generalize to unseen categories when trained with a few labeled samples. To address these limitations, research on few-shot learning has gained significant attention.
large scale incurs substantial labor
Few-shot learning methods (Wang et al., (020) aim to uncover the data structure and model the concept of new categories with only few labeled samples. A popular strategy to tackle few-shot learning is meta-learning which consists of two stages: mela-train and mela-test. In the meta-train stage, a backbone network is trained to classify the base category correctly by leveraging the labeled source data while mimicking few-shot regime where only limited number of samples available per class in each learning episode. In the meta-test stage, with the trained backbone, the novel categories with only few target class samples be added (or enrolled) and tested. Here, the backbone network can be adapted to the target samples.
Nonetheless, most few-shot learning approaches exhibit insufficient generalization capacity when there is a big gap between the source and target data. To investigate this problem, there have been considerable efforts (Triantafillou et al. 2020 Chen et al. 2019 Tseng al. 2020) in establishing cross-domain few-shot learning (CDFSL) benchmarks. Still, these datasets limit their focus to natural
"Qualcomm Al Research initiative of Qualcomm Technologies, Inc.

---

Published as a conference paper at ICLR 2022
images and fail to capture more pragmatic domain shifts where target data may come from more diverse domains such as satellite and medical imagery. Very recently. /uo et al. (2020) introduced challenging benchmark to evaluate generalization capability on distant target domains. Its target domain datasets consist of images from natural, medical and satellite domains with wide variations of context, color, and perspective. Thus, it represents practical applications where the generic model needs to be adapted to a particular use case. On this benchmark, the popular meta-learning
approaches (Vinyals et al. 2016 Finn al., 2017 Snell al. 2017 Sung al. 2018
Lee et al.
2019) have been found to produce poor recognition performance.
This paper proposes a novel contrastive learning and feature selection system (ConFeSS) for single- source cross-domain few-shot learning. Our framework consists of three steps: pre- pre-training a backbone network on single-source dataset, learning a feature masking module on the target dataset, and fine-tuning the backbone network. In the first step, backbone network is trained in an unsupervised fashion, where a self-supervised learning approach is considered with the contrastive loss (Chen et al. 2020 This is in contrast to meta-learning approaches, which use supervision during the pre-training stage. Although the label of the source dataset is given at this step, we consider the unsupervised learning to alleviate the supervision collapse problem (Doersch al.. 2020) and also to generalize better to the distant target domains. In the second step, a feature masking module is learned with target domain data to generate masks for separating task-relevant features from irrelevant features. This step is required because there is a large discrepancy between the source and target datasets, and hence all the features useful for the souree task might not be helpful or even be detrimental to the target task. Furthermore, we expect the generalization performance to be improved (VC) dimension reduction.
with fewer features in the few-shot regime due the Kapnik-Chervonie In the final step, both the pre-trained backbone network and the classifier are fine-tuned to adapt to the target categories by a proper regularization with the relevant features.
Our main contributions can be summarized as follows: (i) Learning a feature masking module with appropriate constraints to select relevant features for few-shot target samples; (ii) Fine-tuning the backbone by regularizing it with the selected relevant features; (iii) Our extensive experimental evaluation and analyses show that our method produces competitive recognition performance on the new CDFSL benchmark (Guo et al. 2020).
2 RELATED WORK
Meta-learning for Few-shot Learning These methods use episodic pre-training to simulate test conditions followed by fast adaptation to novel category samples. One of the earliest meta-learning methods was MatchNet (Vinyals et al. 2016) which learns a mapping function to project labeled and unlabeled samples to their corresponding labels. ProtoNet (Snell et al. 2017) extended this work by learning a representation and assigning a class depending on the distance of query samples to class prototypes while RelationNet (Sung et al., 2018) learns an additional deep metric function. MetaOpt (Lee al. 2019) takes different approach where an SVM-like like classifier is learned on top of the features for better generalization. Finally, MAML (Finn et al.] 2017) is an optimization- method that learns adapt few-shot novel categories in few iterations. All these meta-learning methods have performed poorly on the CDFSL benchmark (Guo et al. 2020). There are many other meta-learning works but they have not been evaluated on the CDFSL benchmark. One can refer (Hospedales et al. 202T) for comprehensive survey on this topic.
Domain Adaptation In this problem, we have source and target domains with the same categories, and the goal is to reduce domain discrepancy between them. Hence, domain adaptation methods cannot be directly used for CDFSL, where the labels between source and target are disjoint. The universal domain adaptation (UDA) setting ( (You et al. 2019) might be more similar to the CDFSL setting because has different source and target categories. However, in UDA, there some overlap between the source and the unknown target categories with lots of unlabeled target data available while CDFSL considers completely novel target categories each containing only few labeled data.
Cross Few-shot Learning There have been very few works on cross-domain few-shot learning. A recent work (Teeng et al. 2020) uses a noisy transformation layer on top of features to simulate cross-domain distributions and produce better generalization. In (Chen et al.) 2019). the authors compare different meta- learning frameworks and propose a competitive fine- mane-tuning-bae based baseline against these methods for the cross-domain setting. However, the datasets used for evaluating
2

---

Published as a conference paper at ICLR 2022
these methods contain only natural images. As a result, there is no significant domain shift between the source and target datasets even though the source and target labels are disjoint. /uo et al.) (2020) introduce novel CDFSL benchmark and show that most meta-learning methods along with the feature-wise transformation (Tseng et al. 2020) approach perform poorly compared to simple fine- tuning methods. In our paper, the fine-tuning step is augmented with feature selection mechanism to select relevant features. More recent methods that evaluate on the CDFSL benchmark include CHEF (Adler et al. 2020), ATA (Wang& & Deng 202T) and STARTUP (Phoo & Hariharan, 2021) CHEF addresses large domain shift by fusion of Hebbian learners applied on different layers. This is done increase the importance of low and mid-level features for distant domain recognition. ATA
plug-and-play method that improves robustness of models through adversarial task augmentation. STARTUP assumes access to large unlabelled data from the target domaín and proposed combining knowledge distillation and contrastive learning to learn the target model. In our framework, we only assume access to few labeled data from the target domain.
Self-supervision for Few-shot Learning Self-supervised learning has been used in the form of different pretext tasks (He et al. 2020 Noroozi & Favaro 2016 Gidaris et al. 2018) to pre-train representations that can be used for down-stream tasks as well. These representations have been able to generalize well the few-shot regime. Recent works (Gidaris et al. 2019; Su et al. 2020 Chen al. 2021) show that adding self-supervised loss functions for representation learning improves few- shot recognition performance. In our paper, we solely use self-supervision in the form of contrastive loss (Chen al. 2020 during pre-training because mitigates supervision collapse as observed in (Doersch et al. 22200). Furthermore, contrastive losses have been theoretically proven (Saunshi
al. 2019)) to produce better representations for few-shot learning but have not been evaluated for their generalization ability on few-shot novel categories with distant domains.
Feature Selection for Few-shot Learning Feature selection is useful for deriving relevant features for a particular task or for preventing overfitting on few-shot samples. (Zhao et al. (2018) separate the features into orthogonal components where the sparse signal component facilitates the feature selection. It is similar to our approach where we use mask to select relevant and irrelevant features, yet we impose different constraints on these decomposed features. [iu et al. (2017) use a greedy feature selection mechanism followed by multiple dropouts to reduce gradient variance of few-shot samples. However, this method is not applicable for transferring to novel tasks with large domain differences. A more recent work (Dvornik et al. 2020) select features from universal representation learned from multiple source domains by optimizing the selection coefficients for different domains. This quite different from our method, which can work even with single source domain by selecting relevant features instead of relevant source domains. [Brriel et al. (2019) use budget-aware mechanism of optimizing a switch vector to select domain-relevan feature channels from a pre- trained architecture. Additionally, masking has been used to adapt single network weights to multiple new tasks (Maneini al. 2018 Mallya al. 2018).
3 PROPOSED FRAMEWORK
Source Feature Extractor
Mask Generator
Source Feature Extractor
Source Feature Extractor
Mask Generator
(1 m)Os
O
Target
I' mQf
Larlf"
Feature Extractor
kaael())
mOf
1'
(a)
(c)
Figure 1: Our framework consists of three steps: (a) Pre- the backbone using a self-supervised contrastive loss; (b) Learning the masking module on the target data to select relevant features, and (c) Fine-tuning the backbone using regularized loss with positively relevant features.
3

---

Published as a conference paper at ICLR 2022
3.1 PROBLEM DESCRIPTON AND NOTATION
For the CDFSL problem, we have a source domain and a target domain. Each domain has an associated joint distribution P over the input space X' and the label space V. The marginal distribution of the input space is denoted as Px. Instances (I, y) can be sampled from P. where æ is the input and is the corresponding label. Accordingly, the source domain can be represented as (Xa, and the target domain as (%) with joint distributions P. and Pt, respectively. Due to the domain difference, the source marginal distribution Px, will be significantly different from the target marginal distribution Px.- Moreover, the target domain classes are novel; hence there is no overlap between V. and Di- The goal is to first learn model from abundant data sampled from the source distribution Pz. Then the model is adapted to few data sampled from the target distribution P.. Finally, the adapted model is evaluated on held-out test data sampled from the target distribution. In our framework, we learn the model from the source distribution, using supervvee contrastive loss function. The adaptation step on the target data involves learning a mask generator followed by regularized fine-tuning. Our framework is depicted in Fig. T and the details are described in the following subsections.
3.2 UNSUPERVISED TRAINING OF BACKBONE
The backbone (the feature extraction module) is trained in an unsupervised manner inspired from recent works on contrastive learning (Chen et al., 2020) and unsupervised pre-training (Doersch) et al., 2020). Contrastive learning has been found to be effective for transfer learning (?). This is because contrastively learned features focus more on mid and low-level features, which are easily adapted. Furthermore, such features produce better reconstruction by learning a holistic representation of images rather than focusing only on discriminative regions. Thus, contrastive learning is an effective pre-training strategy for transferring representations to distant target domains. In our pre- trainininng stage, we augment samples from the existing samples in the training batch using various transformations and use these augmented samples and original samples to determine contrastive loss. Specifically, let there be N₂, training samples in a batch, where the samples are represented as {x. | For each sample x,, we obtain N, random transformations where the gth transformed instance represented as Xit and & {1, 2..., N.). Following the idea of (Doersch et al. 2020), we enforee the the transformed instances X5t to be close to and far from X1, k + i using the following cross-entropy loss,
No N ΣΣ log
((p(-d(o((xa)) (. (xx)))
I T.con
NoN,
No exp( (xxt) (xx)))
Here, (.).) represents the feature extraction module, and d(-) is a distance metric. Snell et al. (017)) showed that Euclidean distances model Bregman divergence of mixture densities, which consistently performs better for the few-shot setting, and so we choose the same metric. The appendix discusses the theoretical support of constrastive learning for few-shot learning.
3.3 LEARNING THE FEATURE MASKING MODULE
The feature masking module is used to generate masks that can select task-relevant and iare-irreeevant features. For simplicity, we call task-relevant and irrelevant features positive and negative features, respectively. It is important to note that we cannot afford a large masking sub-network because it might overfit to few-shot target domain samples. So, we just mask on features fed to the classifier with appropriate regularization during fine-tuning. Let the feature extraction module learned from the source domain be denoted as (.). Given a batch of target domain samples {(x,, y:) } we can obtain the feature f. o. (x²) E Rdd for each sample. We feed the feature into the mask generating module (-) to obtain the mask m, (f.). This mask is then used to produce positive (f, and negative (f. features, such that
F = ④ f.
f- == m.) Ø
where O is the Hadamard product, and 1 is a vector of ones of the appropriate dimension. m; E Rd is mask vector consisting of d clements where the jth element is represented m.j- To generate binary masks m.j, we follow the probabilistic procedure introduced in (Maddison et al.) 2017 Jang et al. 2017). Let zi be the unbounded output logit from the mask module corresponding to the
4

---

Published as a conference paper at ICLR 2022
th sample and the jth dimension. We generate logistic noise such that / = log(u) log(1 u) and u ~ uniform(0,1). The noise is then added to the logits to produce mask Maj. such that myj = =( where a-)) is the sigmoid operation, and is the temperature scale. The noise is added to the logits to explore different binary masks suitable for the target task. To back- pack-propagate discrete masks during training, follow the atraigh-throuuh estimator (Bengio al. (2013) where we use sigmoid during the backward pass and hard-threshold operation during the forward pass. The hard-threshold operation involves setting maj if myj 0.5 otherwise. During inference mode, the hard-threshold operation of the mask is carried out but with the logistic noise =
To train the feature masking module M(-), want make sure that the positive features ft discriminative while the negative features f not. To produce discriminative positive features f,t. we use the cross-entropy criterion such that
poo (f, E (C+ (E,) ),84),
where Lxgr((-) is the cross-entropy criterion, and C+ (-) is linear classifier used for the positive features f,t. To produce negative features f; we use the maximum entropy criterion such that
Ling (f: = L (f,)))
(4)
where (-) is the entropy of the softmax outputs of (FF))) and C-(-) is linear classifier used for the negative features f. The maximum entropy criterion makes sure that output class probabilities are uncertain causing the negative features to be less class discriminative.
The design of the mask only makes positive and negative features apart. However, they can still be statistically similar in arrangement of clusters and higher-order statistics. Hence, a divergence measure to maximize the statistical distance between the positive and the negative features is required. If we let s-)-) be the statistical distance between two sets of features: the positive set F { ) and the negative set F {(f; ) }, then we would minimize the divergence loss,
(F+ F c (F
(5)
The exponent term is used to provide more stable and smaller gradients when close to optimality, The loss terms in Eq and are weighted and combined to obtain
Lmaak
Apoe + Aneg + Adse
(6)
Here, Lpos and Lney are averaged over the batch samples, while Lase is an aggregated loss function over all the batch samples. These loss terms are combined to obtain the final loss Lmask. which is back-propagated across M-), C++ and C-(- update the respective parameters.
3.4 FINE-TUNING
The fine-tuning stage is the final step of adaptation to the target domain. In this step, we train both the feature extractor and the classifier on the target domain data. Since the target domain contains only few labeled data, we regularize the feature extractor to produce positive features using the mask generator that has been trained in the previous step. Let (-r be the target domain feature extractor that is initialized from the parameters of the source domain feature extractor (...). Given batch of target domain samples {xx, ))) for each sample we can obtain the feature f = di (x,) e Rd This feature f is fed into linear classifier C(-) such that obtain the cross-entropy loss,
Lenak
(f,) Lx (C()), vt).
To regularize the network, we want to make sure that the target domain feature f is close to the target relevant (positive) feature f, = . (o,(x)) o ds (x.). This is realized by minimizing the loss,
Toreg
where ||- is the 2-norm. The regularization ensures that the network does not catastrophically forget the positively relevant features and does not allow the negatively relevant features to be transferred. Additionally, distance- baaeed regularization has been shown promote tighter generalization (Gouk
et al. 2021) as discussed the appendix. The loss terms in Eq. 7 and are combined as
Lt Irtak + AregTrreg.
5

---

Published as a conference paper at ICLR 2022
Ly is then averaged over the training samples in batch to compute the final loss, which is back- propagated across (x(-) and C-)) to update the respective parameters. We can choose not to fine- tune (x(+) but ablation studies in Table 2 show that fine-tuni fine-tuning backbone is effective for CDFSL. This completes the fine-tuning stage. All the stages of our proposed framework, including the pre-training and the fine-tuning steps, summarized in Algorithm 1. For test sample Xte, use (Xte) followed by the softmax operation to obtain the class probabilities and the most probable class.
Algorithm 1: ConFeSS framework
Given: Source dataset D, and Target dataset D; alyperparrametes:: Xpos Aneg: Adios Xreg
Step 1: Pre-train backbone (-) on D,
For each sampled batch source data
For each sampled augmentation
Take gradient descent step of Eq. with respect to (.(-)
Step 2: Obtain mask generator M-) from D,
For each sampled batch of target data
Take gradient descent step of Eq. with respect to M(). (++() and C-( (-)
Step 3: Fine-tune backbone (-) on Dr
Initialize (-) from optimized (.)
For each sampled batch of target data
Take gradient descent step of Eq, with respect to dx(-) and C(-)
Step 4: Predict test sample class using optimized (+(-) and C(-)
4
EXPERIMENTAL RESULTS
4.1 DATASET DESCRIPTION
To evaluate our proposed framework, we test on the CDFSL benchmark introduced by Guo et al. (022)). This benchmark uses mini-ImageNet (Vinyals et al. 2016), which is subset of the Im- ageNet (Deng et al.) 2009) dataset as the source domain that contains abundantly labeled natural categories. The model learned the mini-Imagenet dataset is then tested target datasets containing only few labeled training data. These target datasets have large domain differences from the source domain, and in order of increasing dissimilarity, they consist of the following: a) CropDiseases (Mo- hanty et al. 2016)) containing images of different plant disease types. b) EuroSAT AT (Helber et al. 2019), consisting of different classes of satellite imagery, c) ISIC2018 (Tschandl er al., 2018|Cooeee et al. 2018), which contains different dermoscopic images of skin lesions, and d) ChestX (Wang el al. 2017), a collection of chest X-Ray images of different lung disease types.
Table Results our approach (ConFeSS) as compared with previous methods on the ChestX, ISIC, EuroSAT and CropDisease datasets. The best results are shown in holdface. NWKS means N-way K-shot test setting.
ChestX 5W20S
ISIC
EaroSAT SW2US
CropDiseuse 3W SW20SS
SW5S
5W50S
SW3S
5220S
5WSUS
3W5S
SW50S
MatchNet
22.40
23.61
22.12
36.74
45.72
54.5%
64.45
77.10
54,44
66.39
76.38
58.53
MalchNeHFWWT
23.01
63.38
74.90
MAML
23.48
27.53
40.1
52.36
71.70
81.95
78.05
89.75
ProtoNet
24.05
28.21
29.32
39.57
49.50
51. 51.99
73.29
82.27
S0.48
79.72
88.15
90.81
ProtoNet-FWT
23.77
26.87
30.12
38.87
43.78
49.
67.34
75.74
78.66
72.72
85.82
87.17
RelationNet
22.96
26.63
28.45
39.41
41.77
49.32
61.31
74.43
74,91
68.
80.45
85.08
RelstionNet-FWT
27.56
69.40
MetaOpt
22.53
25.53
29.35
36
49.42
$4.80
64.44
79.
83.62
68.41
82, 89
91.76
STARTUP
26.94
33.19
36.91
47.22
58.63
64.16
82.29
89.
91.99
93.02
97.51
98.45
CHEF
24.72
29.71
31.25
54,30
60.86
75,15
83.31
86.55
86.87
94,78
96.77
FT-AII
25.97
31.32
35.49
48. 11
39.31
66.48
79.08
87. 64
90.89
89.25
95.51
97.68
ADA
ConFeSS
27.09
33.57
39.02
48.85
60.10
65.34
84.65
90.40
92.66
88.
95,34
97.56
4.2 IMPLEMENTATION DETAILS
For a fair comparison, we use the ResNet- 10 backbone introduced by Guo et al. (2020), which produces 512 dimension feature space. We use Adam the optimizer with learning rate of 0.001. The statistical distance sx(-) used in Eq. is maximum mean discrepancy (MMD) (retton et al. 2012). MMD between two distributions and Q over feature space X' is defined as
x |
(X))

---

Published as a conference paper at ICLR 2022
By-alo(Y)]ll) where 6 : X + H. and H is a reproducing kernel Hilbert space. The MMD can be easily computed using the kernel trick while we use the Gaussian kernel in our experiments. Unless explicitly mentioned,
use the pre-training batch size N₂ 50 and the augmentation size N₂ = 3. where the augmentations were chosen as in (Chen et al. 2020) For larger values of N₁, found dip in performance probably because the extra augmentations do not represent transformations in the target domain. The results of different values of N₂ are reported and analyzed later in the paper. The masking module (-) consists of small two-layer feed-forward network with hidden layer dimension of 256. We use small subnetwork for masking to prevent overfitting issues. We set temperature r 1. Also, we set Apos 10 Xneg 10-2, Adix 10- 2 and Aveg 10-2 The numbers of training epochs for Step 1, Step 2, and Step in Algorithm are set as 400, 15, and 50, respectively. The
are kept fixed because it is not possible to create a small-to-medium validation set from the few-shot target dataset. The epoch number of 400 for pre-training is kept the same as fine-tuning methods described in (Guo et al. 2020). For all experiments, the average accuracy over 600 episodes of N-way K-shot setting is reported. Each episode contains randomly sampled K-shot samples per class for adaptation and 15 query samples per class for evaluation, where is the number of sampled classes.
4.3 COMPARISONS
We compare our proposed approach against several meta- meta-learning based few-shot learning methods in- troduced in the CDFSL benchmark (Guo et al. 2020)]: MatchNet (Vinyals et al.] 2016). MAML (Finn)
et al. 2017), ProtoNet (Snell et al.. 2017), RelationNet (Sung al.1(2018), and MetaOpt (Lee et al.
2019). Furthermore, Feature-wise Transformation (FWT) (Tseng et al. 2020) was added to the
backbones to simulate the cross-domain setting of the benchmark. We also include the FT-All (Guo
ctt al. 2020) baseline for the comparison that fine-tunes the full network with only the cross-entropy loss. Additionally, we compare with recent methods, STARTUP, CHEF and ATA, which have been evaluated on the CDFSL benchmark. It is to be noted that SENet (Hu et al.) 2017) scales channels similar to the way we select features. However, does not decompose features into relevant and irrelevant ones. Besides, SENet cannot tackle domain-shift because it is only used as module in feature extraction block. Hence, using SENet as the backbone for comparison would not be useful (or even fair) because all the compared baselines use the ResNet-10 backbone. The results of the comparison for 5-way 5-shot, 5-way 20-shot, and 5-way 50-shot test settings are shown in Table [
From Table M we see that our proposed framework ConFeSS outperforms all meta-learmin methods by a large margin. Specifically, for the 5-shot setting, our method produces improvements of 12.64 %, 21.72 %, 15.50 % and 11.49 % over the best meta-learning method the ChestX, ISIC, AT,,SAT, and CropDisease datasets, respectively. Veritably, the improvement margin increases further as the number of shots increases, especially for medical datasets such as ChestX and ISIC. This is because medical datasets have similar classes and require more annotations to perform reasonably well. Also, the FWT module fails to generalize to target datasets and sometimes negatively affects these methods. As expected, the performance of our framework also follows the rank of domain similarity with minilmagenet: least performance for ChestX and best performance for CropDisease.
The meta- meta-lcarning methods use supervision for pre-training and cannot mimic distant domain datasets, which causes them to overfit source data with poor generalization to distant target domains. In com- parison, our contrastively learned backbone only learns the inherent structure of data transformations and can generalize more effectively to other domains. Secondly, the masking module selects only relevant features for fine-tuning on the target domain, thus preventing overfitting, Our method also performs better than CHEF on all settings. Compared to FT-AII, ATA and STARTUP, our method achieves much higher scores in most settings except for the CropDiscase dataset, which is the easiest benchmark for the cross-domand task as is the most similar to minilmageNet. STARTUP uses large amount of unlabelled target domain data while our proposed approach does not use any. Still, our method outperforms STARTUP in more difficult out of 12 settings.
4.4 ADDITIONAL ANALYSES
Ablation study: Table shows ablation study results. The masking module uses losses functions Lneg and Lave- Also, the fine-tuning step uses loss Lreg- w/o Feature Mask implies that the pre- trained network is fine-tuned using only cross-entropy loss, without using a masking module. We can also choose not to fine-tune the whole backbone, which is denoted w/o FT BB in Table 2 for
7

---

Published as a conference paper at ICLR 2022
Table 2: Ablation results. f shows increase in performance with ablation.
5-way EaroSAT
S-way 20-shot
Setting
CropDisease
ISTC
ChestX
CropDivease
EuroSAT
ISIC
ChestX
Framework
85.83
84.65
48.85
27.09
95.34
90.40
00.10
33.37
w/o Cont. Learn.
87.26
83.15
47.66
26.06
95.47 +
88.78
59.96
32.12
w/o FT BB
85.18
83.
42.25
93.52
89.70
52.61
31.12
w/o Feature Mask
87.57
81.87
47.10
26.09
95.49 +
89.93
61.08 T
33.20
Law
87,95
94.74
59.20
32.77
w/o Iacy
89.03
84.41
48.04
26.60
94.72
90.37
60.14 1
32.81
Leee
90.31
59.83
32.69
Direct Positive
87.15
83.94
47.25
26.58
93:65
89.40
59.66
31.92
the 5-way 5-shot and 5-way 20-shot settings. As shown, in most cases, removing these components result in a drop in performance, suggesting that all these loss functions and components are essential. An important step in the framework is fine-tuning the whole backbone, where the most significant drop is observed when it is absent. This indicates that for large domain differences between source and target domain, fine-tuning the backbone is essential. In the 5-shot setting, there is always a drop in performance when removing the feature masking module. However, for the 20-shot setting, performance improves slightly on the ISIC and CropDisease datasets. This demonstrates that the feature masking module is more critical for fewer shot settings. Among Ldies Lieeg and Ineg- there is no clear winner since their order of importance depends on the dataset and the shot. In the table, w/o Cont. Learn. implies when the contrastive pre-training step is replaced by traditional supervised pre-training using cross-entropy loss. The results show that using supervised pre- pre-training produces lower recognition accuracy than contrastive pre- mre-trainnng. However, the standard supervised pre-training along with masking and tie-ttnnng still performs better than most of the other compared methods in Table T We also consider the Direct Positive setting for the ablation study, where instead of using Treg. we directly use the positive features obtained using the mask generator to fine-tune the feature extractor and classifier. The results obtained using this technique are competitive compared to previous work but still worse than the Full framework and w/o Lreg ablation.
The number of features selected: We also analyzed the number of features selected for different datasets in Fig. 2 (a). According to VC theory, the number of features selected for 5-shot setting is less than that of the 20-shot setting to prevent overfitting. The number of features selected for the 50-shot setting is less because the 50-shot setting has lots of training samples and does not require masking module. Also, CropDisease has the highest number of features selected while ChestX has the least number of features selected because CropDisease dataset contains natural images and is more similar to minilmageNet.
neq Feal pos
10d
Feat nea
(a)
(b)
(c)
Figure 2: (a) Average number of positive features selected for different datasets and shots. T-SNE plot of positive and negative features of CropDisease dataset for (b) 5-way 5-shot, (c) 5-way 20-shot setting
Comparison between positive and negative features: We also investigate the difference between the positive and negative features. In practice, the positive features tion task, and therefore expected to be more discriminative than the negative features. We quantify the discrimination ability of features using the metric DS Tr(S.) (S.))TT(SSw) where S, is the between- class scatter matrix, S. is the within-class scatter matrix and Tr(-) is the trace operation. The scatter matrices are defined as S, EM" nu,(,, #) 10)² and S.. Σ Meg (xy 1y,) where μ4 is the sample mean of the ,th class, μ is the mean of all the samples, and n; is the number of samples in the ith class with total of N classes. (I,, is the jth sample-label pair out of M total samples. Higher values of DS indicate better discrimination. We compared DS across different shots and datasets in Table 3 for the 5- way setting. As expected, DS scores for the positive features are higher than that of the negative features for both the 5-shot and 20-shot settings. The DS scores get higher for the positive features of the 20-shot setting since more training samples produce better
relevant to the classifica-
8

---

Published as a conference paper at ICLR 2022
Table 3: Discrimination scores (DS) of positive (+) and negative features (-) for different shots (S) and datasets
ChestX
ISIC
EuroSAT
CropDisuuse
Setting
58+
58-
205+
208-
5S+
58-
205+
20S-
5S+
58.
20S+
20S-
58+
58-
208+
208-
DS
0.018
0.016
0.03
0.02
0.13
0.10
0.17
0.09
0.67
0.42
0.62
0.28
0.63
0.41
0.77
0.31
clusters. The DS difference between positive and negative features for the ChestX dataset is low mostly because the dataset is very hard to cluster. Visualization using t-SNE (Maaten & Hinton] 2008) for 5-shot and 20-shot settings are shown in Fig. (b) and (c) respectively. Results show that positive features produce better clusters compared to negative ones. As expected, the positive features for the 20-shot setting are more discriminative compared the 5-shot setting. Also, the internal statisties of positive and negative features are different because of Ldiv, which maximizes the statistical distance between the two sets of features.
1980
Sc CurtAT
SurdAT
EaudAT
ExedAT GrapSteees
Cheett
-Ches
Chaii
(b)
(c)
(d)
Figure 3: Accuracy with 5W5S setting (a) log10 Aneg (b) log10 Adiv . (c) log10 Areg and (d) logzo Apos vary.
Hyper-parameter sensitivity: We study our framework's performance as the hyper-parameters Aneg. Aposs Adiv, and Aveg are varied. The results for the 5-shot setting are shown in Fig. 3 which shows that the recognition performance is stable with respect Xneg Apos= and X while the performance drops for larger values of Areg. This is because Anegs Apos. and Adio affect the learning of much smaller feature masking network. On the other hand, the value of Areg affects the learning of a much larger network- the target feature extractor, which eventually plays a direct role in the inference stage. The plot in Fig. (c) shows that we should choose \reg < 1 for better performance.
The impact of the number of augmentations: We report how the number of augmentations N₂ used in contrastive pre-training affects cross-domain few-shot recognition performance. Results for 5-way 5-shot, 5-way 20-shot, and 5-way 50 shot settings are shown in Table Results show sharp drop in
Table 4: Recognition performance on the N-way K-shot setting as N₂ is varied during pre-training.
5-way 5-shot
5-way 20-shot
5-way 50-shot
Dataset/N,
3
10
20
30
3
10
20
30
3
10
20
30
CropDisease
88.88
70.11
72.28
70.51
95.34
86.21
86.85
87.08
97.56
92.48
92.10
92.42
AUTSAT
84.65
62.28
60.38
59.93
90.40
72.45
70.40
69.50
92.66
76.87
75.99
75.29
ISIC
48.85
37.59
38.08
36.64
60.10
48.60
48.80
47.99
65.34
53.86
54.78
53.78
ChestX
27.09
23.30
23.29
23.16
33.57
25.68
25.24
25.37
39.02
27.72
27.34
27.25
recognition performance when the number of augmentations is increased beyond This is because additional augmentations in the source dataset do not represent the possible augmentations in the target datasets. With higher N,, there is a propensity to have augmentations that are not valid for target classes. The target datasets consist of specialized domains like medical and satellite imagery, which also exuberate inconsistent categories when the target datasets are transformed using arbitrary source augmentation policies. As result, contrastive representations learned using those augmentations might not generalize well. For example, in ChestX, random cropping or Gaussian blur might affect discriminative regions in images. This phenomenon has also been recently studied in (Xiao et al.) 2021), where color augmented representations do not transfer well for color discrimination tasks.
5 CONCLUSION
We presented a framework called ConFeSS (Contrastive Learning and Feature Selection System) to learn a generalizable representation followed by a feature selection mechanism while fine-tuning on the target domain. We introduce novel loss constraints on selecting relevant and irrelevant features for the target domain. Extensive experiments conducted on the cross-domain few-shot learning benchmark show approach's 's advantages over the meta- meta-learning and other CDFSL methods. Additional analyses also provide insights into the feature selection mechanism and justify the importance of each component of our framework.

---

Published as a conference paper at ICLR 2022
REFERENCES
Thomas Adler, Johannes Brandstetter, Michael Widrich, Andreas Mayr, David Kreil, Michael Kopp, Günter Klambauer, and Sepp Hochreiter. Cross-domain few-shot learning by representation fusion. arXiv preprint arXiv:2010.06-990 2020.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating propagating gradients through stochastic neurons for conditional computation. arXiv:1308. 343. 2013.
Rodrigo Berriel, Stephane Lathuillere, Moin Nabi, Tassilo Klein, Thiago Oliveira-Samie -antos, Nicu Sebe, and Elisa Ricci. Budget-aware adapters for multi-domain learning. In International Conference Computer Vision, pp. 382-391, 2019.
Zhong Cao, Jiang Lu, Jian Liang, and Changshui Zhang. A theory of self-supervised framework for few-shot learning, 2021. URL http:://openneviee.eeeffommm
Da Chen, Yuefeng Chen, Yuhong Li, Feng Mao, Yuan He, and Hui Xue. Self-supervieed learning for few-shot image classification. International Conference on Acoustics, Speech, and Signal Processing, pp. 1745 1749, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, 2020.
Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. closer look at few-shot classification. In International Conference on Learning Representations, 2019.
Noel CF Codella, David Gutman, Celebi, et al. Skin lesion analysis toward melanoma detection. In International Symposium on Biomedical Imaging, pp. 168- 172, 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009.
Anamika Dhillon and Gyanendra K Verma. Convolutional neural network: a review of models, methodologies and applications object detection. Progress in Artificial Intelligence, 9(2):55-112 2020.
Carl Doersch, Ankush Gupta, and Andrew Zisserman. Crosstransformers: spatially- aware few-shot transfer. In Conference on Neural Information Processing Systems, 2020.
Nikita Dvomií, Cordelia Schmid, and Julien Mairal. Selecting relevant features from multi-domaii representation for few-shot classification. In European Conference on Computer Vision, 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, pp. 1126-1135, 2017.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In International Conference on Learning Representations, 2018.
Spyros Gidaris, Andrei Bursuc, Nikos Komodakis, Pérez, et al. Boosting few-shot visual learning with self-supervision. In International Conference on Computer Vision, pp. 8059-8068, 2019.
Henry Gouk, Timothy Hospedales, and massimiliano pontil. Distance- based regularisation of deep networks for fine-tuning, In International Conference on Learning Representutions, 2021.
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. Journal of Machine Learning Research, 13(1):723-773, 2012.
Yunhui Guo, Noel Codella, et al. A broader study of cross-domain few-shot learning. In European Conference Computer Vision, 2020.
Kaiming He, Haoqi Fan, Yuxin Wu. Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Conference on Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
10

---

Published as a conference paper at ICLR 2022
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2235 2019.
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey. Transactions on Pattern Analysis and Machine Intelligence, 2021.
Jie Hu, Li Shen, Samuel Albanie, Gang Sun, and Enhua Wu. Squeeze-and-excitation networks. Transactions on Pattern Analysis and Machine Intelligence, 2017.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax, In International Conference on Learning Representations, 2017.
Myeongjae Jeon et al. Analysis of large-scale multi-tenant GPU clusters for DNN training workloads. In USENIX Annual Technical Conference, pp. 947-9607 2019.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. Conference on Computer Vision and Pattern Recognition, pp. 7482-7491,
Siavash Khodadadeh, Ladislau Boloni, and Mubarak Shah. Unsupervised meta-learning for few-shot image classification. In Conference on Neural Information Processing Systems, 2019.
Kwonjoon Lee, Subhransu Maji. Avinash Ravichandran, al. Meta-learning with differentiable convex optimization. In Conference on Computer Vision and Pattern Recognition, pp. 10657- 10665, 2019.
Bo Liu, Ying Wei, Yu Zhang, and Qiang Yang. Deep neural networks for high dimension, low sample size data. In International Joint Conference on Artificial Intelligence, pp. 2287-2293, 2017.
Laurens Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9Nov): 2559.265,, 2008
Chris Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: continuous relaxation of discrete random variables. In International Conference Learning Representations, 2017.
Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting a single network to multiple tasks by learning to mask weights. In European Conference on Computer Vision, pp. 67-82, 2018
Massimiliano Mancini, Elisa Ricci, Barbara Caputo, and Samuel Rota Bulo. Adding new tasks to single network with weight transformations using binary masks. In European Conference on Computer Vision Workshops, pp. 180-189, 2018.
Carlos Medina, Aroou Devos, and Matthias Grossglauser. Self-supervised prototypical transfer learning for few-shot classification. arXiv preprint arXiv:2006. 1325, 2020.
Sharada P Mohanty, David P Hughes, and Marcel Salathé. Using deep learning for image-based plant disease detection. Frontiers in Plant Science, 7:1419, 2016.
Phuc Nguyen, Ting Liu, Gautam Prasad, and Bohyung Han. Weakly supervised action localization by sparse temporal pooling network. In Conference on Computer Vision and Pattern Recognition, pp. 6752-6761, 2018.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European Conference on Computer Vision, pp. 69-84, 2016.
Cheng Perng Phoo and Bharath Hariharan. Self-training for few-shot transfer across extreme task differences. In International Conference on Learning Representations, 2021.
Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, et al. A theoretical analysis of contrastive unsupervised representation learning. In International Conference on Machine Learning, pp. 5628-5637 2019.
11

---

Published as a conference paper at ICLR 2022
John Shawe-Tayloo and N Cristianini. An introduction support vector machines and other kemnel- based learning methods. 2000.
Jake Snell, Kevin Swersky, et al. Prototypical networks for few-shot learning. In Conference on Neural Information Processing Systems, 2017.
Jong-Chyi Su, Subhransu Maji, and Bharath Hariharan. When does self-supervision improve few-shot learning? In European Conference on Computer Vision, pp. 645-6666 2020.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In International Conference on Computer Vision, pp. 843-852, 2017.
Flood Sung, Yongxin Yang, Li Zhang, et al. Learning to compare: Relation network for few-shot learning. Conference on Computer Vision and Pattern Recognition, pp. 1199-1208, 2018.
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, et al. Meta-dataset: A dataset of datasets for learning to learn from few examples. In International Conference on Learning Representations, 2020.
Philipp Tschandl et al. The ham00000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific Data, 2018
Hung- Yu Tseng, Hsin- Ying Lee, Jia-Bin Huang, and Ming-Hsuan Yang. Cross-domain few-shot classification via learned feature-wise transformation. In International Conference on Learning Representations, 2020.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Conference on Neural Information Processing Systems, pp. 3630-3638, 2016.
Haoqing Wang and Zhi-Hong Deng. Cross-domain few-shot classification via adversarial task augmentation. International Joint Conference on Artificial Intelligence, pp. 1075-1081, 2021.
Yaqing Wang, Quanming Yao, James Kwok, and Lionel Ni. Generalizing from a few examples: A survey few-shot learning. Computing Surveys, 53(3): 2020.
Xiaosong Wang el Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly- supervised classification and localization of common thorax diseases. In Conference on Computer Vision and Pattern Recognition, pp. 2097-2106, 2017.
Tete Xiao, Xiaolong Wang, Alexei Efros, and Trevor Darrell. What should not contrastive in contrastive learning. In International Conference Learning Representations, 2021.
Liang Yao, Chengsheng Mao, and Yuan Luo. Graph convolutional networks for text classification. In Conference Artificial Intelligence, volume 33, pp. 7370-7377, 2019.
Kaichao You, Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I Jordan. Universal domain adaptation. In Conference on Computer Vision and Pattern Recognition, 2720-2729, 2019.
Sungrack Yun, Janghoon Cho, et al. An end-to-end text-independent speaker verification framework with keyword adversarial network. In Conference of the International Speech Communication Association, 2019.
Qingchen Zhang, Laurence T Yang, Zhikui Chen, and Peng Li. A survey on deep learning for big data. Information Fusion, 42:146-13 157,157, 2018
Bo Zhao, Xinwei Sun, et al. Msplit lbi: Realizing feature selection and dense estimation simultane- ously in few-shot and zero-shot learning. In International Conference on Machine Learning, pp- 5912-5921, 2018.
Jian Zhou el al. Deep learning sequence-based ab initio prediction of variant effects on expression and disease risk. Nature genetics, 50(8):117 2018
12

---

Published as a conference paper at ICLR 2022
A ADDITIONAL EXPERIMENTAL DETAILS
All our experiments were conducted in a cluster of nodes where the nodes contain NVIDIA Tesla V100 GPUs with mix of 16GB and 32GB memory. The details of the benchmark used for comparison have been introduced in the following repository: https://githbb.co www.uu..com/BBBAAA M/cdffl -eaeffll-beehmmrk We only consider the single source domain setting where the source domain is mini mmageee and the
target domains are ChestX 2 ISIC EuroS AT [ and CropDisease P Additional experimental details include: (a) Image size: 224 x 224 (b) Batch size during adaptation: 5 (c) MMD kernel numbers:
(d) MMD kernel multiplier: 2.0. The masking module architecture is as follows: [Linear(512,22)
ReLU 12near(256.512) Gumbel Sigmoid]
B IMPLEMENTATION OF MMD
We use a multi-kernel approach implement MMD. Specifically, for the positive and negative feature matrix: F+ E IR xd and F E Rxxd MMD = mean(XX + YY XY - YX).
1 exp(- IIFt -7
|F,
Here, [XX]s
R= Σ
[Y
K== exp( | - bm
[XY];
bm
Ex exp(
F
and [y = Ek=o exp(
is the 8th row of F+.
bm
::
is the kernel multiplier and ny is the number of kernels. b is the bandwidth and is computed as
b (4N= 20 where [D]s = || Fy:ll and E RR2Nx is the concatenation of F+
sumi D)
and F
C DATA AUGMENTATION FOR CONTRASTIVE PRE-TRAINING
The augmentation policies are the same as those used for pre-training the SimCLR framework (Chen) et al. 2020). They include the following transformations:
Random Crop and Resize: The random cropping has scale in the range [0.08, 1.0] and aspect ratio in the range [3/4, 4/3]. The random cropping is always followed by a horizontal flip with each flip type having probability of 0.5. This is followed by resizing of image to the desired size.
Color jitter is applied with probability of 0.8. The jitter strength values of brightness, contrast, saturation and hue are set as 0.8, 0.8, 0.8 and 0.2, respectively. This is followed by color drop operation (convert to grayscale) with probability of 0.2.
Color Distortion:
Gaussian Blur: This is applied with a probability of 0.5, and radius of blur is selected randomly from the range [0.1, 2.0).
D EFFECT OF DIFFERENT WAYS
We also test our framework on higher number of ways i.e. higher number of classes per episodic evaluation. The results of the experiment are shown in Table As expected, higher number of ways leads to drop in performance because of more difficulty in discrimination. Surprisingly, the drop in performance is less for easier datasets like EuroSAT and CropDisease. This might be probably because the additional classes for higher ways leads to less confusion compared to those of ChestX and ISIC.
100 categories. Downlouded from: https://drive woogle.oogggoooo.. com/fileed wttlil/////////lll http:///wwwmo..o77///////ppooooo
27 categories. License and
information available https:///ww www.kkkgggee....o. chest- xrays/data
77 categories. License and dowaload information available at: httpss//challlngeec iscc- www.:chhwwwi..ommaaaaaa11111
10 categories. Downloaded from: htt::/mddm.c diki het..//ttlessnniee/uasssaaaaaa
38 categories. License and download information available https:/www.kkggtt www.aaapgg......aa.a...ooo com/karoo 4/plant disease
13

---

Published as a conference paper at ICLR 2022
Table Recognition results of approach with different ways and shots (S) the ChestX, ISIC. EuroSAT CropDisease datasets, The entries with imply that testing is not possible because of lesser number total categories present.
ChestX
ISIC
EuroSAT
CropDisease
Setting
58
205
50S
205
50S
205
505
205
50S
3-wway
27.09
33.57
39,02
48.85
60.10
65.3 34
84.05
90.40
92.66
88.88
95.34
97.36
20.72
27.15
32.35
40.38
52.57
39.29
80.23
87.66
90.90
85.37
94.00
96.51
10-way
76.03
84.59
88.15
82.10
92.63
95.48
19-way
75.31
88.45
93.39
E COMPARISON IN 1-SHOT SETTING
Of all the compared methods, only STARTUP (Phoo & Hariharan 2021) and ATA (Wang & Deng 202Tb have been evaluated the 1-shot setting. Hence, evaluate model on the 1-shot setting and report the results in Table 6 Results show that our method is still competitive with respect to
existing methods.
Table 6: Comparison on the I-shot setting.
ChestX
ISIC
EuroSAT
CropDisease
Method
18
18
STARTUP
23.09
32.66
63.88
75.93
34.70
ConFesS
23677
33.46
65.51
76.49
F COMPARISON WITH UNSUPERVISED META-TRAINING
We also compare our method with UMTRA (Khodadadeh 2019) unsupervised meta- training framework derived from the popular meta-learming framework MAML (Finn et al. 2017). UMTRA uses similar algorithm as MAML but extends the case of unlabeled training data. Class membership for unlabeled data is determined such that sample and its augmentation belong to the same class. We compare against two versions: UMTRA-PPotoNN and MMMTR-PProo'TT as reported in (Medina et al.) 2020) the CDFSL benchmark (Guo et al. 2020). UMTRA-Prote Tune extends UMTRA-Proto.Y.
by fine-tuning on target domain data. The results in Table show that our method outperforms the two variants of UMTRA on all shots and all datasets.
Table 7: Comparison against unsupervised meta- meta-training training with different shots (S) and 5-way setting on the ChestX, ISIC. EuroSAT and CropDisease datasets.
ChestX
ISIC
EuroSAT
CropDisease
Method
58
205
50S
58
20S
50S
SS
20S
50S
58
20S
50S
UMTRRA-ProeNe
24.34
28.04
29.88
39.21
44.62
46.48
74.91
80.42
82.24
79.81
86.84
88.44
UMTRA-Prote
25.00
38.
12
68.11
82.67
ConFeSS
27.09
33.57
39.02
48.85
60.10
65.34
84.65
90.40
92.66
88.88
95.34
97.56
G EFFECT OF DIFFERENT EPOCH NUMBERS
The numbers of epochs for training the masking module and fine-tuning are fixed 15 and 50 respectively because we do not have validation split from few-shot target domain dataset to set an optimal value. The epoch numbers (15 and 50) are kept substantially low compared to pre-training epoch number (400) so that the network is less prone to ever-fitting on few-shot data. To show the effect of different epoch numbers, we perform the following experiments on the 5-way 5-shot setting. Firstly, the number of epochs for training the masking module is varied and then the number of epochs for fine-tuning is varied. In Table g we show the results for varying epochs for training masking module while keeping epochs for fine-tuning fixed at 50 well the results for varying epochs for fine-tuning while keeping epochs for training masking module fixed at 15. From the results, it seems that the recognition performance is not very sensitive to the number of epochs used for training masking module or for fine-tuning, However, increasing fine-tune tuning epochs beyond 50 tends to decrease performance slightly for ISIC, ET and CropDisease datasets.
14

---

Published as a conference paper at ICLR 2022
Table 8: Effect of varying epochs for training masking module and for fine-tuning. In the first column, number without parantheses is the epoch number for training masking module while that with parantheses the epoch number for fine-tuning The results are shown in corresponding labelled super-columns.
Varying epoch number for training masking module
Varrying epech number for fine-tuning)
Setting
ChestX
ISIC
EuroSAT
CropDisease
ChestX
ISIC
EuroSAT
CropDisease
5 (25)
26.50
48.82
K3.64
87.64
26.82
48.54
84,23
S6.27
(00)
27.26
15 (75)
27.09
48.85
84.65
88.88
27.09
48.23
83.10
88.56
20 (100)
88.01
27.33
88.23
25 (125)
27.27
48.34
83.69
88.04
27.07
47.48
82.31
88.05
H ALTERNATIVE MODEL. DESIGN CHOICES
In this section, we consider the following alternative model designs and report recognition perfor- mance on the 5-way 5-shot, 5-way 20-shot and 5-way 50-shot setting. The evaluation setup is similar to that described in Section 4.2:
L1 norm: In this design of the ConFeSS framework, we jeplacee the L2 norm ||f] f in Eq. with the LI norm f 11-
Source Mask: In this setup, we train the two layer mask module (defined in appendix A) the source dataset rather than the target dataset. The source dataset used is minilmageNet. Specifically, the mask module is trained on top of the contrastively learned pre-trained feature extractor with the minilmageNet dataset. In the final step, the target feature extractor is fine-tuned on the target dataset.
Neg. Reg.: In this setup, we use negative features for the regularization. Specifically, we use ||[] f 112 in Eq. [
instead of [ 1
Dir. Neg.: We also consider the Direct Negative setting for the ablation study, where instead of using Treg: directly use the negative features obtained using the mask generator to fine-tune the feature extractor and classifier.
Layer Mask Mod.: Here, we study the effect of having masking module of different sizes. Here, K stands for the number of layers used for the masking module. Specifically, we study the effect for K 3, 4, The masking module architecture for = 3, 4, 5 are [Lin- Batch Liinarr128,, 512) - Gumbel Sigmoid], [Linear(5 ( 2,256) Batchhorrm D(256) ReLU - Lin- ear(256,128) -aBtchNorm
car(512,256) Batchhorm wtt.:oommmm22500 D(256) ReL.U Linear(256,
ReLU
ReLU Linear( 128, 256) BatchNorm ReLU (256, 512) Gumbel Sigmoid], and [Linear(512,256) BatchNorm!D(256) ReLU Linear(256,128) 128) BatchNorm/ID(12)) ReLU Linear( 128, 64) BatchNorm
ReLU (6ncar(64, 128)- BatchNoom//D12200 ReLU Linear(1255 512) - Gumbel Sigmoid],
respectively.
Joint Training: We consider the setup where the the masking module and the target feature extractor are trained together in one stage instead of the proposed two stages, using combined losses in Eq. 66 and Eq.
The results of comparing these alternative model designs with our proposed framework ConFeSS shown in Table 9 Results show that among all these alternative model designs, especially L1 norm, Source Mask, Neg. Reg., and Dir. Neg. perform poorly compared our original ConFeSS framework. In most of the cases, having larger masking module produces similar or slightly better performance compared ConFeSS because of better representation capacity of output masks. Joint training of masking module and target feature extractor produces poorer recognition performance for 5 shot setting compared to ConFeSS. However, for higher shot setting, the joint training procedure produces similar or better performance compared to ConFeSS. This is because joint training encompasses optimization larger number parameters, which might cause the network to overfit on lower shots while exploit additional amount of training data for higher shots.
I RESULTS WITH CONFIDENCE INTERVAL
In this section, we re-report comparison studies: Table 10 and Table 11 show the performance with 95 % confidence interval for Table and Table 2, respectively.
15

---

Published as a conference paper at ICLR 2022
Table 9: Recognition performance alternative model designs along with 95 % confidence interval shown in parentheses for different shots and datasets.
ChestX
ISIC
EuroSAT
CropDiscase
Method
SWSS
SW20S
5W50S
5WSS
5W20S
SWS50S
5W5S
5W205
5W500
5WSS
SW-20S
SWS5S
25.78
31.79
38.41
45.45
56.40
62.32
81.29
89.81
90.34
83.80
92.65
96.19
LI porm
(0.44)
(0.47)
(0.52)
(0.35)
(0.35)
(0.30)
(0.42)
(0.39)
(0.17)
(0.28)
(0.67)
(0.23)
25,82
30.59
36.55
44.04
53,69
61.17
79.57
85.33
90.78
90.26
94.75
Souree Mask
(0.34)
(0:47)
(0.21)
(0.32)
(0.24)
(0.43)
(0.60)
(0.48)
(0.39)
(0.38)
(0.25)
(0:24)
Neg. Reg.
24.05
28:21
29.32
39.57
49.50
51.99
73.29
82.27
85.48
79.72
88.15
90.81
(0.11)
(0.25)
(0.23)
(0.28)
(0.51)
(0.50)
(0.37)
(0.62)
(0.35)
(0.46)
(0.33)
(0.22)
Dir. Neg.
24.23
27.48
30.32
39.17
48,24
50.72
72.69
81 34
83.98
80.32
86.15
24
(0.41)
(0.22)
(0.44)
(0.38)
(0.51)
(0.36)
(0.55)
(0.17)
(0.41)
(0.12)
(0.53)
Laaer Mask Mod.
(0.36)
26.79
(0.28)
34.12
(0.12)
40.04
(0.27)
48.84
(0.23)
60,26
(0.46)
65. 88
(0.34)
84.13
(0.23)
90.59
(0.27)
91.52
(0.37)
88.36
(0.41)
95.72
(0.10)
97.67
Layer Mask Mod.
27.19
34.11
40.34
48.79
60.79
65.95
84.16
90.63
91.70
38.38
95.56
97.61
(0.38)
(0.22)
(0,20)
(0. 18)
(0.29)
(0.11)
(0.32)
(0.26)
(0.28)
(0.15)
(0.44)
(0.33)
Layer Mask Mod.
39.79
60.24
66.39
83.74
91.20
88.
97.64
(0.42)
(0.21)
(0.20)
(0.37)
(0.15)
(0.39)
(0.23)
(0.14)
(0.14)
(0.50)
(0.56)
(0.12)
Joint Training
25.93
34:33
39.57
47.24
59.62
66.12
83.01
90.23
91.50
88.27
95.71
97.60
(0.41)
(0.32)
(0.30)
(0.41)
(0.23)
(0.34)
(0.30)
(0.12)
(0.38)
(0.43)
(0.24)
(0. (8.18)
ConFeSS
27.09
33.57
39.02
48.85 85
60.10
65.3 34
84,65
90.40
92.66
88 88
95,34
97.56
(0.24)
(0.31)
(0.12)
(0.29)
(0.33)
(0.45)
(0.38)
(0.24)
(0.36)
(0.51)
(0.48)
(0.43)
Table 10: Table results along with 95 % confidence interval shown in parentheses.
ChestX
ISIC
EuroSAT
CropDisease
Method
5W5S
5W20S
5W50S
5W5S
2020
5W50S
5W58
5W20S 20S
5W50S
5WSS
5W
5W50S
MatchNet
22.40
23.61
22.12
36.74
45.72
54.58
64.45
77.10
54.44
66.39
76.38
58.53
(0.7)
(0.86)
(0.88)
(0.53)
(0.53)
(0.65)
(0.63)
(0:57)
(0.67)
(0.78)
(0.67)
(0:73)
MalchNet+FWT
21.26
23.01
30.40
32.01
33.17
56.04
63.38
74.90
(0.31)
(0.37)
(0.34)
(0:48)
(0.48)
(0:43)
(0.65)
(0.69)
(0.76)
(0.90)
(0.71)
(0.78)
MAML
23.48
27.53
40.13
52.36
71.70
81.95
78.05
89.75
(0.96)
(0.43)
(0.58)
(0.57)
(0.72)
(0.55)
(0.68)
(0.42)
ProtoNet
24.05
28.21
29.32
39.57
49.50
51.90 11
73.29
82.27
80.48
79.72
88.15
90.81
(1.01)
(1.15)
(1.12)
(0.57)
(0.55)
(0.52)
(0.71)
(0.57)
(0.57)
(0.67)
(0.51)
ProtoNet-FWT
23.77
26.87
30.12
38. 87
43.78
49 84
67. 34
75.74
78.66 64
72.72
85.85
87.17
(0.42)
(0.43)
(0.46)
(0.52)
(0.47)
(0.51)
(0.76)
701
(0.57)
(0.51)
10. 10.50)
RelationNet
22.96
26.63
28.45
39:41
41.77
49.32
61.31
74.43
74,91
68. 99.99
80.45
85.08
(0.88)
(0.92)
(1.20)
(0.58)
(0.49)
(0.51)
(0.72)
(0.66)
(0,58)
(0:75)
(0.64)
(0.53)
RelationNerePWT
22.74
26.75
27.56
43.31
61.16
69.40
64.91
78.43
(0.40)
(0.41)
(0.40)
(0.55)
(0.51)
(0.53)
(0.70)
(0.64)
(0.60)
(0.79)
(0.59)
(0.56)
22.53
25.53
29.35
36.28
49.42
54.80
64.44
79.
83.62
68.41
82.89
91.76
MetaOpt
(0.91)
(1.02)
(0.99)
(0.50)
(0.60)
(0.54)
(0.73)
(0.62)
(0.58)
(0.73)
(0.54)
(0.38)
STARTUP
26.94
33.19
36.91
47. 22
538.3
64.16
82.29
89 26
91,99
93,02
97,51
98.45
(0.94)
(0.46)
(0.50)
(0.61)
(0.58)
(0.58)
(0.60)
(0.43)
(0.36)
(0.45)
(0.21)
(0.
CHEF
24.72
29.71
31.25
41 26
$4,30
60.86
74,15
83
86.55
86.87
94.78
96.77
(0.14)
(0.27)
(0.20)
(0.34)
(0.34)
(0.27)
(0.88)
IT-AI
25.97
31.32
35.49
48.11
59.31
66.48
79.08
87.6 64
90.89
89.25
95.51
97.68
(0.41)
(0.45)
(0.45)
(0.64)
(0.48)
(0.56)
(0.61)
(0.47)
(0.36)
(0.51)
(0.31)
(0.21)
24.43
45.8
83,75
90.59
ATA
(0.2)
(-)
(-)
(0.3)
(-)
(-)
(0.4)
(-)
(-)
(0.3)
(-)
(-)
ConFeSS
27.09
33.57
39.02
48.85
60.10
84.65
90,40
92.66
88.
95,34
97.56
(0.24)
(0.31)
(0.12)
(0.29)
(0.33)
(0.45)
(0.38)
(0.24)
(0.36)
(0.51)
(0.48)
(0.43)
J FEATURE MASKING AND VC THEORY
The generalization ability of machine learning model is related to the Vapnik-Chervonenkic (VC) theory. The VC dimension Shawe-Taylor & Cristianini 2000) measures the capacity or complexity of a machine learning model. For model family, the VC dimension is the maximum number of training points that can be shattered by that family. The VC dimension of a set of separating hyperplanes is d +1 1 where d is the feature space dimensionality, Vapnik proved that with probability 1 - 17, the test loss (Lte) is upper bounded as
7 + 1oN( log(?)
Lae S Ler +
(10)
N
where fer is the training loss, N is the number of training samples, and y is the VC dimension. For better generalization, the goal is to reduce the upper bound, which can be decreased by having more training samples N. However, when N is small in the few-shot setting, the upper bound increases, triggering generalization performance to drop. If we reduce 7, we can decrease the upper bound. For linear classifier, y is upper bounded by the number of features. Hence, if we reduce the number of features, we also reduce the upper bound of y and subsequently the generalization upper bound. This is realized with the masking module M.),, which selects a fraction of features before forwarding them
16

---

Published as a conference paper at ICLR 2022
Table 11: Table results along with 95 % confidence interval shown in parentheses.
5-way 5-shot
5-way 20-shot
Setting
CropDisease
EuroSAT
ISIC
ChestX
CropDisease
EuroSAT
ISIC
ChestX
84.65
48.85
27.09
90.40
33.57
Full Framework
(0.51)
(0.38)
(0.29)
(0.24)
(0.48)
(0.24)
(0.33)
(0.31)
87.26
83.15
47.66
26.06
95.47
88,78
59.96
32.12
w/o Cont. Learn.
(0.32)
(0.21)
(0.18)
(0.42)
(0.53)
(0.17)
(0.28)
(0.28)
85.18
83.14
42.25
25.76
93.52
89.70
52.61
31.12
w/o BB
(0.42)
(0.25)
(0.73)
(0.52)
(0.56)
(0.25)
(0.34)
(0.24)
47.10
95.49
89,93
33,20
w/o Feature Mask
(0.44)
(0.26)
(0.13)
(0.15)
(0.26)
(0.20)
(0.24)
(0.18)
87.95
84.23
48.62
26.92
94.74
89.31
59.20
32.77
Liiee
(0.24)
(0.21)
(0.22)
(0,50)
(0.34)
(0.25)
(0.25)
(0.09)
89.03
84.41
48.04
26.60
94.72
90.37
60.14
32.81
w/o Lung
(0.12)
(0.36)
(0.18)
(0.30)
(0.17)
(0.38)
(0.18)
(0.34)
87.83
48.34
26.73
90.31
32.69
w/o Leeg
(0.30)
(0.26)
(0.32)
(0.24)
(0.22)
(0.22)
(0.08)
(0.12)
87.15
83.94
47.25
26.58
93.65
89.40
59.66
31,92
Direct Positive
(0.16)
(0.15)
(0.24)
(0.14)
(0.21)
(0.11)
(0.16)
(0.20)
the linear classifier. Thus, feature selection has theoretical support for improving generalization performance in the few-shot setting. Also, empirical results in Fig. 22..22c (a) show different datasets and
shots selecting different number of features and hence realizing different upper bounds of
K
THEORY OF CONTRASTIVE LEARNING AND FEW-SHOT LEARNING
In (Cao et 202T), the authors proved the following bound:
Louu S To + (sx()
(11)
Here, Lxup is the supervised evaluation metric for learned representations. Ly the unsupervised contrastive evaluation metric for true negative samples. s(fa. is the intra-class deviation using the key encoder fk- To and 71 are coefficients depending on class distributions. Leup can be the training loss of any supervised few-shot meta-learning method which generalize to novel categories. Since fūj upper bounds Lyup. decreasing Ly amounts to decreasing L Also, LT can be decreased arbitrarily because it evaluated only on true negative samples. Hence, contrastive losses can be useful for learning representations that are effective for few-shot learning.
L THEORY OF DISTANCE DISTANCE-BASED REGULARIZATION FOR FINE-TUNINE
In (Gouk et al. 2021 the authors proved that with probability - the test loss (Lte) is upper bounded
DF
log(2/n)
Lie S for +
II 2 Vn 33 j=1
(12)
j=1 2B IP.
2m
Here, Lar is the training loss. K. is coefficient depending the properties of dataset. is the number of training samples. B,5 is the upper bound of the Frobenius norm of weight parameter of layer of both pre-trained and fine-tuned model. Df is the upper bound of the Frobenius norm of the difference between weight parameter of layer j of pre-trained and fine-tuned model. n, is the number of columns in weight parameter of layer j. According to the bound, the generalization gap between C,y and Lie decreases if Df 's of all the layers can be decreased. However, minimizing the weights between pre-trained and fine-tunee model for all layers might be cumbersome. Hence, choose to minimize the Frobenius norm of difference in features for our regularization term Lyeg.
M LIMITATIONS OF OUR FRAMEWORK
Although our framework produces competitive performance on the CDFSL benchmark, it has the following limitations: (a) In online setting, when target domain samples arrive in streaming
17

---

Published as a conference paper at ICLR 2022
fashion, our method might not be applicable. This is mainly because of the presence of the mask generator. Even though the mask generator is a small network, it still requires small batch of samples for learning the parameters. In the online setting, samples arrive one at time, and the small masking network might overfit. A workaround to prevent overfitring can be selectively updating only certain parameters during online learning.
(b) Another limitation of framework is the use of large number of hyperparameters in the adaptation step for weighing the loss functions i.c. Apos Aneg, Adio and Xveg- For practical few-shot adaptation, it is difficult number of validation samples to tune the optimal hyperparameter configuration. Hence, we just fixed the hyperparameter values in our experiments. Another possible workaround involves learning hyperparameters themselves within the framework of multi-task uncertainty (Kendall et al. 2018).
aside sufficient
18


Reviews:
[{"review_id": "tHfwJoFGnPr", "reviewer": "Reviewer_MRXL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. The paper is well-written and the easy to read. The organization is also clear.\n2. The idea is interesting and makes sense. The authors have clear motivation on each step of the proposed framework. Pretrain the network unsupervisedly is natural in the case of cross-domain few-shot learning due to the domain difference between the source domain and target domain. The feature selection module is also designed with a clear motivation for cross-domain few-shot learning. The decoupling of positive features and negative features is shown to be helpful. \n3. The authors also conduct extensive analysis of the proposed method. \n\n\nWeaknesses:\n\n1. The proposed framework has a lot of hyperparameters to tune which is an issue for few-shot learning since there is no validation set to use.\n\n2. No standard deviations are reported in the tables which is important for the evaluation of the few-shot learning due to the randomness.\n\n\nMore questions:\n\n1.  How about using negative features for fine-tuning? This can be a good ablation to show the usefulness of positive features.\n\n2. Another ablation is to pre-train supervisedly to show the advantages of self-supervised pretraining. \n\n3. How the mask module is trained? And would a better mask module produces better final performance?\n", "summary_of_the_paper": "In this paper, the authors propose a generic framework for cross-domain few-shot learning. There are three steps: 1. Pre-training the backbone unsupervisedly using a self-supervised contrastive loss. 2. Select relevant features via a mask module. 3. Fine-tune the network with the selected features. The proposed framework is evaluated in the recently proposed CDFSL benchmark. The results show that the proposed framework outperforms the baselines in most of the cases. ", "summary_of_the_review": "In this paper, the authors propose a framework for cross-domain few-shot learning. The idea is reasonable and the authors also provide a good analysis of the components of the framework. The proposed framework outperforms the baselines in most of the cases. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "fyl7sNm1Rm", "reviewer": "Reviewer_nRRt", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The challenge is to surpass those approaches which deal on FSL with meta learning. My main claim is: why do not compare with standard approaches from Multi-task learning, Embedding learning, Learning with External Memory? FSL by Meta Learning is just one way, not necessarily the best one. In the experimental results, some approaches have been considered. They should be mapped with respect to a wider taxonomy of FSL. To help the authors, in the introduction (\"Few-shot learning methods aim to uncover…\"), I suggest to add general references about FSL, such as WANG, Yaqing, et al. Generalizing from a few examples: A survey on few-shot learning. ACM Computing Surveys (CSUR), 2020, 53.3: 1-34. This to justify the meta train-meta test cycle, and to make clear that FSL is not just meta learning. \n\nWhy do we need to stick with a single source domain, when there are plenty of universal representation to start the FSL, as the same authors discuss, with (Dvornik et al., 2020)?\n\nWhy do we need to apply unsupervised training on the single source domain, instead of a supervised one? Single source domain should be known, hence labeled training data should be available to train with supervision. Perhaps this has to create a general vocabulary of features, due to the distance between domain and target, but this should be better explained. \n\nAre the authors sure that the framework can be cast as meta learning? In general, in meta learning there should be an outer loop and an inner loop, where the outer loop modifies the parameters for the inner one. Instead of this nested general algorithm, the present one is more a serial 3-step architecture, where each step is done just one time. I agree that each step has a double loop inside, in fact this is just a question for clarification. \n\nWhat is strange to me is the selection of so many hyper parameters, 8. I'm ok with the Hyper-parameter sensitivity ablation study, which is useful for the lambdas, but how do they have been found, especially the number of training epochs for the three steps, which appear so different (400, 15, 50)? \n\nIn the appendix H, a general reference to introduce the VC dimension should be given (such as the N. Cristianini and J. Shawe-Taylor one) \n", "summary_of_the_paper": "The problem is to learn by few shots (FSL) when the final domain is highly distant from the source base (i.e. natural, medical and satellite), dubbed single source cross-domain few-shot learning. In short, the approach consists of three steps: 1) train a feature extracting backbone with the contrastive loss on a (single source) base category; 2) train a masking module to select relevant features for the target domain; 3) fine-tuned along with the backbone to give features similar to the relevant ones.  The last two are claimed to be novel and, together with the experiments on CDFSL benchmark, sold as the main contributions.", "summary_of_the_review": "The paper is ok to me, even if it not so general, since it cope with a very specific configuration (single-source domain, FSL with meta learning) which could be just one case of a more general situation, which may lead to better results (multi source domain, or domain which are kind of closer to the final ones). Experiments seem convincing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RH5ZPeNj99_", "reviewer": "Reviewer_yTZ7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Paper strengths:\n- The cross-domain few-shot learning problem studied in this paper is a fundamental task that deserves further study.\n- The paper is well written and easy to follow.\n- The proposed method is straightforward.\n- Experiments are sufficient, and ablation studies can bring good observations and analyses for the method.\n\nPaper weaknesses:\n- The proposed method consists of three stages for training, which seems complicated and a little bit tricky in implementations. Could it be simplified in the training process?\n- In experiments, most results of the proposed method are satisfactory. However, on CropDisease, it is significantly worse than the results of STARTUP. Are there any reasons for the observations?\n", "summary_of_the_paper": "This paper proposes a framework, terms as ConFeSS, for dealing with cross-domain few-shot learning problems. Specifically, it firstly trains a feature extracting backbone with the contrastive loss on the base category data for learning better features. Then, it trains a masking module to select relevant features suited to target domain classification. Finally, a classifier is fine-tuned along with the backbone such that the backbone produces features similar to the relevant ones. Experiments are conducted on several cross-domain few-shot learning benchmarks.", "summary_of_the_review": "The authors are encouraged to respond to the comments listed in the paper weaknesses during rebuttals.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0fOk7rrmJxG", "reviewer": "Reviewer_H3UB", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n1. The positive and negative features separation is novel, which serves as a regularization for fine-tuning on the target domain. \n2. The algorithm 1 is clean and easy to understand. \n3. The performance on CDFSL outperforms several SOTA methods. \n\n## Weakness\n1. The feature masking module has too many losses making it difficult to reproduce. Comparing to fine-tuning full network, the improvements are small. \n2. The ablation study presented in Sec 4.4 is either inefficient or unclear. For example, can we replace eq. (8) with || f_i^t - f_i^+ ||^2? \n3. Clarity can be improved: \na) The labels on D_s are discarded? \nb) How do you prevent trivial masks? I.e., all close to 1's or all close to 0's? \nc) The intuition on the losses in eq.(6) and (8) are not very straightforward. It would be better to start from the simplest case without any of these losses and then add them one by one. How did you remove the mask module in Table 2? \nd) Why do you train the mask generator on D_t rather than D_s? \ne) How do you tune hyper-params if no validation set on target domain doesn't exist.  ", "summary_of_the_paper": "This paper proposes a framework for cross-domain few-shot learning. The framework consists of three steps: 1) pre-training backbone; 2) meta-learn the feature masking network; 3) fine-tuning on target domain. Experiments on the CDFSL dataset show this framework outperforms SOTA methods. ", "summary_of_the_review": "The method is technically sound, and the paper is easy to follow. However, there are still many details are either hidden or unclear, which hinders its reproducibility. Besides, the method is only evaluated on CDFSL, which is a relatively small dataset. It would be more convincing if the method is applied and compared on Meta-Dataset (another larger cross-domain dataset). Hence, this is a borderline paper to me. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Summary: 
Paper addresses the cross-domain few-shot learning scenario, where meta-learning data is unavailable, and approaches are evaluated directly on novel settings. Authors propose a 3-step approach: 1) self-supervised pretraining, 2) feature selection, 3) fine-tuning, and demonstrate gains over state-of-art. 

Pros:
- Approach is novel for this setting
- Paper is clear and easy to understand
- Performance beats several prior methods
- Experiments are thorough
- Fundamental problem is worthwhile of investigation 

Cons:
- Some concerns among multiple reviewers on how hyperparameters are selected. Authors have provided more information and tables in the paper.
- Training process is multi-step and not unified. Authors provided additional information about unified training results, which yielded poorer results, likely due to overfitting from training many parameters at once. 

Overall recommendation based on the consensus of reviewers and AC expertise: accept.
Decision: Accept (Poster)