Self-training For Few-shot Transfer Across Extreme Task Differences
Authors: Cheng Perng Phoo
Bharath Hariharan
Venue: ICLR 2021 Oral (2021)
Keywords: few-shot learning
self-training
cross-domain few-shot learning

Abstract: Most few-shot learning techniques are pre-trained on a large, labeled “base dataset”. In problem domains where such large labeled datasets are not available for pre-training (e.g., X-ray, satellite images), one must resort to pre-training in a different “source” problem domain (e.g., ImageNet), which can be very different from the desired target task. Traditional few-shot and transfer learning techniques fail in the presence of such extreme differences between the source and target tasks. In this paper, we present a simple and effective solution to tackle this extreme domain gap: self-training a source domain representation on unlabeled data from the target domain. We show that this improves one-shot performance on the target domain by 2.9 points on average on the challenging BSCD-FSL benchmark consisting of datasets from multiple domains.

Published as a conference paper at ICLR 2021
SELF-TRAINING FOR FEW-SHOT TRANSFER ACROSS EXTREME TASK DIFFERENCES
Cheng Perng Phoo, Bharath Hariharan
Department of Computer Science
Coreell University {cpphoo, bharathh]@cs... cot.eeallees.cccoll........
ABSTRACT
Most few-shot learning techniques are pre-trained on large, labeled "base dataset". In problem domains where such large labeled datasets are not avail- able for pre-training (e.g.. X-ray, satellite images), one must resort to pre-training in different "source" problem domain (e.g., ImageNet), which can be very differ- ent from the desired target task. Traditional few-shot and transfer learning tech- niques fail in the presence of such extreme differences between the source and target tasks. In this paper, we present a simple and effective solution to tackle this extreme domain gap: self-training source domain representation data from the target domain. We show that this improves one-shot performance on the target domain by 2.9 points on average on the challenging BSCD-FSL benchmark consisting of datasets from multiple domains. Our code is available at https:: wtw.g.tthhuug.gggggggggg com wtthho/ STARTUP
unlabeled
INTRODUCTION
Despite progress in visual recognition, training recognition systems for new classes in novel domains requires thousands of labeled training images per class. For example, train recognition system for identifying crop types in satellite images, one would have to hire someone to go to the different locations on earth get the labels of thousands of satellite images. The high cost of collecting annotations precludes many downstream applications.
This issue has motivated research on few-shot learners: systems that can rapidly learn novel classes from few examples. However, most few-shot learners are trained on a large base dataset of classes from the same domain. This problem many domains (such as medical imagery, satellite images), where no large labeled dataset of base classes exists. The only alternative is train the few- shot learner on different domain (a common choice is to use ImageNet). Unfortunately, few-shot learning techniques often assume that novel and base classes share modes of variation (Wang al., 2018). class-distinctive fcatures (Snell et al., 2017). or other inductive biases. These assumptions broken when the difference between base and novel is extreme the difference between object classification in internet photos and pneumonia detection in X-ray images. As such, recent work has found that all few-shot learners fail the face of such extreme task/domain differences, underperforming even naive transfer learning from ImageNet (Guo et al. 2020).
Another alternative comes to light when one considers that many of these problem domains have unlabeled data (e.g., undiagnosed X-ray images, or unlabeled satellite images). This suggests the possibility of using self-suuee techniques on this unlabeled data to produce a good feature representation, which can then be used to train linear classifiers for the target classification task using just few labeled examples. Indeed, recent work has explored self-supervised learning on a variety of domains (Wallace & Hariharan] 2020). However, self-supervised learning starts tabula rasa, and as such requires extremely large amounts of unlabeled data (on the order of millions of images). With more practical unlabeled datasets, self-supervised techniques still struggle to outcompete naive ImageNet transfer (Wallace & Hariharan 2020). We are thus faced with a conundrum: on the one hand, few-shot learning techniques fail to bridge the extreme differences between ImageNet and domains such as X-rays. On the other hand, self-supervised techniques fail when they ignore inductive biases from ImageNet. A sweet spot in the middle, if it exists, is elusive.

---

Published as a conference paper at ICLR 2021
bird
dog
road
road
Labeled base
Unlabeled target domain data
river Fow chatshat lask
river
datasct
Query
(source)
target domain
Adaplation Classifier training
Pre-training
Testing
Representation
Model
Representation Learning
Evaluation
Figure 1:
Problem setup. the representation learning phase (left), the learner has access to a large labeled "base dataset" in the source domain, and some unlabeled data in the target domain, which to pre-train its representation. The learner must then rapidly learn/adapt to few-shot tasks in the target domain in the evaluation phase (right).
In this paper, solve this conundrum by presenting strategy that adapts feature representations trained on source tasks to extremely different target domains, so that target task classifiers can then be trained the adapted representation with very little labeled data. Our key insight is that pre-trained base classifier from the source domain, when applied to the target domain, induces a grouping of images on the target domain. This grouping captures what the pre-trained classifier thinks are similar or dissimilar in the target domain. Even though the classes of the pre-trained classifier are themselves irrelevant in the target domain, the induced notions of similarity and dis- similarity might still be relevant and informative. This induced notion of similarity is in contrast current self-supervised techniques which often function by considering each image as its own class and dissimilar from every other image in the dataset (Wu et al. 2018, Chen et al. 2020). We propose to train feature representations ing. This approach produces a feature representation that is (a) adapted to the target domain, while (b) maintaining prior knowledge from the source task to the extent that reader might observe the similarity of this approach to self-trainn teff-training, except that our goal is to adapt the feature representation to the target domain, rather than improve the base classifier itself.
the novel target domain to replicate this induced group-
relevant. A discerning
We call our approach "Self Training Adapt Representations To Unseen Problems". ", STARTUP. In a recently released BSCD-FSL. benchmark consisting of datasets from extremely different do- mains (Guo al. 2020), show that STARTUP provides significant gains (up 2.9 points average) over few-shot learning, transfer learning and self-supervision state-of-the-art To the best of our knowledge, ours is the first attempt bridge such large task/domain gaps and successfully and consistently outperform naive transfer in cross-domain few-shot learning.
2 PROBLEM SETUP
Our goal is to build learners for novel domains that can be quickly trained to recognize new classes when presented with very few labeled data points ("few-shot"). Formally, the target domain is defined by set of data points (c.g. images) Xv, unknown set of classes (or label space) VAv, and a distribution Dy over Xxy x V/ A "few-shot learning task" in this domain will consist of a set of classes Y yyv very small training set ("support")
S [Zx, y.
Div WEY
and small test set ("query")
Q = {m m
1= D
When presented with such a few-shot learning task, the learner must rapidly learn the classes pre- sented and accurately classify the query images.
2

---

Published as a conference paper at ICLR 2021
As with prior few-shot learning work, we will assume that before being presented with few-shot learning tasks in the target domain, the learner has access to a large annotated dataset Dg known as the base dataset. However, crucially unlike prior work few-shot learning, we assume that this base dataset is drawn from a very different distribution. In fact, we assume that the base dataset is drawn from completely disjoint image space X's and disjoint set of classes YB:
DB= = [1r, w) C AB x V/s
where Ag is the set of data (or the source domain) and V/ is the set of base classes. Because the base dataset is so different from the target domain, we introduce another difference vis-a-vis the conventional few-shot learning setup: the learner is given access to an additional unlabeled dataset from the target domain:
Du = {wi)i ~ DN
Put together, the learner will undergo two phases. In the representation learning phase, the learner will pre-train its representation on Dg and Du; then it goes into the evaluation phase where it will be presented few-shot tasks from the target domain where it learns the novel classes (Figure TT).
3 RELATED WORK
Few-shot Learning (FSL). This paper explores few-shot transfer, and such the closest related work is on few-shot learning. Few-shot learning techniques are typically predicated on some degree of similarity between classes in the base dataset and novel classes. For example, they may assume that features that are discriminative for the base classes are also discriminative for the novel classes, suggesting a metric learning- based approach (Gidaris & Komodakis 2018 Qi et al. 2018 Snell
jet al., 2017; Vinyals et al. 2016; [Sung et al., 2018 Hou et al.] (019) or transfer learning-baad based approach (Chen et al.) 2009b]] Wang al. 2019 Kolesnikov al. 2020 Tian et al. (020). Altema-
tively, they may assume that model initiali iitiiinns that lead to rapid convergence on the base classes
are also good initializations for the novel classes (Finn al. 2017 2018: Ravi & LLrrchelle| [2017] 22017 Nichol & Schulman, Rusu et al. 2019; Sun et | Lee et al. (2019). Other methods assume that modes of intra-class variation are shared, suggesting the possibility of learned, class-agnostic
augmentation policies (Hariharan & Girshick 2017;; Wang et al.] [0188) Chen et al., (019c)). Some- what related is the use of a class- agnostic parametric model that can "denoise" few-shot models, be they from the base or novel classes (Gidaris & Komodakis 2018 2019). In contrast to such strong assumptions of similarity between base and novel classes, this paper tackles few-shot learning prob- lems where base and novel classes come from very different domains, also called cross-domain few-shot learning.
Cross-domain Few-shot Classification (CD-FSL), When the domain gap between the base and novel dataset is large, recent work (Guo al. 2020 2020 Chen et al. 2019b) has shown that existing state- of-the-art art few-shot learners fail generalize. Tseng al. (2020) attempt to address this problem by simulating cross-domain transfer during training. However, their approach assumes access to an equally diverse array of domains during training, and much smaller domain gap at test time: for example. both base and novel datasets are from internet images. Another relevant work (Ngiam et al.) 2018) seeks to build domain-specifie feature extractor by reweighting different classes of examples in the base dataset based on the target novel dataset but their work only investigates transfer between similar domains (both source and target are internet images). Our paper tackles more extreme domain gap. Another relevant benchmark for this problem is (Zhai et al.) (019) but they assume access to more annotated examples (Ik annotations) during test time than the usual FSL setup.
Few-shot learning with unlabeled data. This paper uses unlabeled data from the target domain to bridge the domain gap. Semi-supervised few-shot learning (SS-FSI.) (Ren et al., 2018 Li et al.) 2019 Yu et al.] 2020 Rodríguez al.] 2020, Wang et al. (2000) and transductive few-shot learning (T-FSL) (Liu et al.] 2019; Dhillon et al. 2200 Hou et al. 2019 Wang et al. 2020 Rodríguez et al, 2020) do use such unlabeled data, but only during evaluation, assuming that representations trained on the base dataset are good enough. In contrast our approach leverages the unlabeled data during representation learning. The two are orthogonal innovations and can be combined.
Self- Self-Trainigg Our approach is closely related to self telf-training, which has been shown to be effective for supersupe training and knowledge distillation. In self-training, a teacher model trained on
3

---

Published as a conference paper at ICLR 2021
the labeled data is used to label the unlabeled data and another student model is trained on both the original labeled data and the unlabeled data labeled by the teacher. Xie et al. (2020) and Yalniz et al.] (2019) have shown that using self- self-training can improve ImageNet classification performance. Knowledge distillation (Hinton et al. 2015) is similar but aims to compress a large teacher network by training student network to mimic the prediction of the teacher network. A key difference between these and our work is that self-training / knowledge distillation focus on a single task of interest, i.e, there is no change in label space. Our approach is similar, but we are interested in transferring to novel domains with wholly different label space: an unexplored scenario.
Domain Adaptation. Transfer new domains is also in the purview of domain adaptation (Tzeng et al.. 2017; Hoffman et al., 2018]] Long et al.) 2018: Xu et al.) [019] Laradji & Babanezhad 2020 Wang & Deng, 2018: Wilson Cook 2020) where the goal is to transfer knowledge from the label- abundant source domain to target domain where only unlabeled data is available. In this realm,
self-training has been extensively explored (Zou et al., 2018 Chen et al., 2019a Zou et al.) 2019 Zhang et al., 2019: Mei et al. 2020). However, key assumption in domain adaptation is that the source domain and target domain share the same label space which does not hold for FSL.
Self-supervised Learning. Learning from unlabeled data has seen resurgence of interest with advances in self-supervised learning. Early self-supervised approaches were based on handcrafted "pretext tasks" such as solving jigsaw puzzles (Noroozi & Favaro 2016), colorization (Zhang al.
2016) or predicting rotation (Gidaris et al., 2018). A more recent (and better performing) line of self-supervised learning is contrastive learning (Wu et al. 2018,) Misra & Maaten 2020 He al.) 2020 Chen et al.) 2020) which aims to learn representations by considering each image together with augmentations
separate class. While self supervision has been shown boost few-shot learning (Gidaris et al. 2019 Su et al. 22020), its utility in cases of large domain gaps between base and novel datasets have not been evaluated. Our work focuses on this challenging scenario.
4 APPROACH
Consider a classification model fo = C' o 6 where & embeds input x into Rd and is a (typically linear) classifier head that maps (xx) to predicted probabilities P(y]æz). 0 is vector of parameters. During representation learning, STARTUP performs the following three steps:
1. Learn a teacher model &g on the base dataset DB by minimizing the cross entropy loss
2. Use the teacher model to construct a lobtly- set D. = { Hi where
yi for (z:) Vx, E Du.
Note that Di is a probability distribution as described above.
3. Learn a new student model 6* on DB and D by optimizing:
min è
Σ ICR(Jo(Zs).n)) + N (FaD))DDg
Σ (H5,0))ERH
IKI(Jo(z)), 03) + luntabnted(D,,
NB
(2)
where IcE is the cross entropy loss, (KL is the KL divergence and luntabeled is any unsupervised/self-spprrisss loss function (See below).
The third term, lantabeled, is intended to help the learner extract additional useful knowledge specific to the target domain. We use a sfat--of-the-ar sulf-super loss function based on contrastive learning: SimCLR (Chen et al., 2020). The SimCIR loss encourages two augmentations of the same image to be closer in feature space to each other than to other images in the batch. We refer the reader to the paper for the detailed loss formulation.
The first two terms are similar to those in prior self-training literature (Xie al. 2020). However, while in prior self- teff-training work, the second term (LLL) is thought to mainly introduce noise during training, we posit that [g has more substantial role play here: it encourages the model to learn feature representations that emphasize the groupings induced by the pseudo-labels ÿJ on the target domain. We analyze this intuition in section 5.2.2 a.22ion5.2.22
4

---

Published as a conference paper at ICLR 2021
4.1 EVALUATION
STARTUP is agnostic to inference methods during evaluation; any inference methods that rely on representation (Snell et 2017 Gidaris & Komodakis 2018) be used with STARTUP. For simplicity and based on results reported by Guo et al.J (2020), we freeze the representation & after performing STARTUP and train linear classifier on the support set and evaluate the classifier on the query set.
4.2
INITIALIZATION STRATEGIES
Xie et al. 2020 found that training the student from scratch sometimes yiclds better results for Im- ageNet classification. To investigate, we focused variant of STARTUP where the SimCLR loss omitted and experimented with three different initialization strategies from scratch (STARTUP- Rand (no SS)), from teacher (STARTUP-T (no SS)) and using the teacher's embedding with randomly initialized classifier (STARTUP (no SS)). We found no conclusive evidence that one single initializa- tion strategy superior the others across different datasets (See Appendix (A.4) but we observe that (STARTUP (no SS)) is either the best or the second best in all scenarios. As such, we opt to use teacher's 's embedding with a randomly initialized classifier as the default student initialization.
EXPERIMENTS
We defer the implementation details to Appendix A.1
5.1 FEW-SHOT TRANSFER ACROSS DRASTICALLY DIFFERENT DOMAINS
Benchmark. We experiment with the challenging (BSCD-FSL) benchmark introduced in Guo al. (020). The base datasel in this benchmark is minilmageNet (Vinyals et al. 2016). which is an object recognition task on internet images. There are novel datasets in the benchmark, none of which involve objects, and all of which come from very different domain than internet images: CropDiseases (recognizing plant diseases in leaf images), EuroSAT (predicting land-use from satel- lite images), ISIC2018 8 (identifying melanoma from images of skin lesions) and ChestX (diagnosing chest X-rays). Guo et found that state-of of the-hhead few-shot learners fail on this benchmark.
To construct our setup, we randomly sample 20% of data from each novel datasets form the respective unlabeled datasets D.,. We use the rest for sampling tasks for evaluation. Following Guo et al. (2020), we evaluate 5-way k-shot classification tasks (the support set consists of 5 classes and k examples per class) for E {1, 5} and report the mean and 95% confidence interval over 600 few-shot tasks (conclusions generalize to k [20, 50}. See Appendix A.2).
Baselines. We compare to the techniques reported in Guo et al. (022)), which includes most state- of-the-art approaches well as cross-domain few-shot technique Tseng al. (2020). The top performing among these is naive Transfer which simply trains a convolutional network to classify the base dataset, and the resulting representation learn linear classifier when faced with novel few-shot tasks. These techniques do not use the novel domain unlabeled data.
We also compare to another baseline, Sim that uses the novel domain unlabeled data Da train a representation using SimCLR(Chen et al. 2020). and then uses the resulting representation to learn linear classifiers for few-shot tasks. This builds upon state-of-the-an self-supervised techniques.
To compare to a baseline that uses both sources of data, we establish Transfer + SimCLR. This baseline is similar the SimCLR baseline except the embedding is initialized to Transfer's embed- ding before SimCLR training.
Following the benchmark, all methods use ResNet-10 (He al. 2016) unless otherwise stated.
5.1.1 RESULTS
We present main results minilmageNet + BSCD-FSL in Table
STARTUP vs Few-shot learning techniques. STARTUP performs significantly better than all few- shot techniques in most datasets (except ChestX, where all methods are similar). Compared to
5

---

Published as a conference paper at ICLR 2021
Table 1: 5-way k-shot classification accuracy on mininnageNet-BSCD-SSS.. Mean and 95% con-
fidence interval are reported. (no SS) indicates removal of SimCLR. ProtoNet: (Snell et al., 2017). MAML:(Finn et al. 2017). MetaOpt:(Lee et 2019) ( et al. 2020). *Numbers reported in (Guo al. 2020);; our re-implementation of Transfer uses different batch size and 80% of the original test set for evaluation. Bolded entries are top performing methods that are not different
based on t-test at significant level 0.05.
Methods
ChestX
ISIC
k=1
k=5
k=1
k=5
MAML"
23.48 0.96
40.13 0.58
ProtoNet
24.05 + 1.01
39.57 + 0.57
ProtoNet + FWT
23.77 + 0.42
38.87 + 0.52
MetaOpt*
22.53 + 0.91
36.28 + 0.50
Transfer*
25.35 0.96
43.56 0.60
Transfer
22.71 + 0.40
26.71 + 0.46
30.71 0.59
43.08 + 0.57
SimCLR
22.10 + 0.41
25.02 + 0.42
26.25 0.53
36.09 = 0.57
Transfer + SimCLR
22.70 + 0.40
26.95 + 0.45
32.63 + 0.63
45.96 + 0.61
STARTUP (no SS)
22.87 + 0.41
26.68 + 0.45
32.24 0.62
46.48 0.61
STARTUP
23.09 + 0.43
26.94 + 0.44
32.66 0.60
47.22 + 0.61
Methods
EuroSAT
CropDisease
k=1
k=5
k=1
k=5
MAML
71.70 0.72
78.05 0.68
ProtoNet*
73.29 + 0.71
79.72 0.67
ProtoNet + FWT
67.34 + 0.76
72.72 + 0.70
MetaOpt
64.44 + 0.73
68.41 + 0.73
Transfer*
75.69 0.66
87.48 0.58
Transfer
60.73 + 0.86
80.30 + 0.64
69.97 + 0.85
90.16 + 0.49
SimCLR
43.52 + 0.88
59.05 + 0.70
78.23 0.83
92.57 0.48
Transfer + SimCLR
57.18 + 0.87
77.61 + 0.66
76.90 + 0.78
92.64 0.44
STARTUP (no SS)
62.90 + 0.83
81.81 0.61
73.30 0.82
91.69 0.47
STARTUP
63.88 + 0.84
82.29 + 0.60
75.93 + 0.80
93.02 + 0.45
previous state-of-the-art, Transfer, observe average of 2.9 points improvement on the 1-shot case. The improvement is particularly large on CropDisease, where STARTUP provides almost a point increase for 1-shot classification. This improvement significant given the simplicity of our approach, and given that all meta-learning techniques underperform this baseline.
STARTUP SimCLR. The SimCLR baseline in general tends to underperform naive transfer from minilmageNet, and consequently, STARTUP performs significantly better than SimCLR on ISIC and ATTOSAT. The exception to this is CropDisease, where SimCLR produces a surprisingly good representation. We conjecture that the base embedding is not good starting point for this dataset. However, we find that using SimCLR as an auxilliary loss to train the student (STARTUP vs STARTUP (no SS)) is beneficial.
STARTUP vs Transfer + SimCLR. STARTUP outperforms Transfer + SimCLR in most cases (except 5-shot in ChestX and 1-shot in ISIC). We stress that the strength of STARTUP is not solely from SimCLR but rather from both self-training and SIMCLR. This is especially evident in EuroSAT since the STARTUP (no SS) variant outperforms Transfer and Transfer + SimCLR.
Larger and stronger teachers. To unpack the impact of teacher quality, we experiment with a larger network and transfer from the full ILSVRC 2012 dataset (Deng et al. 2009) BSCD-FSI..

---

Published as a conference paper at ICLR 2021
Table 2:
5-way k-shot classification accuracy on ImageNe((LLSRR -BCD)-BS Bolded entries are top performing methods that are not different based on t-test at significant level 0.05.
Methods
ChestX
ISIC
k=l
k=5
k=1
k=5
Transfer
21.97 + 0.39
25.85 + 0.41
30.27 0.51
43.88 0.56
STARTUP (no SS)
22.90 = 0.40
26.74 = 0.46
30.18 = 0.56
44.19 + 0.57
STARTUP
23.03 + 0.42
27.24 + 0.46
31.69 0.59
46.02 0.59
Methods
EuroSAT
CropDisease
k=1
k=5
k=1
k=5
Transfer
66.08 +: 0.81
85.58 + 0.48
74.17 + 0.82
92.46 + 0.42
STARTUP (no SS)
70.08 + 0.80
87.12 + 0.45
80.13 + 0.77
94.51 + 0.38
STARTUP
73.83 + 0.77
89.70 + 0.41
85.10 + 0.74
96.06 + 0.33
In particular, used the publicly available pre-trained ResNet-18 18 (He et al. 2016) teacher and train a student via STARTUP. We compare this to a transfer baseline that uses the same network and ImageNet the training set. The result be found in table Surprisingly, larger, richer embeddings do not always transfer beller, in contrast to in-domain results reported by Hariharan & Girshick (017). However, STARTUP still useful improving performance: the absolute improve- ment in performance for STARTUP compared to Transfer remains about the same in most datasets except EuroSAT and CropDisease where larger improvements are observed.
5.2 WHIY SHOULD STARTUP WORK?
While it clear that STARTUP helps improve few shot transfer across extreme domain differences, not clear why or how it achieves this improvement. Below, we look at few possible hypotheses.
5.2.1 HYPOTHESIS 1: STARTUP ADDS NOISE WHICH INCREASES ROBUSTNESS.
Xic et al. (2020) posit that self-training introduces noise when training the student and thus yielding more robust student. More robust students may be learning more generalizable representations, and this may be allowing STARTUP to bridge the domain gap. Under this hypothesis, the function of the unlabeled data is only to add noise during training. This in turn suggests that STARTUP should yield improvements on the target tasks even if trained on unlabeled data from a different domain. To test this, train STARTUP ResNet-18 student on EuroSAT and ImageNet and evaluate on CropDisease. This model yields a 5-way 1-shot performance of 70.40 + 0.86 (88.78 + 0.54 for 5-shot), significantly underperforming the naive Transfer bascline (Table See Appendix A.7 for different combinations of unlabeled dataset and target dataset). This suggests that while the hypothesis valid in conventional semi-supervised learning, it is incorrect in the cross-domain few-shot learning setup: unlabeled data are not merely functioning as noise. Rather, STARTUP learning inherent structure in the target domain useful for downstream classification. The question now becomes what inherent structure STARTUP is learning, which leads us to the next hypothesis.
5.2.2 HYPOTHESIS 2: STARTUP ENHANCES TEACHER-INDUCH GROUPINGS
The teacher produces a meaningful grouping of the data from the target domain. The predic- tions made by the teacher essentially induce grouping on the target domain. Even though the base label space and novel label space are disjoint, the groupings produced by the teacher might not be entirely irrelevant for the downstream classification task. To test this, we first assign each example in the novel datasets to its most probable prediction by the teacher (ResNet 18 trained on ImageNet). We then compute the adjusted mutual information (AMI) (Vinh et al. 2010) between the result- ing grouping and ground truth label. AMI ranges from for unrelated groupings to 1 for identical groupings. From Table 3 we see that AT and CropDisease, there is quite bit of agree-

---

Published as a conference paper at ICLR 2021
Table Adjusted Mutual Information (AMI) of the grouping induced by the teacher and the ground truth label. AMI has value from 0 to 1 with higher value indicating more agreement.
ChestX
ISIC
EuroSAT
CropDisease
AMI
0.0075
0.0427
0.3079
0.2969
EuroSAT
CropDisease
Before
After
Before
After
Figure 2: t-SNE plot of EuroSAT and CropDisease prior and after STARTUP.
ment between the induced grouping and ground truth label. Interestingly, these are the two datasets where we observe the best transfer performance and most improvement from STARTUP (Table 2). suggesting correlations between the agreement and the downstream classification task performance.
STARTUP enhances the grouping induced by the teacher. Even though the induced groupings by the teacher can be meaningful, one could argue that those groupings are captured in the teacher model already, and no further action update the representation is necessary. However, we posit that STARTUP encourages the feature representations to emphasize the grouping. To verify, we plot the t-SNE (Maaten & Hinton 2008) of the data prior STARTUP and after STARTUP for the two datasets in figure 2 From the t-SNE plot, we observe more separation after doing STARTUP, signifying a representation with stronger discriminability.
Put together, this suggests that STARTUP works by (a) inducing a potentially meaningful group- ing on the target domain data, and (b) training a representation that emphasizes this grouping.
5.3 FEW-SHOT TRANSFER ACROSS SIMILAR DOMAINS
Is STARTUP still useful when the gap between the base and target is smaller? To answer this, tested STARTUP on two popular within-domain few-shot learning benchmark: minilmageNet [Vinyals et al. 2016) and tieredImageNet (Ren et al., 20118). For minilmageNet, we use 20% of the novel set as the unlabeled dataset and use the same teacher as in section 5.1 For tieredImageNet, use ResNet-12 12 Oreshkin et al. 2018) as our model architecture and evaluate two different setups tieredImageNet-leee that uses 10% of the novel set as unlabeled data (following |enn al. (2018)
and tiered mere that uses 50% of the novel set as unlabeled data. We follow the same evaluation protocols section 5.1
We report the results in table 4 We found that on minilmageNet, STARTUP and its variants neither helps nor hurts in most cases (compared to Transfer), indicating that the representation is already well-matched. On both variants of tieredImageNet, we found that STARTUP, with the right ini- tialization, can in fact outperform Transfer. In particular, in the less data case, it is beneficial to initialize the student with the teacher model whereas in the more data case, training the students from scratch is superior. In sum, these results show the potential of STARTUP variants to boost few-shot transfer even when the base and target domains are close.
8

---

Published as a conference paper at ICLR 2021
Table 5-way 1-shot (top) and 5-way 5-shot (bottom) classification accuracy on minilmagenet and tieredlmageNet. Bolded entries are top performing methods that are not different based on t-test at significant level 0.05.
Methods (k=1)
minilmageNet
lieredlmageNettless
tieredlmageNett-more
Transfer
54.18 = 0.79
57.29 = 0.83
57.68 = 0.89
STARTUP-T (no SS)
53.91 0.79
60.39 0.86
61.00 0.86
STARTUP (no SS)
53.74 0.80
53.49 0.85
57.19 + 0.89
STARTUP-Rand (no SS)
54.15 = 0.81
56.93 + 0.91
63.29 + 0.90
STARTUP-T
54.00 + 0.80
60.19 + 0.86
60.16 + 0.86
STARTUP
54.20 0.81
55.33 0.85
53.88 0.89
STARTUP-Rand
53.89 0.87
56.93 + 0.91
61.95 + 0.93
Methods (k=5)
minilmageNet
tieredlmageNet-lesss
tieredlmageNet-morre
Transfer
76.20 + 0.64
79.05 + 0.65
78.67 + 0.69
STARTUP-T (no SS)
76.26 = 0.64
80.14 = 0.65
79.61 + 0.68
STARTUP (no SS)
76.42 + 0.63
78.36 + 0.66
78.50 + 0.68
STARTUP-Rand (no SS)
73.77 0.66
79.58 0.69
81.60 0.66
STARTUP-T
76.21 + 0.63
79.40 0.67
79.04 + 0.68
STARTUP
76.48 = 0.63
77.78 = 0.67
77.48 + 0.68
STARTUP-Rand
71.08 + 0.72
79.58 + 0.69
81.03 + 0.66
Additional Ablation Studies: We conducted three additional ablation studies: (a) training the student with various amount of unlabeled data, (b) training the student without the base dataset and (c) using the rotation as self-supervision instead of SimCLR in STARTUP. We show that STARTUP benefits from more unlaheled data (Appendix A.5), training student without the base dataset can hurt performance in certain datasets but not all datasets (Appendix (.6) and STARTUP (w/ Rotation) outperforms Transfer certain datasets but underperforms its SimCLR counterparts (Appendix
A.3
CONCLUSION
We investigate the use of unlabeled data from novel target domains to mitigate the performance degradation of few-shot learners due to large domain/task differences. introduce STARTUP simple yet effective approach that allows few-shot learners to adapt feature representations to the target domain while retaining class grouping induced by the base classifier. We show that STARTUP outperforms prior art extreme cross-domain few-shot transfer.
ACKNOWLEDGEMENT
This work is funded by the DARPA LwLL program.

---

Published as a conference paper at ICLR 2021
REFERENCES
Lars Buitinck, Gilles Louppe, Mathieu Blondel, Fabian Pedregosa, Andreas Mueller, Olivier Grisel, Vlad Niculae, Peter Prettenhofer, Alexandre Gramfort, Jaques Grobler. Robert Layton, Jake Van- derPlas, Arnaud Joly, Brian Holt, and Gaël Varoquaux. API design for machine learning software: experiences from the seikit-learn project. In ECML PKDD Workshop: Languages for Data Min- ing and Machine Learning, pp. 108-122, 2013.
Chaoqi Chen, Weiping Xie, Wenbing Huang. Yu Rong, Xinghao Ding, Yue Huang. Tingyang Xu, and Junzhou Huang. Progressive feature alignment for unsupervised domain adaptation. In Pro- ceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 627-636, 2019a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. Proceedings of International Conference on Machine Learning (ICML), 2020.
Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. closer look at few-shot classification. Proceedings of the International Conference on Learning Repre- sentations (ICLR), 2019b.
Zitian Chen, Yanwei Fu, Yu-Xiong Wang, Lin Ma, Wei Liu, and Martial Hebert. Image deformation meta- neetr-n for one-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), June 2019c.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255, Icee, 2009.
Guneet S Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. A baseline for few-shot image classification. In Proceedings of the International Conference on Learning Rep- resentations (ICLR), 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta- leta-lea for fast adaptation of deep networks. In Proceedings of the 34th International Conference Machine Learning- Volume 70, pp. 1126-1135, 2017.
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-learning. In Ad- vances Neural Information Processing Systems, pp. 9516 001-9527,000 2018.
Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pp. 4367- 4375, 2018
Spyros Gidaris and Nikos Komodakis. Generating classification weights with gnn denoising autoen- coders for few-shot learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 21-30, 2019.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In Proceedings of the International Conference on Learning Repre- sentations (ICLR), 2018.
Spyros Gidaris, Andrei Bursuc, Nikos Komodakis, Patrick Pérez, and Matthieu Cord. Boosting few- shot visual learning with self-supervision. In Proceedings of the IEEE International Conference on Computer Vision, pp. 8059-8068, 2019.
Yunhui Guo, Noel CF Codella, Leonid Karlinsky, John R Smith, Tajana Rosing, and Rogerio Feris. A new benchmark for evaluation of cross-domain seo-shot learning. In Proceedings of the Euro- pean Conference on Computer Vision (ECCV), 2020.
Bharath Hariharan and Ross Girshick. Low-shot visual recognition by shrinking and hallucinating features. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3018- 3027, 2017.
10

---

Published as a conference paper at ICLR 2021
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning- In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 9729-9738, 2020.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.0253: 2015.
Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In International conference on machine learning. pp. 1989-1998. PMLR, 2018.
Ruibing Hou, Hong Chang, MA Bingpeng, Shiguang Shan, and Xilin Chen. Cross attention network for few-shot classification. In Advances in Neural Information Processing Systems, pp. 4003- 4014, 2019.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung. Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. 2020.
Issam H Laradji and Reza Babanezhad. M-adda: Unsupervised domain adaptation with deep metric learning. In Domain Adaptation for Visual Understanding, pp. 17-31. Springer, 2020.
Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 10657-10665, 2019.
Xinzhe Li, Qianru Sun, Yaoyao Liu, Qin Zhou, Shibao Zheng, Tat-Seng Chua, and Bernt Schiele. Learning to self-train for semi-supervised few-shot classification. In Advances in Neural Infor- mation Processing Systems, pp. 10276-10286 2019.
Yanbin Liu, Juho Lee, Minseop Park, Saehoon Kim, Eunho Yang, Sung Ju Hwang, and Yi Yang. Learning propagate labels: Transductive propagation network for few-shot learning. In Pro- ceedings of the International Conference on Learning Representations (ICLR), 2019.
Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael Jordan. Conditional adversarial domain adaptation. In Advances in Neural Information Processing Systems, pp. 1640-1650, 2018.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(Nov) 2009,2605,222 2008.
Ke Mei, Chuang Zhu, Jiaqi Zou, and Shanghang Zhang. Instance adaptive self-training for unsu- pervised domain adaptation. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representa- tions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 6707-6717, 2020.
Jiquan Ngiam, Daiyi Peng, Vijay Vasudevan, Simon Kornblith, Quoc Le, and Ruoming Pang. Do- main adaptive transfer learning with specialist models. arXiv preprint arXiv:1811.07055. 2018.
Alex Nichol and John Schulman. Reptile: a scalable metalearning algorithm.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European Conference on Computer Vision, pp. 69-84. Springer, 2016.
Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. In Advances in Neural Information Processing Systems, pp. 721-731, 2018.
11

---

Published as a conference paper at ICLR 2021
Adam Paszke, Sam Gross, Francisco Massa, Adam Lever, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Viti,o Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019. URL http: http:: http://aaperseeee 901 h01s-pytorch-an- pdf
ibrary.
Hang Qi, Matthew Brown, and David Lowe. Low-shot learning with imprinted weights. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 5822-5830, 2018.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In Proceedings of the International Conference Learning Representations (ICIRR) 2017.
Mengye Ren, Eleni Triantafilouu Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B Tenenbaum, Hugo Larochelle, and Richard S Zemel. Meta-leaming for semi- cumi-supervised few-shot classifica- tion. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
Pau Rodríguez, Issam Laradji, Alexandre Drouin, and Alexandre Lacoste. Embedding propagation: Smoother manifold for few-shot classification. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. In Proceedings of the International Conference Learning Representations (ICLR), 2019.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in neural information processing systems, pp. 4077-4087, 2017.
Jong-Chyi Su, Subhransu Maji, and Bharath Hariharan. When does self-supervision improve few- shot learning? In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele. Meta-transfer learning for few-shot learning. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 403-412, 2019.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE Con- ference Computer Vision and Pattern Recognition, pp. 1199-1208, 2018.
Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B Tenenbaum, and Phillip Isola. Rethinking few- shot image classification: a good embedding is all you need? In Proceedings of the European Conference Computer Vision (ECCV), 2020.
Hung- Yu Tseng, Hsin- Ying Lee, Jia-Bin Huang, and Ming Hsuan Yang. Cross-domain few-shot classification via learned fcature-wise transformation. International Conference on Learning Representations, 2020.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7167-7176, 2017.
Nguyen Xuan Vinh, Julien Epps, and James Bailey. Information theoretic measures for cluster- ings comparison: Variants, properties, normalization and correction for chance. The Journal of Machine Learning Research, 1 11:2837-285, 2010.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in neural information processing systems, pp. 3630-3638, 2016.
Bram Wallace and Bharath Hariharan. Extending and analyzing self-supervised learning across domains. In Proceedings of the European Conference Computer Vision (ECCV), 2020.
12

---

Published as a conference paper at ICLR 2021
Mei Wang and Weihong Deng. Deep visual domain adaptation: A survey. Neurocomputingg 312:
135-153, 2018.
Yan Wang, Wei-Lun Chao, Kilian Q Weinberger, and Laurens van der Maaten. Simpleshot: Re- visiting nearest-neighbor classification for few-shot learning. arXiv preprint 2019.
Yikai Wang, Chengming Xu, Chen Liu, Li Zhang, and Yanwei Fu. Instance credibility inference for few-shot learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12836-12845, 2020.
Yu-Xiong Wang, Ross Girshick, Martial Hebert, and Bharath Hariharan. Low-shot learning from imaginary data. In Proceedings of the IEEE conference on computer vision and pattern recogni- fion, pp. 7278-7286, 2018.
Garrett Wilson and Diane J Cook. A survey of unsupervised deep domain adaptation. ACM Trans- actions on Intelligent Systems and Technology (TIST), 11(5):1-46, 2020.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1733-3742, 2018.
Qizhe Xie, Minh- Thang Luong, Eduard Hovy, and Quoe V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 10687-10698, 2020.
Ruijia Xu, Guanbin Li, Jihan Yang, and Liang Lin. Larger norm more transferable: An adaptive feature norm approach for unsupervised domain adaptation. In Proceedings of the IEEE Interna- tional Conference on Computer Vision, pp. 1435, 2019.
I Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan. Billion-scale semi- supervised learning for image classification. arXiv preprint 1xXiv:/905.005444 2019.
Zhongjie Yu, Lin Chen, Zhongwei Cheng, and Jiebo Luo. Transmatch: wansfer-learingg scheme for semi-supervised few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 12864, 2020.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. The visual task adaptation benchmark. arXiv preprint arXiv:1910.000 00.076 2019.
Qiming Zhang, Jing Zhang, Wei Liu, and Dacheng Tao. Category anchor-guided unsupervised domain adaptation for semantic segmentation. In Advances in neural information processing systems, 2019.
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European confer- ence on computer vision, pp. 649-666. Springer, 2016.
Yang Zou, Zhiding Yu, BVK Kumar, and Jinsong Wang. Unsupervised domain adaptation for se- mantic segmentation via class-balanced self-training. In Proceedings of the European conference on computer vision (ECCV), pp. 289-305, 2018.
Yang Zou, Zhiding Yu, Xiaofeng Liu, BVK Kumar, and Jinsong Wang. Confidence regularized self-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 5982-5991, 2019.
13

---

Published as a conference paper at ICLR 2021
A APPENDIX
A.1 IMPLEMENTATION DETAILS
We implemented STARTUP by modifying the the publicly-availabl implementation of BSCD-FSL.
by Guo et al. (2020)
A.1.1 TRAINING THE TEACHER
1. MinilmageNer: We train the teacher model using the code provided in the BSCD-FSI. benchmark. We keep everything the same except setting the batch size from 16 to 256.
2. Net::: We used the same setup minilmageNet except reduce the number of epochs to 90. We do not use any image augmentation for tieredImageNet.
3. ImageNet: We used the pretrained ResNet18 available on PyTorch (Paskke et al. 2019)
A.1.2 TRAINING THE STUDENT
Optimization Details. Regardless of the base and novel datasets, the student model is trained for 1000 epochs where an epoch is defined to be a complete pass on the unlabeled data. We use a batch size of 256 on the unlabeled dataset and batch size of 256 for the base dataset if applicable. We use the SGD with momentum optimizer with momentum 0.9 and weight decay le-4. To pick the suitable starting learning rate, 10% of the unlabeled data and 5% of the labeled data % when using ImageNet as the base dataset) are set aside as our internal validation set. We pick the starting learning rate by training the student with starting learning rate Ir E {le-1, 5e-2, 3e-2, 1e-2, Se-3, 3e-3, 1c-3} for k epochs where k is the smallest epoch that guarantees at least 50 updates to the model and pick the learning rate that yields lowest loss on the validation set as the starting learning rate. We reduce the learning rate by a factor of 2 when the training loss has not decreased by 20 epochs. The model that achieves the lowest loss on the internal validation set throughout the 1000 epochs of training is picked as the final model.
SimCLR. Our implementation of SimCLR's loss function is based on a publicly available imple- mentation of SimCLR? 2 We added the two-layer projection head on top of the embedding function 6. The temperature of Ne-Xent is set to since there is no validation set for BSCD-FSI for hyper- parameter selection and For the stochastic image augmentations for SimCLR, we use the augmentations defined for each novel dataset in Guo al. (2020). These augmentations include the commonly used "randomly resized crop". color jittering, random horizontal flipping. For tieredImageNet and minilmageNet, we use the stochastic transformation implemented for the BSCD-FSL benchmark. We refer readers to the BSCD-FSL implementation for more details.
a temperature of 1 when inferring the soft label of the unlabeled set.
When training the student on the base dataset, we the augmentation used for training the teacher for fair comparison. The batchsize for SIMCLR is set to 256.
A.1.3 TRAINING LINEAR CLASSIFIER.
We use the implementation by BSCD-FSL, i. training the linear classifier with standard cross tropy and SGD optimizer. The linear classifier trained for 100 epochs with learning rate 0.01, momentum 0.9 and weight decay Ie-4.
1.1.4 BASELINES
We use the same evaluation methods- linear classifier. Please see A.1.3 for classifier training.
Transfer. This is implemented using the teacher model as feature extractor. Please see A.1.] for details.
SimCLR. This is implemented similarly to the SimCLR loss described in A.1.2
https://gitub..cmm/BBB hhaps://gtthuh.co///salllss/sse
14

---

Published as a conference paper at ICLR 2021
A.1.5 T-SNE
We use the publicly available scikit-learn implementation t-SNE (Buitinck al. 2013). We used the default parameters except for the perplexity where we set to 50. To speed up the experiment, we randomly sampled 25% of the data used for sampling few-shot tasks (80 % the full dataset) and run t-SNE on this subset.
A.2 FULL RESULTS ON BSCD-FSL
We present the result on minilmageNet + BSCD-FSL for shot = 1, 3, 20, 50 in Table 5 and 6 In addition to STARTUP, (STARTUP-T and STARTUP-T (no SS)) in these tables for reference. Results on ImageNet - BSCD- The conclusions found in still hold for higher shots in
also reported results using teacher model student initialization
FSL can be found in Table 7 and 8 general.
A.3 USING ROTATION FOR SELF-SUPERVISION.
We use rotation Gidaris al. 2018) instead of SimCLR in STARTUPand report the results in in Ta- ble 5 and 6 We observe that STARTUP (w/ Rotation) is able to outperform Transfer in CropDisease
and AuroSAT but generally underperforms its SimCLR counterparts.
A.4 INITIALIZATION STRATEGIES FOR THE STUDENT
We investigate the impact of different initialization strategies for the student STARTUP. For this experiment, we remove SimCLR from STARTUP and consider three initialization strategies for the student from scratch (STARTCP-Rand (no SS)), from teacher embedding with randomly initialized classifier (STARTUP (no SS)), from teacher model (STARTUP-T (no SS)). We repeated the experiment in section 5.1 on minilmageNet - BSCD-FSL and report the results in table g We found that not a single initialization is superior to the others (for instance random initialization is the best on CropDisease but the worst on ISIC) however we did find that initializing the student with the teacher's embedding with randomly initialized classifier for STARTUP is either the best or second best in all scenarios so we set that as our default initialization.
A.5 IMPACT OF DIFFERENT AMOUNT OF UNLABELED EXAMPLES
STARTUP uses unlabeled data to adapt feature representations to novel domains. As with all learning techniques, it should perform better with more unlabeled data. To investigate how the amount of un- labeled examples impacts STARTUP, we repeated the minilmageNNe -> ISIC experiments in(5.] various amount of unlabeled data (20% of the dataset (2003 examples) is set aside for evaluation). The verdict is clear STARTUP benefits from more unlabeled data (Figure
A.6 TRAINING THE STUDENT WITHOUT THE BASE DATASET
STARTUP requires joint training on both the base dataset well the target domain. But in many cases, the base dataset may not be available. Removing the cross entropy loss on the base dataset when training the student essentially boils down to fine-tuning paradigm. For minilmageNet BSCD-FSL (Table [O), we found no discernible difference between all datasets except on the ISIC where we observe significant degradation in 5-shot performance.
A.7 STARTUP ON DIFFERENT UNLABELED DATA
We consider the ImageNet + CD-FSL experiment. We perform STARTUP unlabeled data differ- ent from the target domain and present the result in Table II We found that is crucial that the unlabeled data to perform STARTUP on should be from the target domain of interest.
15

---

Published as a conference paper at ICLR 2021
Table 5: 5-way k-shot classification accuracy on minlmageNett-BSC--SS.. Mean and 95% con- fidence interval are reported. (no SS) indicates removal of SimCLR. ProtoNet: (Snell et al. 2017). MAML:(Fimn al., 2017), MetaOpt:(Lee al. 2019). ( et al. 2020p. *Numbers re- ported in (Guo et al. 2020): our re- re-implementation of Transfer uses a different batch size and 80%
of the original test set for evaluation. Bolded entries are top performing methods that are not differ- ent based on t-test at significant level 0.05.
Methods
ChestX
ISIC
k=1
k=5
k=1
k=5
MAMI.
23.48 0.96
40.13 0.58
ProtoNet
24.05 + 1.01
39.57 0.57
ProtoNet + FWT
23.77 + 0.42
38.87 + 0.52
MetaOpt"
22.53 0.91
36.28 + 0.50
Transfer*
25.35 0.96
43.56 0.60
Transfer
22.71 0.40
26.71 +: 0.46
30.71 0.59
43.08 + 0.57
SimCLR
22.10 + 0.41
25.02 0.42
26.25 0.53
36.09 0.57
Transfer + SimCLR
22.70 + 0.40
26.95 + 0.45
32.63
0.63
45.96 + 0.61
STARTUP-T (no SS)
22.79 + 0.41
26.03 + 0.43
32.37
0.61
45.20 0.61
STARTUP (no SS)
22.87 + 0.41
26.68 + 0.45
32.24 0.62
46.48 + 0.61
STARTUP-T
22.75 + 0.40
26.47 + 0.43
32.16 0.60
45.75 - 0.60
STARTUP
23.09 + 0.43
26.94 + 0.44
32.66 0.60
47.22 + 0.61
STARTUP (w/ Rotation)
22.83 + 0.42
26.38 + 0.43
31.54 0.61
45.68 0.60
Methods
EuroSAT
CropDisease
k=1
k=5
k=1
k=5
MAML
71.70 + 0.72
78.05 + 0.68
ProtoNet*
73,29 0.71
79.72 0.67
ProtoNet + FWT*
67.34 + 0.76
72.72 0.70
MetaOpt*
64.44 + 0.73
68.41 + 0.73
Transfer*
75.69 + 0.66
87.48 + 0.58
Transfer
60.73 + 0.86
80.30 + 0.64
69.97 + 0.85
90.16 + 0.49
SimCLR
43.52 + 0.88
59.05 + 0.70
78.23 + 0.83
92.57 0.48
Transfer SimCLR
57.18 = 0.87
77.61 0.66
76.90 0.78
92.64 0.44
STARTUP-T (no SS)
63.00 + 0.84
81.25 + 0.62
71.11 + 0.83
90.79 + 0.49
STARTUP (no SS)
62.90 + 0.83
81.81 0.61
73.30 0.82
91.69 0.47
STARTUP-T
63.49 + 0.85
81.54 + 0.63
72.85 + 0.83
91.49 0.48
STARTUP
63.88 + 0.84
82.29 + 0.60
75.93 + 0.80
93.02 + 0.45
STARTUP (w/ Rotation)
62.18 + 0.86
81.37 + 0.65
70.53 + 0.84
90.59 + 0.48
16

---

Published as a conference paper at ICLR 2021
Table 6: 5-way k-shot classification accuracy on minlmaggeNet-BSCD-FFSL for higher shots. Mean and 95% confidence interval are reported. are methods reported in (Guo et al. 2020). De- spite using their code, difference in batch size and test set (80% of the original test set) have resulted in discrepancies between our Transfer and their Transfer'. (no SS) indicates removal of SimCLR. Bolded entries are top performing methods that are not different based on t-lest at significant level 0.05.
Methods
ChestX
ISIC
k=20
k=50
k=20
k=50
MAMI
27.53 + 0.43
52.36 0.57
ProtoNet"
28.21 + 1.15
29.32 1.12
49.50 0.55
51.99 0.52
ProtoNet + FWT
26.87 + 0.43
30.12 + 0.46
43.78 + 0.47
49.84 + 0.51
MetaOpt"
25.53 + 1.02
29.35 + 0.99
49.42 0.60
54.80 = 0.54
Transfer*
30.83 + 1.05
36.04 0.46
52.78 0.58
57.34 0.56
Transfer
31.99 + 0.46
35.74 + 0.47
54.28 0.59
60.26 + 0.56
SimCLR
29.62 + 0.44
32.69 0.42
47.17 0.58
52.55 0.56
Transfer + SimCLR
32.73 + 0.46
36.64 + 0.47
57.33 + 0.59
62.84 + 0.60
STARTUP-T (no SS)
31.77 + 0.44
35.57 + 0.47
55.80 0.59
61.15 0.56
STARTUP (no SS)
33.02 + 0.47
36.72 + 0.47
57.41 + 0.57
62.71 + 0.56
STARTUP-T
32.79 + 0.46
36.66 - 0.47
56.43 0.60
61.76 + 0.58
STARTUP
33.19 + 0.46
36.91 0.50
58.63 0.58
64.16 + 0.58
STARTUP (w/ Rotation)
31.94 + 0.47
36.33 + 0.46
57.97 0.60
63.44 0.57
Methods
EuroSAT
CropDisease
k=20
k=50
k=20
k=50
MAML
81.95 + 0.55
89.75 + 0.42
ProtoNet*
82.27 + 0.57
80.48 + 0.57
88.15 0.51
90.81 0.43
ProtoNet + FWT*
75.74 + 0.70
78.64 + 0.57
85.82 - 0.51
87.17 0.50
MetaOpt*
79.19 = 0.62
83.62 + 0.58
82.89 + 0.54
91.76 + 0.38
Transfer*
84.13 + 0.52
86.62 + 0.47
94.45 + 0.36
96.62 + 0.25
Transfer
88.31 + 0.45
91.09 + 0.37
96.10 + 0.28
97.69 + 0.20
SimCLR
72.25 + 0.58
78.64 + 0.50
97.26 - 0.23
98.44 + 0.17
Transfer SimCLR
87.48 = 0.45
91.43 0.35
97.41 0.22
98.44 0.17
STARTUP-T (no SS)
88.44 + 0.46
91.13 + 0.38
96.31 + 0.28
97.80 + 0.20
STARTUP (no SS)
89.29 + 0.43
91.94 0.35
96.91 0.25
98.20 0.17
STARTUP-T
88.39 + 0.46
91.27 = 0.38
96.69 + 0.26
98.05 0.19
STARTUP
89.26 + 0.43
91.99 + 0.36
97.51 + 0.21
98.45 + 0.17
STARTUP (w/ Rotation)
88.78 = 0.46
91.63 + 0.37
96.49 + 0.27
98.01 + 0.19
17

---

Published as a conference paper at ICLR 2021
Table 7: 5-way k-shot classification accuracy ImageNer(ILSVRC 2012) +BSCD-FSL. Bolded entries are top performing methods that are not different based on t-test at significant level 0.05.
Methods
ChestX
ISIC
k=5
k=1
k=5
Transfer
21.97 + 0.39
25.85 + 0.41
30.27 0.51
43.88 0.56
STARTUP (no SS)
22.90 + 0.40
26.74 + 0.46
30.18 + 0.56
44.19 + 0.57
STARTUP
23.03 + 0.42
27.24 + 0.46
31.69 0.59
46.02 0.59
Methods
AT
CropDisease
k=1
k=5
k=1
k=5
Transfer
66.08 + 0.81
85.58 + 0.48
74.17 + 0.82
92.46 + 0.42
STARTUP (no SS)
70.08 = 0.80
87.12 + 0.45
80.13 - 0.77
94.51 0.38
STARTUP
73.83 + 0.77
89.70 + 0.41
85.10 + 0.74
96.06 + 0.33
Table 8: 5-way k-shot classification accuracy on ImageNe((LLSRRC 2012) for higher shots. Bolded entries top performing methods that are not different based on t-test at significant level 0.05.
Methods
ChestX
ISIC
k=20
k=50
k=20
k=50
Transfer
30.28 + 0.45
32.55 + 0.46
55.14 0.60
60.99 0.60
STARTUP (no SS)
31.98 = 0.47
34.22 + 0.47
55.54 + 0.57
61.54 + 0.55
STARTUP
32.40 + 0.45
34.95 0.48
57.06 0.58
62.94 0.56
Methods
EuroSAT
CropDisease
k=20
k=50
k=20
k=50
Transfer
91.78 + 0.33
93.76 + 0.29
96.96 + 0.25
98.10 + 0.19
STARTUP (no SS)
92.60 + 0.31
94.53 + 0.26
97.94 + 0.20
98.62 + 0.16
STARTUP
94.27 + 0.26
95.61 + 0.23
98.55 + 0.17
99.07 + 0.13
Table 9: 5-way k-shot classification accuracy on minimmageNet-BSCD for different initializa- tion strategies. Mean and 95% confidence interval are reported. Bolded entries are top performing methods that are not different based on t-test at significant level 0.05.
Methods
ChestX
ISIC
k=1
k=5
k=1
k=5
STARTUP-Rand (no SS)
22.38 + 0.41
24.96 + 0.41
29.76 + 0.60
40.45 0.59
STARTUP-T (no SS)
22.79 + 0.41
26.03 + 0.43
32.37 + 0.61
45.20 + 0.61
STARTUP (no SS)
22.87 + 0.41
26.68 + 0.45
32.24 + 0.62
46.48 + 0.61
Methods
EuroSAT
CropDisease
k=1
k=5
k=1
k=5
STARTUP-Rand (no SS)
63,44 + 0.89
81.05 + 0.60
74.44 + 0.83
92.04 + 0.47
STARTUP-T (no SS)
63.00 + 0.84
81.25 + 0.62
71.11 + 0.83
90.79 + 0.49
STARTUP (no SS)
62.90
0.83
81.81 + 0.61
73.30 + 0.82
91.69 0.47
18

---

Published as a conference paper at ICLR 2021
Top1 Accuracy
47.5
47.0
46.5
46.0
45.5
20
40
60
80
Percentage of Unlabeled Data (%)
5-way 5-shot Classification Accuracy of STARTUP for minilmageNet ISIC with various amount of unlabeled data. Mean and 95% confidence interval over 600 tasks are plotted.
Figure 3:
5-way k-shot classification accuracy mininageNNe-BBSCD-SS.. We compare STARTUP to fine-tuning. Bolded entries are top performing methods that are not different based on 1-test at significant level 0.05.
Table 10:
Methods
ChestX
ISIC
k=1
k=5
k=1
k=5
STARTUP
23.09 + 0.43
26.94 + 0.44
32.66
0.60
47.22 + 0.61
Fine-tuning
22.76 + 0.41
27.05 + 0.45
32.45
0.61
45.73 + 0.61
Methods
EuroSAT
CropDisease k=5
k=l
k=5
k=1
STARTUP
63.88 + 0.84
82.29 + 0.60
75.93 + 0.80
93.02
0.45
Fine-tuning
62.86 + 0.85
82.36 + 0.61
76.13 + 0.78
93.01 + 0.44
Few-shot classification accuracy ImageNet(ILSVRC 2012)= with STARTUP on different datasets. STARTUP-X represents the STARTUP student trained on ImageNet and dataset X. The top table presents the results for 5-way 1-shot and the bottom table presents the results for 5-way 5-shot. Bolded entries are top performing methods that are not different based on 1-test at significant level 0.05.
Table 11:
Methods
ChestX
ISIC
EuroSAT
CropDisease
STARTUP-ChesIX
23.03 + 0.42
31,02 + 0.55
65.20 0.87
70.36 0.86
STARTUP-ISIC
21.98 + 0.39
31.69 + 0.59
61.31 + 0.78
69.27 + 0.87
SUTTST
22.23 = 0.38
30.38 + 0.59
73.83 + 0.77
70.40 + 0.86
STARTUP--ropDissase
22.51 + 0.40
30.59 0.55
62.56 0.83
85.10 0.74
Methods
ChestX
ISIC
EuroSAT
CropDisease
STARTUP-ChesIX
27.24 +: 0.46
44.14 + 0.59
83.76 + 0.56
89.95 +:2 0.49
STARTUP-ISIC
25.05 + 0.42
46.02 + 0.59
81.57 + 0.55
89.24 + 0.50
STARTUP-EUrOSAT
25.15 +: 0.43
43.90 + 0.58
89.70 0.41
88.78 + 0.54
STARTue-CroDDisease
25.21 + 0.44
44.34 0.60
83.13 0.54
96.06 0.33
19


Reviews:
[{"review_id": "qY5S-4tFSkq", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This nicely written paper explores the situation in which one tries to do domain transfer from a large dataset with many labels to a partially labeled dataset with relatively few unlabeled and even less labeled examples, if the domain gap is very large.\n\nThe paper presents a pre-training methodology in which the dataset is first co-trained on the large, labeled dataset and the unlabeled examples. They use the SimCLR objective for pretraining on the unlabeled examples.\n\nThis situation explored here is extremely common in practice, especially in medical imaging: even unlabeled training data is relatively rare, however, it is even rarer to have high quality training data.\n\nThis approach show premise in tackling the domain transfer problem from image-net images to chest X-rays. The paper reports significant progress on chest-ray images and therefore providing important practical contributions in this domain. \n\nWhile the novelty of the approach is somewhat limited: it just combines an existing pretraining approach with co-training on another dataset, its simplicity makes up for it. Also, I am not aware of any work that tackles this important issue so elegantly and directly.\n\nAlso the paper presents two hypothesis on why this approach works. One just posits that the additionally unsupervised data helps independently of the target domain. The second posits that the pretraining helps grouping in the target domain. The paper presents convincing ablation analysis to back the latter claim: training on the target domain is helpful for transferring to that specific domain.\n\nStrength: \n- High quality writing. Easy to read paper\n- Good practical results\n- Tackles a practically highly relevant problem\n- Convincing ablation analyses\n- Simplicity of the approach\n\nWeaknesses:\n- Somewhat limited novelty\n- No theoretical analysis\n- Evaluated on a single scenario\n\nDespite the fact that the approach is somewhat straightforward and of limited novelty and while it was evaluated on one particular use-case, the approach seems to be highly relevant in practice, it is relatively simple and practical and exhibits good performance in an extremely important scenario.\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "YO0lZSy-Joy", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Summary\n\nThe submission tackles the problem of cross-domain few-shot learning in a setting where unlabeled data is available for the test domains.\n\nIt introduces an approach called \"Self Training to Adapt Representations to Unseen Problems\" (STARTUP) which first pre-trains a teacher model on the (labeled) base dataset, and then distills the teacher model into a student model (initialized with the teacher parameters and a random output layer) on the base set of classes as well as the (unlabeled) target domain dataset. For the target domain examples, the loss is a combination of the KL-divergence between logits output by the student and teacher models and an additional unsupervised/self-supervised loss function (SimCLR in this instance). For test episodes, STARTUP is free to choose any applicable inference approach, and the paper opts to fit a linear classifier on top of the frozen feature extractor.\n\nThe proposed approach is evaluated on BSCD-FSL using a random 20% subset of each target domain to form unlabeled datasets and the remaining 80% to form evaluation episodes. STARTUP is compared with recent cross-domain few-shot classification approaches (which do not use unlabeled target domain data) and a purely self-supervised baseline which ignores the base dataset and learns a representation from the target domain using SimCLR.\n\nThe submission also investigates two hypotheses to explain why STARTUP improves performance, and concludes that its purpose is *not* to simply introduce noise during training and instead appears to emphasize the natural groupings induced by the base classifier on the target domain data.\n\n#### Strengths and weaknesses\n\n* **+** Clarity: the paper is well-written and easy to follow.\n* **+** Cross-domain few-shot classification is very relevant to the few-shot learning research community, and the availability of unlabeled data for the test domains is a plausible assumption.\n* **+** The proposed approach is sound and straightforward.\n* **+** The paper makes an effort at explaining why STARTUP provides a performance improvement.\n* **-** The paper should be more rigorous when reporting which approach performs best in a given setting.\n\n#### Recommendation\n\nI recommend acceptance. Overall the paper is clear, the problem being tackled is relevant to the research community, the proposed idea is sound, and evaluation is rigorous. My main concern has to do with the way in which best-performing approaches are reported, but that’s easily fixable in a subsequent version of the paper.\n\n#### Detailed justification\n\nI appreciate the quality of the writing. The proposed idea is straightforward and makes intuitive sense. The baselines chosen for comparison are reasonable.\n\nThe main concern I have is with the way in which results are reported in Table 1. I believe the authors bolded the best-performing entry in each setting without taking the 95% confidence intervals into account. As an example, can we say that STARTUP performs significantly better than Transfer in the ChestX 5-way 1-shot/5-shot settings when their 95% confidence intervals overlap as much as they do? In my opinion the more rigorous way to determine that would be to run a 95% confidence statistical test on the difference between the means and bold all entries for which the test is inconclusive in rejecting the hypothesis that the difference in mean to the best-performing entry is zero, like is done in the Meta-Dataset paper. This doesn’t change the main conclusions drawn by the paper, but should nevertheless be addressed.\n\n#### Questions\n\n1. When training the student model, would there be a benefit to compute the loss as the KL-divergence between the label distributions output by the teacher and student models instead of the cross-entropy with the true label? Have the authors investigated this?\n2. The proposed approach has the downside that it requires training and storing a separate student model for each target domain. Have the authors thought of more compact approaches, like domain-conditional self-training?\n\n#### Additional feedback\n\n1. The abstract mentions evaluating on a \"challenging benchmark with multiple domains\" but does not name it (BSCD-FSL). This is information that would be helpful to the reader.\n1. The insistence on \"several hours of compute\" in the introduction as being a drawback of current recognition systems could be toned down. To me, training a model for several hours doesn’t sound unreasonable.\n1. The Visual Task Adaptation Benchmark (VTAB) and Big Transfer (BiT) papers would be relevant to mention in the related work section. In particular, the VTAB paper investigates various representation learning strategies (including self-supervision) when transferring between very different base and novel domains.\n1. The strategy to use logits on the label set of the base dataset as targets (and therefore leveraging the natural groupings induced by the base dataset classifier) reminds me in a way of Ngiam et al. (2018)’s work on \"Domain Adaptive Transfer Learning with Specialist Models\", which uses this to inform the weighting of classes used to pre-train a target domain-aware classifier on the base dataset. This paper’s proposed approach differs in many ways, but the similarities would be interesting to expand upon.", "extra_scores": {}, "extra_text": {}}, {"review_id": "AVmnv9TF8q", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary of the paper**\n\nThis paper studies transfer learning in an episode learning setting, where at evaluation time a few-shot few-example task is generated. In contrast to the standard setting, two modifications are made:\n1) large domain differences (base dataset is (mini)ImageNet, target datasets are plant crops, xrays, satellite images) from the BSCD-FSL benchmark (Guo et al., 2020);\n2) the learner is given unlabeled target images for domain adaptation.\nThe technical novelty of the paper is to use the unlabeled dataset by (a) obtaining soft-labels from miniImageNet classes, and (b) using SimCLR (Chen et al, 2020) as self-supervised loss.\n\n**Strengths**\n\nThis setting is interesting especially due to the large domain differences. Adding self-supervision under large domain shift seems a logical choice.\n\n**Weaknesses**\n\n- There is no comparison to other self-supervised methods / unsupervised domain adaptation methods for few-shot learning.  In this paper only a single self-supervised method (SimCLR) is evaluated, while there are other methods also focussing on self-supervision for few-shot learning (Gidaris et al., 2019, Su et al., 2020). While these approaches are not evaluated on the large domain shift, does not tell anything about the relevance for this setting.\n- There is no comparison on a standard benchmark, while a benchmark dataset is used, the benchmark evaluation is omitted, only per dataset / task scores are provided. \n- There is no comparison on other domain transfer tasks, for example on more standard MAML types of domain transfers. To allow to compare to other self-supervised methods for few-shot learning.\n- The title might be misleading: Self-Training might no do so much, according to table 1. The performance difference between Transfer and SimCLR baselines are either about equal (ChestX, CropDisease) or Transfer is much better (ISIC +7, EuroSAT +21). Startup shows a small (in the range +.2 -  +4) performance increase over Transfer, which might suggest that SimCLR acts only as a small regularizer. This is confirmed by the ablation of different startup strategies. Unfortunately, the plain combination of Transfer + SimCLR is not explored.\n\n**Summary**\n\nThis paper presents an interesting line of research on few-shot learning on large domain shifts. The technical novelty is minor: it propose to use SimCLR as auxiliary loss in a student-teacher setup. Without comparing to other work or related methods. Experimentally this setup is evaluated on the BSCD-FSL benchmark dataset (yet, without using the benchmark evaluation). Therefore I give an 'OK but not good enough' rating. It seems a good start, but more comparisons, and evaluations are needed for publication.\n\n**Post Rebuttal**\nMy main concerns for the paper are not withdrawn, yet not shared with the other reviewers. I think that is fine, it shows that perspectives differ, and that it is in part why multiple reviewers should read a paper. I also do see that the paper has been improved based on the provided feedback. Given that the paper does not contain major flaws, I upgrade my vote to 6. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "9KqsGQoMYk3", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Problem: The paper introduces the problem of few-shot transfer when there is an extreme difference between the base task and the target task.  The usual few-shot learning setup considers a representation that is trained on a large amount of labeled data. This base representation is then fine-tuned for the target task (that has a few examples, say 1 or 5 labeled examples per class). This strategy works well when the data distribution of the base and target task is similar. However, few-shot learners fail when the data distribution for the two domains are different (e.g., imagenet and crop-diseases) as shown by Guo et al., 2020. \n\nSolution: In this work, authors intuit that there may be a large amount of unlabeled data for the target domain that can be used to learn a good representation of the target task. This is done by using a teacher model trained on a large dataset to generate soft-labels on the unlabeled dataset. A new model is trained using: (1) base dataset; (2) soft-labels on unlabeled data; and (3) contrastive learning on unlabeled data. The new model is initialized using the base representation of the teacher model and the classifier (last layer) is trained from scratch. At test time, *this new representation* is used for learning a classifier on the few labeled examples. This approach has been demonstrated to improve the performance by an average 2.9 points on four tasks: (1) X-Ray; (2) EuroSat; (3) Crop Diseases; and (4) ISIC. \n\nPros: \n\n+ simple approach in yielding better performance!\n\n+ use of unlabeled data from the target domain to learn a better representation.\n\n+ while authors position themselves as \"self-training\" -- I think it is not self-training and is basically about \"conditioning the existing representation on the target task\". To the best of my knowledge, the authors have a very unique and interesting perspective that I have not seen previously.\n\nCons: \n\nFollowing are my concerns with the submission: \n\n1. Chest X-Ray and ISIC dataset vs. Crop-Disease and Satellite Images - The proposed approach works for Crop Disease and Satellite Images which are partly in the domain of base task. It does not show improvement for Chest X-Ray and ISIC dataset that is extremely different from the base task. Is it not against the premise of this paper (specifically the first paragraph of Sec.1)?\n\n2. Section - 4 (Training a new student model): The optimization consists of three parts: (a) supervised loss using base task; (b) pseudo-labels on target task; and (c) contrastive learning on unlabeled images. The evaluation does not show a meaningful difference in the use of these losses except for Crop Diseases dataset. \n\nNote: The authors mention the similarity of (a) and (b) in optimization used in Xie at al., 2020. The statement is nuanced. Strictly speaking, Xie et al., 2020 did not use (a) and (b). They rather trained using the hard pseudo-labels generated on the unlabeled images (using the teacher model).\n\n3. The student model is initialized using the base representation and classifier from scratch. Different initialization strategies are explored. However, conclusions in Section 4.2 are not in sync with Table 7 in Appendix.\n\n4. Section-5.2.1: Hypothesis-1 -- The conclusion of Xie et al., 2020 are valid for their setup (which is strictly for ImageNet classification). \n\n5. Section-5.3: This experiment is inconclusive because of the limited unlabeled data used here. \n\nNow zooming-out and some high-level concerns: \n\n1. First line of the Abstract: \"All few-shot learning techniques *must* be pre-trained on large, labeled dataset.\" --> Why? \n\nI can buy that currently all few-shot learning techniques *are* pre-trained on large, labeled dataset, but I don't see why it is a \"must\"?\n\n2. First paragraph of Section.1: What has few-shot learning to do with computational time? If it is true, then the proposed formulation does not come any close to solving it because it requires training a new model on a large labeled and unlabeled dataset for a new target task. \n\nPOST-REBUTTAL: I have updated the score from 5 to 7 following the clarifications from the authors.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces an approach to self-train a source domain classifier on unlabeled data from the target domain, considering the few-shot learning setting when there is significant discrepancy between the source and target domains. While the reviewers pointed out a few weaknesses, such as somewhat limited methodological novelty  and lack of comparisons with other methods, they all recommend acceptance as final decision. The paper is beautifully written. The proposed method is very simple, but yields excellent results in a very practical problem, which should be of wide interest to the ICLR community. The experimental evaluation is rigorous and the ablation studies are convincing. The AC agrees with the decision made by the reviewers and recommends acceptance.
Decision: Accept (Oral)