Dynamic Distillation Network for Cross-Domain Few-Shot Recognition with Unlabeled Data
Authors: Ashraful Islam
Chun-Fu Chen
Rameswar Panda
Leonid Karlinsky
Rogerio Feris
Richard Radke
Venue: NeurIPS 2021 Poster (2021)
Keywords: Few-shot-learning
Knowledge distillation

Abstract: Most existing works in few-shot learning rely on meta-learning the network on a large base dataset which is typically from the same domain as the target dataset. We tackle the problem of cross-domain few-shot learning where there is a large shift between the base and target domain. The problem of cross-domain few-shot recognition with unlabeled target data is largely unaddressed in the literature. STARTUP was the first method that tackles this problem using self-training. However, it uses a fixed teacher pretrained on a labeled base dataset to create soft labels for the unlabeled target samples. As the base dataset and unlabeled dataset are from different domains, projecting the target images in the class-domain of the base dataset with a fixed pretrained model might be sub-optimal. We propose a simple dynamic distillation-based approach to facilitate unlabeled images from the novel/base dataset. We impose consistency regularization by calculating predictions from the weakly-augmented versions of the unlabeled images from a teacher network and matching it with the strongly augmented versions of the same images from a student network. The parameters of the teacher network are updated as exponential moving average of the parameters of the student network. We show that the proposed network learns representation that can be easily adapted to the target domain even though it has not been trained with target-specific classes during the pretraining phase. Our model outperforms the current state-of-the art method by 4.4% for 1-shot and 3.6% for 5-shot classification in the BSCD-FSL benchmark, and also shows competitive performance on traditional in-domain few-shot learning task.

Dynamic Distillation Network for Cross-Domain Few-Shot Recognition with Unlabeled Data
Ashraful Islam Rensselaer Polytechnic Institute islama6erpi. edu
Chun-Fu Chen
Rameswar Panda MIT-IBM Watson AI Lab rpanda @imm...iimmm..... com
MIT-IBM Watson AI Lab chenrichêus ibm. com
Leonid Karlinsky
Rogerio Feris
Richard J. Radke Rensselaer Polytechnic Institute r rjradkeeecse, rri. edu
IBM Research Leonidkal il. ibm. com
IBM Research rsferis@us. ibm. com
Abstract
Most existing works in few-shot learning rely on meta- meta-learning the network on a large base dataset which is typically from the same domain the target dataset. We tackle the problem of cross-domain few-shot learning where there large shift between the base and target domain. The problem cross-domain few-shot recog- nition with unlabeled target data is largely unaddressed in the literature. STARTUP the first method that tackles this problem using self-training However, it fixed teacher pretrained the unlabeled target samples. As base dataset and unlabeled dataset are from different domains, projecting the target images in the class-domain of the base dataset with a fixed pretrained model might be sub-optimal. We propose simple dynamic distillation-based approach facilitate unlabeled images from the noy- el/base dataset. We impose consistency regularization by calculating predictions from the weakly-augmented versions
labeled base dataset create soft labels for
unlabeled images from teacher net- work and matching it with the strongly augmented versions of the same images from student network. The parameters of the teacher network are updated exponential moving average of the parameters of the student network. We show that the proposed network learns representation that can be easily adapted the target domain even though it has not been trained with target-specific classes during the pretraining phase. Our model outperforms the current statc-of-the art method by 4.4% for 1-shot and 3.6% for 5-shot classification in the BSCD- CSL- benchmark, and also shows competitive performance on traditional in-domain few-shot learning task. Our code is available at: htttttt://// :ttp::///gt... io/Jilge
1 Introduction
The tremendous success of deep learning visual recognition tasks is, to a great extent, attributed to the availability of large scale labeled datasets. While humans can recognize object by looking only at few examples, modern deep neural networks require hundreds or thousands of images for each category to achieve human-level visual recognition capability. This has led to the research on few-shot learning which aims at learning from a much smaller dataset. In a typical few-shot learning setting, there two stages: meta-tran tetttatrriinng and meta-testing. In the meta- meta-trainin training stage. base dataset with labeled images is provided train the model. In the meta-testing stage, the learned model is quickly adapted set of novel classes with only few examples per class (the support set) and evaluated on a set test images from the same novel classes (the query set). The base classes and novel classes typically disjoint, but the images world settings, training the model on a base dataset from the same domain as the target dataset is difficult and infeasible. Guo al. proposed a cross-domain few-shot benchmark, BSCD-FFSL, which contains datasets from extremely different domains. In this benchmark, the meta-training is
obtained from the same domain. However, in many real
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Few-shot setup
Cross-domain few-shot setup
Our setup
Meta-train
Labeled base data
Labeled base dala
Labeled base unlabeled target data
K=3
K+3
K=3
Meta- -tst
Support
Query/(?
Support
Query(?)
Support
Query(()
Figure I: Problem setup. (Left) In typical few-shot learning task, model is trained base dataset first during meta-training stage. meta-lesting stage, a few examples from novel classes, referred as support set, are provided, and the network predicts the categories of different samples from the same classes as support set. The base dataset and the target dataset generally come from the same domain with disjoint categories. (Middle) In cross-domain few-shot learning, there is domain gap between the base dataset and the target dataset. For example, the figure, the base dataset contains natural images from minilmageNett (2)) and the target dataset consists satellite images from EuroSAT dataset ( (Right) Our setting is similar to cross-domain fow-shot learning setup. However, additional unlabeled images are also available during meta-training stage. Although the unlabeled dataset comes from the same domain as the target dataset, it does not contain any images either from the support set query set.
done on a labeled source dataset, and the few-shot evaluation performed on a target dataset which from different domain than the source dataset. The benchmark shows that traditional pretraining and finetuning outperforms more complicated meta-learning based few-shot learning methods by significant margin.
In the real-world scenarios, the target domain should have many unlabeled images, and it might be beneficial use the unlabeled data to learn more target domain specific representations. We hypothesize that using both labeled base data and unlabeled target data during training provides a common embedding for both base and target domain. Then the natural question could be why not use the unlabeled target data only, it might provide more target-specific representation. One issue with this approach that self-supervised learning generally requires a large amount of unlabeled data to work, and, as pointed out by Phoo and Hariharan [88), plain self-supervised learning struggles to outperform the naive transfer learning baseline few-shot learning setup. Secondly, has been shown that combining supervised and unsupervised learning during training provides more transferable representation We argue that similar conclusion holds for cross- cross-domain few-shot learning, i.c.. combining supervised and unsupervised loss provides better representation for the downstream task. Figure Jiluustratt our experimental setup contrast traditional few-shot learning or dosain few-shot learning setup. We show that labeled images from the base dataset are still important to learn generic image features, and images from the target domain, even if unlabeled, can help developing more target domain specific representations.
Figure 2 illustrates our approach. Our goal is train feature extractor which will be used to evaluate few-shot learning performance on the target dataset. We propose a dynamic distillation- based approach to this end. The student network consists of an encoder fs and classifier J.. and the teacher network shares similar architecture as the student network (denoted as ft and gt). The classifier 9s linear layer that predicts the class-logits of the samples from the base dataset. We calculate supervised cross tross-entropy loss between the student's predictions and ground-trum labels on the base dataset. For the unlabeled target data, we compute the teacher's prediction for a weakly- augmented version of image and the student's 's prediction for a strongly augmented version of the same image, and optimize a distillation loss match the predictions. We also apply sharpening in the teacher prediction to encourage low-entropy prediction from the student. Both the supervised loss and distillation loss are used learn the student's weights. The teacher network is updated moving average of the student network. During few-shot evaluation, only use the student

---

encoder fo as feature extractor, learn classifier head on the labeled support images consisting of few examples per category, and calculate the class predictions of the query images.
Our main contributions are:
We propose a simple method for few-shot learning across extreme domain difference. We use dynamic distillation based approach that uses both labeled source data and unlabeled target data to learn a better representation for the few-shot evaluation on the target domain.
Our method significantly outperforms the current state of the art in the BSCD-FSL bench- mark with unlabeled images by 4.4% for 1-shot and 3.6% for 5-shot classification in terms average top-1 accuracy. It even shows superior performance for in-domain few-shot classification on minilmageNet and tieredImageNet datasets.
2 Related Work
Few-shot classification Few-shot learning methods can be divided into three broad categories
and adaptation-based [TO] 14 Early work on few-shot Matching Networks 29 uses cosine similarities on feature vectors produced by independently learned feature extractors, while Relation Networks
generative [36], metric-base [2]
28
23
learning was based on meta-learning 229
23
10
13
23 learn its own similarity metric. MAML [0]] learns good initialization parameters that can be quickly adapted to a new task. Prototypical Networks [2T] learn a feature extractor that is used to calculate distances between features of test images and the mean features of support images. MetaOptNet [4 uses a discriminatively trained linear predictor to learn representations for few-shot learning.
Self-training Self-training trains a student model that mimicks the predictions of a teacher model. Self-training can improves ImageNet classification (33. It is also dominant approach in semi- supervised learning, where the teacher network is used to create pseudo [55 34 or soft labels (3
for huge set of unlabeled images, and the student network is trained to mimic the teacher.
Semi-supervised Learning Our method is inspired from recent developments in semi-supervised learning. Both [6] and (27] uses supervised cross-entropy loss with unsupervised regularization loss. Pseudo-labeling based approaches first train a model on a labeled dataset, use the trained model create pseudo-labels of the unlabeled samples, and retrain the model with both labeled and pseudo-labeled samples [12] I FixMatch (22) proposes simplified model which simultaneously
optimizes cross-entropy loss on the labeled samples and generates pseudo labels using the model's prediction on weakly-augmented unlabeled images. If the pseudo labels are confident enough, the model trained to predict the pseudo labels with a strongly augmented version of the same images. We adopt similar approach by imposing consistency regularization. However, while FixMatch a semi-supervised technique where the unlabeled data is assumed to be from the same domain, our approach is applicable to the cross-domain few-shot learning problem. We also calculates the prediction from a mean teacher network instead of using the same network as FixMatch.
Cross-domain few-shot learning Guo al. 7 proposed cross-domain few-shot learning bench- mark, and noted that existing state-of-the-ar approaches fail to achieve good accuracy on this benchmark. One potential solution could be to use an unlabeled dataset from the target learn representations that are adaptable to a completely different domain. Many approaches also explored few-shot learning with unlabeled data smaller gap between the base and target domains. Our method shares some similarity with the recently developed STARTUP 18] method for cross-domain few-shot learning. STARTUP also uses unlabeled data for learning a better representation. However, STARTUP uses a fixed pretrained model to produce pseudo labels for the unlabeled samples, and then train the network with the labeled base dataset and pseudo-labele a self- sup-rvueee contrastive loss on the unlabeled images to improve accuracy, where our method does not require additional contrastive loss. Actually, we argue that our distillation loss works like self-supervised non-contrastive loss, similar to BYOL [6, for which we might not need to add any extra self-supervised loss. We propose dynamic distillation approach, where the parameters of the teacher network are updated during training. We obtain the prediction for the auggeedd version of unlabeled image from the teacher network, and optimizes the model such that the
15 19:: however, most of these works still assume
target dataset. Additionally, STARTUP also incorporates
3

---

Support Images (labeled)
Bestilaionn aationlionn loss
Sharpening
9
9s
heyd
f&
ft
Lohm
Prdiction
Labeled base data
strong aug
weak aug
Unlabeled target
Query Images
Figure 2: Diagram of our approach. Given labeled basc data and unlabeled target data, our goal is to train a feature extractor which will be used to evaluate few-shot learning performance on the target dataset. The student network consists of an encoder f. and classifier Is, and the teacher network share similar architecture as the student network. We use the labeled base dataset optimize the supervised cross-entropy loss. For target image, we compute
se and student's 's prediction for strongly augmented version of the image, and optimize the distillation loss to match the predictions. We also apply sharpening in the teacher prediction to encourage low- low-entropy prediction from the student. Both the supervised loss and distillation loss are used learn student's 's weights. The teacher network is updated moving average of the student network. During few-shot evaluation, simply learn a new classifier header the few-shot support images, and evaluate on the query images.
teacher's ' prediction for
prediction of the strongly augmented version of the same image obtained from the student network matches that of teacher network. Note that FixMatch [22] also uses similar consistency regularization loss for semi-supervised learning. To our knowledge, we the first to use consistency regularization and dynamic distillation for cross-domain few-shot learning.
3 Methodology
3.1 Preliminary
Few-shot Learning Formulation A few-shot learning task consists of a support set S, which containing K data points from classes for N-way K-shot task, and a query Q = (x; )m consisting of data points only from the N classes of the support set. The goal is to classify the query points with the help of the labeled support set. In the typical few-shot learning setting, (1) an embedding is learned from the base/source dataset Ds. (2) linear classifier is learned top of the fixed embedding on the support set, and (3) the classifications of the query data points are determined.
Cross-domain Few-shot Learning The difference between the typical few-shot learning setup and cross-domain few-shot learning is that the base/source dataset is drawn from a very different domain than the target domain. Additionally, in our setting, we are provided unlabeled data points Du = [x;) b from the target domain. The unlabeled dataset contains more classes than the support set. Given the base dataset Dy, and an unlabeled set Dy, we need to learn an embedding that can extract a representation that can be used for few-shot learning evaluation in the target-domain.
3.2 Proposed Method
Encoder We facilitate knowledge distillation to train our base encoder on both source datatset and target dataset. Denote the embedding network as ʃ. that embeds an input image to d-dimensional vector fs (x)). We add classifier header gs on top of Ss. which predicts n. logits from the embeddings. where n. is the total number of classes in the base dataset. Since the labels of the data points of the

---

base dataset are provided, we calculate the supervised cross eross-entropy loss:
lcE(y. = Hyy,p)
where p = So Softmax((s(([(())) and Ha, b) = -a log
Dynamic distillation Additionally, also have teacher encoder and teacher classifier ge- The task of the teacher network is to produce pseudo labels for the unlabeled images. Given an image z; from the unlabeled set Du, compute the model's prediction p; from weakly-augmented version (denoted as z;") and D from a wrongly-augmentee version (denoted as x;)) of the unlabeled image. The prediction of the weakly-augmented version is produced from the teacher network, which serves as soft-target for the pf for the trongly-augmenten images. Specifically,
images. We use the student network to get the prediction
P Softmax((
P =foottaax((((((4")))
where т is a sharpening parameter. Note that we do not let gradient pass through the teacher network. We calculate the distillation loss by the cross-entropy function
(p .p.) H (pp" p.)
Eq. works like consistency regularizer so that the network predicts similar scores for different augmented versions of the image. We can also consider Eq. 3 as self-supervised loss, similar to BYOL (6] or DINO (2). However, one major difference is that in BYOL or DINO the projection head is a random linear layer, where, we are using the supervised classification head as the projection head.
The total loss function is:
C
Σ
(CE(y,P)
Σ 2,EDU
Ns (BLD))EDs
Nu
where À is a perrmmerrrettrr.. The loss function is used to update the parameters of the student network. For the teacher network, we use mean teacher approach (24), i.c., we update the teacher weights Og from the student weights 0g by: 0g = m0, + (1 m)),, where m is the momentum parameter. Note that when m 1, we are essentially using fixed teacher, and when teacher and student share the same model. When m > 0, the teacher network is moving average of the student network denoting the distillation process as dynamic. Please refer to the Appendix for Py Torcch-like pseudo-code of our method.
O, the
4 Experiments
4.1 Experimental Setup
Dataset We follow the evaluation protocol of the BSCD-FSL benchmark 77.. which contains
novel data from CropDisease [17], EuroSAT [8), ISIC [4. and ChestX [32]. The base dataset is minilmageNet 29). which contains 100 classes from ImageNet dataset 5 where each class has 600 images. The novel datasets are chosen based on increasing dissimilarity from the minilmageNet dataset. More details about the datasets sample 20% of the data from each novel dataset to construct the unlaheled set Du. and the remaining images used for evaluation, where perform 5-way 1-shot and 5-way 5-shot classification. For evaluation metric, we report top-1 accuracy and 95% confidence interval over 600 runs. We also report evaluation results on the larger tieredImageNet dataset [19].
provided in the Appendix. Following (18) randomly
Implementation details We use ResNet-10 the backbone network 7 Our pretraining has two steps. In the first step. we train our network only on the minilmageNet dataset for 200 epochs. use SGD with momentum 0.9, weight decay le-4, learning rate 0.01, batch size 32, and the cosine learning rate scheduler. In the next step, we use the minilmageNet-pretraiice network, and use both the base dataset and the unlabeled dataset to optimize the loss function in Eq. 4 for 60 epochs. During training, increase the weight of distillation loss from 0 to until 40 epochs
5

---

Table I: 5-way 1-shot and 5-shot scores the BSCD-FSL benchmark datasets. The mean and 95% confidence interval of 600 runs are reported. The indicates that the numbers are reported from 7 where no unlabeled data used. The are the numbers reported from [8]. which uses 20% of the original set as the unlabeled dataset. We also use similar number of unlabeled images as [8 however, the splits might be different for random sampling.
1-shot
5-shot
Model
EuroSAT
CropDisease
ISIC
ChestX
EurSAT
CropDisease
ISIC
ChestX
MAML"
71.70+772
78.054.70
40.13+.58
23.48:+488
ProtoNet"
73.29+.71
79.724.79
39.57+-57
2..05+1001
MetaOpe'
64. 74
68.41
36.28:.50
22.53+-911
STARTUP!
63.882.84
75.93=.80
32.66:.60
43.09=443
82.29
93.02:1.45
47.22
26.94::45
ProtoNet
55.32-+.8
52.94+.81
29.58:t.57
21.32=.37
76.92+.67
61.844.68
42.49
24.72:4.43
MatchingNet
54.881.90
46.862.88
21.374.51
20.65
68.001.68
63.944.84
33,96#.54
22. 12.62.1336
Transfer
58.14+.83
68.78=.84
32.12-.59
22.60=.39
80.09+.61
89.79.t.52
43.88=.57
26.51+:43
SimCLR((ass)
58.28::90
61.582.88
32.43:.56
22.37=:22
80.83:2.4
83.44:1.61
44.04
26.63:1.6
SimCLR
62.63++87
69.22+.93
31.45++599
23.59+ 44
82,76+.59
89.31+..8
52.18+:54
29.56+:.9
STARTUP
64.322288
74.45=.86
31.73:2.57
22.27=
83.58+.60
92.41=1.77
45.73+.62
26.21+:46
Trans
63.914.83
70.35+.85
31.674.55
23.72=-44
85.78+.51
91.10+..9
45.974.54
29.45+.10
Ours
73.142.84
82.14
34.66:2.58
23.38=
89.07::47
95.54:1.8
49,36:.59
28.31#:46
using cosine scheduling. The sharpening temperature and teacher momentum parameter are set to 0.1 and 0.99 respectively. For the base images and weakly-augmented unlabeled images, we use the random-resize-crop. horizontal flip and normalization augmentations. For strong augmentation, we additionally use the color jitter, Gaussian blur, and random gray scale transformations. The other hyperparameters are kept the same. Refer to the supplementary for more details about hyper-parameter selection. For few-shot evaluation, we learn logistic regression classifier on the support set, and evaluate the query set.
4.2 Main Results
Comparison to State-of-the-ards Table shows the performance comparison of our approach with other methods on the BSCD-FSL benchmark. All models are trained on the minilmageNet dataset. "Transfer" denotes the baseline trained by cross-entropy loss on the base dataset. "SimCLR" is trained only on the unlabeled images. As noted by (9. self-supervised contrastive learning learns better transferable representation for different downstream domain. Hence, we also create "SimCLR(Basee" baseline that trains the encoder by optimizing self-supervised contrastive loss on the base dataset. "Transfer+SimnLL" refers to the model which is trained with supervised cross-entropy loss from the base dataset and self-supervised contrastive loss from the unlabeled target dataset, which has been reported to show superior transferability across domain (. STARTUP [8] is trained with three losses: cross-entropy loss the base dataset, KL-divergence loss the unlabeled dataset similar, and self-supervised contrastive loss on unlabeled images. STARTUP, Transfer+SimCLR and approach use both the base dataset and additional unlabeled dataset during the representation learning phase.
Our method outperforms all meta-learning beaeingeeand approaches by significant margin at all settings. Moreover, compared to Transfer, we achieve more than ~5.5% improvement for 5-shot classifi- cation on average. The performance improvement on 1-shot is more significant; we achieve 7.9 % improvement on average.
We outperform STARTUP by 5.5% on EuroSAT, 3.1 % on CropDisease, 2.1 % on ChestX, and 3.6 % on ISIC for 5-way 5-shot classification. The performance improvement is also quite significant for 1-shot classification; specifically, our method achieves 8.8 % improvement on EuroSAT and 7.7 % improvement on CropDisease dataset over STARTUP. We only perform slightly worse on the ChestX dataset. For ChestX, it seems that pure unsupervised learning performs pretty well. Considering that our method does not use any self-supervised training or distillation, the performance improvement is impressive. Note that STARTUP uses fixed teacher to extract pseudo-labels for the unlabeled images, whereas we extract pseudo labels from the weakly-augmented images from the same network that is being trained. In that sense, our model works like dynamic teacher, where the pseudo labels get more refined as training progress. We hypothesize that the superior performance might be attributed to the dynamic approach of our model over STARTUP.

---

Table 2: 5-way I-shot and 5-shot scores the BSCD-FSLL benchmark datasets when tieredImageNet used as base dataset. All models use ResNet-18 for backbone 25]. The mean and 95% confidence interval of 600 runs are reported. For EuroSAT and CropDisease dataset, our method achieves significant performance improvement over other models. The improvement for 1-shot learning is huge (over 7% for EuroSAT and 10% for CropDisease).
s-shot
5-shot
Model
EuroSAT
CropDiscase
ISIC
ChestX
EuroSAT
CropDiscase
ISIC
ChestX
Transfer
58.07.1.86
69.94.1.7
29.76.1.55
22.46.1.11
81.341.53
90.12.1.99
41.27:1.58
26.33.1.45
SimCLR(Base)
62.142..9
62.45:.90
31.03::55
22.28:1.40
81.85
84.111.6
42.91
25.962.44
STARTUP
64.32.1.87
70.091.86 86
29.73.151
22.10.1.0
85.191.50
90.81
43.55...6
26.03 44
Mhusfer+SimCLR
58.08+.83
71.25+.89
31.71=.55
23.81=.46
86.081.47
91.31
45.08:.56
30.26::50
Ours
72.15:+.75
84.41::75
33.87=.56
22.70++:2
89,444.42
95.904.34
47.21+.56
27.674:46
Table Few-shot evaluation on the same domain terms of 5-way 5-shot and 5-way 1-shot accuracy on minilmageNet and migredl datasets. We use ResNet-10 backbone for minilmageNet and ResNet-18 backbone for tieredImageNet.
minilmageNet 5-shot
Iaaredl 5-shot
1-shot
1-shot
ProtoNet
51.06.1.83
73.491.63
MatchingNet
52.34
67.28+.67
Transfer
53.401..0
74.26.1.64
58.61
81.42.1.65
Transfer+SinCLE
51.63+
74.65
61.33+.90
82,89
STARTUP
51.681.84
74.05=166
..
82.11
Ours
53.71
76.02+1.61
69.00+.96
85.93+..0
Results with tieredImageNet base data tieredImageNet [19] is a subset of ImageNet dataset with 608 classes. The classes are grouped into 34 super-categories, from which 20 training categories (351 classes). 6 validation categories (97 classes), and 8 testing categories (160 classes) are selected.
We use larger backbone ResNet-18 for meta-training with tieredImageNet 25). Table 2 shows the performance comparison with other models. We get similar conclusion as we get from Table I. Particularly, we get 6.72% average improvement for 1-shot and 3.66% average improvement for 5-shot over STARTUP. However, we do not see much better accuracy than minilmageNet pretrained models even though we are using a much larger base dataset, which suggests that the size of base dataset is not as important as other transfer learning task in cross domain few-shot learning.
Few-shot performance on similar domain It has been shown that self-training improves ImageNet classification 333. Given the distillation approach of our model, one could expect that it might improve the few-shot accuracy even when the target data come from the same domain. Note that STARTUP does improve the in-domain accuracy [88), even though it is also self-training based model. We evaluate on minilmageNet and mageed dataset in terms of 5-way 1-shot and 5-way 5-shot performance. We use the official training split base dataset, and the unlabeled target data are obtained from 20% of the novel (test) set and the rest are used for evaluation. For backbone, we use ResNet-10 for minilmageNet and ResNet-18 for tieredImageNet. Table reports the results for in-domain few-shot performance. We see that our method achieves the best performance among the baselines. Particularly, for 1-shot learning in tieredImageNet, our model outperforms the best one by 7.7%. We infer that our method can be safely applied to few-shot learning task when the domain gap between base and target dataset is small, which is in contrast with STARTUP that does not show improvement over "Transfer" for few-shot learning on similar domain.
4.3 Analysis
Table 4: V-measure cluster score (%) (20 on the KMeans
Efffec of dynamic distillation To under- stand how distillation helps to learn better representation, we use the pretrained mod- els to extract features of the target dataset. Then we use KMeans algorithm to create clusters from the features. The number of clusters in the KMaans is set to be the num- ber of classes of the target dataset. In Table
clustering of the extracted features with the ground-truth clustering. The backbone is ResNet- 10 pretrained on the minilmageNet dataset and/or the unlabeled target dataset.
BuroSAT
CropDisease
ISIC
ChestX
Transter SimCLR
57.01
62.58
14.67
2.45
60.06
62.02
12.12
3.84
STARTUP
62.02
69.50
14.05
2.71
Ours
69.58
73.27
14.32
3.32
7

---

the V-measurement cluster scores [20] between the KMeans clusters and original ground-trut are shown. The V-score has 100% value when there is maximum agreement between ground truth and predicted clusters, and 0% when there's 's no agreement. Table 4 shows that our method achieves higher v-scores for EuroSAT and CropDisease dataset, and the v-scores for ISIC and ChestX are also very competitive. It suggests that our model learns a good clustering of the target data even when we are not using any target labels. The clustering is much better when the domain gap is not extreme.
Fig. shows t-SNE plots [26] from 10 representative classes from the CropDisease and EuroSAT datasets. We compare the embeddings extracted from "Transfer" and our approach. We see that our method creates better grouping on the embeddings of the target datasets, even though we do not use any labels for the target dataset during pretraining.
(a) Transfer
(b) Ours
(c) Transfer
(d) Ours
Figure 3: L-SNE plot of 10 classes from CropDisease (a & b) and EuroSAT (c & d) test sets with features
obtained from Transfer and our method.
Comparison with superviee learning If we ignore the supervised loss, model has simi- larity with sulf-super non con-contra loss similar to BYOL or DINO. However, the projection head we using to calculate the final predictions of the two different views of unlabeled image is the same classification head that is used to predict the classification logits of the labeled base samples. separate projection head for
In Table "Ours (distillation head)" represents the model where the predictions of the unlabeled images. We see that separate projection head performs much worse. We found that the distillation loss is simply converging trivial solution, we add recently developed tricks in self-supervised learning, namely, centering and strong augmentation which turns the unlabeled branch similar to ResNet DINO (2). "Ours (DINO head)" achieves better accuracy than "Ours (distillation head)", suggesting that it alleviates the issue of trivial solution. However, our original method still achieves a sigmificant 3.07% more improvement. It is interesting to note that a separate projection head causes trivial solution for the unlabeled images, whereas our model does not converge to trivial solution with similar settings. We infer that using supervised classifier linear layer the projection head can solve the issue of trivial solution for the self-supervised learning to some extent without requiring extra tricks like BatchNorm [6] or centering 12/. Additionally, it provides better clustering of the unlabeled features, even if the unlabeled samples come from different domain than the labeled samples. TableB also shows results for "Ours + SimCLR", which simply adds a SimCLR loss for the unlabeled samples. It performs slightly better only in ChestX dataset. On average, the performance similar to "Ours", which signifies that there is no clear benefit using self-supervised contrastive loss to achieve better transferability for our method. Note that STARTUP comes to a different conclusion reporting that adding SimCLR loss consistently improves the performance.
trivial solution in this case. To discourage
Table 5: Our method with self-supervised approaches. The evaluation is performed on BSCD-FSL benehmark in terms of 5-way 5-shot accuracy (%). "Ours (distillation head)" refers to the model where we use separate projection head for the distillation loss, which achieves much worse scores. In "Ours DINO", we use separate projection head with centering and strong augmentation as in DINO 2 It achieves better performance than naive transfer, but still under-performe in comparison our approach. "Ours+SimCLRR" simply adds sulf-super contrastive loss for the unlabeled samples. See Appendix for more details.
EuroSAT
CropDisease
ISIC
ChestX
Ours
89.07
95.54
49.36
28.31
Ours (distillation head)
80.06
89.31
46.63
25.29
Ours (DINO head)
85.74
90.55
46.24
25.42
Ours SimCLR
88.48
93.80
49.10
29.45
8

---

Experiment of few-shot classification on fine-grained dataset CUB [30 contains 200 classes and 11,788 images of different bird species. 'minimageNett>>UB' is an interesting experiment to show the transferability of different models to fine-grained dataset. We report the results in Table in terms of 5-way 5-shot scores. For CUB, we found that vanilla Transfer performs surprisingly well (also reported by B)), and adding SimCLR with Transfer (Transfer+SimmLL actually decreases the accuracy. Wallace and Hariharan [1 also experimented with different self-supervised methods on smaller domain and found that all of them under-perfone for fine- grained task [2]. However, our method still performs the best, demonstrating the effectiveness of our approach in a fine-grained downstream task.
Table 6: Experiment on mini-ImageNet >> CUB. All the scores are reproduced by us.
ProtoNet Transfer
SimCLR
Transfer+SimCLR
STARTUP
Ours
63.19
68.72
62.84
67.82
66.10
69.50
Experiment of distillation with unlabeled datasets from different domain Table reports the few-shot accuracy when our model is trained on different unlabeled datasets. The best accuracy is achieved when the unlabeled data and target data are from the same domain. Even if the unlabeled data consists of images from multiple domains including the target domain (denoted as "Ours-all"), still significantly under-performa the base model.
Table 7: Effect of unlabeled datasets from a different domain than the target dataset in terms of 5-way base and "X" dataset during pretraining. "Ours-all" denotes
S-shot accuracy (%). "Ours-X" denoles that that we use unlabeled images from all four target datasets.
FuroSAT
CropDisease
ChestX
ISIC
AAT
89.07
90.43
26.02
46.82
Ours-CropDiseare
81.86
95.54
26.17
45.12
Ours-ISIC 81.94
89.69
26.70
49.36
Ours-ChestX
81.87
90.38
28.31
45.20
Ours-All
82.75
91.31
26.01
46.43
Effect of data augmentation On the unlabeled images, we apply two types of augmentation: weak augmentation to extract pseudo labels and strong augmentation to impose consistency regularization. This setting is denoted as "weak-strong" (w-s), where 'weak' (w) augmentation is applied to the image that fed into the teacher network and 'strong' (s) augmentation is applied to the image that is fed into the student network. We also show results with "weak-weat" (w-w), "strong-weak" weak" (s-w) (s-s) augmentation settings in Figure 4al Both "weak-weak" and "strong-strong"
and "strong-strong perform worse than the other augmentations. We also note that in self-supervised learning, generally strong augmentation is applicd to all training images to get good performance. In our experiment, we find that applying weak augmentation in one of the image pairs improves the performance.
More unlabeled data To measure the effect of amount of unlabeled dataset during pretraining, we divide the target dataset by 80% and 20% splits. The 20% split is used for evaluation. From the 80% split, we vary the amount of unlabeled data and then pretrain with source and the unlabeled set. Fig. shows the average 5-shot accuracy for different amounts of the unlabeled dataset during the pretraining phase. As expected, more information from the unlabeled dataset helps to learn better representations on the targel domain. However, the performance saturates later, and we get diminishing return for more unlabeled data. It also signifies that there scopes improve the performance by using more unlabeled data denoting potential future research direction.
4.4 Addition Ablation Studies
We perform several ablation studies of different components of our approach. All scores are reported for 5-way 5-shot evaluation. More ablations are provided in the Appendix.
Longer training We use the base dataset pretrained network as initialization for our network, and then train on both base dataset and unlabeled dataset for 60 epochs. Figure[4dreportss 5-shot few-shot on the BSCD-FSL. benchmark performance for our pretrained with longer training epoch. Training for more epochs can result in minor improvement (0.2%).
4c reports average 5-way

---

65.0
63.5
Figure 4: Ablation studies for (a) data augmentation on unlabeled images, (b) effect of longer training on base and un- labeled data, (c) amount of unlabeled data. The Y-axis represents average top- accuracy (%) on the four benchmark datasets for 5-shot classification for 600 episodes. We use minilmageNet for la- beled source dataset and ResNet- backbone.
06-
63.0
64.5
67
200
Unlabel Augmentation
Percentage of Unlabeled Data
Number of Training Epochs
(a)
(b)
(c)
Table 8: Ablation studies on different settings.
Mean over 600 runs.
EuroSAT
CropDisease
ISIC
ChestX
Ours
89.07
95.54
49.36
28.31
Ours(w/o base)
82.11
90.52
39.76
26.83
Ours(I-step)
86.16
87.28
46.10
25.11
Effect of base dataset Here, perform experiments without the first term in Eq. 4 i.e., train the network on the base dataset first and then re-train only on the unlabeled images (without joint training on the base dataset), denoted as "Ours (w/o base)". Table T shows that the performance of "Ours (w/o base)" is poor, suggesting that the representation related to the labeled base dataset is still helpful the target domain.
Training without pretrained model on base dataset We perform 2-step training during the representation learning phase we first train the model on mini-IN only, and then jointly train on denotes training on the unlabeled images
mini-IN IN and unlabeled images. In Table@] "Ours(I-step" and Mini-IN from scratch for 300 epochs, which performs worse than 2-step training. Our assumption is that the proposed model is also like self-training where well-trained teacher is needed.
5 Conclusion
We introduced a novel approach to utilize unlabeled data from the target domain for cross- few- shot learning. Experiments show that method achieves sfate-of-hhe-ee the results in the BSCD-FSI benchmark for both 1-shot and 5-shot classification. Our model also outperforms other approaches in the same-domain few-shot learning. Future work can be focused applying approach in each episode during meta-testing so that the model can learn more category-specins representations.
Broader Impact
The approach tackles practical problem of the existing few-shot learning setup that the base dataset and the novel samples generally come from different domain. Our work uses the unlabeled samples from the target dataset learn more target specific representation. Like any other machine learning tool, the final impact depends on the intention of the people or institution applying it. However, it can be useful in drug discovery or medical image analysis where labeled dataset is limited.
Acknowledgments
This material is based upon work supported by the U.S. Department of Homeland Security, Science and Technology Directorate, Office of University Programs, under Grant Award 2013-ST-061. ED0001. The views and conclusions contained in this document those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the U.S. Department of Homeland Security.
10

---

References
[1] E. Arazo, D. Ortego, P. Albert, N. E. O'Connor, and K. McGuinness. Pseudo-labeling and confirmation bias in deep semi-supervised learning. 2020 International Joint Conference on Neural Networks (IJCNN), pages 1-8, IEEE, 2020,
[2] M. Caron, H. Touvron, Misra, H. Jégou, J. Mairal, Bojanowski, and A. Joulin. Emerging properties in sule--spe vision transformers, 2021.
[3] W.-Y. Chen, Y.-C. Liu, 7. Kira, Y.-C. F. Wang, and J.-B. Huang. A closer look at few-shot classification. arXiv preprint arXiv:1904.00 04232, 2019.
[4] N. Codella, V. Rotemberg, P. Tschandl, M. E. Celebi, S. Dusza, D. Gutman, B. Helba, A. Kalloo, K. Liopyris, M. Marchetti, et al. Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic). arXiv preprint arXiv: wtt.::922.0330 03368, 2019.
[5] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, Imagenet: A large-scale hierarchical image datahase. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, IEBE, 2009.
[6] J.BB. Grill, F. Strub, F. Altché, C. Tallec, P. H. Richemond, E, Buchatskaya, C. Doersch, B. A. Pires, 7. D. Guo, M. G. Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.072:
2020.
[7] Y. Guo, N. C. Codella, L. Karlinsky, J. V. Codella, J. R. Smith, K. Sacnko, T. Rosing, and R. Feris. A broader study of cross-domain few-shot learning. Eur. Conf. Comput. Vis., 2020.
[8] P. Helber, B. Bischke, A. Dengel, and D. Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEFE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, (2(7)22117222M) 2019.
[9] A. Islam, C.-F. Chen, R. Panda, L. Karlinsky, R. Radke, and R. Feris. broad study on the transfcrability of visual representations with contrastive lcarning, 2021.
[10] T. Jeong and H. Kim. Ood-maml: Meta-learning few shot Advances in Neural Information Processing Systems, 33, 2020.
detection and classification.
[11] J. Kim. T. Kim, S. Kim, and C. D. Yoo. Edge-labeling graph neural network for few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11-20. 2019.
[12] D.HII Lee al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. Workshop on challenges representation learning, ICML, volume 3. 2013.
[13] K. Lee, S. Maji, A. Ravichandran, and S. Soatto. Meta-learning with differentiable convex optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10657- 10665, 2019.
[14] K. Lee, S. Maji, A. Ravichandran, and S. Soatto. Meta-lcariing with differentiable convex optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10657- 10665, 2019.
[15] Y. Liu, J. Lee, M. Park. S. Kim, E. Yang. S. J. Hwang, and Y. Yang. Learning to propagate labels: Transductive propagation network for few-shot learning. arXiv preprint 1005:::8800.00000 2018.
[16] T. Miyato, S.-i. Maeda, M. Koyama, and S. Ishii. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEFF transactions on pottern analysis and machine intelligence, 1179 2018.
[17] S. P. Mohanty, D. P. Hughes, and M. Salathé. Using deep lcarning for image-based plant discase detection. Frontiers in plant science, 7:1419, 2016.
[18] C. P. Phoo and B. Hariharan. Self-training for few-shot transfer across extreme task differences. arXiv preprint arXiv:001000734, 2020.
[19] Ren, E. Triantafillou, S. Ravi, J. Snell, K. Swersky, J. B. Tenenbaum, H. Larochelle, and S. Zemel. Meta-lean Meta-learning for semi cumi-supervised shw-shot classification. arXiv preprint arXiv: frXiv:1803.00676 2018.
[20] A. Rosenberg and J. Hirschberg. V-measure: A conditional entropy- based external cluster evaluation measure. In Proceedings of the 2007 joint conference on empirical methods in natural language processing and computational natural language learning SAMNLP-CoNLAL pages 410-420, 2007.
[21] J. Snell, K. Swersky, and R. Zemel. Prototypical networks for few-shot learning. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, Fergus, S. Vishwanathan, and R. Gamett, edi- tors, Advances in Neural Information Processing Systems, volume 30, pages 4077-4087 Curran Associates, Inc., 2017. URL ht http:pppp//pprccccinngg. ptw.rrpp..ccppppp////11////////e cb9da6767466 220da177//0122214444444a0aa

---

[22] K. Sohn, D. Berthelot, C.-L. Li, Z. Zhang, N. Carlini, E. D. Cubuk. A. Kurakin, 11 Zhang. and C. Raf- fel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. arXiv preprint urXiv:2001.06835, 2020.
[23] F. Sung. Y. Yang. L. Zhang. T. Xiang. H. Torr, and T. M. Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE conference computer vision and pattern recognition. pages 1199- 1208, 2018.
[24] A. Tarvainen and H. Valpola. Mean teachers are hetter role models: Weight-averaged consistency targets improve sepi-supervised deep learning results, 2018.
[25] Y. Tian, Y. Wang, D. Krishnan, J. B. Tenenbaum, and P. Isola. Rethinking fow-shot image classification: a good embedding is all you need? arXiv preprint arXiv:2003. 1539, 2020.
[26] L.. van der Maaten and G. Hinton. Visualizing high-dimensional data using (2008). Reference Source [Google Scholar), 2008.
[27] V. Verma, K. Kawaguchi, A. Lamb, J. Kannala, Y. Bengio, and D. Lopez-Paz. Interpolation consistency training for semi-super learning, arXiv preprint arXiv:1903.03822 2019.
[28] O. Vinyals, C. Blundell, T. Lillicrap. k. kavukcuoglu, and D. Wierstra. Matching networks for one shot learning. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, edi- tors, Advances Neural Information Processing Systems, volume 29, pages 3630 3638. Curran Associates, Inc., 2016. URL
[29] O. Vinyals, C. Blundell, T. Lillicrap, D. Wierstra, al. Matching networks for one shot learning. In Advances in neural information processing systems, pages 3630-3638, 2016.
[30] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The caltech-uesd birds-200- 200-2011 dataset. 2011.
[31] B. Wallace and B. Hariharan. Extending and analyzing self-supervised learning across domains. In European Conference on Computer Vision, pages 717-734. Springer, 2020.
[32] X. Wang, Peng, L. Lu, Lu, M. Bagheri, and Summers. Chestx-ray8: Hospital-scabe chest x-ray database benchmarks on weakly-supervised classification and localization of common thorax diseases. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2097-2106,
2017.
[33] Q Xie, M.-T. Luong, E. Hovy, and Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 10687-10698, 2020.
[34] Q. Likhomanenko, Kahn, A. Hannun, Synnaeve, and R. Collobert, Iterative pseudo-labeling for speech recognition. arXiv preprint wxXiv2205.09266 2020.
[35] Z. Yalniz, H. Jégou, K. Chen, M. Paluri, and D. Mahajan. Billion-scale semi-supervised learning for image classification. arXiv preprint arXiv:1905,0m 000466 2019.
[36] R. Zhang, T. Che, Z. Ghahramani, Y. Bengio, and Y. Song. Metagan: An adversarial approach to few-shot
learning. NeurlPS, 2:8, 2018
12


Reviews:
[{"review_id": "Z6kxJ5ht5jd", "reviewer": "Reviewer_3Ucv", "summary": "This paper tackles the cross-domain few-shot learning, by utilizing the unlabeled target data to encourage the feature extractor to learns the representations that is effective for target domain. They propose a dynamic distillation where the teacher is dynamically updated via exponential moving average during training. The dynamic distillation works similar to recent self-supervised learning method of BYOL, and successfully enhance the generalization capability of learned representation using unlabeled data. Dynamic distillation network achieves the state-of-the-art performance in BSCD-FSL benchmark.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Strength]\n1.\tDynamic distillation method utilizing the exponential moving average model as a teacher is technically sounds and interesting concept, considering the exponential moving average encoder or momentum encoder is widely used to generate the target feature in self-supervised learning, and successfully learns the more general representations. \n2.\tThe authors provided various analysis and ablation experiment results, which is helpful to understand the proposed method. Analysis on the source of unlabeled data (Table 6), and effect of the data augmentation is interesting, not only from the point of view of cross-domain FSL but also from the point of view of self-supervised learning.\n3.\tThe performance gain over prior work (STARTUP) is remarkable. Also, the authors provided various baselines related to transfer learning and self-supervised learning. \n\n\n [Question]\n\n1.\tIn Tables 1 and 2, the performance of the proposed method is compared with the STARTUP and SimCLR-based baselines. However, BYOL is the most similar SSL method to the proposed method. How is the performance of the BYOL or Transfer + BYOL in cross-domain FSL?\n2.\tIn ablation experiments, the authors presented the performance of Ours (w/o base), that pretrain the model on miniImageNet (base dataset) and then learn representation without labeled base dataset. What will happen if someone pretrains the model on base dataset, and then learn representation using the unlabeled images and images from base dataset (use the images only, and not use the label. Consider the images from base dataset as unlabeled data)? It can be an interesting experiment to figure out the effect of supervised loss in the proposed method.\n3.\tIn line 212 to 214, the authors stated that the proposed method does not use any self-supervised training or distillation. However, the proposed method is dynamic distillation, and it utilizes the unlabeled samples like the self-supervised learning model, BYOL. Can you explain the exact meaning of the sentence in line 212 to 214?\n\n* Typo: In line 136, \"datatset\" ->\"dataset\"", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors addressed the limitations of their work. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "ORwKmDFi06A", "reviewer": "Reviewer_pL1o", "summary": "The author propose using teacher-student knowledge distillation for improving the performance of cross-domain few-shot learning. \nImproved results on cross-domain few-shot learning benchmarks are observed and the performance on standard few-shot learning benchmarks are shown to be comparable to state-of-the-art.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The method introduced in this paper is well-presented. \nThe empirical results are somehow strong.\n\nConcerns:\n- My main concern to this paper is the motivation. Using student-teacher knowledge distillation is not a new idea in training deep neural networks. It remains unclear to me why such method is strong on cross-domain few-shot learning without seeing the experimental results. I expect the authors to provide more discussion on the advantages of the proposed methods, e.g., what specific natures of cross-domain few-shot learning is exploited? The current discussion shows only 'another way of using unlabeled data' in cross domain few-shot learning, yet this is something can be exploited in many tasks like standard few-shot learning, domain generalizations, etc.. \n\n- The paper is somehow poorly organized. It is surprising to see that the main discussion (methodology) is not even 1.5 page long with a Figure in it. I think this is also part of the reason that the motivation of this paper is not well presented. \n\n- Some widely used cross-domain few-shot learning settings were set in [1], and are not included in the paper. \n\n- The performance on standard few-shot classification datasets are actually not comparable to SOTA. E.g., according to the leaderboard [2], with the standard inductive setting, many methods can achieve over 54% with simple Conv-4 architecture on miniImageNet 5way 1shot. While in-domain few-shot classification is obviously less challenging, it is confusing to me why the proposed method performs poorly. \n\n- Can the authors provide more discussion on how the proposed method is 'dynamic'?\n\n[1] Chen, Wei-Yu, et al. \"A closer look at few-shot classification.\" ICLR.\n[2] https://few-shot.yyliu.net/miniimagenet.html", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "MRRrVKaHyvR", "reviewer": "Reviewer_zcQX", "summary": "The submission attempts to address the cross-domain few-shot learning problem by utilizing dynamic distillation from additional unlabeled target data. It simplifies the STARTUP framework by removing the contrastive loss and using a similar network for teacher and student. Moreover, it introduces different data augmentations to learn better representations. The teacher network is also trained dynamically to enhance the training of the student network.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n+ The submission is well-written and easy to follow.\n+ Comprehensive experiments and promising results.\n+ Detailed analysis and convincing ablation study. \n\nWeakness:\n- Limited novelty since the approach combines the mean teacher approach from [23] and self-supervised loss from [5] and [2], yet achieving good performance in the cross-domain few-shot learning context.\n- A question about the distillation loss l_u. The cross-entropy function cannot guarantee that the prediction of the student is equal to the prediction of the teacher. Please clarify this. \n- It would be better to explain why and how to choose the weak and strong data augmentations.\n\nMinor suggestions:\n- Please highlight the numbers in Tables 5 and 6. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The proposed approach doesn't achieve the best performance on ChestX data.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "Je1va04ToLl", "reviewer": "Reviewer_eC3H", "summary": "The paper proposes a method for training on “cross-domain few-shot recognition”. This is a task in which the goal is to transfer representations learned on a base, labeled dataset into a target few-shot dataset that is of a different domain. In particular, the paper focuses on the setting where unlabeled data is available for the target domain.\n\nThe method proposed comprises of training a “student model” on the base data in a supervised manner, while also utilizing a distillation loss on the target data, using a “teacher network”’s predictions as the objective. The teacher is set as an exponential moving average of the student’s parameters. After this initial training phase, the student gets a new randomly initialized head to learn on the few-shot labeled target data.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The method proposed is very interesting and seems to yield good results on the BSCD-FSL benchmark. I found the insights in lines 42-44, 266-276 particularly insightful.\n\n# Major Comments\nIt is very related to many similar techniques, such as MoCo, SimCLR, BYOL, etc. In particular, it seems like it differs from MoCo only by the addition of a supervised head and the downstream finetuning on few-shot datasets. I found it disappointing to not see it in the baselines compared. Other baselines (line 111) seem to have been left out because they “assume smaller gap [between domains]”, but this means that this claim isn’t shown empirically. Overall, this decreases my confidence in the high performance of the proposed method.\n\nI also am not convinced by the conclusion of lines 226-227, as another possible explanation is that the target tasks are simply too easy for extra large data to show any benefit.\n\n# Minor Comments\nI found it surprising that training with Ours-All in Table 6 did not yield the best results in all target tasks. Why do the authors think that is?\n\nOverall I found the paper very interesting and well explained, and would consider raising my score if my main concerns are addressed.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper received mixed ratings, with three reviewers recommending acceptance and one rejection. The reviewers' main concerns include the novelty of the method, its motivation, missing comparison to some baselines, and the clarity of some parts of the paper. The paper went through several rounds of questions/answers, and the authors feedback addressed the concerns of most reviewers. The most negative reviewer did not update their score, but we believe that the latest feedback from the authors, on which the reviewer did not comment, better addressed some of the concerns raised by the reviewer. As such, we believe that the paper can be accepted to NeurIPS but strongly recommend the authors to incorporate their feedback in the final version.
Decision: Accept (Poster)