Dynamic Distillation Network for Cross-Domain Few-Shot Recognition with Unlabeled Data
Year: 2022

Abstract:
Most existing works in few-shot learning rely on meta-learning the network on a large base dataset which is typically from the same domain as the target dataset. We tackle the problem of cross-domain few-shot learning where there is a large shift between the base and target domain. The problem of cross-domain few-shot recognition with unlabeled target data is largely unaddressed in the literature. STARTUP was the first method that tackles this problem using self-training. However, it uses a fixed teacher pretrained on a labeled base dataset to create soft labels for the unlabeled target samples. As the base dataset and unlabeled dataset are from different domains, projecting the target images in the class-domain of the base dataset with a fixed pretrained model might be sub-optimal. We propose a simple dynamic distillation-based approach to facilitate unlabeled images from the novel/base dataset. We impose consistency regularization by calculating predictions from the weakly-augmented versions of the unlabeled images from a teacher network and matching it with the strongly augmented versions of the same images from a student network. The parameters of the teacher network are updated as exponential moving average of the parameters of the student network. We show that the proposed network learns representation that can be easily adapted to the target domain even though it has not been trained with target-specific classes during the pretraining phase. Our model outperforms the current state-of-the art method by 4.4% for 1-shot and 3.6% for 5-shot classification in the BSCD-FSL benchmark, and also shows competitive performance on traditional in-domain few-shot learning task. Our code is available at: https://git.io/Jilgs.

heading: 1 Introduction
text: The tremendous success of deep learning in visual recognition tasks is, to a great extent, attributed to the availability of large scale labeled datasets. While humans can recognize an object by looking only at a few examples, modern deep neural networks require hundreds or thousands of images for each category to achieve human-level visual recognition capability. This has led to the research on few-shot learning which aims at learning from a much smaller dataset. In a typical few-shot learning setting, there are two stages: meta-training and meta-testing. In the meta-training stage, a base dataset with labeled images is provided to train the model. In the meta-testing stage, the learned model is quickly adapted to a set of novel classes with only a few examples per class (the support set) and evaluated on a set of test images from the same novel classes (the query set). The base classes and novel classes are typically disjoint, but the images are obtained from the same domain. However, in many real world settings, training the model on a base dataset from the same domain as the target dataset is difficult and infeasible. Guo et al. [7] proposed a cross-domain few-shot benchmark, BSCD-FSL, which contains datasets from extremely different domains. In this benchmark, the meta-training is
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
done on a labeled source dataset, and the few-shot evaluation is performed on a target dataset which is from different domain than the source dataset. The benchmark shows that traditional pretraining and finetuning outperforms more complicated meta-learning based few-shot learning methods by a significant margin.
In the real-world scenarios, the target domain should have many unlabeled images, and it might be beneficial to use the unlabeled data to learn more target domain specific representations. We hypothesize that using both labeled base data and unlabeled target data during training provides a common embedding for both base and target domain. Then the natural question could be - why not use the unlabeled target data only, it might provide more target-specific representation. One issue with this approach is that self-supervised learning generally requires a large amount of unlabeled data to work, and, as pointed out by Phoo and Hariharan [18], plain self-supervised learning struggles to outperform the naive transfer learning baseline in few-shot learning setup. Secondly, it has been shown that combining supervised and unsupervised learning during training provides more transferable representation [9]. We argue that similar conclusion holds for cross-domain few-shot learning, i.e., combining supervised and unsupervised loss provides better representation for the downstream task. Figure 1 illustrates our experimental setup in contrast to traditional few-shot learning or cross-domain few-shot learning setup. We show that labeled images from the base dataset are still important to learn generic image features, and images from the target domain, even if unlabeled, can help developing more target domain specific representations.
Figure 2 illustrates our approach. Our goal is to train a feature extractor which will be used to evaluate few-shot learning performance on the target dataset. We propose a dynamic distillationbased approach to this end. The student network consists of an encoder fs and classifier gs, and the teacher network shares similar architecture as the student network (denoted as ft and gt). The classifier gs is a linear layer that predicts the class-logits of the samples from the base dataset. We calculate a supervised cross-entropy loss between the student’s predictions and ground-truth labels on the base dataset. For the unlabeled target data, we compute the teacher’s prediction for a weaklyaugmented version of an image and the student’s prediction for a strongly augmented version of the same image, and optimize a distillation loss to match the predictions. We also apply sharpening in the teacher prediction to encourage low-entropy prediction from the student. Both the supervised loss and distillation loss are used to learn the student’s weights. The teacher network is updated as a moving average of the student network. During few-shot evaluation, we only use the student
encoder fs as a feature extractor, learn a classifier head on the labeled support images consisting of few examples per category, and calculate the class predictions of the query images.
Our main contributions are:
• We propose a simple method for few-shot learning across extreme domain difference.
• We use dynamic distillation based approach that uses both labeled source data and unlabeled target data to learn a better representation for the few-shot evaluation on the target domain.
• Our method significantly outperforms the current state of the art in the BSCD-FSL benchmark with unlabeled images by 4.4% for 1-shot and 3.6% for 5-shot classification in terms of average top-1 accuracy. It even shows superior performance for in-domain few-shot classification on miniImageNet and tieredImageNet datasets.

heading: 2 Related Work
text: Few-shot classification Few-shot learning methods can be divided into three broad categories - generative [36], metric-base [21, 28, 23] and adaptation-based [10, 14]. Early work on few-shot learning was based on meta-learning [29, 23, 10, 13]. Matching Networks [29] uses cosine similarities on feature vectors produced by independently learned feature extractors, while Relation Networks [23] learn its own similarity metric. MAML [10] learns good initialization parameters that can be quickly adapted to a new task. Prototypical Networks [21] learn a feature extractor that is used to calculate distances between features of test images and the mean features of support images. MetaOptNet [14] uses a discriminatively trained linear predictor to learn representations for few-shot learning.
Self-training Self-training trains a student model that mimicks the predictions of a teacher model. Self-training can improves ImageNet classification [33]. It is also a dominant approach in semisupervised learning, where the teacher network is used to create pseudo [35, 34] or soft labels [33] for a huge set of unlabeled images, and the student network is trained to mimic the teacher.
Semi-supervised Learning Our method is inspired from recent developments in semi-supervised learning. Both [16] and [27] uses supervised cross-entropy loss with unsupervised regularization loss. Pseudo-labeling based approaches first train a model on a labeled dataset, use the trained model to create pseudo-labels of the unlabeled samples, and retrain the model with both labeled and pseudo-labeled samples [12, 1]. FixMatch [22] proposes a simplified model which simultaneously optimizes cross-entropy loss on the labeled samples and generates pseudo labels using the model’s prediction on weakly-augmented unlabeled images. If the pseudo labels are confident enough, the model is trained to predict the pseudo labels with a strongly augmented version of the same images. We adopt a similar approach by imposing consistency regularization. However, while FixMatch is a semi-supervised technique where the unlabeled data is assumed to be from the same domain, our approach is applicable to the cross-domain few-shot learning problem. We also calculates the prediction from a mean teacher network instead of using the same network as FixMatch.
Cross-domain few-shot learning Guo et al. [7] proposed a cross-domain few-shot learning benchmark, and noted that existing state-of-the-art approaches fail to achieve good accuracy on this benchmark. One potential solution could be to use an unlabeled dataset from the target to learn representations that are adaptable to a completely different domain. Many approaches also explored few-shot learning with unlabeled data [11, 15, 19]; however, most of these works still assume a smaller gap between the base and target domains. Our method shares some similarity with the recently developed STARTUP [18] method for cross-domain few-shot learning. STARTUP also uses unlabeled data for learning a better representation. However, STARTUP uses a fixed pretrained model to produce pseudo labels for the unlabeled samples, and then train the network with the labeled base dataset and pseudo-labeled target dataset. Additionally, STARTUP also incorporates a self-supervised contrastive loss on the unlabeled images to improve accuracy, where our method does not require additional contrastive loss. Actually, we argue that our distillation loss works like a self-supervised non-contrastive loss, similar to BYOL [6], for which we might not need to add any extra self-supervised loss. We propose a dynamic distillation approach, where the parameters of the teacher network are updated during training. We obtain the prediction for the weakly-augmented version of an unlabeled image from the teacher network, and optimizes the model such that the
prediction of the strongly augmented version of the same image obtained from the student network matches that of teacher network. Note that FixMatch [22] also uses similar consistency regularization loss for semi-supervised learning. To our knowledge, we are the first to use consistency regularization and dynamic distillation for cross-domain few-shot learning.

heading: 3 Methodology

heading: 3.1 Preliminary
text: Few-shot Learning Formulation A few-shot learning task consists of a support set S, which containing K data points from N classes for N -way K-shot task, and a query Q = {xi}mi=1 consisting of data points only from the N classes of the support set. The goal is to classify the query points with the help of the labeled support set. In the typical few-shot learning setting, (1) an embedding is learned from the base/source dataset DS , (2) a linear classifier is learned on top of the fixed embedding on the support set, and (3) the classifications of the query data points are determined.
Cross-domain Few-shot Learning The difference between the typical few-shot learning setup and cross-domain few-shot learning is that the base/source dataset is drawn from a very different domain than the target domain. Additionally, in our setting, we are provided unlabeled data points DU = {xi}NUi=1 from the target domain. The unlabeled dataset contains more classes than the support set. Given the base dataset DS , and an unlabeled set DU , we need to learn an embedding that can extract a representation that can be used for few-shot learning evaluation in the target-domain.

heading: 3.2 Proposed Method
text: Encoder We facilitate knowledge distillation to train our base encoder on both source datatset and target dataset. Denote the embedding network as fs that embeds an input image x to a d-dimensional vector fs(x). We add a classifier header gs on top of fs, which predicts nc logits from the embeddings, where nc is the total number of classes in the base dataset. Since the labels of the data points of the
base dataset are provided, we calculate the supervised cross-entropy loss:
lCE(y, p) = H(y, p) (1)
where p = Softmax(gs(fs(x))), and H(a, b) = −a log b.
Dynamic distillation Additionally, we also have a teacher encoder ft and teacher classifier gt. The task of the teacher network is to produce pseudo labels for the unlabeled images. Given an image xi from the unlabeled set DU , we compute the model’s prediction pwi from a weakly-augmented version (denoted as xwi ) and p s i from a strongly-augmented version (denoted as x s i ) of the unlabeled image. The prediction of the weakly-augmented version is produced from the teacher network, which serves as a soft-target for the strongly-augmented images. We use the student network to get the prediction psi for the strongly-augmented images. Specifically,
psi = Softmax(gs(fs(x s i ))); p w i = Softmax(gt(ft(x w i ))/τ) (2)
where τ is a sharpening parameter. Note that we do not let gradient pass through the teacher network. We calculate the distillation loss by the cross-entropy function
lU (p w i , p s i ) = H(p w i , p s i ) (3)
Eq. 3 works like a consistency regularizer so that the network predicts similar scores for different augmented versions of the image. We can also consider Eq. 3 as self-supervised loss, similar to BYOL [6] or DINO [2]. However, one major difference is that - in BYOL or DINO the projection head is a random linear layer, where, we are using the supervised classification head as the projection head.
The total loss function is:
L = 1 NS ∑ (xi,yi)∈DS lCE(y, p) + λ 1 NU ∑ xi∈DU lU (p w i , p s i ) (4)
where λ is a hyper-parameter. The loss function is used to update the parameters of the student network. For the teacher network, we use mean teacher approach [24], i.e., we update the teacher weights θt from the student weights θs by: θt = mθt + (1 − m)θs, where m is the momentum parameter. Note that when m = 1, we are essentially using fixed teacher, and when m = 0, the teacher and student share the same model. When m > 0, the teacher network is a moving average of the student network denoting the distillation process as dynamic. Please refer to the Appendix for PyTorch-like pseudo-code of our method.

heading: 4 Experiments

heading: 4.1 Experimental Setup
text: Dataset We follow the evaluation protocol of the BSCD-FSL benchmark [7], which contains novel data from CropDisease [17], EuroSAT [8], ISIC [4], and ChestX [32]. The base dataset is miniImageNet [29], which contains 100 classes from ImageNet dataset [5] where each class has 600 images. The novel datasets are chosen based on increasing dissimilarity from the miniImageNet dataset. More details about the datasets are provided in the Appendix. Following [18], we randomly sample 20% of the data from each novel dataset to construct the unlabeled set DU , and the remaining images are used for evaluation, where we perform 5-way 1-shot and 5-way 5-shot classification. For evaluation metric, we report top-1 accuracy and 95% confidence interval over 600 runs. We also report evaluation results on the larger tieredImageNet dataset [19].
Implementation details We use ResNet-10 as the backbone network [7, 18]. Our pretraining has two steps. In the first step, we train our network only on the miniImageNet dataset for 200 epochs. We use SGD with momentum 0.9, weight decay 1e-4, learning rate 0.01, batch size 32, and the cosine learning rate scheduler. In the next step, we use the miniImageNet-pretrained network, and use both the base dataset and the unlabeled dataset to optimize the loss function in Eq. 4 for 60 epochs. During training, we increase the weight of distillation loss λ from 0 to 1 until 40 epochs
using cosine scheduling. The sharpening temperature and teacher momentum parameter are set to 0.1 and 0.99 respectively. For the base images and weakly-augmented unlabeled images, we use the random-resize-crop, horizontal flip and normalization augmentations. For strong augmentation, we additionally use the color jitter, Gaussian blur, and random gray scale transformations. The other hyperparameters are kept the same. Refer to the supplementary for more details about hyper-parameter selection. For few-shot evaluation, we learn a logistic regression classifier on the support set, and evaluate on the query set.

heading: 4.2 Main Results
text: Comparison to State-of-the-arts Table 1 shows the performance comparison of our approach with other methods on the BSCD-FSL benchmark. All models are trained on the miniImageNet dataset. “Transfer” denotes the baseline trained by cross-entropy loss on the base dataset. “SimCLR” is trained only on the unlabeled images. As noted by [9], self-supervised contrastive learning learns better transferable representation for a different downstream domain. Hence, we also create a “SimCLR(Base)” baseline that trains the encoder by optimizing self-supervised contrastive loss on the base dataset. “Transfer+SimCLR” refers to the model which is trained with supervised cross-entropy loss from the base dataset and self-supervised contrastive loss from the unlabeled target dataset, which has been reported to show superior transferability across domain [9]. STARTUP [18] is trained with three losses: cross-entropy loss on the base dataset, KL-divergence loss on the unlabeled dataset similar, and self-supervised contrastive loss on unlabeled images. STARTUP, Transfer+SimCLR and our approach use both the base dataset and additional unlabeled dataset during the representation learning phase.
Our method outperforms all meta-learning-based approaches by a significant margin at all settings. Moreover, compared to Transfer, we achieve more than ∼5.5% improvement for 5-shot classification on average. The performance improvement on 1-shot is more significant; we achieve 7.9% improvement on average.
We outperform STARTUP by 5.5% on EuroSAT, 3.1% on CropDisease, 2.1% on ChestX, and 3.6% on ISIC for 5-way 5-shot classification. The performance improvement is also quite significant for 1-shot classification; specifically, our method achieves 8.8% improvement on EuroSAT and 7.7% improvement on CropDisease dataset over STARTUP. We only perform slightly worse on the ChestX dataset. For ChestX, it seems that pure unsupervised learning performs pretty well. Considering that our method does not use any self-supervised training or distillation, the performance improvement is impressive. Note that STARTUP uses a fixed teacher to extract pseudo-labels for the unlabeled images, whereas we extract pseudo labels from the weakly-augmented images from the same network that is being trained. In that sense, our model works like a dynamic teacher, where the pseudo labels get more refined as training progress. We hypothesize that the superior performance might be attributed to the dynamic approach of our model over STARTUP.
Results with tieredImageNet base data tieredImageNet [19] is a subset of ImageNet dataset with 608 classes. The classes are grouped into 34 super-categories, from which 20 training categories (351 classes), 6 validation categories (97 classes), and 8 testing categories (160 classes) are selected. We use a larger backbone ResNet-18 for meta-training with tieredImageNet [25]. Table 2 shows the performance comparison with other models. We get similar conclusion as we get from Table 1. Particularly, we get 6.72% average improvement for 1-shot and 3.66% average improvement for 5-shot over STARTUP. However, we do not see much better accuracy than miniImageNet pretrained models even though we are using a much larger base dataset, which suggests that the size of base dataset is not as important as other transfer learning task in cross domain few-shot learning.
Few-shot performance on similar domain It has been shown that self-training improves ImageNet classification [33]. Given the distillation approach of our model, one could expect that it might improve the few-shot accuracy even when the target data come from the same domain. Note that STARTUP does not improve the in-domain accuracy [18], even though it is also a self-training based model. We evaluate on miniImageNet and tieredImageNet dataset in terms of 5-way 1-shot and 5-way 5-shot performance. We use the official training split as base dataset, and the unlabeled target data are obtained from 20% of the novel (test) set and the rest are used for evaluation. For backbone, we use ResNet-10 for miniImageNet and ResNet-18 for tieredImageNet. Table 3 reports the results for in-domain few-shot performance. We see that our method achieves the best performance among the baselines. Particularly, for 1-shot learning in tieredImageNet, our model outperforms the best one by 7.7%. We infer that our method can be safely applied to few-shot learning task when the domain gap between base and target dataset is small, which is in contrast with STARTUP that does not show improvement over “Transfer" for few-shot learning on similar domain.
4.3 Analysis
Effect of dynamic distillation To understand how distillation helps to learn better representation, we use the pretrained models to extract features of the target dataset. Then we use KMeans algorithm to create clusters from the features. The number of clusters in the KMeans is set to be the number of classes of the target dataset. In Table
4, the V-measurement cluster scores [20] between the KMeans clusters and original ground-truth are shown. The V-score has 100% value when there is maximum agreement between ground-truth and predicted clusters, and 0% when there’s no agreement. Table 4 shows that our method achieves higher v-scores for EuroSAT and CropDisease dataset, and the v-scores for ISIC and ChestX are also very competitive. It suggests that our model learns a good clustering of the target data even when we are not using any target labels. The clustering is much better when the domain gap is not extreme.
Fig. 3 shows t-SNE plots [26] from 10 representative classes from the CropDisease and EuroSAT datasets. We compare the embeddings extracted from “Transfer” and our approach. We see that our method creates better grouping on the embeddings of the target datasets, even though we do not use any labels for the target dataset during pretraining.
Comparison with self-supervised learning If we ignore the supervised loss, our model has similarity with self-supervised non-contrastive loss similar to BYOL or DINO. However, the projection head we are using to calculate the final predictions of the two different views of an unlabeled image is the same classification head that is used to predict the classification logits of the labeled base samples. In Table 5, “Ours (distillation head)” represents the model where we use separate projection head for the predictions of the unlabeled images. We see that separate projection head performs much worse. We found that the distillation loss is simply converging to a trivial solution in this case. To discourage trivial solution, we add recently developed tricks in self-supervised learning, namely, centering and strong augmentation - which turns the unlabeled branch similar to ResNet DINO [2]. “Ours (DINO head)” achieves better accuracy than “Ours (distillation head)”, suggesting that it alleviates the issue of trivial solution. However, our original method still achieves a significant 3.07% more improvement. It is interesting to note that a separate projection head causes trivial solution for the unlabeled images, whereas our model does not converge to trivial solution with similar settings. We infer that using a supervised classifier linear layer as the projection head can solve the issue of trivial solution for the self-supervised learning to some extent without requiring extra tricks like BatchNorm [6] or centering [2]. Additionally, it provides a better clustering of the unlabeled features, even if the unlabeled samples come from different domain than the labeled samples. Table 5 also shows results for “Ours + SimCLR”, which simply adds a SimCLR loss for the unlabeled samples. It performs slightly better only in ChestX dataset. On average, the performance is similar to “Ours”, which signifies that there is no clear benefit using a self-supervised contrastive loss to achieve better transferability for our method. Note that STARTUP comes to a different conclusion reporting that adding SimCLR loss consistently improves the performance.
Experiment of few-shot classification on fine-grained dataset CUB [30] contains 200 classes and 11,788 images of different bird species. ‘miniImageNet->CUB’ is an interesting experiment to show the transferability of different models to a fine-grained dataset. We report the results in Table 6 in terms of 5-way 5-shot scores. For CUB, we found that vanilla Transfer performs surprisingly well (also reported by [3]), and adding SimCLR with Transfer (Transfer+SimCLR) actually decreases the accuracy. Wallace and Hariharan [31] also experimented with different self-supervised methods on smaller domain and found that all of them under-perform for fine-grained task [2]. However, our method still performs the best, demonstrating the effectiveness of our approach in a fine-grained downstream task.
Experiment of distillation with unlabeled datasets from different domain Table 7 reports the few-shot accuracy when our model is trained on different unlabeled datasets. The best accuracy is achieved when the unlabeled data and target data are from the same domain. Even if the unlabeled data consists of images from multiple domains including the target domain (denoted as “Ours-all”), it still significantly under-performs the base model.
Effect of data augmentation On the unlabeled images, we apply two types of augmentation: weak augmentation to extract pseudo labels and strong augmentation to impose consistency regularization. This setting is denoted as “weak-strong” (w-s), where ‘weak’ (w) augmentation is applied to the image that is fed into the teacher network and ‘strong’ (s) augmentation is applied to the image that is fed into the student network. We also show results with “weak-weak” (w-w), “strong-weak” (s-w) and “strong-strong” (s-s) augmentation settings in Figure 4a. Both “weak-weak” and “strong-strong” perform worse than the other augmentations. We also note that in self-supervised learning, generally strong augmentation is applied to all training images to get good performance. In our experiment, we find that applying weak augmentation in one of the image pairs improves the performance.
More unlabeled data To measure the effect of amount of unlabeled dataset during pretraining, we divide the target dataset by 80% and 20% splits. The 20% split is used for evaluation. From the 80% split, we vary the amount of unlabeled data and then pretrain with source and the unlabeled set. Fig. 4b shows the average 5-shot accuracy for different amounts of the unlabeled dataset during the pretraining phase. As expected, more information from the unlabeled dataset helps to learn better representations on the target domain. However, the performance saturates later, and we get diminishing return for more unlabeled data. It also signifies that there are scopes to improve the performance by using more unlabeled data denoting potential future research direction.

heading: 4.4 Addition Ablation Studies
text: We perform several ablation studies of different components of our approach. All scores are reported for 5-way 5-shot evaluation. More ablations are provided in the Appendix.
Longer training We use the base dataset pretrained network as initialization for our network, and then train on both base dataset and unlabeled dataset for 60 epochs. Figure 4c reports average 5-way 5-shot few-shot on the BSCD-FSL benchmark performance for our pretrained with longer training epoch. Training for more epochs can result in minor improvement (0.2%).
Effect of base dataset Here, we perform experiments without the first term in Eq. 4, i.e., we train the network on the base dataset first and then re-train only on the unlabeled images (without joint training on the base dataset), denoted as “Ours (w/o base)”. Table 8 shows that the performance of “Ours (w/o base)” is poor, suggesting that the representation related to the labeled base dataset is still helpful to the target domain.
Training without pretrained model on base dataset We perform 2-step training during the representation learning phase - we first train the model on mini-IN only, and then jointly train on mini-IN and unlabeled images. In Table 8, “Ours(1-step)” denotes training on the unlabeled images and mini-IN from scratch for 300 epochs, which performs worse than 2-step training. Our assumption is that the proposed model is also like self-training where a well-trained teacher is needed.

heading: 5 Conclusion
text: We introduced a novel approach to utilize unlabeled data from the target domain for cross-domain fewshot learning. Experiments show that our method achieves state-of-the-art results in the BSCD-FSL benchmark for both 1-shot and 5-shot classification. Our model also outperforms other approaches in the same-domain few-shot learning. Future work can be focused on applying our approach in each episode during meta-testing so that the model can learn more category-specific representations.

heading: 6 Broader Impact
text: The approach tackles a practical problem of the existing few-shot learning setup that the base dataset and the novel samples generally come from different domain. Our work uses the unlabeled samples from the target dataset to learn more target specific representation. Like any other machine learning tool, the final impact depends on the intention of the people or institution applying it. However, it can be useful in drug discovery or medical image analysis where labeled dataset is limited.

heading: 7 Acknowledgments
text: This material is based upon work supported by the U.S. Department of Homeland Security, Science and Technology Directorate, Office of University Programs, under Grant Award 2013-ST-061ED0001. The views and conclusions contained in this document are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the U.S. Department of Homeland Security.

Review summaries:
The submission attempts to address the cross-domain few-shot learning problem by utilizing dynamic distillation from additional unlabeled target data. It simplifies the STARTUP framework by removing the contrastive loss and using a similar network for teacher and student. Moreover, it introduces different data augmentations to learn better representations. The teacher network is also trained dynamically to enhance the training of the student network.

The paper proposes a method for training on “cross-domain few-shot recognition”. This is a task in which the goal is to transfer representations learned on a base, labeled dataset into a target few-shot dataset that is of a different domain. In particular, the paper focuses on the setting where unlabeled data is available for the target domain.  The method proposed comprises of training a “student model” on the base data in a supervised manner, while also utilizing a distillation loss on the target data, using a “teacher network”’s predictions as the objective. The teacher is set as an exponential moving average of the student’s parameters. After this initial training phase, the student gets a new randomly initialized head to learn on the few-shot labeled target data. 

This paper tackles the cross-domain few-shot learning, by utilizing the unlabeled target data to encourage the feature extractor to learns the representations that is effective for target domain. They propose a dynamic distillation where the teacher is dynamically updated via exponential moving average during training. The dynamic distillation works similar to recent self-supervised learning method of BYOL, and successfully enhance the generalization capability of learned representation using unlabeled data. Dynamic distillation network achieves the state-of-the-art performance in BSCD-FSL benchmark.