Dynamic Distillation Network for Cross-Domain Few-Shot Recognition with Unlabeled Data

1 Introduction . The tremendous success of deep learning in visual recognition tasks is , to a great extent , attributed to the availability of large scale labeled datasets . While humans can recognize an object by looking only at a few examples , modern deep neural networks require hundreds or thousands of images for each category to achieve human-level visual recognition capability . This has led to the research on few-shot learning which aims at learning from a much smaller dataset . In a typical few-shot learning setting , there are two stages : meta-training and meta-testing . In the meta-training stage , a base dataset with labeled images is provided to train the model . In the meta-testing stage , the learned model is quickly adapted to a set of novel classes with only a few examples per class ( the support set ) and evaluated on a set of test images from the same novel classes ( the query set ) . The base classes and novel classes are typically disjoint , but the images are obtained from the same domain . However , in many real world settings , training the model on a base dataset from the same domain as the target dataset is difficult and infeasible . Guo et al . [ 7 ] proposed a cross-domain few-shot benchmark , BSCD-FSL , which contains datasets from extremely different domains . In this benchmark , the meta-training is 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . done on a labeled source dataset , and the few-shot evaluation is performed on a target dataset which is from different domain than the source dataset . The benchmark shows that traditional pretraining and finetuning outperforms more complicated meta-learning based few-shot learning methods by a significant margin . In the real-world scenarios , the target domain should have many unlabeled images , and it might be beneficial to use the unlabeled data to learn more target domain specific representations . We hypothesize that using both labeled base data and unlabeled target data during training provides a common embedding for both base and target domain . Then the natural question could be - why not use the unlabeled target data only , it might provide more target-specific representation . One issue with this approach is that self-supervised learning generally requires a large amount of unlabeled data to work , and , as pointed out by Phoo and Hariharan [ 18 ] , plain self-supervised learning struggles to outperform the naive transfer learning baseline in few-shot learning setup . Secondly , it has been shown that combining supervised and unsupervised learning during training provides more transferable representation [ 9 ] . We argue that similar conclusion holds for cross-domain few-shot learning , i.e. , combining supervised and unsupervised loss provides better representation for the downstream task . Figure 1 illustrates our experimental setup in contrast to traditional few-shot learning or cross-domain few-shot learning setup . We show that labeled images from the base dataset are still important to learn generic image features , and images from the target domain , even if unlabeled , can help developing more target domain specific representations . Figure 2 illustrates our approach . Our goal is to train a feature extractor which will be used to evaluate few-shot learning performance on the target dataset . We propose a dynamic distillationbased approach to this end . The student network consists of an encoder fs and classifier gs , and the teacher network shares similar architecture as the student network ( denoted as ft and gt ) . The classifier gs is a linear layer that predicts the class-logits of the samples from the base dataset . We calculate a supervised cross-entropy loss between the student ’ s predictions and ground-truth labels on the base dataset . For the unlabeled target data , we compute the teacher ’ s prediction for a weaklyaugmented version of an image and the student ’ s prediction for a strongly augmented version of the same image , and optimize a distillation loss to match the predictions . We also apply sharpening in the teacher prediction to encourage low-entropy prediction from the student . Both the supervised loss and distillation loss are used to learn the student ’ s weights . The teacher network is updated as a moving average of the student network . During few-shot evaluation , we only use the student encoder fs as a feature extractor , learn a classifier head on the labeled support images consisting of few examples per category , and calculate the class predictions of the query images . Our main contributions are : • We propose a simple method for few-shot learning across extreme domain difference . • We use dynamic distillation based approach that uses both labeled source data and unlabeled target data to learn a better representation for the few-shot evaluation on the target domain . • Our method significantly outperforms the current state of the art in the BSCD-FSL benchmark with unlabeled images by 4.4 % for 1-shot and 3.6 % for 5-shot classification in terms of average top-1 accuracy . It even shows superior performance for in-domain few-shot classification on miniImageNet and tieredImageNet datasets . 2 Related Work . Few-shot classification Few-shot learning methods can be divided into three broad categories - generative [ 36 ] , metric-base [ 21 , 28 , 23 ] and adaptation-based [ 10 , 14 ] . Early work on few-shot learning was based on meta-learning [ 29 , 23 , 10 , 13 ] . Matching Networks [ 29 ] uses cosine similarities on feature vectors produced by independently learned feature extractors , while Relation Networks [ 23 ] learn its own similarity metric . MAML [ 10 ] learns good initialization parameters that can be quickly adapted to a new task . Prototypical Networks [ 21 ] learn a feature extractor that is used to calculate distances between features of test images and the mean features of support images . MetaOptNet [ 14 ] uses a discriminatively trained linear predictor to learn representations for few-shot learning . Self-training Self-training trains a student model that mimicks the predictions of a teacher model . Self-training can improves ImageNet classification [ 33 ] . It is also a dominant approach in semisupervised learning , where the teacher network is used to create pseudo [ 35 , 34 ] or soft labels [ 33 ] for a huge set of unlabeled images , and the student network is trained to mimic the teacher . Semi-supervised Learning Our method is inspired from recent developments in semi-supervised learning . Both [ 16 ] and [ 27 ] uses supervised cross-entropy loss with unsupervised regularization loss . Pseudo-labeling based approaches first train a model on a labeled dataset , use the trained model to create pseudo-labels of the unlabeled samples , and retrain the model with both labeled and pseudo-labeled samples [ 12 , 1 ] . FixMatch [ 22 ] proposes a simplified model which simultaneously optimizes cross-entropy loss on the labeled samples and generates pseudo labels using the model ’ s prediction on weakly-augmented unlabeled images . If the pseudo labels are confident enough , the model is trained to predict the pseudo labels with a strongly augmented version of the same images . We adopt a similar approach by imposing consistency regularization . However , while FixMatch is a semi-supervised technique where the unlabeled data is assumed to be from the same domain , our approach is applicable to the cross-domain few-shot learning problem . We also calculates the prediction from a mean teacher network instead of using the same network as FixMatch . Cross-domain few-shot learning Guo et al . [ 7 ] proposed a cross-domain few-shot learning benchmark , and noted that existing state-of-the-art approaches fail to achieve good accuracy on this benchmark . One potential solution could be to use an unlabeled dataset from the target to learn representations that are adaptable to a completely different domain . Many approaches also explored few-shot learning with unlabeled data [ 11 , 15 , 19 ] ; however , most of these works still assume a smaller gap between the base and target domains . Our method shares some similarity with the recently developed STARTUP [ 18 ] method for cross-domain few-shot learning . STARTUP also uses unlabeled data for learning a better representation . However , STARTUP uses a fixed pretrained model to produce pseudo labels for the unlabeled samples , and then train the network with the labeled base dataset and pseudo-labeled target dataset . Additionally , STARTUP also incorporates a self-supervised contrastive loss on the unlabeled images to improve accuracy , where our method does not require additional contrastive loss . Actually , we argue that our distillation loss works like a self-supervised non-contrastive loss , similar to BYOL [ 6 ] , for which we might not need to add any extra self-supervised loss . We propose a dynamic distillation approach , where the parameters of the teacher network are updated during training . We obtain the prediction for the weakly-augmented version of an unlabeled image from the teacher network , and optimizes the model such that the prediction of the strongly augmented version of the same image obtained from the student network matches that of teacher network . Note that FixMatch [ 22 ] also uses similar consistency regularization loss for semi-supervised learning . To our knowledge , we are the first to use consistency regularization and dynamic distillation for cross-domain few-shot learning . 3 Methodology . 3.1 Preliminary . Few-shot Learning Formulation A few-shot learning task consists of a support set S , which containing K data points from N classes for N -way K-shot task , and a query Q = { xi } mi=1 consisting of data points only from the N classes of the support set . The goal is to classify the query points with the help of the labeled support set . In the typical few-shot learning setting , ( 1 ) an embedding is learned from the base/source dataset DS , ( 2 ) a linear classifier is learned on top of the fixed embedding on the support set , and ( 3 ) the classifications of the query data points are determined . Cross-domain Few-shot Learning The difference between the typical few-shot learning setup and cross-domain few-shot learning is that the base/source dataset is drawn from a very different domain than the target domain . Additionally , in our setting , we are provided unlabeled data points DU = { xi } NUi=1 from the target domain . The unlabeled dataset contains more classes than the support set . Given the base dataset DS , and an unlabeled set DU , we need to learn an embedding that can extract a representation that can be used for few-shot learning evaluation in the target-domain . 3.2 Proposed Method . Encoder We facilitate knowledge distillation to train our base encoder on both source datatset and target dataset . Denote the embedding network as fs that embeds an input image x to a d-dimensional vector fs ( x ) . We add a classifier header gs on top of fs , which predicts nc logits from the embeddings , where nc is the total number of classes in the base dataset . Since the labels of the data points of the base dataset are provided , we calculate the supervised cross-entropy loss : lCE ( y , p ) = H ( y , p ) ( 1 ) where p = Softmax ( gs ( fs ( x ) ) ) , and H ( a , b ) = −a log b . Dynamic distillation Additionally , we also have a teacher encoder ft and teacher classifier gt . The task of the teacher network is to produce pseudo labels for the unlabeled images . Given an image xi from the unlabeled set DU , we compute the model ’ s prediction pwi from a weakly-augmented version ( denoted as xwi ) and p s i from a strongly-augmented version ( denoted as x s i ) of the unlabeled image . The prediction of the weakly-augmented version is produced from the teacher network , which serves as a soft-target for the strongly-augmented images . We use the student network to get the prediction psi for the strongly-augmented images . Specifically , psi = Softmax ( gs ( fs ( x s i ) ) ) ; p w i = Softmax ( gt ( ft ( x w i ) ) /τ ) ( 2 ) where τ is a sharpening parameter . Note that we do not let gradient pass through the teacher network . We calculate the distillation loss by the cross-entropy function lU ( p w i , p s i ) = H ( p w i , p s i ) ( 3 ) Eq . 3 works like a consistency regularizer so that the network predicts similar scores for different augmented versions of the image . We can also consider Eq . 3 as self-supervised loss , similar to BYOL [ 6 ] or DINO [ 2 ] . However , one major difference is that - in BYOL or DINO the projection head is a random linear layer , where , we are using the supervised classification head as the projection head . The total loss function is : L = 1 NS ∑ ( xi , yi ) ∈DS lCE ( y , p ) + λ 1 NU ∑ xi∈DU lU ( p w i , p s i ) ( 4 ) where λ is a hyper-parameter . The loss function is used to update the parameters of the student network . For the teacher network , we use mean teacher approach [ 24 ] , i.e. , we update the teacher weights θt from the student weights θs by : θt = mθt + ( 1 − m ) θs , where m is the momentum parameter . Note that when m = 1 , we are essentially using fixed teacher , and when m = 0 , the teacher and student share the same model . When m > 0 , the teacher network is a moving average of the student network denoting the distillation process as dynamic . Please refer to the Appendix for PyTorch-like pseudo-code of our method .

Summary: The paper proposes a method for training on “cross-domain few-shot recognition”. This is a task in which the goal is to transfer representations learned on a base, labeled dataset into a target few-shot dataset that is of a different domain. In particular, the paper focuses on the setting where unlabeled data is available for the target domain.  The method proposed comprises of training a “student model” on the base data in a supervised manner, while also utilizing a distillation loss on the target data, using a “teacher network”’s predictions as the objective. The teacher is set as an exponential moving average of the student’s parameters. After this initial training phase, the student gets a new randomly initialized head to learn on the few-shot labeled target data.