Dynamic Distillation Network for Cross-Domain Few-Shot Recognition with Unlabeled Data

1 Introduction . The tremendous success of deep learning in visual recognition tasks is , to a great extent , attributed to the availability of large scale labeled datasets . While humans can recognize an object by looking only at a few examples , modern deep neural networks require hundreds or thousands of images for each category to achieve human-level visual recognition capability . This has led to the research on few-shot learning which aims at learning from a much smaller dataset . In a typical few-shot learning setting , there are two stages : meta-training and meta-testing . In the meta-training stage , a base dataset with labeled images is provided to train the model . In the meta-testing stage , the learned model is quickly adapted to a set of novel classes with only a few examples per class ( the support set ) and evaluated on a set of test images from the same novel classes ( the query set ) . The base classes and novel classes are typically disjoint , but the images are obtained from the same domain . However , in many real world settings , training the model on a base dataset from the same domain as the target dataset is difficult and infeasible . Guo et al . [ 7 ] proposed a cross-domain few-shot benchmark , BSCD-FSL , which contains datasets from extremely different domains . In this benchmark , the meta-training is 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . done on a labeled source dataset , and the few-shot evaluation is performed on a target dataset which is from different domain than the source dataset . The benchmark shows that traditional pretraining and finetuning outperforms more complicated meta-learning based few-shot learning methods by a significant margin . In the real-world scenarios , the target domain should have many unlabeled images , and it might be beneficial to use the unlabeled data to learn more target domain specific representations . We hypothesize that using both labeled base data and unlabeled target data during training provides a common embedding for both base and target domain . Then the natural question could be - why not use the unlabeled target data only , it might provide more target-specific representation . One issue with this approach is that self-supervised learning generally requires a large amount of unlabeled data to work , and , as pointed out by Phoo and Hariharan [ 18 ] , plain self-supervised learning struggles to outperform the naive transfer learning baseline in few-shot learning setup . Secondly , it has been shown that combining supervised and unsupervised learning during training provides more transferable representation [ 9 ] . We argue that similar conclusion holds for cross-domain few-shot learning , i.e. , combining supervised and unsupervised loss provides better representation for the downstream task . Figure 1 illustrates our experimental setup in contrast to traditional few-shot learning or cross-domain few-shot learning setup . We show that labeled images from the base dataset are still important to learn generic image features , and images from the target domain , even if unlabeled , can help developing more target domain specific representations . Figure 2 illustrates our approach . Our goal is to train a feature extractor which will be used to evaluate few-shot learning performance on the target dataset . We propose a dynamic distillationbased approach to this end . The student network consists of an encoder fs and classifier gs , and the teacher network shares similar architecture as the student network ( denoted as ft and gt ) . The classifier gs is a linear layer that predicts the class-logits of the samples from the base dataset . We calculate a supervised cross-entropy loss between the student ’ s predictions and ground-truth labels on the base dataset . For the unlabeled target data , we compute the teacher ’ s prediction for a weaklyaugmented version of an image and the student ’ s prediction for a strongly augmented version of the same image , and optimize a distillation loss to match the predictions . We also apply sharpening in the teacher prediction to encourage low-entropy prediction from the student . Both the supervised loss and distillation loss are used to learn the student ’ s weights . The teacher network is updated as a moving average of the student network . During few-shot evaluation , we only use the student encoder fs as a feature extractor , learn a classifier head on the labeled support images consisting of few examples per category , and calculate the class predictions of the query images . Our main contributions are : • We propose a simple method for few-shot learning across extreme domain difference . • We use dynamic distillation based approach that uses both labeled source data and unlabeled target data to learn a better representation for the few-shot evaluation on the target domain . • Our method significantly outperforms the current state of the art in the BSCD-FSL benchmark with unlabeled images by 4.4 % for 1-shot and 3.6 % for 5-shot classification in terms of average top-1 accuracy . It even shows superior performance for in-domain few-shot classification on miniImageNet and tieredImageNet datasets . 2 Related Work . Few-shot classification Few-shot learning methods can be divided into three broad categories - generative [ 36 ] , metric-base [ 21 , 28 , 23 ] and adaptation-based [ 10 , 14 ] . Early work on few-shot learning was based on meta-learning [ 29 , 23 , 10 , 13 ] . Matching Networks [ 29 ] uses cosine similarities on feature vectors produced by independently learned feature extractors , while Relation Networks [ 23 ] learn its own similarity metric . MAML [ 10 ] learns good initialization parameters that can be quickly adapted to a new task . Prototypical Networks [ 21 ] learn a feature extractor that is used to calculate distances between features of test images and the mean features of support images . MetaOptNet [ 14 ] uses a discriminatively trained linear predictor to learn representations for few-shot learning . Self-training Self-training trains a student model that mimicks the predictions of a teacher model . Self-training can improves ImageNet classification [ 33 ] . It is also a dominant approach in semisupervised learning , where the teacher network is used to create pseudo [ 35 , 34 ] or soft labels [ 33 ] for a huge set of unlabeled images , and the student network is trained to mimic the teacher . Semi-supervised Learning Our method is inspired from recent developments in semi-supervised learning . Both [ 16 ] and [ 27 ] uses supervised cross-entropy loss with unsupervised regularization loss . Pseudo-labeling based approaches first train a model on a labeled dataset , use the trained model to create pseudo-labels of the unlabeled samples , and retrain the model with both labeled and pseudo-labeled samples [ 12 , 1 ] . FixMatch [ 22 ] proposes a simplified model which simultaneously optimizes cross-entropy loss on the labeled samples and generates pseudo labels using the model ’ s prediction on weakly-augmented unlabeled images . If the pseudo labels are confident enough , the model is trained to predict the pseudo labels with a strongly augmented version of the same images . We adopt a similar approach by imposing consistency regularization . However , while FixMatch is a semi-supervised technique where the unlabeled data is assumed to be from the same domain , our approach is applicable to the cross-domain few-shot learning problem . We also calculates the prediction from a mean teacher network instead of using the same network as FixMatch . Cross-domain few-shot learning Guo et al . [ 7 ] proposed a cross-domain few-shot learning benchmark , and noted that existing state-of-the-art approaches fail to achieve good accuracy on this benchmark . One potential solution could be to use an unlabeled dataset from the target to learn representations that are adaptable to a completely different domain . Many approaches also explored few-shot learning with unlabeled data [ 11 , 15 , 19 ] ; however , most of these works still assume a smaller gap between the base and target domains . Our method shares some similarity with the recently developed STARTUP [ 18 ] method for cross-domain few-shot learning . STARTUP also uses unlabeled data for learning a better representation . However , STARTUP uses a fixed pretrained model to produce pseudo labels for the unlabeled samples , and then train the network with the labeled base dataset and pseudo-labeled target dataset . Additionally , STARTUP also incorporates a self-supervised contrastive loss on the unlabeled images to improve accuracy , where our method does not require additional contrastive loss . Actually , we argue that our distillation loss works like a self-supervised non-contrastive loss , similar to BYOL [ 6 ] , for which we might not need to add any extra self-supervised loss . We propose a dynamic distillation approach , where the parameters of the teacher network are updated during training . We obtain the prediction for the weakly-augmented version of an unlabeled image from the teacher network , and optimizes the model such that the prediction of the strongly augmented version of the same image obtained from the student network matches that of teacher network . Note that FixMatch [ 22 ] also uses similar consistency regularization loss for semi-supervised learning . To our knowledge , we are the first to use consistency regularization and dynamic distillation for cross-domain few-shot learning . 3 Methodology . 3.1 Preliminary . Few-shot Learning Formulation A few-shot learning task consists of a support set S , which containing K data points from N classes for N -way K-shot task , and a query Q = { xi } mi=1 consisting of data points only from the N classes of the support set . The goal is to classify the query points with the help of the labeled support set . In the typical few-shot learning setting , ( 1 ) an embedding is learned from the base/source dataset DS , ( 2 ) a linear classifier is learned on top of the fixed embedding on the support set , and ( 3 ) the classifications of the query data points are determined . Cross-domain Few-shot Learning The difference between the typical few-shot learning setup and cross-domain few-shot learning is that the base/source dataset is drawn from a very different domain than the target domain . Additionally , in our setting , we are provided unlabeled data points DU = { xi } NUi=1 from the target domain . The unlabeled dataset contains more classes than the support set . Given the base dataset DS , and an unlabeled set DU , we need to learn an embedding that can extract a representation that can be used for few-shot learning evaluation in the target-domain . 3.2 Proposed Method . Encoder We facilitate knowledge distillation to train our base encoder on both source datatset and target dataset . Denote the embedding network as fs that embeds an input image x to a d-dimensional vector fs ( x ) . We add a classifier header gs on top of fs , which predicts nc logits from the embeddings , where nc is the total number of classes in the base dataset . Since the labels of the data points of the base dataset are provided , we calculate the supervised cross-entropy loss : lCE ( y , p ) = H ( y , p ) ( 1 ) where p = Softmax ( gs ( fs ( x ) ) ) , and H ( a , b ) = −a log b . Dynamic distillation Additionally , we also have a teacher encoder ft and teacher classifier gt . The task of the teacher network is to produce pseudo labels for the unlabeled images . Given an image xi from the unlabeled set DU , we compute the model ’ s prediction pwi from a weakly-augmented version ( denoted as xwi ) and p s i from a strongly-augmented version ( denoted as x s i ) of the unlabeled image . The prediction of the weakly-augmented version is produced from the teacher network , which serves as a soft-target for the strongly-augmented images . We use the student network to get the prediction psi for the strongly-augmented images . Specifically , psi = Softmax ( gs ( fs ( x s i ) ) ) ; p w i = Softmax ( gt ( ft ( x w i ) ) /τ ) ( 2 ) where τ is a sharpening parameter . Note that we do not let gradient pass through the teacher network . We calculate the distillation loss by the cross-entropy function lU ( p w i , p s i ) = H ( p w i , p s i ) ( 3 ) Eq . 3 works like a consistency regularizer so that the network predicts similar scores for different augmented versions of the image . We can also consider Eq . 3 as self-supervised loss , similar to BYOL [ 6 ] or DINO [ 2 ] . However , one major difference is that - in BYOL or DINO the projection head is a random linear layer , where , we are using the supervised classification head as the projection head . The total loss function is : L = 1 NS ∑ ( xi , yi ) ∈DS lCE ( y , p ) + λ 1 NU ∑ xi∈DU lU ( p w i , p s i ) ( 4 ) where λ is a hyper-parameter . The loss function is used to update the parameters of the student network . For the teacher network , we use mean teacher approach [ 24 ] , i.e. , we update the teacher weights θt from the student weights θs by : θt = mθt + ( 1 − m ) θs , where m is the momentum parameter . Note that when m = 1 , we are essentially using fixed teacher , and when m = 0 , the teacher and student share the same model . When m > 0 , the teacher network is a moving average of the student network denoting the distillation process as dynamic . Please refer to the Appendix for PyTorch-like pseudo-code of our method .

Summary: The submission attempts to address the cross-domain few-shot learning problem by utilizing dynamic distillation from additional unlabeled target data. It simplifies the STARTUP framework by removing the contrastive loss and using a similar network for teacher and student. Moreover, it introduces different data augmentations to learn better representations. The teacher network is also trained dynamically to enhance the training of the student network.