TOHAN: A One-step Approach towards Few-shot Hypothesis Adaptation

1 Introduction . In domain adaptation ( DA ) [ 7 , 21 , 41 , 42 , 50 ] , we aim to train a target-domain classifier with data in source and target domains . Based on the availability of data in the target domain ( e.g. , fully-labeled data , partially-labeled data and unlabeled data ) , DA is divided into three categories : supervised DA ( SDA ) [ 43 ] , semi-supervised DA [ 20 ] and unsupervised DA ( UDA ) [ 56 ] . Since SDA methods outperform UDA methods for the same quantity of target data [ 33 ] , it becomes attractive if we can train a good target-domain classifier using labeled source data and few labeled target data [ 46 ] . Hence , few-shot domain adaptation ( FDA ) methods [ 33 ] are proposed to train a target-domain classifier with accessible labeled data from the source domain and few labeled data from the target ˚Equal contribution . Work done when Haoang Chi remotely visited HKBU . : Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Closed-set DA Open-set DA Partial DA . Few-shot DA . Source-free . domain . Compared to SDA and UDA methods , FDA methods only require few data in the target domain , which is suitable to solve many problems , e.g. , medical image processing [ 48 ] . Existing FDA methods involve many approaches and applications . Structural casual model [ 46 ] has been proposed to overcome the problem caused by apparent distribution discrapancy . Since deep neural networks tend to overfit the few-labeled data in the training process , a meta-learning method becomes an effective solution to the FDA problem [ 45 ] . Besides , FDA methods perform well in face generation [ 51 ] and virtual-to-real scene parsing [ 54 ] . However , it is risky to directly access source data for training a target-domain classifier ( required by FDA methods ) due to the private information contained in the source domain . In the current era , labeled data are distributed over different physical devices and usually contain private information , e.g. , data on personal phones or from surveillance cameras [ 26 ] . Since FDA methods [ 46 ] require abundant labeled source data to train a target-domain classifier , they may leak private information in the training process , which may result in massive loss [ 19 ] . In this paper , to prevent the private data leakage of the source domain in existing FDA methods , we propose a novel and very challenging problem setting , where the classifier for the target domain has to be trained using few labeled target data and a well-trained source-domain classifier , named few-shot hypothesis adaptation ( FHA , see Figure 1 ) . In the literature [ 26 ] , researchers have adapted a source-domain hypothesis to be a target-domain classifier when abundant unlabeled target data are available . However , since these methods require abundant target data , they can not address the FHA problem well , which has been empirically verified in Table 1 and Table 2 . The key benefit of FHA is that we do not need to access the source data , which wisely avoids private-information leakage of source domain under mild assumptions . Besides , since the size of datasets of most domains is large in the real world , existing FDA methods will take a long time to train a target-domain classifier . However , in FHA , we train a target-domain classifier only with a source classifier and few labeled target data , reducing the computation cost greatly . To address FHA , we first revisit the theory related to learning from few labeled data and try to find out if FHA can be addressed in principle . Fortunately , we find that , in semi-supervised learning ( SSL ) where only few labeled data available , researchers have already shown that , a good classifier can be learned if we have abundant unlabeled data that are compatible with the labeled data . Thus , motivated by the SSL , we aim to address FHA via gradually generating highly compatible data for the target domain . To this end , we propose a target-oriented hypothesis adaptation network ( TOHAN ) to solve the FHA problem . TOHAN maintains two deep networks simultaneously , in which one focuses on learning an intermediate domain ( i.e. , learning compatible data ) and the other takes care of the intermediate-to-target distributional adaptation ( Figure 2 ) . Specifically , due to the scarcity of target data , we can not directly generate compatible data for the target domain . Thus , we first generate an intermediate domain where data are compatible with the given source classifier and the few labeled target data . Then , we conduct the intermediate-to-target distributional adaptation to make the generated intermediate domain close to the target domain . Eventually , we embed the above procedures into our one-step solution , TOHAN , to enable gradual generation of an intermediate domain that contains highly compatible data for the target domain . According to the learnability of SSL , with the generated “ target-like ” intermediate domain , TOHAN can learn a good target-domain classifier . We conduct experiments on 8 FHA tasks on 5 datasets ( MNIST , SVHN , USPS , CIFAR-10 and STL-10 ) . We compare TOHAN with 5 competitive baselines . Experiments show that TOHAN effectively transfers knowledge of the source hypothesis to train a target-domain classifier when we only have few labeled target data . In other words , our paper opens a new door to the domain adaptation field , which solves private-data leakage and data shortage simultaneously . 2 Few-shot Hypothesis Adaptation . In this section , we formalize a novel and challenging problem setting , called few-shot hypothesis adaptation ( FHA ) . Let X Ă Rd be a feature ( input ) space and Y : “ t1 , . . . , Nu be a label ( output ) space , and N is the number of classes . A domain [ 9 ] for the FHA problem is defined as follows . Definition 1 ( Domains for FHA ) . Given random variables Xs , Xt P X , Ys , Yt P Y , the source and target domains are joint distributions P pXs , Ysq and P pXt , Ytq , respectively , where the joint distributions P pXs , Ysq ‰ P pXt , Ytq and X is compact . Then the FHA problem is defined as follows . Problem 1 ( FHA ) . Given a model ( consisting of an encoder gs and a classifier hs ) trained on the source domain P pXs , Ysq and independent and identically distributed ( i.i.d . ) labeled data Dt “ ` xit , y i t ˘ ( nt i “ 1 ( nt ď 7N , following [ 37 ] ) drawn from the target domain P pXt , Ytq , the aim of FHA is to train a classifier ht : X Ñ Y with gs , hs and Dt such that ht can accurately classify target data drawn from P pXt , Ytq . Remark 1 . In FHA , there exists an assumption : malicious attackers can not easily find sourcedomain-like data from the Internet and via some other ways . Otherwise , attackers may use the attack methods [ 57 ] to recover the training data , leading to data leakage . Possible Privacy-leakage Issues in FHA . The assumption in Remark 1 is derived from the attack methods that aim to recover training data from a well-trained model . According to recent modelinversion attack methods [ 57 ] , they need to access auxiliary data whose background is similar to the training data to help recover input data . There also exists a white-box inference attack method [ 34 ] that determines a data point ’ s membership in the training set of the model . Therefore , FHA belongs to passive protection , requiring the training data of source model are sufficiently different from public data . To thoroughly avoid this issue , data owners might utilize the defending techniques ( against the model-inversion attacks ) to train their source models . Comparison with Few-shot Learning . The main difference between FHA and few-shot learning ( FSL ) is the representation of source domain . For FHA , source domain is represented by a model trained with source data . While , for FSL , source domain is represented by labeled data themselves [ 29 , 30 ] . Besides , the data used to train source classifiers come from different domains from target data in FHA , while source data and target data come from the same domain in FSL . The works [ 14 , 49 ] propose to hallucinate additional training examples to solve few-shot visual recognition , inspired by human ’ s visual imagination . Meta-learning [ 11 , 40 ] also performs well in FSL by learning the distribution of tasks with high generalization ability . As using few data for training easily leads to overfitting , there are works [ 10 , 55 ] trying to constrain the hypothesis space to avoid it . Data augmentation generative adversarial network ( DAGAN ) [ 2 ] aims to augment target data through a conditional generative adversarial network to enhance the few-shot learning procedure . Comparison with UDA . The main differences between FHA and UDA lie in the amount and label of data in the two domains . For the source domain , UDA requires a large amount of labeled data [ 6 , 59 ] , while FHA only requires a well-trained model . For the target domain , UDA requires a large amount of unlabeled data [ 44 , 60 ] , while FHA requires few labeled data . Comparison with FDA . With the development of FSL , researchers also apply ideas of FSL into domain adaptation , called few-shot domain adaptation ( FDA ) . FADA [ 33 ] is a representative FDA method , which pairs data from the source domain and data from the target domain and then follows the adversarial domain adaptation method . Casual mechanism transfer [ 46 ] is another novel FDA method dealing with a meta-distributional scenario , in which the data generating mechanism is invariant among domains . Nevertheless , FDA methods still need to access many labeled source data for training , which may cause the private-information leakage of the source domain . Comparison with Hypothesis Transfer Learning . In hypothesis transfer learning ( HTL ) , we can only access a well-trained source-domain classifier and small labeled or abundant unlabeled target data . [ 24 ] requires small labeled target data and uses the Leave-One-Out error to find the optimal transfer parameters . Later , SHOT [ 26 ] is proposed to solve the HTL with many unlabeled target data by freezing the source-domain classifier and learning a target-specific feature extraction module . As for the universal setting , a two-stage learning process [ 23 ] has been proposed to address the HTL problem . Compared with FHA , HTL still requires at least small target data ( e.g. , at least 12 samples in binary classification problem [ 24 ] , or at least 10 % target data are labeled [ 1 ] ) . In FHA , we focus on a more challenging situation : only few data ( e.g. , one sample per class ) are available . Besides , previous solutions to HTL mainly focus on mortifying existing hypotheses or loss functions used for fine-tuning . However , our solution stems from the learnability of semi-supervised learning ( Section 3 ) and try to generate more compatible data , which is quite different from previous works .

Summary: As the increasing need for privacy protection, previous few-shot DA (FDA) has limited applications in the real world. To overcome this drawback of the FDA, this paper considers a very challenging problem setting, few-shot hypothesis adaptation (FHA), which is the most difficult part of hypothesis transfer learning (HTL). Compared to previous solutions to HTL, the authors argue that generating target samples is a good solution. The proposed method is validated on common datasets.