TOHAN: A One-step Approach towards Few-shot Hypothesis Adaptation
Year: 2021

Abstract:
In few-shot domain adaptation (FDA), classifiers for the target domain are trained with accessible labeled data in the source domain (SD) and few labeled data in the target domain (TD). However, data usually contain private information in the current era, e.g., data distributed on personal phones. Thus, the private data will be leaked if we directly access data in SD to train a target-domain classifier (required by FDA methods). In this paper, to prevent privacy leakage in SD, we consider a very challenging problem setting, where the classifier for the TD has to be trained using few labeled target data and a well-trained SD classifier, named few-shot hypothesis adaptation (FHA). In FHA, we cannot access data in SD, as a result, the private information in SD will be protected well. To this end, we propose a targetoriented hypothesis adaptation network (TOHAN) to solve the FHA problem, where we generate highly-compatible unlabeled data (i.e., an intermediate domain) to help train a target-domain classifier. TOHAN maintains two deep networks simultaneously, in which one focuses on learning an intermediate domain and the other takes care of the intermediate-to-target distributional adaptation and the target-risk minimization. Experimental results show that TOHAN outperforms competitive baselines significantly.

heading: 1 Introduction
text: In domain adaptation (DA) [7, 21, 41, 42, 50], we aim to train a target-domain classifier with data in source and target domains. Based on the availability of data in the target domain (e.g., fully-labeled data, partially-labeled data and unlabeled data), DA is divided into three categories: supervised DA (SDA) [43], semi-supervised DA [20] and unsupervised DA (UDA) [56]. Since SDA methods outperform UDA methods for the same quantity of target data [33], it becomes attractive if we can train a good target-domain classifier using labeled source data and few labeled target data [46].
Hence, few-shot domain adaptation (FDA) methods [33] are proposed to train a target-domain classifier with accessible labeled data from the source domain and few labeled data from the target ˚Equal contribution. Work done when Haoang Chi remotely visited HKBU. :Corresponding author.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).

heading: Closed-set DA Open-set DA Partial DA

heading: Few-shot DA

heading: Source-free
text: domain. Compared to SDA and UDA methods, FDA methods only require few data in the target domain, which is suitable to solve many problems, e.g., medical image processing [48]. Existing FDA methods involve many approaches and applications. Structural casual model [46] has been proposed to overcome the problem caused by apparent distribution discrapancy. Since deep neural networks tend to overfit the few-labeled data in the training process, a meta-learning method becomes an effective solution to the FDA problem [45]. Besides, FDA methods perform well in face generation [51] and virtual-to-real scene parsing [54].
However, it is risky to directly access source data for training a target-domain classifier (required by FDA methods) due to the private information contained in the source domain. In the current era, labeled data are distributed over different physical devices and usually contain private information, e.g., data on personal phones or from surveillance cameras [26]. Since FDA methods [46] require abundant labeled source data to train a target-domain classifier, they may leak private information in the training process, which may result in massive loss [19].
In this paper, to prevent the private data leakage of the source domain in existing FDA methods, we propose a novel and very challenging problem setting, where the classifier for the target domain has to be trained using few labeled target data and a well-trained source-domain classifier, named few-shot hypothesis adaptation (FHA, see Figure 1). In the literature [26], researchers have adapted a source-domain hypothesis to be a target-domain classifier when abundant unlabeled target data are available. However, since these methods require abundant target data, they cannot address the FHA problem well, which has been empirically verified in Table 1 and Table 2.
The key benefit of FHA is that we do not need to access the source data, which wisely avoids private-information leakage of source domain under mild assumptions. Besides, since the size of datasets of most domains is large in the real world, existing FDA methods will take a long time to train a target-domain classifier. However, in FHA, we train a target-domain classifier only with a source classifier and few labeled target data, reducing the computation cost greatly.
To address FHA, we first revisit the theory related to learning from few labeled data and try to find out if FHA can be addressed in principle. Fortunately, we find that, in semi-supervised learning (SSL) where only few labeled data available, researchers have already shown that, a good classifier can be learned if we have abundant unlabeled data that are compatible with the labeled data. Thus, motivated by the SSL, we aim to address FHA via gradually generating highly compatible data for the target domain. To this end, we propose a target-oriented hypothesis adaptation network (TOHAN) to solve the FHA problem. TOHAN maintains two deep networks simultaneously, in which one focuses on learning an intermediate domain (i.e., learning compatible data) and the other takes care of the intermediate-to-target distributional adaptation (Figure 2).
Specifically, due to the scarcity of target data, we cannot directly generate compatible data for the target domain. Thus, we first generate an intermediate domain where data are compatible with the given source classifier and the few labeled target data. Then, we conduct the intermediate-to-target
distributional adaptation to make the generated intermediate domain close to the target domain. Eventually, we embed the above procedures into our one-step solution, TOHAN, to enable gradual generation of an intermediate domain that contains highly compatible data for the target domain. According to the learnability of SSL, with the generated “target-like” intermediate domain, TOHAN can learn a good target-domain classifier.
We conduct experiments on 8 FHA tasks on 5 datasets (MNIST, SVHN, USPS, CIFAR-10 and STL-10). We compare TOHAN with 5 competitive baselines. Experiments show that TOHAN effectively transfers knowledge of the source hypothesis to train a target-domain classifier when we only have few labeled target data. In other words, our paper opens a new door to the domain adaptation field, which solves private-data leakage and data shortage simultaneously.

heading: 2 Few-shot Hypothesis Adaptation
text: In this section, we formalize a novel and challenging problem setting, called few-shot hypothesis adaptation (FHA). Let X Ă Rd be a feature (input) space and Y :“ t1, . . . , Nu be a label (output) space, and N is the number of classes. A domain [9] for the FHA problem is defined as follows.
Definition 1 (Domains for FHA). Given random variables Xs, Xt P X , Ys, Yt P Y , the source and target domains are joint distributions P pXs, Ysq and P pXt, Ytq, respectively, where the joint distributions P pXs, Ysq ‰ P pXt, Ytq and X is compact.
Then the FHA problem is defined as follows.
Problem 1 (FHA). Given a model (consisting of an encoder gs and a classifier hs) trained on the source domain P pXs, Ysq and independent and identically distributed (i.i.d.) labeled data Dt “ ` xit, y i t ˘(nt
i“1 (nt ď 7N , following [37]) drawn from the target domain P pXt, Ytq, the aim of FHA is to train a classifier ht : X Ñ Y with gs, hs and Dt such that ht can accurately classify target data drawn from P pXt, Ytq. Remark 1. In FHA, there exists an assumption: malicious attackers cannot easily find sourcedomain-like data from the Internet and via some other ways. Otherwise, attackers may use the attack methods [57] to recover the training data, leading to data leakage.
Possible Privacy-leakage Issues in FHA. The assumption in Remark 1 is derived from the attack methods that aim to recover training data from a well-trained model. According to recent modelinversion attack methods [57], they need to access auxiliary data whose background is similar to the training data to help recover input data. There also exists a white-box inference attack method [34] that determines a data point’s membership in the training set of the model. Therefore, FHA belongs to passive protection, requiring the training data of source model are sufficiently different from public data. To thoroughly avoid this issue, data owners might utilize the defending techniques (against the model-inversion attacks) to train their source models.
Comparison with Few-shot Learning. The main difference between FHA and few-shot learning (FSL) is the representation of source domain. For FHA, source domain is represented by a model
trained with source data. While, for FSL, source domain is represented by labeled data themselves [29, 30]. Besides, the data used to train source classifiers come from different domains from target data in FHA, while source data and target data come from the same domain in FSL. The works [14, 49] propose to hallucinate additional training examples to solve few-shot visual recognition, inspired by human’s visual imagination. Meta-learning [11, 40] also performs well in FSL by learning the distribution of tasks with high generalization ability. As using few data for training easily leads to overfitting, there are works [10, 55] trying to constrain the hypothesis space to avoid it. Data augmentation generative adversarial network (DAGAN) [2] aims to augment target data through a conditional generative adversarial network to enhance the few-shot learning procedure.
Comparison with UDA. The main differences between FHA and UDA lie in the amount and label of data in the two domains. For the source domain, UDA requires a large amount of labeled data [6, 59], while FHA only requires a well-trained model. For the target domain, UDA requires a large amount of unlabeled data [44, 60], while FHA requires few labeled data.
Comparison with FDA. With the development of FSL, researchers also apply ideas of FSL into domain adaptation, called few-shot domain adaptation (FDA). FADA [33] is a representative FDA method, which pairs data from the source domain and data from the target domain and then follows the adversarial domain adaptation method. Casual mechanism transfer [46] is another novel FDA method dealing with a meta-distributional scenario, in which the data generating mechanism is invariant among domains. Nevertheless, FDA methods still need to access many labeled source data for training, which may cause the private-information leakage of the source domain.
Comparison with Hypothesis Transfer Learning. In hypothesis transfer learning (HTL), we can only access a well-trained source-domain classifier and small labeled or abundant unlabeled target data. [24] requires small labeled target data and uses the Leave-One-Out error to find the optimal transfer parameters. Later, SHOT [26] is proposed to solve the HTL with many unlabeled target data by freezing the source-domain classifier and learning a target-specific feature extraction module. As for the universal setting, a two-stage learning process [23] has been proposed to address the HTL problem. Compared with FHA, HTL still requires at least small target data (e.g., at least 12 samples in binary classification problem [24], or at least 10% target data are labeled [1]). In FHA, we focus on a more challenging situation: only few data (e.g., one sample per class) are available. Besides, previous solutions to HTL mainly focus on mortifying existing hypotheses or loss functions used for fine-tuning. However, our solution stems from the learnability of semi-supervised learning (Section 3) and try to generate more compatible data, which is quite different from previous works.

heading: 3 How to Learn from Few-shot Data in Principle
text: From the view of statistical learning theory [47], it is unrealistic to directly learn an accurate targetdomain classifier only with few labeled data. However, the amount of labeled data in semi-supervised learning (SSL) [61] is also few (e.g., one sample per class), but SSL methods still achieves good performance across various learning tasks, which motivates us to consider solving FHA in the view of SSL. First, we will show theoretical analysis regarding learnability of SSL.
Learnability of SSL. For simplicity, we consider the 0-1 semi-supervised classification problem. Let c˚ : X Ñ t0, 1u be the optimal target classifier and H “ th : X Ñ t0, 1uu is a hypothesis space. Let errphq “ Ex„P rhpxq ‰ c˚pxqs be the true error rate of a hypothesis h over a distribution P . In SSL, its learnability mainly depends on the compatibility χ : Hˆ X ÞÑ r0, 1s that measures how “compatible” h is to an unlabeled data x. Let χph, P q “ Ex„P rχph, xqs be the expectation of compatibility of data from P on a classifier h. If the unlabeled data and c˚ are highly compatible (i.e., χpc˚, P q closes to 1), then, in theory, we can learn a good classifier with few labeled data and sufficient unlabeled data. Specifically, we have the following theorem (see proof in Appendix B).
Theorem 1. Let χ̂ph, Sq “ 1|S| ř xPS χph, xq be the empirical compatibility over unlabeled dataset S. Let H0 “ th P H : xerrphq “ 0u. If c˚ P H and χpc˚, P q “ 1´ t, then mu unlabeled data and ml labeled data are sufficient to learn to error with probability 1´ δ, for
mu “ O ˆ V CdimpχpHqq 2 log 1 ` 1 2 log 2
δ
˙
(1)
and
ml “ 2
„
lnp2HP,χpt` 2 qr2ml, P sq ` ln 4
δ

, (2)
where χpHq “ tχh : h P Hu, χhp¨q “ χph, ¨q, and HP,χpt` 2 qr2ml, P s is the expected number of splits of 2ml data drawn from P using hypotheses in H of compatibility more than 1´ t´ 2 . In particular, with probability at least 1´ δ, we have errpĥq ď , where
ĥ “ argmax hPH0 χ̂ph, Sq. (3)
Remark 2. If the unlabeled data are highly compatible to c˚, t is small, which results in a smaller ml. Namely, with the smaller ml, we can still achieve a low error rate. In view of Theorem 1, it is clear that SSL will be learnable if many compatible unlabeled data are available. Motivated by SSL, we wonder if we can generate compatible data to help our learning task. The answer is affirmative.
Solving FHA in Principle. Motivated by Theorem 1, finding many highly compatible unlabeled data is a breakthrough point for FHA. Hence, generating unlabeled target data is a straightforward solution. However, due to the shortage of existing target data, directly generating them is unrealistic. To solve this problem, we can ask for help from the source classifier. In our paper, we first try to generate intermediate domain Pm containing knowledge of source and target domains, which are compatible with both the source classifier and target classifier, i.e.,
Pm “ argmax P rχphs, P q ` χpht, P qs, (4)
where χphs, P q (resp. χpht, P q) measures how compatible hs (resp. ht) is with the data distribution P . Then, we will adapt intermediate domain Pm to the target domain via distributional adaptation with the training procedure going on. Finally, we can obtain many unlabeled data that are compatible with hs and ht (more compatible with ht), meaning that, based on Theorem 1, we can address FHA in principle. According to Eq. (4), it can be seen that we can have two straightforward solutions: maximizing χphs, P q or χpht, P q, corresponding to S+FADA and T+FADA in benchmark solutions. The results in Table 1 and Table 2 indicate that these two straightforward solutions cannot address FHA well, which motivates us to maximize them simultaneously, which is realized below.

heading: 4 Target-Oriented Hypothesis Adaptation Network for FHA Problem
text: This section presents a powerful one-step approach: target-oriented hypothesis adaptation network (TOHAN, see Figure 2). TOHAN can generate data that are highly compatible with both the source classifier and target classifier and adapt the knowledge of these data to the target domain gradually.
Intermediate domain generation. The first step of TOHAN is to generate the intermediate domain data (Figure 2a). We input Gaussian random noise z to a generator Gn (taking the nth class for an example), then the generator outputs generated data. We aim to generate data satisfying (1) the generated data Gnpzq can be correctly classified by the given source classifier fs “ hs ˝ gs, and (2) Gnpzq becomes closer to the target domain with training procedure going on. Thus, there are two loss functions regarding the intermediate domain generation. The first one is as follows.
Without loss of the generality, we assume Gnpzq generates B images, where B is the batchsize in the training process of TOHAN. When Gnpzq is inputted to the source-domain classifier fs, we will obtain an B ˆN matrix GMn , where the ith row in GMn represents probability of the ith generated image belonging to each class. Thus, the nth column in GMn represents the probability that the B generated images belongs to the nth class, and we denote the nth column in GMn as ln. Since Gnpzq aims to generate data belonging to the nth class, we should update parameters of GMn to make each element in ln close to 1. Namely, the first loss function to train the Gn can be defined as
LsGn “ 1
B }ln ´ 1}22 , (5)
where 1 is a B-by-1 vector whose elements are 1.
As discussed before, we also want to reduce the distance between the generated data Gnpzq and the target data whose labels are n. In this way, we can make the generated data close to the target
domain and attain an intermediate domain Pm. Following [27], we adopt an augmented L1 distance }X ´ Y }1 “ ř i ωi |Xi ´ Yi|, where ωi “ |Xi ´ Yi| 2 {}X ´ Y }2. Compared to the ordinary `1 norm, the augmented L1 distance encourages larger gradients for feature dimensions with higher residual error [27]. Compared to the `2 norm, since L1 distance is more robust to outliers [36], it is better to measure the distance between generated images and target images. Thus, the second loss to train Gn is defined as follows,
LtGn “ 1
MBK
B ÿ
i“1
K ÿ
k“1
› ›xim ´ xkt › › 1 , (6)
where M “ max x1,x2PX }x1 ´ x2}1 (X is compact and } ¨ }1 is continuous) and Gnpzq :“ tximuBi“1. Combining Eq. (5) and Eq. (6), we obtain the total loss to train the generator Gn:
LGn “ LsGn ` λL t Gn “
1 B }ln ´ 1}22 ` λ MBK
B ÿ
i“1
K ÿ
k“1
› ›xim ´ xkt › › 1 , (7)
where λ is a hyper-parameter between two losses to tradeoff the weight of knowledge of source and target domains. To ensure that the generated data are high-quality images, we train the generator Gn (n “ 1, . . . , N ) for some steps all alone. Note that, Eq. (7) corresponds to Eq. (4), and Eq. (5) (resp. Eq. (6)) is corresponding to χphs, Pmq (resp. χpht, Pmq). Then we conduct intermediate-to-target distributional adaptation (see the next paragraph) and generation simultaneously.
Intermediate-to-target distributional adaptation. Now, we focus on how to construct domaininvariant representations (DIP) between the intermediate domain and the target domain. Through DIP, a classifier for the intermediate domain can be used to classify target data well [28, 59].
Since we only have few target data per class, we aim to “augment” them. Following [33], we can overcome the shortage of target data by pairing them with the corresponding intermediate data. Specifically, we create 4 groups of data pairs: G1 consists of data pairs from the same domain with the same label, G2 consists of pairs from different domains (one from the intermediate and one from the target domain) but with the same label, G3 consists of pairs from the same domain with different labels, and G4 consists of pairs from different domains (one from the intermediate and one from the target domain) and with different labels.
Based on the above four groups, we construct a four-class group discriminator D to decide which of the four groups a given data pair belongs to, which differs from classical adversarial domain adaptation [12, 20]. The group discriminator D aims to classify the data pair groups. As a classification problem, we train D with the standard categorical cross-entropy loss:
LD “ ´Ê « 4 ÿ
i“1 yGi log pD pφ pGiqqq
ff
, (8)
where Êr¨s represents the empirical mean value, yGi is the label of group Gi, and φpGiq :“ rgtpx1q, gtpx2qs, px1, x2q P Gi, and gt is the encoder on target domain. Note that we freeze gt when minimizing the above loss function (see Figure 2b).
Next, we turn to train gt and ht with the group discriminator D fixed, which confuses D between G1 and G2 (also G3 and G4). However, we need D to correctly discriminate positive pairs (G1, G2) from negative pairs (G3, G4). This means that domain confusion and classification are realized at the same time. We firstly initialize gt and ht with the same weight as gs and hs, respectively. Motivated by the non-saturating game [13], we minimize the following loss to update gt and ht (see Figure 2c):
Lh˝g “ ´βÊ ryG1 log pD pφ pG2qqq ´ yG3 log pD pφ pG4qqqs ` Ê r` pft pXtq , f˚t pXtqqs , (9)
where β is a hyper-parameter to tradeoff confusion and classification and ` is the cross-entropy loss. ft :“ gt ˝ ht is the target model and f˚t is the optimal target model. Corresponding to Theorem 1, optimizing the first term in Eq. (9) increases compatibility of the target model with the intermediate data, and optimizing the second term in Eq. (9) reduces xerrphtq, resulting in a smaller errphtq. Compared to [33], Eq. (9) means that we train the target model by confusing D and improving classification accuracy simultaneously.
Algorithm 1 Target-oriented hypothesis adaptation network (TOHAN) Input: encoder gs, classifier hs, Dt “ xit, y i t (nt
i“1, learning rate γ1, γ2, γ3 and γ4, total epoch Tmax, pretraining D epoch Td, adaptation epoch Tf , network parameter tθGnuNn“1, θh˝g , θD . 1: Initialize tθGnuNn“1 and θD; for t “ 1, 2, ....., Tmax do
2: Initialize Dm “ ∅ for n “ 0, 1, . . . , N ´ 1 do
3: Generate random noise z; 4: Generate data Gnpzq then add them to Dm 5: Update θGn Ð θGn ´ γ1∇LGn pz,Dtq using Eq. (7);
end if t “ Tmax ´ Tf then
for i “ 1, 2, . . . , Td do 6: Sample G1, G3 from Dm ˆDm; 7: Sample G2, G4 from Dm ˆDt; 8: Update θD Ð θD ´ γ2∇LD ` tGiu4i“1 ˘
using Eq. (8); end
end if t ě Tmax ´ Tf then
9: Sample G1, G3 from Dm ˆDm; 10: Sample G2, G4 from Dm ˆDt; 11: Update θh˝g Ð θh˝g ´ γ3Lh˝gptGiu4i“1, xm, xtq using Eq. (9); 12: Update θD Ð θD ´ γ4∇LD ` tGiu4i“1 ˘
using Eq. (8); end
end Output: the neural network ht ˝ gt.
TOHAN: A one-step solution to FHA. Although we can sequentially combine the above two steps to solve the FHA problem (i.e., a two-step solution), the fixed intermediate domain (generated by the first step) may have large distributional discrepancy with the target domain. As a result, such two-step solution may not obtain a good target-domain classifier. To address this issue, we introduce a one-step solution TOHAN. The ablation study verifies that TOHAN outperforms such two-step solution (see ST+F and TOHAN in Table 3).
The entire training procedure of TOHAN is shown in Algorithm 1. Since the convergence speed of generator G is relatively slow, the quality of generated data is poor at the beginning of the training process of G. Thus, we will train the generator G for a certain number of epochs before performing intermediate-to-target distributional adaptation (lines 2 to 5). When the generator can generate high-quality images, we train the generator and conduct adaptation together.
We train every generator Gn (n “ 1, 2, . . . , N ) separately, and we generate the intermediate domain data using the latest generators. Then, we pair the intermediate data with the target data and pretrain the group discriminator D (lines 6 to 8). Next, we pair the intermediate data with target data and conduct the adaptation (lines 9 to 12). After conducting intermediate-to-target distributional adaptation, we obtain better gt and ht, i.e. classifying the target data more accurately. With the better target-domain classifier, we can make the generated intermediate data get closer to the target domain, in turn, these generated intermediate data further promote adaptation performance.

heading: 5 Experiments
text: We compare TOHAN with benchmark solutions on five standard supervised DA datasets: MNISTpMq, SYHNpSq, USPSpUq, CIFAR-10 pCFq, STL-10 pSLq. We follow the standard domain-adaptation protocols [39] and compare average accuracy of 5 independent repeated experiments. For digital datasets (i.e., M, S, and U), we choose the number of target data (per class) from 1 to 7 [33]. For objects datasets (i.e., CF and SL), we choose the number of target data as 10. Details regarding these datasets can be found in Appendix C. The code is available at github.com/Haoang97/TOHAN.
Benchmark solutions for FHA. Although the FHA is a new problem setting, we still design 5 benchmark solutions to this new problem. (1) Without adaptation (WA): to classify the target domain
with the source classifier (encoder gs and classifier hs). (2) Fine-tuning (FT): to train the classifier gs with few owned target data. (3) SHOT: a novel HTL method, where we modify it to use the labeled target data instead of only using the unlabeled target data. [26]. (4) S+FADA (S+F): to generate faked source data with the source classifier then apply them to DANN [12]. (5) T+FADA (T+F): to generate fake target data with few real target data then apply them to DANN. We demonstrate details of 5 benchmark solutions in Appendix D. Experimental details can be found in Appendix E. Moreover, we conduct additional experiments to compare existing HTL method named dkdHTL [53], and the related results and analysis can be found in Appendix F.
Results on digits FHA tasks. We conduct experiments on 6 digits FHA tasks: MÑS, SÑM, MÑU, UÑM, SÑU and UÑS. Table 1 reports the target-domain classification accuracy of 6 methods on 6 digits FHA tasks. It is clear that TOHAN performs the best on almost every task. On MÑS, SÑM, MÑU and SÑU, TOHAN outperforms all benchmark solutions ob-
viously. However, on the tasks UÑM and UÑS, the accuracy of TOHAN is slightly lower than SHOT when the amount of target data is too small (n “ 1, 2). This abnormal phenomenon shows that TOHAN cannot generate intermediate domain data effectively with very little target data, especially when the resolution of source data is much smaller than that of target data. In this case, the data we generate is close to the source domain, so TOHAN degrades to S+FADA.
In Appendix G, we use t-SNE to visualize the features extracted by TOHAN and 5 benchmark solutions on MÑU task (see Figure 8 in Appendix G). When we use WA and FT methods, nearly all classes mix together. Although the classification accuracies of SHOT, S+F and T+F are relatively high, there are still some mixing among classes. For TOHAN, it can be seen that all classes are separated well, which demonstrates that TOHAN works well for solving the FHA problem.
Results on objects FHA tasks. Following [39], we also evaluate TOHAN and benchmark solutions on 2 objects FHA tasks: SLÑ CF and CF Ñ SL, and the results are shown in Table 2. Considering the complexity of datasets and the difficulty of our problem setting, we do not have amazing results like digits tasks. In SL Ñ CF , we achieve of 4.8% improvement over WA and a performance accuracy of 56.9%. Note that because the numbers of pixels per image of CF and SL are quite different, the images from SL lose a lot of information when inputted to the pre-trained model of CF , thus making the effects of TOHAN and benchmark solutions are not obvious for CF Ñ SL.
Comparing TOHAN with FSL methods. As mentioned above, FHA is a difficult case of FSL where the prior knowledge is a pre-trained model of another domain. To test the effectiveness of FSL methods in FHA, we compare TOHAN with a novel FSL method called domain-adaptive few-shot learning (DAPN) [58]. Note that we use the same pre-trained model in both TOHAN and DAPN. Taking CF Ø SL with five target data (per class) as an example, we solve FHA with TOHAN and DAPN and show the results in Figure 3. It is clear that TOHAN outperforms DAPN when the training epoch (t) is relatively large.
Ablation Study. Finally, we study the advantages of one-step method over other two-step methods. We consider the following baselines: S+F, T+F and ST+FADA (ST+F). We have explained S+F and T+F previously. ST+F denotes the two-step version of TOHAN, i.e., to conduct intermediate domain generation and intermediate-totarget distributional adaptation separately. We make ablation study on three digital datasets mentioned before as an example.
As shown in Table 3, it is clear that TOHAN works better than the other baselines. The generator of S+F uses the loss LsGn , which merely contains knowledge from the source domain. The generator of T+F uses the loss LtGn and ignores the knowledge contained in the source-domain classifier. In contrast, TOHAN uses both LsGn and
LtGn . As a result, TOHAN achieves higher accuracy than S+F and T+F. Besides, the generators and classifiers in TOHAN promote each other in the training process, which results in that TOHAN performs better than the ST+F. In Figure 4, we visualize the data generated by S+FADA and TOHAN. It is clear that data generated by S+FADA are chaotic that contain little useful information. However, data generated by TOHAN contain many target-domain high-level visual features, and they can be classified by the source classifier accurately, resulting in a better performance in FHA. The detailed analysis of ablation study can be found in Appendix G.
Verification of No Source-data Leakage in Intermediate Domain. As a key contribution, TOHAN solves FHA through generating intermediate data. To guarantee that no source data are leaked, we need to verify that there is no source-domain features in the intermediate data. We determine this by calculating the PSNR values [15, 52] between each intermediate sample and all source samples.
PSNR indicates the generation quality of an image f given a standard image g, and is defined as
PSNRpf, gq “ 10 log10 2552 MSEpf, gq , where MSEpf, gq “ 1 MN
M ÿ
i“1
N ÿ j“1 pfij ´ gijq2.
The larger PSNR value indicates the two images are more similar. Therefore, taking MÑS as an example, we report the top-5 largest PSNR values in Table 4. That is, we check whether the worst case satisfies our claim. For comparison, we also compute the PSNR values of (source data, source data), (target data, target data), (source data, target data), and (intermediate data, target data), and we report the average PSNR values of the above four cases in Figure 5.
As can be seen, the intermediate data are much closer to the target data, and they are very different from the source data. The average PSNR between source data and target data is 9.3209. The top-5 largest PSNR values between each intermediate data and all source data (« 17.89) are obviously smaller than 50.1722 (the average PSNR between intermediate data and target data). Through this result, we can state that intermediate data are similar to the target data and very different from the source data. Therefore, the above evidence shows that the generated intermediate data contain no source domain features, and the source data do not leak when generating the intermediate data.

heading: 6 Conclusion
text: This paper presents a very challenging problem setting called few-shot hypothesis adaptation (FHA), which trains a target-domain classifier with only few labeled target data and a well-trained sourcedomain classifier. Since we can only access a well-trained source-domain classifier in FHA, the private information in the source domain are protected well. To this end, we propose a novel onestep FHA method, called target-oriented hypothesis adaptation network (TOHAN). Experiments conducted on 8 FHA tasks confirm that TOHAN effectively adapts the source-domain classifier to the target domain and outperforms competitive benchmark solutions to the FHA problem.

heading: Acknowledgments and Disclosure of Funding
text: This work was partially supported by the National Natural Science Foundation of China (No. 91948303-1, No. 61803375, No. 12002380, No. 62106278, No. 62101575, No. 61906210) and the National Grand R&D Plan (Grant No. 2020AAA0103501). FL would also like to thank Dr. Yanbin Liu for productive discussions.

Review summaries:
This paper proposes a novel problem in domain adaptation - Few-shot Hypothesis Adaptation (FHA) - that combines Hypothesis Transfer Learning and the Few-Shot Domain Adaptation problem. Specifically, it assumes a lack of source domain data but the presence of a classifier (hypothesis) pretrained on the source domain and a few labelled data points (<=10 per class) from the target domain (with no additional data). The paper also adapts the FADA method and proposes a novel method, called TOHAN, which uses the source-domain hypothesis and the few target-domain data points to generate the intermediate-domain data compatible with the two domains.

This paper focuses on a problem setting where data are not available but classifiers are. The possible benefits of this setting lie in the privacy reservation, i.e., data used in previous few-shot learning may leak the private information. The proposed solution does not use a fine-tuning-based method but a sample-generation-based method, which is motivated by the learnability of semi-supervised learning. 

As the increasing need for privacy protection, previous few-shot DA (FDA) has limited applications in the real world. To overcome this drawback of the FDA, this paper considers a very challenging problem setting, few-shot hypothesis adaptation (FHA), which is the most difficult part of hypothesis transfer learning (HTL). Compared to previous solutions to HTL, the authors argue that generating target samples is a good solution. The proposed method is validated on common datasets.