TOHAN: A One-step Approach towards Few-shot Hypothesis Adaptation
Authors: Haoang Chi
Feng Liu
Wenjing Yang
Long Lan
Tongliang Liu
Bo Han
William Cheung
James Kwok
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: transfer learning
domain adaptation
few-shot learning

Abstract: In few-shot domain adaptation (FDA), classifiers for the target domain are trained with \emph{accessible} labeled data in the source domain (SD) and few labeled data in the target domain (TD). However, data usually contain private information in the current era, e.g., data distributed on personal phones. Thus, the private data will be leaked if we directly access data in SD to train a target-domain classifier (required by FDA methods). In this paper, to prevent privacy leakage in SD, we consider a very challenging problem setting, where the classifier for the TD has to be trained using few labeled target data and a well-trained SD classifier, named few-shot hypothesis adaptation (FHA). In FHA, we cannot access data in SD, as a result, the private information in SD will be protected well. To this end, we propose a target-oriented hypothesis adaptation network (TOHAN) to solve the FHA problem, where we generate highly-compatible unlabeled data (i.e., an intermediate domain) to help train a target-domain classifier. TOHAN maintains two deep networks simultaneously, in which one focuses on learning an intermediate domain and the other takes care of the intermediate-to-target distributional adaptation and the target-risk minimization. Experimental results show that TOHAN outperforms competitive baselines significantly.

TOHAN: A One-step Approach towards Few-shot Hypothesis Adaptation
Haoang Chil.2,6* Feng Liu Wenjing Yang'! Long Lan Tongliang Liu",
Bo Han William K. Cheung?. James T. Kwok
State Key Laboratory of High Performance Computing, College of CS, NUDT CS Department, HKBU
DeSI Lab, AAII, Faculty of Engineering and ITT, UTS
TML Lab, School of CS, Faculty of Engineering, USYD 5 CSE Department, HKUST
Peng Cheng Laboratory. Shenzhen
shoangchh6688gmaill com, feng liuuuts...... edu. au, {wenjing yang, long. lnnddnutt...... www.ddteeuuu.....eemmeeoo cn, (bhanml, williami filliam>@compp hkbu. edu. hk jamesk@cse ust hk
Abstract
In few-shot domain adaptation (FDA), classifiers for the target domain are trained with accessible labeled data in the source domain (SD) and few labeled data in the target domain (TD). However, data usually contain private information in the current era, e.g., data distributed on personal phones. Thus, the private data will be leaked if we directly access data in SD train a sarget-domain classifier (required by FDA methods). In this paper, to prevent privacy leakage in SD, we consider a very challenging problem setting, where the classifier for the TD has to trained using few labeled target data and well-trained SD classifier, named few-shol hypothexis udaptation (FHA). FHA, we cannot access data in SD, as a result, the private information in SD will be protected well. To this end, we propose target- oriented hypothesis adaptation network (TOHAN) to solve the FHA problem, where generate cohbllccomatible unlabeled data (i.e., intermediate domain) to help train a target-domain classifier. TOHAN maintains two deep networks simultaneously. in which one focuses learning an intermediate domain and the other takes care of the intermediate-te sermmediate-to-tageee target distributional adaptation and the target-risk minimization. Experimental results show that TOHAN outperforms competitive baselines significantly.
Introduction
In domain adapiation (DA) [7, 21, 41, 42, 50], we aim to train target-domain classifier with data in source and target domains. Based on the availability of data in the target domain (e.g., fably-labeled data, partially-labeled data and unlabeled data), DA is divided into three categories: supervised DA (SDA) [43], semi-supervised DA [20] and unsupervised DA (UDA) [56]. Since SDA methods outperform UDA methods for the same quantity of target data [33], it becomes attractive if we can train a good target-domain classifier using labeled source data and few labeled target data [46].
Hence, few-shot domain adaptation (FDA) methods [33] are proposed to train a target-domain classifier with accessible labeled data from the source domain and few labeled data from the target
* Equual contribution. Work done when Haoang Chi remotely visited HKBU.
Corresponding author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Closed-scl
Source domain
Target domain
Open-set DA
Partial DA
Few-shot DA
Labeled dataset
Labeled&
Targen dataset
Source-free
DA
Few-shot
Source madell (Encuder +Classifiett
hypothesis
Few Labeled Tagget dataset
adaptation
Figure 1: The few-shot kypothesis adaptation (FHA) and existing domain adaptation problem settings. In FHA, we aim to train classifier for the target domain only using few labeled target data and well-trained source-domain classifier. Namely, we do not access any source data when training the target-domain classifier. This setting prevents the data leakage of the source domain passively. The lock means we cannot access data in the source domain.
domain. Compared SDA and UDA methods, FDA methods only require few data in the target domain, which is suitable to solve many problems, e.g., medical image processing [48]. Existing FDA methods involve many approaches and applications. Structural casual model [46] has been proposed to overcome the problem caused by apparent distribution discrapancy. Since deep neural networks tend to overfit the few-labeled data the training process, letanleg method becomes an effective solution to the FDA problem [45]. Besides, FDA methods perform well in face generation [51] and virtual-te to-real scene parsing [54].
However, it is risky to directly access source data for training a target-domain classifier (required by FDA methods) due the private information contained in the source domain. In the current era, labeled data are distributed over different physical devices and usually contain private information, e.g.. data on personal phones or from surveillance cameras [26]. Since FDA methods [46] require abundant labeled source data to train a target-domaij classifier, they may leak private information in the training process, which may result massive loss [19].
In this paper, to prevent the private data leakage of the source domain in existing FDA methods, we propose novel and very challenging problem setting, where the classifier for the target domain has to be trained using few labeled target data and a well-trained source-doman classifier, named few-shot hypothesis adaptation (FHA, Figure 1). In the literature [26], researchers have adapted source-domain hypothesis to be a target-domain classifier when abundant unlabeled target data are available. However, since these methods require abundant target data, they cannot address the FHA problem well, which has been empirically verified in Table I and Table 2.
The key benefit of FHA is that do not need to access the source data, which wisely avoids private-information leakage of source domain under mild assumptions. Besides, since the size of datasets of most domains is large in the real world, existing FDA methods will take a long time to train a target-domain classifier. However, in FHA, we train a target-domain classifier only with a source classifier and few labeled target data, reducing the computation cost greatly.
To address FHA, we first revisit the theory related to learning from few labeled data and try to find out if FHA can be addressed in principle. Fortunately, we find that, in semi-supervised learning (SSL) where only few labeled data available, researchers have already shown that, a good classifier can be learned if we have abundant unlabeled data that compatible with the labeled data. Thus, motivated by the SSL, we aim to address FHA via gradually generating highly compatible data for the target domain. To this end, we propose target-oriented hypothesis adaptation network (TOHAN) to solve the FHA problem. TOHAN maintains two deep networks simultaneously. in which one focuses on learning intermediate domain (i.e., learning compatible data) and the other takes care of the intermediate-Fe wwweenediitttt-ttraeeee to-tget distributional adaptation (Figure 2).
Specifically, due to the scarcity of target data, we cannot directly generate compatible data for the target domain. Thus, we first generate an intermediate domain where data are compatible with the given source classifier and the few labeled target data. Then, conduct the intermediate-to-te tt-tagtt

---

CNN
CNN
Trrpelion
ONN
Casalier
Encadrr 9
Cliaalier
CNN
Gropg
CNN
Dastzrnderr
Goap
Beb
Scatce
lraczeditt Data
Dasa
Candtirr bg
(a) Geserate interneddate domann.
Train group disctimnnttorD.
(c) Train clossifier It.
Figure 2: Overview of target-oriented hypothesis adaptation network (TOHAN), It consists of gener- ator G, encoder Da- St (initialize g(=g,), classifier has h2 (initialize 11, and group discriminator D. (a) Firstly, we train a generator G using the source classifier Ds- h2 and target data D,. Then we generate intermediate data between the two domains. (b) We freeze It and h2 and update group discriminator D. (c) We freeze D and update and h₁ In subfigures (b) and (c), they show data pair from G2, where the two data points come from the same class but different domains.
distributional adaptation to make the generated intermediate domain close the target domain. Eventually, we embed the above procedures into our one-step solution, TOHAN, to enable gradual generation of an intermediate domain that contains highly compatible data for the target domain. According to the learnability of SSL, with the generated "target-like" intermediate domain, TOHAN can learn good larget-domain classifier.
We conduct experiments on FHA tasks on 5 datasets (MNIST, SVHN, USPS, CIFAR-10 and STL-100). We compare TOHAN with 5 competitive baselines. Experiments show that TOHAN effectively transfers knowledge of the source hypothesis to train target-domain classifier when we only have few labeled target data. In other words, paper opens new door the domain adaptation field, which solves private-data leakage and data shortage simultaneously.
2 Few-shot Hypothesis Adaptation
In this section, we formalize a novel and challenging problem setting, called few-shot hypothesis adaptation (FHA). Let X' Rd be feature (input) space and y : (1,.-., .N be label (output) space, and N the number of classes. A domain [9] for the FHA problem is defined as follows.
Definition 1 (Domains for FHA). Given random variables X X, E X, Y,, Y₁ E Y, the source and target domains are joint distributions P (X,, Ys) and P (Xe, Yi), respectively, where the joint distributions P (X,, Y.) * (X.,, Yi) and X is compact.
Then the FHA problem is defined as follows.
Problem 1 (FHA), Given a model (consisting of an encoder gs and classifier h2) trained on the source domain P (X Y.) and independent and identically distributed (i.id.) labeled data Dj = (z), vi) The (ng & 7N, following [37]) drawn from the target domain P PX,, Yt), the aim of FHA is train classifier b4 : - with gs, h2 and D; such that h2 can accurately classify target data drawn from P Y.).
Remark 1. In FHA, there exists an assumption: malicious attackers cannot easily find source- domain-like like data from the Internet and via some other ways. Otherwise, attackers may use the attack methods [57] to recover the training data, leading to data leakage.
Possible Privacy-leakage Issues FHA. The assumption in Remark is derived from the attack methods that aim to recover training data from well-trained model. According recent model- inversion attack methods [57], they need to access auxiliary data whose background similar to the training data to help recover input data. There also exists white-box inference attack method [34] that determines a data point's 's membership in the training set of the model. Therefore, FHA belongs to passive protection, requiring the training data of source model sufficiently different from public data. To thoroughly avoid this issue, data owners might utilize the defending techniques (against the model-inversion attacks) train their source models.
Comparison with Few-shot Learning. The main difference between FHA and few-shot learning (FSL) is the representation of source domain. For FHA, source domain is represented by model
3

---

trained with source data. While, for FSL, source domain is represented by labeled data themselves [29, 30]. Besides, the data used to train source classifiers come from different domains from target data in FHA, while source data and target data come from the same domain in FSL. The works [14, 49] propose to hallucinate additional training examples to solve few-shot visual recognition, inspired by human's 's visual imagination. Meta-learning [11, 40] also performs well in FSL by learning the distribution of tasks with high generalization ability. As using few data for training easily leads to overfitting, there are works [10, 55J trying to constrain the hypothesis space to avoid it. Data augmentation generative adversarial network (DAGAN) [2] aims to augment target data through a conditional generative adversaria network to enhance the few-shot learning procedure.
Comparison with UDA. The main differences between FHA and UDA lie in the amount and label of data in the two domains. For the source domain, UDA requires a large amount of labeled data [6, 59], while FHA only requires a well-trained model. For the target domain, UDA requires large amount of unlabeled data [44, 60], while FHA requires few labeled data.
Comparison with FDA. With the development of FSL, researchers also apply ideas of FSL into domain adaptation, called few-shot domain adaptation (FDA). FADA [33] is representative FDA method, which pairs data from the source domain and data from the target domain and then follows the adversarial domain adaptation method. Casual mechanism transfer [46] is another novel FDA method dealing with a distribisiiau scenario, in which the data generating mechanism is invariant among domains. Nevertheless, FDA methods still need to access many labeled source data for training. which may cause the private-information leakage of the source domain.
Comparison with Hypothesis Transfer Learning. In hypothesis transfer learning (HTL), we can only access a well-trained source-domain classifier and small labeled or abundant unlabeled target data. [24] requires small labeled target data and uses the Leave-One-One error to find the optimal transfer parameters. Later, SHOT [26] is proposed solve the HTL with many unlabeled target data by freezing the source-domain classifier and learning a target-specific feature extraction module. As for the universal setting, a two-stage learning process [23] has been proposed address the HTL problem. Compared with FHA, HTL still requires at least small target data (e.g., at least 12 samples in binary classification problem [24], or at least 10% target data are labeled [i]). In FHA, we focus on more challenging situation: only few data (c.g.. one sample per class) are available. Besides, previous solutions to HTL. mainly focus on mortifying existing hypotheses or loss functions used for fine-tuning. However, our solution stems from the learnability of semi- remi-supervised learning (Section 3) and try to generate more compatible data, which quite different from previous works.
3 How to Learn from Few-shot Data in Principle
From the view of statistical learning theory [47], it is unrealistic to directly learn an accurate target- domain classifier only with few labeled data. However, the amount of labeled data in semi-supervised learning (SSL) [61] is also few (c.g., one sample per class), but SSL methods still achieves good performance across various learning tasks, which motivates us to consider solving FHA in the view of SSL. First, we will show theoretical analysis regarding learnability of SSL.
Learnability of SSL. For simplicity, we consider the 0-1 semi-supervised classification problem.
Let c² X x {0, 13 be the optimal target classifier and H = (h : X - {0, 13} is a hypothesis space. Let err P. In SSL, its lcarnability mainly depends on the compatibility X H. x + [0. 1] that measures how "compatible" h is to unlabeled data x. Let x(h, P) = Er- p[x,,, x)] be the expectation of compatibility of data from P on a classifier h. If the unlabeled data and c* are highly compatible (i.e., x(c* ; P) closes to 1), then, in theory, we can learn a good classifier with few labeled data and sufficient unlabeled data. Specifically, we have the following theorem (see proof in Appendix B).
Ex- p(((z) # c* cxx)] be the true error rate of a hypothesis h over distribution
Theorem 1. Let Xh, = J/ E Eves x(h, x) be the empirical compatibility over unlabeled dataset S. Let Ho = {h e H: @r(hh) = 0}. cf e H and x(c". P) = 1 - t, then m. unlabeled data and my labeled data are sufficient learn error with probability 1 - for
my = O [CCimm(((H))
1
1
2 5
log + -
log
(I)
62
4

---

and
2
4 8
In In(2Hp 2e) Pl) + In
(2)
where x(H) [xi h E HJ. Xx(-) x(h,)) and H.P.x (t + 2e) P] is the expected number of splits of 2m, data drawn from P using hypotheses H. of compatibility more than 1 1- 2e. In particular, with probability at least 1 6, we have err( h)) c, where
h = arg max {(h, S))
(3)
ho
Remark 2. If the unlabeled data are highly compatible to c* , I small, which results a smaller my. Namely, with the smaller my, can still achieve low error rate. In view of Theorem 1, it is clear that SSL will be learnable if many compatible unlabeled data are available. Motivated by SSL, we wonder if we can generate compatible data to help our learning task. The answer is affirmative.
Solving FHA in Principle. Motivated by Theorem I, finding many highly compatible unlabeled data is breakthrough point for FHA. Hence, generating unlabeled target data is a straightforward solution. However, due to the shortage of existing target data, directly generating them is unrealistic. To solve this problem, can ask for help from the source classifier. In paper, first try to generate intermediate domain Pm containing knowledge of source and target domains, which are compatible with both the source classifier and target classifier, i.e.,
Pm = arg max[x(hs, P) + x(he, P)],
(4)
where x(h,, P) (resp. X(,,, P)) measures how compatible he (resp. h₂) is with the data distribution P. Then, we will adapt intermediate domain Pm. to the target domain via distributional adaptation with the training procedure going on. Finally, we can obtain many unlabeled data that are compatible with h, and h, (more compatible with he), meaning that, based on Theorem 1, we can address FHA in principle. According to Eq. (4), it can be seen that we can have two straightforward solutions: maximizing xhh,, P) or x(h, P), corresponding to S+FADA and T+FADA in benchmark solutions. The results in Table and Table indicate that these two straightforward solutions cannot address FHA well, which motivates us to maximize them simultaneously, which is realized below.
4 Target-Oriented Hypothesis Adaptation Network for FHA Problem
This section presents a powerful one-step approach: target-oriented hypothesis adaptation network (TOHAN, see Figure 2). 'OHAN can generate data that are highly compatible with both the source classifier and target classifier and adapt the knowledge of these data to the target domain gradually.
Intermediate domain generation. The first step of TOHAN is to generate the intermediate domain data (Figure 2a). We input Gaussian random noise Z to a generator G,, (taking the thth class for an example), then the generator outputs generated data. We aim to generate data satisfying (1) the generated data (2) can be correctly classified by the given source classifier Sa = ht o Ja and (2) Gz) becomes closer to the target domain with training procedure going on. Thus, there are two loss functions regarding the intermediate domain generation. The first one is as follows.
Without loss of the generality, we assume (z(z) generates B images, where B is the batchsize in the training process of TOHAN. When G.,(z) is inputted to the source-domain classifier fs, we will obtain an B x N matrix G.M. where the 5th row in G.M represents probability of the 7th generated image belonging each class. Thus, the th column in G, represents the probability that the B generated images belongs to the th class, and we denote the nh column in G.M ln- Since G.(z)) aims to generate data belonging to the th class, we should update parameters of G to make each element in 4, close to 1. Namely, the first loss function to train the G, can be defined
LG
B ||, - 11
(5)
where 1 is B-by-1 vector whose elements 1.
As discussed before, we also want to reduce the distance between the generated data Gx)))) and the target data whose labels are In this way, make the generated data close to the target
5

---

domain and attain an intermediate domain Pm. Following [27]. we adopt an augmented L1 distance ||x - 1 = E wi X; Yil. where w/ = |X; Y.I /| 22 Compared to the ordinary & norm, the augmented L1 distance encourages larger gradients for feature dimensions with higher residual error [27]. Compared to the & norm, since L1 distance is more robust to outierss [36], it is better to measure the distance between generated images and target images. Thus, the second loss to train G,, is defined as follows,
C.
(6)
MBK
m
where M max ||a1 r2l11 (X' is compact and 1 1
is continuous) and Gn((2)
Combining Eq. (5) and Eq. (6), we obtain the total loss to train the generator Gn:
B
K
La. = LG. + AC.. = B 142 - 11 MBK
Σ
(7)
where À is a hyper-parameter between two losses to tradeoff the weight of knowledge of source and target domains. To ensure that the generated data are high-quality images, we train the generator G, (n = -..... N) for some steps all alone. Note that, Eq. (7) corresponds to Eq. (4), and Eq. (5) (resp. Eq. (6)) corresponding to xhh,, Pm) (resp. x(he, Pm)).. Then we conduct
distributional adaptation (see the next paragraph) and generation simultaneously.
Intermediate--to-trgee distributional adaptation. Now, we focus on how to construct domain- invuriant representutions (DIP) between the intermediate domain and the target domain. Through DIP, a classifier for the intermediate domain can be used to classify target data well [28, 59].
Since we only have few target data per class, we aim to "augment" them. Following [33]. we can overcome the shortage of target data by pairing them with the corresponding intermediate data, Specifically, we create 4 groups of data pairs: G1 consists of data pairs from the same domain with the same label, G2 consists of pairs from different domains (one from the intermediate and one from the target domain) but with the same label, G3 consists of pairs from the same domain with different labels, and G4 consists of pairs from different domains (one from the intermediate and one from the target domain) and with different labels.
Based the above four groups, construct four-class group discriminator D to decide which of the four groups a given data pair belongs to, which differs from classical adversarial domain adaptation [12, 20). The group discriminator D aims classify the data pair groups. As a classification problem. we train D with the standard categorical cross-entropy loss:
LD = E Σ yg. log (D (D(O
(8)
where [8| represents the empirical mean value, yG. is the label of group Gi and (GG) = [9a(x1), de (. (22 Gz, and ge is the encoder target domain. Note that freeze g when minimizing the above loss function (see Figure 2b).
Next, we turn to train Sr and ht with the group discriminator D fixed, which confuses D between Gi and G2 (also G3 and G4). However, we need D to correctly discriminate positive pairs (Gi, G2) from negative pairs (G3, G1). This means that domain confusion and classification are realized at the same time. We firstly initialize Se and fú with the same weight as gs and he, respectively. Motivated by the non-saturating game [13], minimize the following loss to update gt and hú (see Figure 2c):
Chog - B [vc, log (D (ob (G2)) Ug. log (D ( (9+))) + (X) (z
where 3 is hhper--paree syper-parameter to tradeoff confusion and classification and é is the cross cross-entropy loss. fi := h2 is the target model and f is the optimal target model. Corresponding to Theorem 1, optimizing the first term in Eq. (9) increases compatibility of the target model with the intermediate data, and optimizing the second term in Eq. (9) reduces (Fr(),) resulting in smaller err(h)) Compared [33], Eq. (9) means that we train the target model by confusing D and improving classification accuracy simultaneously.

---

Algorithm/ Target-oriented hypothesis adaptation network (TOHAN)
Input: encoder 9s. classifier ht,, D. = {ai, - learning rate Yı 72. 73 and 74. total epoch Taa pretraining D epoch Ta. adaptation eptch I',, network parameter (0c., )m= Ohegs Op.
1: Initialize (6G, | and 0p:
for = 1,2, 1.2....... maa do
2: Initialize Dm.
for 0, 0.1.....
do 3: Generate random noise
4: Generate data Gz(z) then add them Dm Update ta.. Ag. - ? VLa.. (z, De) using Eq. (7);
end
if = Tmaz T then Ta
for i= 1, =,,,...... ........Taa. do
6: Sample 91, Gs from Dm x Dm:
7: Sample 92, from Dm Dr;
Update On - 0n- - 7/ VDD (191)2-1) using Fq. (8);
end
end
if
> Tmaz 7'y then
9: Sample G1, Go from Dm x Dmi
10: Sample G2, Ga from Dm x Di;
11: Update Onog Onog 7/3 Cneg ((:) Im; 21) using Eq. (9);
12: Update On- OD - 74 VD (( using Eq. (8);
end
end
Output: the neural network he
one-step solution FHA. Although we can sequentially combine the above two steps to solve the FHA problem (i.e., two-step solution), the fixed intermediate domain (generated by the first step) may have large distributional discrepancy with the target domain. As result, such two-step solution may not obtain good target-domain classifier. To address this issue, we introduce a one-step solution TOHAN. The ablation study verifies that TOHAN outperforms such two-step solution (see S and TOHAN in Table 3).
TOHAN:
The entire training procedure of TOHAN is shown in Algorithm Since the convergence speed of generator G is relatively slow, the quality of generated data is poor at the beginning of the training process of G. Thus, we will train the generator for certain number of epochs before performing intermediatee-o-targe distributional adaptation (lines 2 to 5). When the generator can generate high-quality images, we train the generator and conduct adaptation together.
We train every generator Gn (n = 1,2,.... ... N) separately, and we generate the intermediate domain data using the latest generators. Then, we pair the intermediate data with the target data and pre- train the group discriminator D (lines 6 to 8). Next, we pair the intermediate data with target data and conduct the adaptation (lines 9 12). After conducting intermediate-to- to-taagge distributional adaptation, we obtain better and ht, i.e. classifying the target data more accurately. With the better target-domain classifier, we can make the generated intermediate data get closer to the target domain, in turn, these generated intermediate data further promote adaptation performance.
Experiments
We compare TOHAN with benchmark solutions on five standard supervised DA datasets: MNIST (M), SYHN(S), USPS(U), CIFAR-10 (CF), STL-10 (SL). We follow the standard domain-adappation protocols [39] and compare average accuracy of 5 independent repeated experiments. For digital datasets (i.e., M, S, and U), we choose the number of target data (per class) from to 7 [33]. For objects datasets (i.e., CF and ST.), we choose the number of target data as 10. Details regarding these datasets can be found in Appendix C. The code is available github. Haw.ub....moHHHooggg///HHHAA
Benchmark solutions for FHA Although the FHA a new problem setting, we still design benchmark solutions to this new problem. (1) Without adaptation (WA): to classify the target domain
7

---

Table I: Classification atanracy-standard deviation (%) on 6 digits FHA tasks. Bold value represents the highest accuracy on each column.
FHA Methods
Number Target Data per Class
Tasks
WA
5
6
26.7+110
26.8+2.1
26.8+1.6
27.0+0.7
1.2
27.5+00.8
1..3+1.5
SHOT
25.7+2.2
26.9+1.2
27.9+2.6
29.1+0.4
29.1+1.4
29.6+1.7
19.
M-S
24.1
S+F
1.3
27.710.5
27.810.7
1.
18.41
29.011.0
19.6
T+F
1.0
26.3100.8
28.911.0
1.3
29.2.1.33
31.910.4
3.
TOHAN
26.7.1.11
28.611.11
29.5114
2..6.0.4
30.5
32.110.2
33.2.1.8
FT 70.2+0.0
70.6
70.7.0.11
70.8
70,910.2
71.1
71.1
SHOT
12.621.
73.642.0
74.110.6
74.61112
74.900.7
75.4:0.3
16.
S-+M 70.2
S+F 1..4
83.1
83.311.
85.9
86.0
87.642.66
1.0
T+F 74.2+1.8
81.6+4.0
83.4+0.8
82.0+2.3
86,240.7
87.2+0.8
88.2
TOHAN
76.0=1.9
83.3+0.3
84.210.4
86.5+111
8..1+13
88.0
0..7
FT 7..4=0.7
7..7+119
76.9
77.3+111
77.6+1.
78.3+2.1
1..
SHOT
87.2+0.2
87.91..3
87.8+0.4
88,0=0.4
87.9
88.0+0.3
88.4+0.3
M-UU 69.7
S+F 83.7=0.9
86.010.4
86.1
86.5
86.8114
87.010.6
87.2+0.8
T+F
84.2+0.1
84.210.3
85.240.9
85.2-0.6
8..0
8..8
87.2+0.5
TOHAN
87.7+0.7
88.3+0.5
88.5+1.2
89.3+0.9
89,400.8
90.0+1.0
90.4+1.2
FT 83.5=0.4
84.312.4
84.510.7
8.
86.611.0
87.210.7
88.1
SHOT
83.1
85.5
85881066
86.0
86.610.2
86.710.2
87.010.1
U-M 82.9
S+F 83.2+0.2
84.010.3
85.011.2
85.6-0.5
85.7
86.210.6
87.2
T+F 82.9+0.7
83,9+0.2
84.7+0.8
85.4+0.6
85.6+0.7
86.3+0.9
86.6+0.7
TOHAN
84.0=0.5
85.2+0.3
85.6+0.7
86.5
87.3+0.6
88.2+0.77
89.2-0.5
64.9
66.511.5
66.711.7
67.311.1
68.1
68.3
69.7
SHOT
74.7+0.3
1.4
7.06+100
75.8=0.7
77.12211
77.8+1.6
79.6+0.6
S-U 64.3
S+F 1.
7..6+1.4
7.
76.2+1.3
7..2+1.7
77.8+3.0
1..7+
T+F 71.7=0.6
7..3+1.9
74.5+0.8
75.9+2.1
1..
76.8+1.8
79.7+1.9
TOHAN
7.9
7.28
79.4.0.9
80.2.1.6
10
81.11-111
1.96
FT
23.4=1.8
23.6+2.7
23.811.6
24.6=1.4
24.611.2
24.8
25.5+1.8
SHOT
3..
31.6+0.4
29.8+0.5
29.4+0.3
29.7+
29.8+0.8
30.1
U-S
17.3
S+F 28.1=1.2
2..7+1.3
29.0+
30.1+1.1
30.3+1.
30.7+1.0
30.9+1.5
T+F 1..
27.9+009
28.4+1..
19.4
29.5+0.7
30.2+1.0
30.4
TOHAN
29.9+1.2
30.5+1.2
31.4+1.1
3.98
33.1+100
34.0+1..0
35.1+1.8
with the source classifier (encoder 9. and classifier h2). (2) Fine-tuning (FT): to train the classifier gs with few owned target data. (3) SHOT: novel HTL method, where we modify it to use the labeled target data instead only using the unlabeled target data. [26]. (4) S+FADA (S+F): to generate faked source data with the source classifier then apply them to DANN [12]. (5) T+FADA (T+F): to generate fake target data with few real target data then apply them to DANN. We demonstrate details of benchmark solutions in Appendix D. Experimental details can be found in Appendix E. Moreover, we conduct additional experiments to compare existing HTL method named dkdHTL. [53], and related results and analysis can be found in Appendix F.
Results on digits FHA tasks. We conduct experiments digits FHA tasks: M S- M- U-M, S-U and S. Table reports the target-domain classification accuracy of 6 methods It is clear that TOHAN performs the best on almost every task. On M-SS S-M, M-U and S-U, TOHAN out- performs all benchmark solutions ob- viously. However, on the tasks U-M and U-S, the accuracy of TOHAN is slightly lower than SHOT when the amount of target data is too small (n = 1, This abnormal phenomenon shows that TOHAN cannot generate intermediate domain data effectively with very little target data, especially when the resolution of source data much smaller than that of target data. In this case, the data we generate is close to the source domain, TOHAN degrades to S+FADA
725
DAPN
DAPN
06.6
Accormoy
Accuracy
04.5
71
22
6 digits FHA tasks.
70
52.
12
20
30
42
50
10
20
30
53
Epoch
Epoch
CF
SL
SL CF
Figure 3: TOHAN vs DAPN.

---

Table 2: Classification accuracyIssanddard deviation (%) on 2 objects FHA tasks: CIFAR-10 STL-10 (CF-SL) and STL-10 - CIFAR-10 (SL-CF), Bold value represents the highest accuracy (%) among TOHAN and benchmark solutions.
Methods
WA
FT
SHOT
S+F
T+F
TOHAN
CF-
70.6
7..5+
71.900.4
72.1
71.3+0.5
72.8+0.11
SL--F
51.8
54.310.5
53.910.2
56.910.5
55.8
56.610.3
In Appendix G, use t-SNE to visualize the features extracted by TOHAN and 5 benchmark solutions on M-U task (see Figure 8 in Appendix G). When we use WA and FT methods, nearly all classes mix together. Although the classification accuracies of SHOT, S+F and T+F are relatively high, there are still some mixing among classes. For TOHAN, it can be seen that all classes are separated well, which demonstrates that TOHAN works well for solving the FHA problem.
Results on objects FHA tasks. Following [39]. we also evaluate TOHAN and benchmark solutions on 2 objects FHA tasks: SL - CF and CF - SL, and the results are shown in Table 2. Considering the complexity of datasets and the difficulty of our problem setting, we do not have amazing results like digits tasks. In SL - CFF, we achieve of 4.8% improvement over WA and a performance accuracy of 56.9%. Note that because the numbers of pixels per image of CF and SL are quite different, the images from SL lose a lot of information when inputted to the pre-trained model of C.F, thus making the effects of TOHAN and benchmark solutions are not obvious for CF SL.
Comparing TOHAN with FSL methods. As mentioned above, FHA is a difficult case of FSL where the prior knowledge is pre-trained model of another domain. To test the effectiveness of FSL methods FHA, we compare TOHAN with a novel FSL method called domain alaptive few-shot learning (DAPN) [58]. Note that
domain-aulqptive
the same pre-trained model in both TOHAN and DAPN. Taking CF + SL with five target data (per class) as an example, we solve FHA with TOHAN and DAPN and show the results in Figure 3. It is clear that TOHAN outperforms DAPN when the training epoch (t) is relatively large.
Ablation Study. Finally, we study the advantages of one-step method over other two-step methods. We consider the following baselines: S+F, T+F and ST+FADA (ST+F). We have explained S+F and T+F previously. ST+F denotes the two-step version of TOHAN, i.e, to conduct intermediate domain generation and intermediate-to- target distributional adaptation separately. We make ablation study on three digital datasets mentioned before as an example.
(w) SHADAA
S
As shown in Table 3, it is clear that TOHAN works better than the other baselines. The generator of S+F uses the loss LG.. which merely contains knowledge from the source domain. The generator of T+F uses the loss EG and ignores the knowledge contained in the source-domain classifier. In contrast, TOHAN uses both LG and Co.. As a result, TOHAN achieves higher accuracy than S+F and T+F. Besides, the generators and classifiers in TOHAN promote each other in the training process, which results in that TOHAN performs better than the ST+F In Figure 4, we visualize the data generated by S+FADA and TOHAN. It is clear that data generated by S+FADA are chaotic that contain little useful information. However, data generated by TOHAN contain many torget-domain high-level visual features, and they can be classified by the source classifier accurately, resulting in a better performance in FHA. The detailed analysis of ablation study can be found in Appendix G.
(b) TOHAN
Figure 4: Visualization of
S+FADA and TOHAN.
Verification of No Source-data Leakage in Intermediate Domain. As key contribution, TO- HAN solves FHA through generating intermediate data. To guarantee that source data are leaked, we need to verify that there is no source-domain features in the intermediate data. We determine this by calculating the PSNR values [15, 52] between each intermediate sample and all source samples.

---

Table Ablation study. We show the average accuracy of the tasks on digits datasets in this table. Bold value represents the highest accuracy (%) on each column. See full results in Appendix G.
FHA
Number of Target Data per Class
Methods
3
5
7
S+F
61.2
63.0
64.3
65.4
65.7
66.4
67.2
T+F
61.0
63.0
64.2
64.5
65.7
66.5
67.4
ST+F
61.8
64.5
64.9
65.8
66.5
67.3
68.4
TOHAN
63.3
65.4
66.4
67.5
68.0
68.9
70.0
Table 4: The top-5 largest PSNR values between intermediate samples and all source samples.
Ranking
3
PSNR value
17.8951
17.8948
17.8948
17.8947 17,8947
PSNR indicates the generation quality of an image f given standard image 9, and is defined as
255²
1
M N
PSNR(J.g) 10
where MSE(S.9)
Σ
U3 9(5)
MSE(f.g)
MN
The larger PSNR value indicates the two images more similar. Therefore, taking M-S as an example, we report the top-5 largest PSNR values in Table 4. That is, we check whether the worst case satisfies our claim. For com- parison, we also compute the PSNR values of (source data, source data), (target data, target data), (source data, target data), and (intermedi- ate data, target data), and we report the average PSNR values of the above four cases Figure As can be seen, the intermediate data are much closer to the target data, and they are very different from the source data. The average PSNR between source data and target data is 9.3209. The top-5 largest PSNR values between each intermediate data and all source data (= 17.89) are obviously smaller than 50.1722 (the average PSNR between intermediate data and target data). Through this result, we can state that intermediate data are similar to the target data and very different from the source data. Therefore, the above evidence shows that the generated intermediate data contain no source domain features, and the source data do not leak when generating the intermediate data.
50.1722
52,3372
54.3006
Source
Intermedlate
Target
9.3209
The average PSNR values of (source data, source data), (target data, target data), (source data, target data), and (intermediate data, target data).
Figure 5:
Conclusion
This paper presents very challenging problem setting called few-shot hypothesis adaptation (FHA), which trains a target-domain classifier with only few labeled target data and a well-trained source- domain classifier. Since private information in the source domain are protected well. To this end, we propose novel one- step FHA method, called target-oriented hypothesis adaptation network (TOHAN). Experiments conducted on FHA tasks confirm that TOHAN effectively adapts the source-domain classifier to the target domain and outperforms competitive benchmark solutions the FHA problem.
few-shothyotthestt
only access well-trained source-domain classifier in FHA, the
Acknowledgments and Disclosure of Funding
This work was partially supported by the National Natural Science Foundation of China (No. 91948303-1, No. 61803375, No. 12002380, No. 62106278, No. 62101575, No. 61906210) and the National Grand R&D Plan (Grant No. 2020AAA0103501), would also like to thank Dr. Yanbin Liu for productive discussions.
10

---

References
[1] Sk Miraj Ahmed, Aske R. Lejbølle, Rameswar Panda, and Amit K. Koy-Chowdhury. Camera on-boarding for person re-identification using hypothesis transfer learning. In CVPR, 2020.
[2] Antreas Antoniou, Amos Storkey, and Harrison Edwards. Data augmentation generative adversarial networks. In ICLR Workshop, 2018.
[3] Sercan Ömer Arik, Jitong Chen, Kainan Peng, Wei Ping, and Yanqi Zhou. Neural voice cloning with few samples. In NeurlPS, 2018.
[4] Adam Coates, Andrew Ng, and Honglak Lee. An Analysis of Single Layer Networks in Unsupervised Feature Learning. In AISTATS, 2011.
[5] Luc Devroye, László Györfi, and Gábor Lugosi. A Probublistic Theory of Paltern Recognition. Springer, 1996.
[6] Jiahua Dong, Yang Cong, Gan Sun, and Dongdong Hou. Semaatic-transferbble weakly- supervised endoscopic lesions segmentation. In ICCV, 2019.
[7] Jiahua Dong, Yang Cong, Gan Sun, Bineng Zhong, and Xiaowei Xu. What can be transferred: Unsupervised domain adaptation for endoscopic lesions segmentation. In CVPR, 2020.
[8] Harrison Edwards and Amos Storkey. Towards neural statistician. In ICLR, 2017.
[9] Zhen Fang, Jie Lu, Feng Liu, Junyu Xuan, and Guangquan Zhang. Open set domain adaptation: Theoretical bound and algorithm. IEEE Transactions on Neural Networks and Learning Systems, 32(10):4309-453 4322, 2021.
[10] Li Fei-Fei, Robert Fergus, and Pietro Perona. One-shot learning of object categories. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(4):594-611, 2006.
[11] Chelsea Finn, Picter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adapta- tion of deep networks. ICML, 2017.
[12] Y. Ganin, F. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and
V. Lempitsky. Domain-adversarii training of neural networks. Journal of Machine Learning Research, 17(59)1-35, 2016.
[13] Ian Goodfellow. NeurIPS 2016 tutorial: Generative adversarial networks. arXiv preprint arXiv:1001.00660, 2016.
[14] Bharath Hariharan and Ross B. Girshick. Low-shot visual recognition by shrinking and hallucinating features. In ICCV, 2017.
[15] Alain Horé and Djemel Ziou. Image quality metries: PSNR vs. SSIM. In ICPR, 2010.
[16] Yunzhong Hou and Liang Zheng. Source free domain adaptation with image translation. arXiv preprint arXiv:2008. 2000000000.444 14, 2020.
[17] Gao Huang. Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, 2017.
[18] Jonathan J. Hull. A database for handwritten text recognition research. IEEE Transactions on Pattern Analysis and Machine Intelligence, 16(5):550-554, 1994.
[19] Bargav Jayaraman and David Evans. Evaluating differentially private machine learning in practice. In USENIX, 2019.
[20] Pin Jiang, Aming Wu, Yahong Han, Yunfeng Shao, Meiyu Qi, and Bingshuai Li. Bidirectional adversarial training for semi-supervised domain adaptation. In IJCAI, 2020.
[21] Yongcheng Jing, Xiao Liu, Yukang Ding, Xinehao Wang, Errui Ding, Mingli Song, and Shilei Wen. Dynamic instance normalization for arbitrary style transfer. In AAAI, 2020.
11

---

[22] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical Report TR-2009, University of Toronto, Toronto, 2009.
[23] Jogendra Nath Kundu, Naveen Venkat, R Venkatesh Babu, et al. Universal source-free domain adaptation. In CVPR, 2020.
[24] Ilja Kuzborskij and Francesco Orabona. Stability and hypothesis transfer learning. In ICML, 2013.
[25] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 1):2278-2324, 1998.
[26] Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In ICML, 2020.
[27] Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, 2017.
[28] Feng Liu, Wenkai Xu, Jie Lu, Guangquan Zhang, Arthur Gretton, and Danica J. Sutherland. Learning deep kernels for non-parametric two-sample tests. In ICML, 2020.
[29] Yanbin Liu, Juho Lee, Minseop Park, Sachoon Kim, Eunho Yang, Sung Ju Hwang, and Yi Yang. Learning to propagate labels: Transductive propagation network for few-shot learning. In ICLR, 2019.
[30] Yanbin Liu, Juho Lee, Linchao Zhu, Ling Chen, Humphrey Shi, and Yi Yang. A multi-mode modulator for multi-domain feo-shot classification. In ICCV, 2021.
[31] Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael Jordan. Conditional adversarial domain adaptation. In NeurIPS, 2018.
[32] Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta-learner. In ICLR, 2018
[33] Saeid Motiian, Quinn Jones, Seyed Iranmanesh, and Gianfranco Doretto. Few-shot adversarial domain adaptation. In NeurIPS, 2017.
[34] Milad Nasr, Reza Shokri, and Amir Houmansadr. Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning. In Proceedings of the 2019 IEEE Symposium on Security and Privacy, 2019.
[35] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In NeurIPS, 2011.
[36] Feiping Nie, Heng Huang, Xiao Cai, and Chris H. Q. Ding, Efficient and robust feature selection viajoint 2, 1-norms minimization. In NeurIPS, 2010.
[37] Seonwook Park, Shalini De Mello, Pavlo Molchanov, Umar lqbal, Otmar Hilliges, and Jan Kautz. Few-shot adaptive gaze estimation. In ICCV, 2019.
[38] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In ICLR, 2015.
[39] Rui Shu, Hung H. Bui, Hirokazu Narui, and Stefano Ermon. A DIRT-T approach to unsupervised domain adaptation. In ICLR, 2018.
[40] Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning. In NewrlPS, 2017.
[41] Jie Song, Yixin Chen, Xinchao Wang, Chengchao Shen, and Mingli Song. Deep model transferability from attribution maps. In NeurlPS, 2019.
[42] Jie Song, Yixin Chen, Jingwen Ye, Xinchao Wang, Chengchao Shen, Feng Mao, and Mingli Song. DEPARA: deep attribution graph for deep knowledge transferability. In CVPR, 2020.
12

---

[43] S. Sukhija, N.C. Krishnan, and G. Singh. Supervised heterogeneous domain adaptation via random forests. In IJCAI, 2016.
[44] Gan Sun, Yang Cong, Jiahua Dong, Yuyang Liu, Zhengming Ding, and Haibin Yu. What and how: Generalized lifelong spectral clustering via dual memory. IEEE Transactions Pattern Analysis and Machine Intelligence, 2021.
[45] Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele. Meta-transfer learning for few-shot learning. In CVPR, 2019.
[46] Takeshi Teshima, Issei Sato, and Masashi Sugiyama. Few-shot domain adaptation by causal mechanism transfer. In ICML, 2020.
[47] Vladimir N. Vapnik. Statistical Learning Theory. Wiley, 1998.
[48] Dong Wang, Yuan Zhang, Kexin Zhang, and Liwei Wang. Focalmix: Semi-supervised learning for medical image detection. In CVPR, 2020.
[49] Yu-Xiong Wang, Ross B. Girshick, Martial Hebert, and Bharath Hariharan. Low-shot learning from imaginary data. In CVPR, 2018.
[50] Ying Wei, Yu Zhang, Junzhou Huang, and Qiang Yang. Transfer learning via learning transfer. In ICML, 2018.
[51] Chao Yang and Ser-Nam Lim. One-shot domain adaptation for face generation. In CVPR, 2020.
[52] Hongxu Yin, Arun Mallya, Arash Vahdat, Jose M. Alvarez, Jan Kautz, and Pavlo Molchanov. See through gradients: Image batch recovery via gradinversion. CVPR, 2021.
[53] Yiqin Yu, Xu Min, Shiwan Zhao, Jing Mci, Fei Wang, Dongsheng Li, Kenney Ng, and Shaochun Li. Dynamic knowledge distillation for black-box hypothesis transfer learning. arXiv:2007. 12355, 2020.
[54] Junyi Zhang, Ziliang Chen, Junying Huang, Liang Lin, and Dongyu Zhang. Few-shot structured domain adaptation for virtual-to-real scene parsing. In ICCV, 2019.
[55] Yabin Zhang, Hui Tang, and Kui Jia. Fine-grained visual categorization using meta- learning optimization with sample selection of auxiliary data. In ECCV, 2018.
[56] Yiyang Zhang, Feng Liu, Zhen Fang, Bo Yuan, Guangquan Zhang, and Jie Lu. Clarinet: A one-step approach towards budget-friendly unsupervised domain adaptation. IJCAI, 2020.
[57] Yuheng Zhang, Ruoxi Jia, Hengzhi Pei, Wenxiao Wang, Bo Li, and Dawn Song. The secret revealer: Generative model-inversion attacks against deep neural networks. In CVPR, 2020.
[58) An Zhao, Mingyu Ding, Zhiwu Lu, Tao Xiang, Yulei Niu, Jiechao Guan, Ji-Rong Wen, and Ping Luo. Domain-adaptive few-shot learning. In WACV, 2021.
[59] Li Zhong. Zhen Fang, Feng Liu, Jie Lu, Bo Yuan, and Guangquan Zhang. How does the combined risk affect the performance of unsupervised domain adaptation approaches? In AAAI. 2021.
[60] Li Zhong, Zhen Fang, Feng Liu, Bo Yuan, Guangquan Zhang, and Jie Lu. Bridging the theoretical bound and deep algorithms for open set domain adaptation. IEEE Transactions on Neural Networks and Learning Systems, 2021.
[61] Xiaojin Zhu. Semi-supervised learning. In Encyclopedia of Machine Learning, pages 892 392-897. 2010.
13


Reviews:
[{"review_id": "b3g4sjkIyCN", "reviewer": "Reviewer_6L9x", "summary": "This paper focuses on a problem setting where data are not available but classifiers are. The possible benefits of this setting lie in the privacy reservation, i.e., data used in previous few-shot learning may leak the private information. The proposed solution does not use a fine-tuning-based method but a sample-generation-based method, which is motivated by the learnability of semi-supervised learning. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed solution thought few-shot learning in a different way. The most of current methods address few-shot learning via finding/digging relation among available data and few-shot data. Data can be from different distributions (although they should be related) [r1]. In this paper, generating samples for the target domain is proposed and validated. If the samples are suitable for the target domain, a good classifier should be achievable (like SSL).\n\nMajor concerns/questions: \n\n   1. Although generating samples look novel and interesting, how to ensure a reliable generating process is unclear to me. There are many ways to influence the process and make the generated sample unreliable. Regarding this concern, I am not sure why the source domain can help as well. If the source cannot help, the generated samples must be unreliable, is that right? \n\n   2. I notice that HTL might be a very similar setting to the proposed setting. What are the major differences between both? Have you tried to compare TOHAN with HTL methods?\n\n   3. A large pre-trained classifier also seems suitable for solving the proposed setting. Such a classifier contains enough knowledge that is useful for the target domain as well. If so, what is the meaning to consider the proposed setting and solution?\n\n  4. Since you use a generator (DC-GAN) for each class, I was worried about the computational cost when scaling the method to larger datasets. 10 generators might be acceptable, but when the number becomes 200 or even 1000, is there any way to alleviate the computational cost? \n\n  5. I do not understand $\\mathcal{H}_{P,X}$ in Eq. (2). How is it calculated? Also, what is the meaning of the $t$ in the definition of Theorem 1? I guess it is some kind of form/assumption but I think the authors should elaborate on it here.  The subscript of $M$ in line 71 appears abruptly without any description. \n\n  6. Why do you introduce M in Eq. (6)? Does this M influence the final results?\n\n[r1] Model-agnostic meta-learning for fast adaptation of deep networks. ICML 2017.\n\n----\n## Post-Rebuttal Update\n----\n\nAfter viewing the rebuttal and other reviewers' comments, I feel more confident in the authors' methods. I think the authors did a great job in addressing my questions, so I have decided to raise my score to 7. There is something the authors need to address more but I do think it does not diminish the contribution of this work. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations and potential negative societal impact of their work have been discussed well.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "NLA99sF7Lnm", "reviewer": "Reviewer_FhwF", "summary": "This paper focuses on solving few-shot domain adaptation problem without accessing to source data during the training time, i.e., hypothesis transfer. The proposed method \"TOHAN\" uses two DNNs: one learns intermediate domain while another minimize the target risk and intermediate-target discrepancy.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Hypothesis transfer that this paper focuses on is an interesting and important problem which can protect pravicy. Also, only get access to a few labeled target data makes this problem more challenging. By solving the few-shot problem, the proposed method generates intermediate data using a generator and fixed source model. Then the distribution discrepancy between intermediate data and the target data are minimized.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Generally, the problem setting is interesting and the solution is reasonable. My major concerns are:\n1. How much time does it take running one epoch?\n2. Since there are only a few target data, is there a balanced problem between target data and the intermediate data?\n3. Since the distance between intermediate data and the target data is minimized in the following step, does it important to make the intermediate data look like target data in the generation step?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "H1dVWzZlmc2", "reviewer": "Reviewer_4XQu", "summary": "As the increasing need for privacy protection, previous few-shot DA (FDA) has limited applications in the real world. To overcome this drawback of the FDA, this paper considers a very challenging problem setting, few-shot hypothesis adaptation (FHA), which is the most difficult part of hypothesis transfer learning (HTL). Compared to previous solutions to HTL, the authors argue that generating target samples is a good solution. The proposed method is validated on common datasets.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n1.  The considered problem setting is practical and is not solved well in the literature. In the current era, how to protect privacy is quite important. If we can use well-trained classifiers (from other domains) to help obtain a good classifier for the target domain, many more available well-trained classifiers can be used, which can save a lot of energy to train a deep network for the target domain or to train a very big generalized deep network. \n\n2. Compared to existing fine-tuning methods, the proposed solution considers the domain gap between the backbone network and the target domain, which is a research gap in the current literature. Besides, generating unlabeled data for the target domain sounds interesting, especially in few-shot situations.\n\n3. Experiments are conducted on five data sets. Baselines, including FDA methods and HTL methods, are used to validate if the proposed method can handle FHA well.\n\nCons:\n\n1. In Problem 1, we have two domains: source and target. However, if the source domain is a random domain, can we still obtain a useful classifier for the target domain? The key assumption that makes FHA solvable is missing. What is the key to ensuring that FHA can be solved? This is the major concern about the considered setting.\n\n2. The proposed solution, TOHAN, is a hybrid one that repeats generating and adapting. However, how could you know the initial steps being good? At the initial stage, we might not get good target samples, right? How could we ensure we can get good results from possible bad target samples in the loop mentioned above?\n\n3. SHOT is actually not a classical HTL method. Network-based HTL methods should be also considered as a baseline. Besides, more HTL methods should be reviewed, such as multi-source HTL methods.\n\n4. What is the performance of the method that does not use the group information to implement the domain alignment? I did not see the results of this kind of method.\n\n5. In Problem 1, n_t is less than 7 seems too extreme?\n\n6. Grammar issues should be addressed in the revised version.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors have adequately addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12 hours"}, "extra_text": {}}, {"review_id": "04kb4qSobXi", "reviewer": "Reviewer_1Jpd", "summary": "This paper proposes a novel problem in domain adaptation - Few-shot Hypothesis Adaptation (FHA) - that combines Hypothesis Transfer Learning and the Few-Shot Domain Adaptation problem. Specifically, it assumes a lack of source domain data but the presence of a classifier (hypothesis) pretrained on the source domain and a few labelled data points (<=10 per class) from the target domain (with no additional data). The paper also adapts the FADA method and proposes a novel method, called TOHAN, which uses the source-domain hypothesis and the few target-domain data points to generate the intermediate-domain data compatible with the two domains.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\\+ means strength, \\- means weakness\n\n**Originality**\n\n\\+ *Novel setting*. As far as I am aware, the paper proposes a novel setting - Few-shot Hypothesis Adaptation (FHA) - a combination of existing problems - Hypothesis Transfer Learning and the Few-Shot Domain Adaptation.\n\n\\+/\\- *Somewhat novel method*. As far as I am aware, the paper also proposes a novel method - TOHAN - which is a minor adaptation of FADA [23] into the setting. The method architecture seems to be heavily inspired by FADA [23]. Unlike FADA, TOHAN generates and uses the generated intermediate domain instead of the original source domain. However, apart from that, it is not entirely clear what the technical differences are between these methods. Which line in Algorithm 1 would be different for FADA / S-FADA / T-FADA / ST-FADA?\n\n\\- *The relation between FHA and FSL is poorly explained, and FSL is poorly cited.* I found the sentence in lines 88-89 particularly vague and poorly written. In addition, line 90 that compares TSN [37] and ProtoNets [29] (\"which is relatively weaker than the former\") brings little value to the paper. In my humble opinion, these methods were designed for two different problems (i.e. video action classification [37] and single image classification [29]), and it is inappropriate to compare them in this way. The authors might find the following works from FSL literature more related to their work: (Antreas et al., 2018), (Hariharan & Girshick., 2017), (Wang et al., 2018). Generally, these methods also generate/hallucinate samples from limited target-domain data and should be cited. \n\n**Quality**\n\n\\+ *Some good experiments.* The paper performs a solid number of comparisons with representative baselines from HTL and FDA literature ([19] and [23], respectively) and basic fine-tuning baseline proposed by the authors. \n\n\\+/\\- *Work seems grounded in some theoretical work*. However, I did not attempt to verify the proof, so I cannot comment on its correctness.\n\n\\- *Only marginal improvements over baselines, mostly within the error bar range.* Although the authors claim the method performs better than the baselines, the error range is rather high, suggesting that the performance differences between some methods are not very significant. \n\n\\- *There might be a possible fundamental flaw to the claim of full data privacy.* The authors claim that the TOHAN method (and therefore also the FHA problem) \"strictly\" protects the privacy of the source domain by using the source hypothesis rather than the source data itself (lines 245-247). However, the claim that knowledge is \"completely\" inaccessible may be false. For instance, Ateniese et al. (2015) have shown that it is possible to extract certain types of information from pretrained classifiers and models. In a more recent privacy analysis of deep learning, Nasr et al. (2019) suggest that even well-trained models might leak a significant amount of information about the data they were trained on. Moreover, the proposed TOHAN relies on the leaked source-domain knowledge to generator appropriate source-domain data. Therefore, it seems to me that claim that TOHAN is privacy secure is completely false.\n\n\\- *Lack of sufficient evidence for no source domain features in intermediate data.* The authors claim that \"high-level, visual and useful features of source domain are rare in the generated intermediate data (Figure 6)\" (lines 243-244); however, no empirical value is provided to support this claim (i.e. what do authors mean by \"rare\" in this context? how rare is \"rare\"?). Figure 6a/b does support this claim as it shows only 4 intermediate-domain images and no examples of the original source-domain data. This is insufficient evidence to draw the conclusion in lines 243-244, and in fact, it points to the contrary. \n\n**Clarity**\n\n\\- *The paper is poorly written.* The paper is not particularly easy to read. It contains many spelling and grammatical errors (see below for proposed corrections). There is a large number of acronyms (e.g. FHA, SHOT, HTL, ST+F etc..) and some confusing mathematical notation (e.g. $D$, $\\mathcal{D}$, $X$ and $\\mathcal{X}$ refer to different entities) which make this work more confusing to read. The notation in the figure is inconsistent with the main text (uses $\\mathcal{X}$ instead of $\\mathcal{D}$). \n\n**Significance**\n\n\\- *The paper presents a novel and interesting problem but could be flawed.* This novel problem and method could have important consequences in the context of data privacy - however, to me, the idea seems fundamentally flawed (see my comments in the \"Quality\" subsection, or  \"Limitations And Societal Impact\"). \n\n\\- *The TOHAN improvements over baselines are mostly marginal.* Although the authors claim the method performs better than the baselines, the error range is rather high, suggesting that the performance differences between some methods are not very significant and the improvements are marginal. \n\n\n**Spelling, Grammer, and Other Possible Mistakes.**\n\n\\- Line 32, grammar: \"face of generation\" --> \"face generation\"\n\n\\- Figure 2 caption, grammar: \"which two data come from ... \" --> \"where the two data points come from ... \"\n\n\\- Line 128, wrong word: \"dependents\" --> \"depends\"\n\n\\- Line 167, redundant word: \"regarding to the ...\" --> \"regarding the ... \"\n\n\\- Line 210, redundant words: \"which confuses D unable to distinguish between ...\" --> \"which confuses D between ... \"\n\n\\- Line 213, wrong word: \"initial\" --> \"initialize\"\n\n**References**\n\nAteniese et al., 2015, \"Hacking smart machines with smarter ones: How to extract meaningful data from machine learning classifiers\", International Journal of Security and Networks, Volume 10, Issue 3\n\nNasr et al., 2019, \"Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learning,\" 2019 IEEE Symposium on Security and Privacy\n\nAntoniou et al., 2018, \"Data Augmentation Generative Adversarial Networks\", (ICLR 2018 Workshop)\n\nHariharan & Girshick., 2017, \"Low-shot Visual Recognition by Shrinking and Hallucinating Features\", (ICCV 2017)\n\nWang et al., 2018, \"Low-shot learning from imaginary data\" (CVPR 2018)\n\n\n\n_____\n### POST-REBUTTAL\n_____\n\nAfter a detailed discussion with the authors, I decided to increase my original rating from 3 to 7. The initial low rating was due to initially hidden assumptions and a poorly defined scope of data privacy which are central in the paper. These have been discussed and clarified by the authors during the rebuttal. The authors also addressed my concerns regarding the novelty and source-domain leakage into the intermediate domain. The authors have agreed to improve the clarity, literature review, dampen down on the privacy claims, and include additional experiments, and I am happy to increase the rating. Although the privacy claims are now not as strong as originally claimed (e.g. the method does not guard against source-information leakage, but rather shelters individual source data points from a possible data leakage), the paper still opens up an interesting area of research and presents a novel method that will likely attract attention in the community. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors claim that their method allows for full privacy of the source domain by relying on a well-trained source domain classifier. However, this seems to be based on the false assumption that the source domain classifier does not leak any private information. Recent studies (for example, Nasr et al. (2019) and Ateniese et al. (2015)) suggest that there might be a significant amount of source-domain information that pre-trained models leak. Moreover, TOHAN relies on leaking information from the source domain to generate realistic/compatible intermediate-domain data. This completely invalidates one of the main claims of the paper that private information is protected. ", "review_text": "", "extra_scores": {"ethical_concerns": "n/a", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "15"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper provides a few-shot learning technique that avoids looking at the data obtained to train previous tasks. As such it has promise for privacy preserving settings. The techniques seem solid, as well as the experimental setting. There where a few concerns related to over-promising in the area of differential privacy, however the reviewers agree that this can be fixed by the authors towards a camera-ready version. I recommend accepting the paper - it should be a good addition to Neurips
Decision: Accept (Spotlight)