Diversity of Generated Unlabeled Data Matters for Few-shot Hypothesis Adaptation
Authors: Ruijiang Dong
Feng Liu
Haoang Chi
Tongliang Liu
Mingming Gong
Gang Niu
Masashi Sugiyama
Bo Han
Venue: Submitted to ICLR 2023 (2023)
Keywords: Hypothesis Adaptation

Abstract: Generating unlabeled data has been recently shown to help address the few-shot hypothesis adaptation (FHA) problem, where we aim to train a classifier for the target domain with a few labeled target-domain data and a well-trained source-domain classifier (i.e., a source hypothesis), for the additional information of the highly-compatible unlabeled data. However, the generated data of the existing methods are extremely similar or even the same. The strong dependency among the generated data will lead the learning to fail. In this paper, we propose a diversity-enhancing generative network (DEG-Net) for the FHA problem, which can generate diverse unlabeled data with the help of a kernel independence measure: the Hilbert-Schmidt independence criterion (HSIC). Specifically, DEG-Net will generate data via minimizing the HSIC value (i.e., maximizing the independence) among the semantic features of the generated data. By DEG-Net, the generated unlabeled data are more diverse and more effective for addressing the FHA problem. Experimental results show that the DEG-Net outperforms existing FHA baselines and further verifies that generating diverse data plays an important role in addressing the FHA problem. 

Under review as a conference paper at ICLR 2023
DIVERSITY OF GENERATED UNLABELED DATA MAT- TERS FOR FEW-SHOT HYPOTHESIS ADAPTATION
Anonymous authors Paper under double-blind review
ABSTRACT
Generating unlabeled data has been recently shown to help address the few-shot hypothests adaptation (FHA) problem, where we aim train a classifier for the target domain with a few labeled target-domain data and a well-trained source- domain classifier (i.e., a source hypothesis), for the additional information of the highly-compatible unlabeled data. However, the generated data of the existing methods are extremely similar or even the same. The strong dependency among the generated data will lead the learning generative network (DEG-Nec -et) for the FHA problem, which can generate diverse unlabeled data with the help of a kernel independence measure: the HlibbrrtSShhiil independence criterion (HSIC). Specifically, DEG-Net will generate data via minimizing the HSIC value (i.e., maximizing the independence) among the semantic features of the generated data. By DEG-Net, the generated unlabeled data are more diverse and more effective for addressing the FHA problem. Experimental results show that the DEG-Net outperforms existing FHA baselines and further verifies that generating diverse data plays an important role addressing the FHA problem.
fail. In this paper, we propose
INTRODUCTION
Data and expert knowledge are always scarce in newly-emerging fields, thus it is very important and challenging study how to leverage knowledge from other similar fields help complete tasks in the new fields. To cope with this challenge, transfer learning methods were proposed to leverage knowledge of source domains (e.g., data in source domains or models trained with data in source domains) to help complete the tasks in other similar domains (a.k.a. the target domains) (Fang et al., 2020; Jing et al., 2020; Pan & Yang, 2009; Sun et al., 2019; Teshima et al., 2020; Zamir et al., 2018). Among many transfer learning methods, hypothesis transfer learning (HTL) methods have received lot of attention since it does not require access to the data in source domains, which prevents the data leakage issue and protects the data privacy (Chi et al., 2021a; Du et al., 2017; Liang al., 2020; Yang et al., 2021a;b). Recently, the few-shol hypothesis adapiation (FHA) problem has been formulated to make HTL more realistic, which is suitable to solve many problems (Liu al., 2021; Snell et al., cource-domain classifier
2017; Wang et al., 2020; Yang et al., 2020). In FHA, only a (i.e., source hypothesis) and few labeled domaen data are available (Chi al., 2021a).
Similar to HTL, FHA also aims to obtain a good target-domain classifier with the help of a source hypothesis and few target-domain data (Chi et al., 2021a; Motiian et al., 2017). Recently, generating unlabeled data has been shown to be an effective strategy to address FHA (Chi et al., 2021a). The target-oriented hypothesis adaptation network (TOHAN), a one-step solution to the FHA problem, constructed an intermediate domain to enrich the training data. The data in the intermediate domain are highly compatible with both source domain and target domain (Balcan & Blum, 2010). By the generated unlabeled data in the intermediate domain, TOHAN partially overcame the problems caused by data scarce in the target domain.
However, the existing methods ignore the diversity of the generated data or the independence among the generated data, so that the generated data are extremely similar or even the same. Lack of diversity leads to less effective data for addressing the FHA problem. Taking the FHA task of digits datasets as an example, we found that the data generated by TOHAN has an issue that the generator tends to copy target data (Figure (a)). To show how diversity matters in the FHA problem, conduct the

---

Under review as a conference paper at ICLR 2023
larget data
generated data
Source Data
Target Data
Generated Data
45
The accuracy (%)
35
39
20
50
40
60
The Number of Data
(a) The copy issue of gener- ated data on M -> S task
(b) Classification accuracy different amount of unlabeled data drawn from different domains on M - S task
The low-diversity issue of generated unlabeled data when solving the FHA problem. Subfigure (a) illustrates the labeled data (left) drawn from the target domain and unlabeled data (right) generated by TOHAN on the MNIST-SVHN (M + 5) task. It clear that the generated data are similar to each other and seem to copy the original target data (middle, left), Subfigure (b) illustrates the accuracy of the data drawn from different domains with different data volumes on the task M + S. For the source data and target data, the accuracy of the model trained by them is higher as the number of the data increases. For the generated data, the growth of data volume only helps to improve the accuracy of the model when it is small.
Figure 1:
experiments in the digits datasets. We use a few target labeled data and the increasing unlabeled data to train the target model. The result is shown in Figure 1(b). For the source data and target data, is clear that the accuracy of the model trained is higher as the number of the data increases. For the generated data, the growth of data volume only helps to improve the accuracy of the model when is small (e.g., less than 45 in Figure ((b)). However, the accuracy of the model fluctuates around 33% regardless of the increase in the unlabeled data, when the number of data exceeds 35. This result shows that the model trained by generate data converge faster than those trained by the source data and target data, since the generated data have less diversity.
In this paper, show how the diversity of unlabeled data (i.e., the independence among unlabeled data) affects the FHA methods, we theoretically analyze the affect of the sample complexity regarding the FHA problem (Theorem 1). In this analysis, we adopt the log-coefficient score (Dagan et al., 2019) to measure the dependency among unlabeled data. Our results show that we can still count on the unlabeled data to help address the FHA problem as long as the unlabeled data are weakly dependent (a < 0.5). Nevertheless, once a 2 0.5, the results in Theorem 1 may not hold, resulting to fail theoretically. In addition, we find that high dependency among unlabeled data usually means that we need more unlabeled data to obtain a good target-domain classifier. From the above analysis and Figure 1, we argue that diversity matters in addressing the FHA problem.
To this end, we propose the enhraninn generative network (EEG-Ne) for the FHA problem, which is a weight-shared conditional generative method equipped with a kernel independence measure: the Hilberr-Schmidt independence criterion (HSIC) (Gretton al., 2005; Ma et al., 2020; Pogodin & Latham, 2020), which is used in various situations, e.g., clustering (Song et al., 2007; Blaschko Gretton, 2008), independence testing (Gretton et al., 2007), and self-supervised classification (Li et al., 2021). Although the log-coefficient score is used to analyze the affect of the sample complexity regarding the FHA problem, its calculation requires know the distribution regarding the target-domait data, which is unknown. in practice. Yet, HSIC can be estimate easily by the data sample. Thus, we adopt the HSIC to calculate the dependency among generated unlabeled data.
The overview of DEG-Net is in Figure 2, showing that there are two modules in DEG-Net: the generation module and the adaptation module. In the generation module, we train the conditional generator with well-trained source classifier and few target domain data. To train the generator with both the source domain and target domain knowledge and improve the diversity of generated data simultaneously, the generative loss of DEG-Net consists of 3 parts: the classification loss, similarity loss and diversity loss. More specifically, DEG-Net generates data via minimizing the HSIC value (i.e., maximizing the independence) between the semantic features of the target data and generated data, where the semantic features are the hidden-layer layer outputs of the well-trand -ra--tr source hypothesis. To use the generalization knowledge in the semantic features of data that is shared by different classes
2

---

Under review as a conference paper at ICLR 2023
(Chen et al., 2020; Chi al., 2021b; Yao et al., 2021), the generator is a weight-shared network. As for the adaptation module, the source classifier is trained to work well over the target domain. The adaptation module consists of a classifier and group discriminator (Chi et al., 2021at Motiian et al., 2017). With the help of the group discriminator which tries to confuse the classifier to distinguish data from different domains, the classifier is trained classify the data over the target domain using the generated data and few label target data.
We verify the effectiveness of DEG-Net on 8 FHA benchmark tasks (Chi al., 2021a). Experimental results show that DEG-Net outperforms existing FHA methods and achieves the state-of-the art performance. Besides, due to the weight-shared mechanism, DEG-Net is much faster than previous generative FHA methods in training. We also conduct an ablation study to verify that each component the DEG-Net is useful, which shows that diverse generated data can help improve the performance when addressing the FHA problem, which lights up novel road for the FHA problem.
2 PRELIMINARY AND RELATED WORKS
Problem Setting. this section, we formalize FHA problem mathematically. Denoting by X' C T the input space and by )/ (1, .... K} the output space, where K' is the number of classes. The source domain (target domain) can be defined as a joint probability distribution jı8 on X' x y (n' on X x y). Besides,
assume that there is well-trained model f* X + y. The f is trained with data {a. MI Yi=1 drawn independently and identically distributed (i.i.d) from μr" with the aim of minimizing ((m,)))== [((h(z), y)], where é y x y is loss function to measure if two elements in y are close, and h is an element a hypothesis set H = (h: X - V). Thus, f can be defined as
f == arg min (xx,,),,(((((x')) y*)]-
NEW
fe is also called the source hypothesis in our paper. Hence, the FHA problem is defined follow: Problem 1. (FHA) Given the source hypothesis S defined in Eq. (1) and the labeled dataset St = {(x,, (ol.:()) He (my S 7K, according to (Chi et al., 202la; Park et al., 2019)), drawn i..dd from target domain p². the FHA methods aim to train a classifier f e H with f and St such that we minimize the value of ((xex))((((((")) Y"},, where h E H. Namely, aarggmnng min E(xxrr)((h((X)) (r))((((X*)))")) Y.).
f arg
Hypothesis Transfer Learning Methods. Hypothesis transfer learning (HTL) aims train a classifier with only a well-trained classifier and small labeled data or abundant unlabeled data over the target domain (Kuzborskij & Orabona, 2013; Tommasi et al., 2010; Liang et al., 2020). In (Kuzborskij & Orabona, 2013), they used the leave-one-our error to find the optimal transfer parameters based on the regularized least squares with biased regularization. SHOT (Liang et al., 2020) froze the source hypothesis and trained a domian-specific encoding module using the abundant unlabled data. Later, neighborhood reciprocity clustering (Yang et al., 202 was proposed to address HTL by encouraging reciprocal neighbors to concord in their label prediction. Different from the FHA problem, the HTL problem does not have the limitation on the number of the target domain labeled data.
Target-oriented Hypothesis Adaptation Network. TOHAN (Chi et al., 202 is one-step solution for the FHA problem. It has a good performance due to using the generated unlabeled data in the adaptation process. Motivated by the learnability in comi--spervised learning (SSL), TOHAN found that unlabeled data in the intermediate domain, which is compatible with both the source classifier and target classifier, can address the FHA problem for providing the additional information in the training. Guided by this principle, the key module of TOHAN is to generate the unlabeled data drawn from the probability distribution pt"
nT = arg min [x(h*, + x(h 10]
where (hh,, (1) (resp. x(h,, (() measures how compatible 1:" (resp. h²) is with unlabeled data distribution y (Balean & Blum, 2010).
3 THEORETICAL ANALYSIS REGARDING THE DATA DIVERSITY IN FHA
In previous works, researchers have shown that generated high-compatible data can help address the FHA problem. However, as discussed in Section 1, the diversity of the generated data matters in
3

---

Under review as a conference paper at ICLR 2023
addressing the FHA. Besides, the previous studies assume that the generated data is independent in their theory, and is inconsistent with their methods. In this section, we will show how the dependency among the generated data affects the performance of FHA methods. Similar to (Chi et al., 2021a), our theory is also based on the theory regarding SSL (Webb & Sammut, 2010).
Dependency Measure. Following Dagan et al. (2019), we use the log-coefficient that measures the dependency among observations from a random variable Z to theoretically analyze the data diversity.
Definition 1 (Log-influence and log-coefficient (Dagan et al., 2019)). Let Z = (Zi...., 4m) be a random variable over (X V) and let pz denote either its probability distribution if discrete its density if continuous. Assume that pz > O on all (X x y)m For any i + j E f1,2..... m. define the log- inffuencee between j and
1
pz 12l2,,,---1luzz10 1z 1-1
sup
log
(3)
4
2-1-1 E x V1
4212/2/2---1142226
2,
2, 24 23 xxy
Then the log-coefficient of Z is defined as alog ( ( = max Σ iifs flog (Z).
From Definition I, is clear that Oliog (Z) will be zero if Zi and Z₁ are independent (for any o j).
Sample Complexity Analysis for FHA. Since the generated data are unlabeled, we follow the theory regarding SSL to analyze how the generated unlabeled data can help address the FHA problem. More importantly, we will analyze how the dependency among the generated data affects the performance of FHA methods. For simplicity, we consider binary SSL problem (i.c., K 2).
Let f" + {0, } be the optimal target classifier. Let err(h) = Ex-na [M(x) # f'(x]] be the true error rate of hypothesis h E H. over marginal distribution 2 In FHA, its learnability mainly depends on the compatibility X : H x X' + [0, 1] measuring how "ccompatble"" h is to one x(h,
- [x) to represent the expectation of compatibility of data from uix on a classifier h, and let S s((m)) be an observation of random variable t.m_ = (Xi...., X m. where the distribution regarding X is μxx i = 1,... My, The following theorem shows that, under some conditions, we can still learn good f even when the dependency among unlabeled the target data exists.
unlabeled data æ. In the following,
I Exesioa x(h,z) be the empirical compatibility over mmm) Theorem 1. Let 8(h, SX s(mma) ma
and Ho (h E H @Fr(h) 0}. ff H. x(f",1%x = 1 - t, and mlg (Xt, ) < 1 1/2, then my unlabeled data and my labeled data are sufficient to learn to error with probability 1 8, for
my = mmax (o
(o
1
2) 88)
( VCdim(((HH)
log
(4)
Olog (X (mmma)) 1)c2
(1
200ug (1-200 Xt.mu
and
mn = 1n(2HHggxxx 26) xxx) + In 41
5
(5)
where xHH) = {xx : h E H} is assumed to have a finite VC dimension, (a(.) = x(h, ), and
Hyp x (1+26)[2m, /xx is the expected number of splits 2m, data drawn from p'x using hypotheses in H of compatibility more than 1 t 2c. In particular, with probability at least 1 5, we have (h) S 6, where h arg max Ho X(h, s((m))
The proof of Theorem is presented in Appendix A, which mainly follows the recent result in the problem of learning from dependent data (Dagan et al., 2019).
Theorem shows that when the dependency among the unlabeled data is weak (i.e., Oting (X L. 1/2), we can obtain a similar result compared to classical result in the SSL theory (Balcan & Blum, 2010). Namely, if we can generate unlabeled data that are highly compatible to f", which means that t is very small and thus H H (t + 2c) [2m;, u'x] is small, thus we do not need lot of labeled data to learn good fr (Chi et al., 2021a).
Diversity Matters in FHA. Theorem I also shows that diversity of unlabeled data matters in FHA. There are two reasons. The first reason is that Theorem 1 might not be true if there is strong
4

---

Under review as a conference paper at ICLR 2023
Senmantic Teature
labeled target domain z!
puine data group G.
discriminator
rindom
noise X
calegorical Information
Freeze while training
classtffir .....ooe........................
gemerated data
Freeze
Freeze while training
Step Train the ciscriminator
discriminator
Step 2: Train the clossifier
Generation module
(b) Adaptation module
Figure 2: Overview of the iiversiy--nhanncinn generative network (DEG-Net), It consists of generator G, classifier f = hy o St (initialize f = I't) and group discriminator D. (a) In the generation module, we train generator G' with freezing the classifier f f (b) In the adaptation module, we first pair the generated data and labeled data and use the paired data to train the discriminator D while freezing the encoder h, Then, we freeze the discriminator D to train the classifier ft.
dependency among the unlabeled data (e.g.. Olog ( > 1/2). This will directly make the previous work lose the theoretical foundation to address the FHA problem. The second reason is that we need more unlabeled data to reach the same error if the dependency among the unlabeled data increases. Specifically, if mog (Xt.m.) is very close to 1 1/2, then m. could be a very large number. The above reasons motivate us take care of the dependency among the generated data. To weaken such dependency, we propose our diversity-cnhancing generative network for the FHA problem.
4 DIVERSITY-ENHANCING GENERATIVE NETWORK FOR FHA PROBLEM
In this section, we propose the problem. DEG-Net has two modules: the generation module to generate diverse unlabeled data; the adaptation module to train the classifier for the target domain.
generative nelwork (DEG-Net) for the FHA
4.1 DIVERSITY-ENHA GENERATION
In order overcome the shortcomings of the current generative method for FHA problem, come up with solutions for both the generative architecture and the loss function. As for the generative architecture, we propose the weight-shared conditional generative network for generating the data with the specific category. As for the generative loss function we design novel loss function to constrain the similarities and diversity of the semantic information of the generated data.
Weight-shared Conditional Generative Network. As discussed before, for using generalized features which are shared among different classes to improve the quality of generated data and reduce the time of training, the weight-shared conditional generative network is promising. Following Chi al. (2021a), 1a), the generator aims to generate data specific categories from Gaussian random noise. The encoder outputs the semantic feature and the class probability distribution of the generated data. To achieve the aim of the generative method, we design the two-part loss functions: the classification loss function and the similarity of semantic feature loss function.
We assume that 2."" = G(z*, (n) is the generated data with specific category, where the inputs of generator are the Gaussian random noise 2 and the specific categorical information e, Specifically, we use the one-hot encoded label as the categorical information. The generated data xx"" inputs the well-trained source-domain classifier f hts ga, where the output of h1 is the group discriminator feature, which will be used in the adaptation module and the output of g, is probability feature P, = .p...... .......,p" K) where p." is the probability of the generated data belonging to the th class respectively. The semantic feature " used in the similarity loss and diversity loss is the hidden-laye output of h, (details the hidden-layer selection can be found in Appendix C.). We aim update parameters of the generator to force the generated data with the categorical information

---

Under review as a conference paper at ICLR 2023
Algorithm I Diversity-enhancinn Generative Network (DEG-Net)
conditional generator Geg parameterized by 8c. group discriminator Døp parameterized by 0p. classifier fa, parameterized by 0,. kernel function generation batch size B., learning rate ac. an and a,. total epoch Tma,, pertraining group discriminator epoch Ta.
Input:
for = 1,,,,..... maa do
for 0, 1,... K de
1: Generate random noise and categorical information Ca:
Generate data (=(=) and then add them Dm:
Calculate the semantic feature a classification probability p:
4: Calculate the kernel matrice of semantic feature S using kernel function
end
5: Update 0G +- 8G aG VGVL using Eq. (11);
if
T Tmaz - then
for = 1, 1.2....
Sample Gs from Pm X Dm:
7: Sample G2 from Dm. x DD;
8: Update 0p <- 0D ap V VLD (((Tt))) using Eq. (12);
end
end
i
2 lmax Ty then
9: Sample GG from Dm X Dm:
10: Sample G2.94 from Pm Dí;
11: Update 8 t 0y as (19 h2) using Eq. (13);
12: Update @D + 0D ap (D((()))))) using Eq. (12);
end end
Output: a well-trained classifier fs.
x belonging the th class, i.e., making Pn, close to 1. Specifically, we minimize the following
loss to generate the data of specific category n:
K
B
C.
Ip
- -112
(6)
K n=
Bn
==1
where B,, is the batch-size of the generator.
To make the generated data closer to data in the target domain, we need to define the loss function to measure the difference between data of two different domains. Motivated by Zheng & Zhou (2021), DEG-Net uses the semantic features to calculate the similarities. To avoid the copy issue, we decided to use the & distance | y11 = Σ, wi I yil. where W; = [z, 1 /|| y since it encourages larger gradients for feature dimensions with higher residual errors. Compared to the f--mrmm it better to measure the similarity of the semantic features between the generated images and the target images, since & distance is more robust to outliers (Oneto al., 2020). Thus, the similarity loss is defined as following:
K
B mi
C. =
ΣΣ
2D
K
MB,
where M max EX ||
-2l11 (X' is compact and ||: is continuous), my is the number of labeled data drawn from the target domain, 2i and sthh are the semantic features of target data and generated data, respectively. Combining Eq. (6) and Eq. (7), we obtain the loss to train the weight-shared conditional generative network:
XL.,
(8)
where À 2 0 a hyper- ppper-parameter between two losses. Note that optimizing Eq. (8) is corresponding to TOHAN's principle Eq. (2), where Eq. (6) (resp. Eq. (7)) is corresponding to x(h", (1) (resp. ((h", p)). To ensure that the conditional generator can generate the image with the correct class label, pretrain the generator using the well-trained source model for some epochs.
Generative Function with Diversity. As discussed above, the weak dependence of unlabeled data is an important condition for using generated unlabeled data to address the FHA problem. To ensure

---

Under review as a conference paper at ICLR 2023
that the unlabeled data are weakly dependent among unlabeled data (i.e., generate more diverse unlabeled data), it is necessary to use diversity regularization to train the generator. Unfortunately. the rogcoffficient score, dependence measure used to analyze the sample complexity, is hard to calculate, since its calculation requires the unknown distribution regarding the target-domain data. HSIC, kernel independence measure can also measure the dependency of the generated data. Different from the log-coefficient score, HSIC can be easily estimated (Gretton et al., 2005; Song al., 2012):
1
HSIC(,, (N 1)2 Tr(KHLH),
where K (k:,y) = k(, 2j) (L (11y) k(y., y,)) is the kernel matrix (le(-.) is the kernel function) and H = J 11 the centering matrix. We minimize the HSIC measure of the generated data's 's semantic features to obtain weakly dependent data. Specifically, we use the Gaussian kernel as the kernel function, and minimize the following loss to generate more diversity data:
K
K
1 (B.. 1)2 HS,
La
HSIC 8
(10)
K
where S" = (s - k( (( vn)) is the kernel matrice of the semantic features of the generated data with specific class. Hence, we obtain the total loss to train the generator with diversity enhancing:
La La + 3La,
(11)
where 3 2 0 a parere-rrreeter between the generative loss and diversity regularization. Note that the diversity regularization and the similarity loss restrict themselves.
4.2 ADAPTATION MODULE USING GENERATED DATA
Following Chi et al. (2021a), we create paired data using the labeled data in the target domain and the generated data, and assign the group labels to the paired groups under the following rules: Gi pairs the generated data with the same class label; 92 pairs the generated data and the data in the target domain with the same class label; G3 pairs the generated data but with different class label; Gı pairs the generated data and the data in the target domain and also with different class label. By using adversarial learning, we train discriminator D which could distinguish between the data in different domains while maintaining high classification accuracy on generated data. The discriminator D four-class classifier with the inputs of the above paired group data. Different from classical adversarial domain adaptation (Ganin et al., 2016; Jiang et al., 2020), the group discriminator D decides which of the four groups given data pair belongs to. By freezing the encoder, we train D with the cross-entropy loss:
LD = - Σ ug. log (D((())))))))
(12)
where (B()) represents the empirical mean value, Vo, the group label of group G. and ((9.) [9(x1), ((22) is the output of the encoder with the paired data input.
=
Next, we will train the classifier ft h4 o ge while freezing the group discriminator, which is initialized with the same weight as that in the source classifier f = h2 gs. Motivated by non- saturating games (Goodfellow, 2016), we minimize the following loss to update f":
(13) where > 0 a hyper-parameter, is the cross-entropy loss, and ft is the optimal target model. Note that, the label information of generated data has as certain noise. As demonstrated in Theorem 1, it is only necessary to use generated unlabeled data for addressing the FHA problem. Thus, we only use labeled target data for target supervised loss in Eq. (13).
Cj - luc, log (D (o (G2))) yg, log (D (6 (G+))) + E (1t) ft (z1)),,
5
EXPERIMENTS
We compare DEG-Net with previous FHA methods on digits datasets (i.e. MNIST UM), (U). SVHN (S)) and objects datasets (i.e. CIFAR-10 (CF) and STL-110 (ST.)), following (Chi
7

---

Under review as a conference paper at ICLR 2023
Table I: Classification accuracy standard deviation (%) 6 digits FHA tasks. Bold value represents the highest accuracy on each column.
Number of Target Data per Class
Tasks
WA
FHA Methods
5
70.2.10.00
70.6110.3
70.710..1
70.8.0.3
7..9...2
71.110.3
71.110.4
SHOT
72.6.:1.9
73.6:1:0.0
7..1
74.6::12
7..9
75.4::0.3
1.5
S+FADA
74.4+15
83.1+007
83.3=1.1
85.940.
86.0=1.2
87.6+22.6
19.1
- U
69.7
T4FADA
74.24:188
81.64-4.0
83.4=088
82.0+:233
86.2=0.7
87.2::0.8
88,2
TOHAN
76.0:1199
83.3::0.3
84.2:004
86.511111
87.11:1.3
88.0.10.5
89.7
DEG-Net
83.1:10.9
86.211..8
8..51:066
88.7:10.9
0..6
91.5.110.6
92.1
FT
26.71:100
26.8:12.11
1.681:166
27.0:10.7
27.31:12
27.5::0.8
28.31:
SHOT
25.7:12.2
26.9:11.2
27.91:2.6
29.1:104
1..1
29.6.11.7
2.58
S+FADA
25.61:1.3
27.7:100.5
27.8=0077
28.2::13
28
29.04::.0
29.621.9
M 8 24.1
T+FADA
25.31110
26.3.10.8
28.9
29.111.3
1.3
31.904
32.41
TOHAN
26.7.20.1
28.6:11.1
29.5
29.6::0.4
30.5
32.1::022
33.2.10.8
DEG-Net
27.24:0.3
1.
29.7=009
30.7
32.9=155
33.7
14.9+16
6494111
66.5-1..5
66.7-1177
67.341.11
68.122.3
68.3400.5
69.7
SHOT
74.7+-0.3
75.5+114
75.6=1.0
75.8+0.77
77.1=2.1
17.8
79,6=0.66
S+FADA
72.24:14
73.6
7..7==14
76.24:1
77.2=117
718433000
19.7
8 - -
64.3
T+FADA
71.7100.6
7.
7..5
75.912.11
1..
76.8:1188
1.97
TOHAN
758-1099
76.8+1122
7.94=0.9
80.2
80.5=
811111111
12
DEG-Net
75.2110.3
81.71.77
84.3
FT
70.24:0.0
70.6+:0.3
70.7=
70.8+0.33
70,9=00.2
71.1::0.3
71.1=004
SHOT
72.61199
73.6:120
7..1
74611:2
7..9
75.4:10.3
76.1
S+FADA
74.41115
83.1
83.32 1.1
85.910.
86.0
87.6.1266
8.01
8 M
70.2
T+FADA
74
81.6:4-0
83.4-08
82.04233
86,2-0.7
87.240088
88.2
TOHAN
76.0-t:99
83.3:10.3
84.2==044
86.5+111
87.1
88.0:20.55
89.7
DEG-Net
76.2:11.3
78.2:11.3
8.66
9.,2
FT
83.5.0.4
84.3.12
0.7
85.5.11.
86.61
87.2.10.7
88.1 2.7.
SHOT
83.14:0.5
85.5:10.3
85.
86.0::02
86.6-0.2
86.710022
87.0=0.1
S+FADA
83.2-:0.2
84.0-20.3
85.0=
85.6:005
85.7=0.6
86.2::0.66
87.2
U M
82.9
T+FADA
82.9-0.77
83.9:10.2
85.4:0066
05.6:00.7
86.6
TOHAN
8..0
85.2.10.3
85.6
86.5110.5
0..
88.2.10.7
89.2.10.5
DEG-Net
0.9
91.6
FT
23A118
23.6112.7
1.
24.61144
2..61:12
24.8100.7
2..5
SHOT
M0331
31.6:1.4
29.820.5
29.4:033
29.710.5
29.8:10.8
30.1
S+FADA
28.11:1.
28.7:11.33
29.0::12
30.11:1.1
30.3:1..3
30.7-1:100
30.9
U
17.3
T+FADA
27.5114
29.5.10.7
TOHAN
29.9
30.5+11.2
314=1.1
32.8+0099
33.1=1.0
34.0==:1.0
3..1
DEG-Net
29.1+:1.
30.7+1111
31.8=0.7
33.0+:1.6
33.5=114
35.1+:1.3
36.2-1.2
ct al., 2021a). Following the standard domain adatiaaioatttiionn protocols (Shu et al., 2018), we compare
DEG-Net with 4 baselines: (1) Without adaptation (WA); (2) Fine funing (FT); (3) SHOT (Liang et al., 2020); (4) S+FADA (Chi et al., 2021a); (5) T+ FADA (Chi et al., 2021a); and (6) TOHAN (Chi al., 2021a). Details regarding datasets, baselines and implementation are in Appendixes B and C.
Digits Datasets. Following Chi et al. (2021a); Motian et al. (2017), We conduct 6 tasks of the adaptation among the 3 digital datasets and choose the number of target data from to 7 per class. The classifier accuracy on the target domain of our method over tasks is shown in Table 1. The results show that the performance of DEG-Net the best almost all the tasks. It is clear that the accuracy of DEG-Net is lower than TOHAN when the amount of target data is too small. The diversity regularization and the similarity loss restrict each other, avoid the copy issue. However, when the amount of target data is too small, the target domain information is few, so the generator is less likely generate similar data with the target domain. Diversity loss enhances this adversarial effect, resulting DEG-Net degrading to TOHAN and SHOT. Another improvement of DEG-Net over TOHAN that the faster training process of the generator. We need 0.93s to complete the training within each epoch in DEG-Net, while needing 1.35s in TOHAN.
Objects Datasets. Following Chi al. (2021a), we examine the performance of DEG-Net on object tasks and choose the number of target data as 10 per class. The classification accuracies on object tasks are shown in Table 2. It is clear that we outperform baselines. CF + SL, we achieve 1.5% improvement over TOHAN. In SL - CF, we achieve performance aceracy of 57.2%, 0.3% improvement over S+EADA. is clear that the effect DEG-Net not obvious in objective tasks. may be caused by the simple structure of generative networks and complexity of datasets.
8

---

Under review as a conference paper at ICLR 2023
Table 2: Classification accuracy--standard deviation (%) on 2 objects FHA tasks. Bold value
represents the highest accuracy on each row.
PHA Methods
Tasks
WA
FT
SHOT
S+FADA
T4RADA
TOHAN
DFG-Net
CF +
71.5110
719104
72.110.4
713105
72.810.1
7.3
SL + CF
51,8
54,3405
53.94022
$6.94:555
55.80088
56.640..3
57.2:10.5
Table 3: Ablation study. Classification accuracy--standard deviation(%) on M - U. Bold value represents the highest accuracy on each column.
Number Target per Class
FIA Methods
2
5
TOHAN
76.0: 1.9
83.3
84.2:10.4
86.5:1111
87 17.1
88.0:10.5
09.7:20.5
separate gencrative DEG-Net
75.7+00.7
84.7=00.5
85.0=1.2
85.9=0.9
87,4
89.1+10
90.4 +1.
DEG-Net w/o diversity
87.2++1.99
0,3
89.2::0.4
90.2-:1111
90.31:1.3
91.1=00.5
91.2:+0.5
DEG-Net
87.3--099
89.2=0.88
90.1-:0.6
90.8:10099
90.6++0.5
91.5--0.6
92.1-00.6
DEG-Net Generates More Diverse Data Than TOHAN. In this part, we analyze the diversity of the generated data by DEG-Net and TOHAN to see if our generation process can produce more diverse data than TOHAN's. We choose the square root of the HSIC to measure the diversity of the generated data in the task M - S, and calculate the HSIC value among the target-domain data as a reference value that is 0.0013. After the calculation, the average diversity measure of DEG-Net is 0.0019, and the average diversity measure of TOHAN is 0.0027. It is clear that DEG-Net can generate more diverse data than TOHAN. The detailed diversity analysis can be found in Appendix D.
Ablation Study. To show the advantage of weight-shared architecture and the diversity loss, we conduct experiments: (1) The architecture of weight-shared is the same the DEG-Net but Eq. (8) to train the generator (DEG-Net without diversity). (2) The separate generative method, which similar TOHAN, has K generators and use the semantic features calculate the similarity loss for training each generator:
N
1
CE
1pp- 112 + NyMB, B,
Ip.
(14)
As shown in Table 3, DEG-Net works better than both methods introduced above, and the weight- shared architecture works better than the separate generative method, which reveals that both the weight-shared architecture and the diversity loss improve the quality of generated data and thus achieve the higher accuracy. Specifically, compared to modified DEG-Net without the diversity loss, the separate generative method ignores the generalization knowledge in the semantic features of data which is shared with all the classes. Modified DEG-Net discards the diversity loss, and thus generates the low diverse data and results in the worse performance. However, the HSIC diversity loss does not work for all the situations. It is clear that DEG-Net achieves the similar accuracy with mo dified DEG-Net without diversity or even worse if the amount of the labeled data is very few (i.e., my S 2). This phenomenon may be caused by worse data generated by diversity method. Since the diversity loss restricting to the similarity loss, the generator is less likely to generated similar data over target domain (i.e., the distribution of generated data is far from the target domain).
CONCLUSION
In this paper, we focus on generating more diverse unlabeled data for addressing the FHA problem. We experimentally and theoretically prove that the diversity of generated data (i.e., the independence among the generated data) matters in addressing the FHA problem. For addressing FHA problem, we propose a diversity-enhancing generative network (DEG-Net), which consists of the generation module and the adaptation module. With the weight-shared conditional generative method equipped with a kernel independence measure: HSIC, DEG-Net can generate more diverse unlabeled data and achieve the better performance. Experiments show that the generated data of DEG-Net more diverse, and thus DEG-Net achieves the state-of-the art performance when addressing the FHA problem, which lights up novel and theoretical-guaranteed road to the FHA problem in the future.

---

Under review as a conference paper at ICLR 2023
7 REPRODUCIBILITY STATEMENT
We implement all methods by PyTorch 1.7.1 and Python 3.7.6, and conduct all the experiments on two NVIDIA RTX 2080Ti GPUs. We use the standard DCGAN network (Radford al., 2015) as the generator network architecture. We adopt the backbone network of LeNet-5 (LeCun et al., 1998) with batch normalization and dropout as the encoder. We employ connected layers with softmax function as the classifier. The semantic feature in digital tasks is the output of first fully connection layer. We adopt connected layers with softmax function as the group discriminator D. We choose the Gaussian kernel the kernel funcion to calculate the HSIC measure. The hyper-parameter settings details can be found in Appendix C. The details regarding to the datasets used in the paper can be found in Appendix B. For the theoretical resultsclear explanations, we proof the Theorem 1 in Appendix A.
REFERENCES
Maria-Florina Balcan and Avrim Blum. A discriminative model for semi-supervised learning, Journal of ACM, 19:1:19 2010.
Matthew Blaschko and Arthur Gretton. Learning taxonomies by dependence maximization. In NeurIPS, 2008.
Xiaohan Chen, Zhangyang Wang, Siyu Tang, and Krikamol Muandet. Mate: plugging in model awareness to task embedding for meta learning. In NeurlPS, 2020.
Haoang Chi, Feng Liu, Wenjing Yang, Long Lan, Tongliang Liu, Bo Han, William Cheung, and James Kwok. Tohan: A one-step approach towards few-shot hypothesis adaptation. 2021a.
Haoang Chi, Feng Liu, Wenjing Yang, Long Lan, Tongliang Liu, Bo Han, Gang Niu, Mingyuan Zhou, and Masashi Sugiyama. Demystifying assumptions in learning to discover novel classes. In ICLR, 2021b.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In AISTATS, 2011.
Yuval Dagan, Constantinos Daskalakis, Nishanth Dikkala. and Siddhartha Jayanti. Learning from
weakly dependent data under dobrushin's 's condition. In COLT, volume 99, pp. 914-928 2019.
Simon S Du, Jayanth Koushik, Aarti Singh, and Barnabás Póczos. Hypothesis transfer learning via transformation functions. NeurlPS, 2017.
Tongtong Fang, Nan Lu, Gang Niu, and Masashi Sugiyama. Rethinking importance weighting for deep learning under distribution shift. 2020.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The journal of machine learning research, 17(1):209--200 20009, 2016.
Goodfellow. Nips 2016 tutorial:
Generative adversarial networks.
arXiv preprint
arXiv:1001.001600 2016.
Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statistical dependence with hilbert-schmidt norms. In ALT, pp. 63-77, 2005.
Arthur Gretton, Kenji Fukumizu, Choon Teo, Le Song. Bernhard Schölkopf, and Alex Smola. A kernel statistical test of independence. In NeurIPS, 2007.
Jonathan J. Hull. A database for handwritten text recognition research. IEEE Transactions on pattern analysis and machine intelligence, 16(5):550-554, 1994.
Pin Jiang, Aming Wu, Yahong Han, Yunfeng Shao, Meiyu Qi, and Bingshuai Li. Bidirectional adversarial training for semi-supervised domain adaptation. In IJCAI, 2020.
10

---

Under review as a conference paper at ICLR 2023
Yongcheng Jing, Xiao Liu, Yukang Ding, Xinchao Wang, Errui Ding, Mingli Song, and Shilei Wen. Dynamic instance normalization for arbitrary style transfer. In AAAI, 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Iija Kuzborskij and Francesco Orabona. Stability and hypothesis transfer learning. In ICML, 2013.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 (0(11))2278-23) 1998.
Yazhe Li, Roman Pogodin, Danica Sutherland, and Arthur Gretton. Self-supervised learning with kemel dependence maximization. 2021.
Jian Liang, Dapeng Hu, and Jiashi Feng. Do really need access the source data? source hypothesis transfer for unsupervised domain adaptation. In ICML, 2020.
Lu Liu, William Hamilton, Guodong Long, Jing Jiang, and Hugo Larochelle. A universal representa- tion transformer layer for few-shot image classification. In ICLR, 2021.
Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael Jordan. Conditional adversarial
domain adaptation. In NeurIPS, 2018.
Wan-Duo Kurt Ma, JP Lewis, and Bastiaan Kleijn. The HSIC bottleneck: Deep learning without back-propagation. In AAAI, 2020.
Saeid Motiian, Quinn Jones, Seyed Iranmanesh, and Gianfranco Doretto. Few-shot adversarial domain adaptation. 2017.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In NewrlPS, 2011.
Luca Oneto, Michele Donini, Giulia Luise, Carlo Ciliberto, Andreas Maurer, and Massimiliano Pontil. Exploiting mmd and sinkhorn divergences for fair and transferable representation learning. In NeurIPS, 2020.
Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering. 22(10):1345-135) 2009.
Sconwook Park, Shalini De Mello, Pavlo Molchanov, Umar lqbal, Otmar Hilliges, and Jan Kautz. Few-shot adaptive gaze estimation. In /CCV, 2019.
Roman Pogodin and Peter Latham. Kernelized information bottleneck leads to biologically plausible 3-factor hebbian learning in deep networks. In NewrlPS, 2020.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06433 2015.
Connor Shorten and Taghi M Khoshgoftaar. survey on image data augmentation for deep learning. Journal of big data, 2019.
Rui Shu, Hung H Bui, Hirokazu Narui, and Stefano Ermon. A dirt-t approach to unsupervised domain adaptation. In ICLR, 2018.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In
NeurlPS, 2017.
Le Song, Alex Smola, Arthur Gretton, and Karsten M Borgwardt. A dependence maximization view of clustering. In /CML, 2007.
Le Song, Alex Smola, Arthur Gretton, Justin Bedo, and Karsten Borgwardt. Feature selection via dependence maximization. Journal of Machine Learning Research, 13(5), 2012.
Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele. Meta-transfed learning for few-shot learning. In CVPR, 2019.
11

---

Under review as a conference paper at ICLR 2023
Takeshi Teshima, Issci Sato, and Masashi Sugiyama. Few-shot domain adaptation by causal mecha- nism transfer. In NeurIPS, 2020.
Tatiana Tommasi, Francesco Orabona, and Barbara Caputo. Safety in numbers: Learning categories from few examples with multi model knowledge transfer. In CVPR, 2010.
Yaqing Wang, Quanming Yao, James Kwok, and Lionel Ni. Generalizing from a few examples: A survey on few-shot learning. ACM computing surveys (csur), 53(3):1-34, 2020.
Geoffrey I Webb and Claude Sammut. Encyclopedia of machine learning. Springer, 2010.
Shiqi Yang. Joost de Weijer, Luis Herranz, Shangling Jui, al. Exploiting the intrinsic neighbor- hood structure for source-free domain adaptation. 2021a.
Shiqi Yang, Yaxing Wang, Joost van de Weijer, Luis Herranz, and Shangling Jui. Generalized source-free domain adaptation. In ICCV, 2021b.
Shuo Yang, Lu Liu, and Min Xu. Free lunch for few-shot learning: Distribution calibration. In ICLR, 2020.
Huaxiu Yao, Long- Kai Huang, Linjun Zhang, Ying Wei, Li Tian, James Zou, Junzhou Huang, et al. Improving generalization in meta-learning via task augmentation. ICML, 2021.
Amir R Zamir, Alexander Sax, William Shen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. CVPR, 2018.
Huangjie Zheng and Mingyuan Zhou, Exploiting chain rule and bayes' theorem compare probability distributions. 2021.
12

---

Under review as a conference paper at ICLR 2023
A
PROOF OF THEOREM
Before proving the Theorem 1, we first introduce McDiarmid-like inequality under the log- coefficient of a random vector Z.
Lemma (McDiarmid-like Inequality under the Log-coefficient Z). Let μ p(m) be distribution defined over 2m, and Z (Zi...., Zm.) mm be random vector, and g. 2 - R with the following bounded differences property with parameters AI...., Am 0:
m
4Z Zzz 10 (.) g(2,))
2: A/=
(15)
t=1
where z = (X x m. If alog Z) < 1, then, for all t > 0.
Olog (2)) (1 (-
Pr(lg()) Elg Z) S 2exp
(16)
2in X2
Proof. Based on Definition 2.2 and Lemma 5.2 in (Dagan et al., 2019), we know that μ (mm) satisfies Dobrushin's 's condition with coefficient < 1. Thus, based on Theorem 2.3 in (Dagan et al., 2019), we have
(1 a1-
Pr[1g(Z) [[g(Z)]| 2 S 2 exp
(17)
2 2 x3
Since, a S Oglg (Z), we prove this lemma.
Then, introduce recent result regarding bounding the expected suprema of empirical process using the corresponding Gaussian complexity.
Theorem (Dagan et al., 2019)). Let be a random vector over some domain 2nh and let G be class of functions from Z to IR If Mlog (Z) 1/2, then
m
m
CBz(9)
E sup S~Z
g(s))
(18)
gEg
m
m
VI
20riog Z
where C > is a universal constant, and S = (8) 8m) is a sample of Z.
Note that, the above result very general, does not assume that the m marginals of the distribution of Z are identical. Based on the above theorem and lemma,
prove the following lemma. Lemma 2. Let Z be random vector over some domain n"h and let G be class of functions from Z to R. If alog ( < 1/2, and there exists L > O such that for any E G and Zi, (g(g,))))))) s L, then, for any t 0,
-
(8z(9)
(i-Biig (2))m²
Pr S~Z
sup gEG
m Σ (s:)
+6 Se
(19)
m
/I 20 Z
for some universal constants C, C' 0.
Proof. We first consider proving the non-absolute-valuu version. Let
1 Σ g(s))
M(S) sup gEg
g(s))
(20)
i=1
For any and S" Z. we have that M(S) - M(S) < Σm 2L1,,ts/mm
According to
Lemma 1, we have
Pr [M(S) B(M(S) 2 S exp (- (1- 0lin m))
(21)
13

---

Under review as a conference paper at ICLR 2023
for some universal constant C' > 0. Then, combining ([M( (based on Theorem 2), we have
Pr S-Z
sup 9
1 m
m
(s))
m
s(si)
CBz(9)
t E
(1 11-0lox (2)ma²
m
200log (
(22)
For the opposite incquality, (M( part, following (Dagan et al., 2019), we can apply the same arguments on -g. Note that ( (G)), which concludes the bound.
The above lemma is slightly general version of Theorem 6.7 in (Dagan et al., 2019) by considering the influence of Olog (Z). Based on Lemma 2, we can prove the Theorem below.
Theorem 1. Let x(h,
1 Exes(mma x(h,,)) be the empirical compatibility over s(m")) 5 s(me)) ma
and Ho (h E H (Fr(h) 0}. If H. x(5", x(f%,1x) =1 t, and mog ( (X < 1/2, then ma unlabeled data and my labeled data are sufficient learn to ervor with probability 1 for
(
2) .0 olog (X (x.mmm))e log )).0 ( VCdim(x(HH)
my = .max
(4)
(10200
200og (X (.imm)))e
and
In(22, 13x 22 x) + In 4
my
In
(5)
where x(H) = (xx : E H} is assumed to have a finite VC dimension, xa(+) = x(h, )) and
Hpp..x (t+2e)t+)))))) [2m], p'x is the expected number of splits of 2m, data drawn from μ using hypotheses in H of compatibility more than 1 - t- 2e. In particular, with probability at least 1 6, we have err(h) S e, where h arg ma Ho X(h, slmm)).
Proof. Let S be the set of ma unlabeled data. Based the relation between VC dimension and the Gaussian complexity, Lemma gives that, with probability at least 1 ê
2 we have
Pr (x,))s(x(x>) = 1] - Pr Pr [xx(x) = 1ll S for all Xa E x(1),
where S denotes the uniform distribution over S. Since (x(x) = x(h, x). this implies that we have (x, D) (hh S) < for all E H.
Therefore, the set of hypotheses with Xhh, S) 2 1 - t-c - is contained in Hpp. (t + 2e).
The bound on the number of labeled data now follows directly from known concentration results using the expected number of partitions instead of the maximum in the standard VC-dimension bounds. This bound ensures that with probability 2. none of the functions E Hy (1++ 2c) with err(h) 2 have err(h) = 0.
The above two arguments together imply that with probability 1- 8, all h E H. with (h) and (h, 8) 2 1-t-c have err(h) 2 and furthermore f" has S(f*,S) 2 1-t-e. This in turn implies that with probability at least 1 8, we have err(h) where h arg ma Xh, S).
B DATASETS
Digits. Following TOHAN(Chi et al., 2021a), we conduct 6 adaptation experiments on digits datasets: M + U, M + S U.S + M, U - M and U - S. MNIST (M) (LeCun al., 1998) is the handwritten digits dataset, which have been size-normalized adn centered in 28 x 28 pixels. SVHN (S) (Netzer et al., 2011) is the real-world image digits dataset, of which images are 32 32 pixels with 3 channels. USPS (U) (Hull, 1994) data are 16x16 grayscale pixels. The SVHN and USPS images are resized to 28 28 grayscale pixels in the adaptation task (Chi et al., 2021a).
Objects. Following (Sun et al., 2019), we compared DEG-Net and benchmark on CIFAR-10 and STL-10. The CIFAR-10 (Krizhevsky et al., 2009) dataset contains 60, 000 32 x 32 color images in 10 categories, while the STL-10 (Coates al., 2011) 1) dataset is inspired by the CIFAR- 10 dataset with some modifications. However, these two datasets only contain nine overlapping classes. We removed the non-overlapping classes ("frog" and "monkey") (Shu al., 2018).
14

---

Under review as a conference paper at ICLR 2023
C DETAILS REGARDING EXPERIMENTS
Baselines. We follow the standard domain-adaptation protocols (Shu et al., 2018) and compare DEG- Net with baseline: (1) Without adaptation (WA): to classify target domain with the well-trained source domain calssifier. (2) Fine tuning (FT); to train the last connected layer of the classifier with few accessible labeled data. (3) SHOT' an HTL mehtod, where we modify it to use both the labeled target data and unlabeled target data (Liang et al., 2020). (4) S+ tAAA:to generate unlabeled data using the loss L. with the well-trained source clasifier and apply them into DANN (Ganin et al., 2016). (5) :A generate unlabeled data using the loss Ls with the few labeled target data and apply them into DANN. (6) TOHAN: a novel FHA method, which generate the specific category unlabeled data separately (Chi al., 2021a).
Implementation Details. We implement all methods by PyTorch 1.7.1 and Python 3.7.6, and conduct all the experiments on NVIDIA RTX 2080Ti GPUs. Due to the limitation of the accessible computing resources, we can not choose more complex networks as the backbone of the generator.
Our conditional generator G uses the standard DCGAN network (Radford et al., 2015). We adopt the backbone network of LeNet-5 with batch normalization and dropout to extarct the group discriminator feature. We employ connected layers with softmax function the classifier to obtain the probability. The semantic feature in digital tasks is the output of first fully connection layer. We adopt 3 connected layers with softmax function as the group discriminator D.
syper-parameter Settings. Following the common protocol of domain adaptation (Shu et al., 2018), we set fixed hyper-parameters for the different datasets. We pretrain the conditional generator for 300 epochs and pretrain the group discriminator for 100 epochs. The training step of the classifier (i.e. the adaptation module) are set 50. As for the generator and the group discriminator, the learding rate of adam optimizer is set to 1 10 10-3, As for the classifier, the learding rate of adam optimizer is set to
1 x 10-2, The tradeoff parameter À in Eq. (8) set to 0.9 and the tradeoff parameter /3 in Eq. (11) is set to 0.1. Following (Long et al., 201 8) the radeoff parameter in Eq. (13) is set to 1+exp(-10g) 1.
2
D ADDITIONAL ANALYSIS
Augmentation Techniques on the FHA Problem In this section, we compare the accuracy of the target classifier trained by TOHAN and that of TOHAN with the basic geometric data augmentation for FHA problem over the digit tasks. The geometric data augmentation technique has been widely- explored diversify the image data (Shorten & Khoshgoftaar, 2019). In our experiment, we randomly choose one or more the following augmentation technique: resizing, shifting, cropping and slight rotations (1 and 20 and to -00) fot the generated data in TOHAN. The classifier accuracy on the target domain of our method over 4 experiments and the average accuracy is shown in Table 4.
is clear that the performance of the augmentation techniques is worse than our method in general. It may be caused by the fact that the generated image are similar and even the same as the few target data. The diversity of is still low with data augmentation. The accuracy of the augmentation is basically the same as TOHAN's. The improvement brought by the augmentation is more obvious while the number of the target data is increasing.
Diversity Analysis of DEG-Net In this section, we compare the diversity of generated data of DEG-Net with that of TOHAN and target data. Because of the difficulty of calculating log-influence, we use the HSIC to measure the diversity of data. Considering that the generated batch in the training process is 32, we calculate the HSIC measure with the 32 sample data. Table 5 shows the diversity of the different data. It is clear that the diversity loss in DEG-Net works well to make the generated data data more diverse.
Data Efficiency Analysis of DEG-Net In this section, conduct the experiments in the taskS M + S and M + U to analyze the efficiency of the generated data. Following the architecture of the DEG-Net, we use the Eq. (11) to train the conditional generator and obtain the following loss to
15

---

Under review as a conference paper at ICLR 2023
Table 4: Classification accuracy standard deviation (%) digits FHA tasks of the data augmentation. Bold value represents the highest average accuracy on each column.
Number of Target Data per Class
Method
Tasks
M +
77.1=0.4
83. 03.6
84.0=20.7
86.7+1.1
8755406
88.11114
19.4+
26.7=10
1.
29.7#13
29.410.7
30.3::12
32440088
13.5+:15
TOHAN with augmentation
S M
764=0.5
08.6+
82.7=0.1
86.540.3
87.94022
88 88.2+003
89.6
+
82.1=0.7
84.9
85.3+20.6
86.741.5
874408
87.940077
89.8+0.44
Average of tasks
65.6:20.7
6.07:11.0
70.4::0.7
72.4:099
73.3:077
74.110.8
75.6::0..8
76.0=199
83.3::0..3
84.2#0.4
86.5+:111
87.1+133
88 88.0:0.5
89.7+00.5
M + 9
26.7=001
28.6:1:111
29.5:114
29.6::0.4
30.5:1122
32.11:022
33.2:10.88
TOHAN
8 +
84.2:10.4
86.511.1
U
M
84.0=0.5
25.2-+0.3
85.6+20.7
86.540.5
87.3+:066
88.240.77
89.2-+00.5
Average of tasks
65.7=1.1
70.1:10.5
70.9:00.7
72.240.8
73.0=+1.1
74.0+:0.5
75.5+00.
M -
83.1=0099
86 2:
86.5:0.66
88.740.99
89.6:20.5
91..60066
92, 12.16
27.2:10.3
29.7:10.9
30.71.0.8
DEG-Net
S M
76.2=
78.24:
85.7-106
85.94-0.8
88.6:11.6
89.5
90.2-10.7
82.2=0.7
85.94-0.6
86.5+1.5
87.8+099
88,940099
90.3+:0.5
91.6+112
Average of tasks
67.1=088
69.7:::100
72.1
73.3
75.0:t1111
76.3+::000
77.21000
Table 5: The Diversity of the target data and generated data by different methods.
Task
Target Data
TOHAN
DEG-Net
M +
0.0027
0.0019
U 5
0.0013
0.0025
0.0021
5 5 M
0.0016
0.0013
U + M
0.0004
0.0014
0.0008
S + U M +U
0.0012
0.001
0.0002
0.0009
0.0005
update classifier ft:
; - lug. log (+ yo, log (D (6 (9+)))
(23)
+ E (J. (xx)), 17 (X.)) + [e (Je (ra)), Ji (x,)),,
where Ty is the generated data and fi (xgg) is the label of the generated data. We use the different numbers of the generated data by TOHAN (Chi al., 2021a) and DEG-Net to train the classifier and the classification accuracy is shown in Table 6. It is clear that the performance of using data generated by TOHAN is almost the same as just using labeled data. addition, DEG-Net can not improve the performance of the model while the number of the target data per class small may be caused by that the generated data is similar to the label target data, so that add the almost same data for the training will bring little improvement. However, is worth nothing that the improvement will be large the number data generated by DEG-Net more than per class. This phenomenon indicates that that the data generated by DEG-Net is more independent the existing target data and could be treated as the new ones in some degree.
data generated by
16

---

Under review as a conference paper at ICLR 2023
Table 6: Classification accuracy (%) on digits FHA tasks using the generated data. Bold value represents the highest average accuracy on each column
Number of Target Data per Class
Task
Method
Number of Generated Data Class
2
3
6
7
76,0
83.3
84.2
86.5
87.1
88.0
89.7
75.8
83.7
84.3
84.3
85.0
87.5
88.1
TOHAN
76.0
83.3
84.3
84.0
85.1
87.2
89.5
83.1
86.2
82.6
86.0
85.9
88.2
88.7
91.9
92.3
26.7
28.6
29.5
29.6
30.5
32.1
33.2
26.2
28.4
TOHAN
20
25.8
26.9
29.8
27.4
29.8
32.7
32.8
M
27.2
28.5
29.7
30.7
32.9
33.7
34.9
27.3
28.2
29.6
29.
32.8
33.8
DEG-Net
20
26.4
27.3
30.2
28.9
33.5
35.0
36.4
17


Reviews:
[{"review_id": "tdRZsypdcMT", "reviewer": "Reviewer_1mfd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper generally follows a previous publication TOHAN (NeurIPS) to solve the few-shot hypothesis adaptation (FHA) problem. Like TOHAN, this paper proposes a method called DEG-Net consisting of two modules, i.e., the generation module and the adaptation module. The generation module generates diverse data from random noise and given the generated data and labeled target data, the adaptation module adapts the classifier to the target domain via adversarial training. The only difference between DEG-Net and TOHAN is the data generation module. DEG-Net uses a weight-shared conditional generative network instead of K (class number) generators. Besides, DEG-Net encourages the diversity of generated data by minimizing the HSIC measure of the generated data’s features. Experiments on Digits and CIFAR10-STL10 show that DEG-Net outperforms existing FHA methods in most cases.", "summary_of_the_review": "This paper focuses on the challenging few-shot hypothesis adaptation problem and proposes to improve an existing method TOHAN by enhancing the diversity of generated data. My concern lies in the weak novelty, weak empirical evaluation, and large complexity of the proposed method.", "strength_and_weaknesses": "### Strength\n- The targeted problem of few-shot hypothesis adaptation is meaningful and challenging.\n\n- Enhancing the diversity of generated data is reasonable and theoretically motivated.\n\n- The paper is well-written and well-organized.\n\n### Weakness\n- The proposed method lacks novelty. The only differences compared to TOHAN are the weighted-shared generator, feature-based similarity loss, and diversity loss. The two former designs are common in previous works [1, 2]. As a result, the only novelty is the diversity loss. However, there exist very related works proposing to solve the diversity problem in image generation for better target task performance, such as [3].\n\n- The proposed method includes many hyper-parameters ($\\lambda$, $\\beta$, $\\gamma$). Since it is challenging to tune them since we do not have a labeled target validation set for hyper-parameter tuning in FHA or other unsupervised domain adaptation settings. I am also curious how authors determine such hyper-parameters in all adaptation tasks because the paragram \"Hyper-parameter Settings\" do not include the validation method.\n\n- The empirical results do not show that \"DEG-Net outperforms existing FHA methods and achieves the state-of-the art performance\", as claimed by the authors before Section 2. When labeled samples are very few like 1 or 2, DEG-Net usually underperforms TOHAN, as shown in Table 2. \n\n- The experiments only consider toy datasets like digits and CIFAR10-STL10. I understand that this paper generally follows TOHAN in the method design and experiment design. But in a very related topic called source-free domain adaptation, it is common to perform the empirical evaluation on popular domain adaptation benchmarks like Office, Office-Home, VisDA, and DomainNet. FHA is a very practical problem setting as aware by the authors, evaluation on toy benchmarks is not convincing.\n\n- More baselines should be considered for comparison, such as recent test-time adaptation methods like Tent [4].\n\n[1] Model Adaptation: Unsupervised Domain Adaptation without Source Data, CVPR 2020\n\n[2] Perceptual Losses for Real-Time Style Transfer and Super-Resolution, ECCV 2016\n\n[3] Contrastive Model Inversion for Data-Free Knowledge Distillation, IJCAI 2021\n\n[4] Tent: Fully Test-time Adaptation by Entropy Minimization, ICLR 2021\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MTcXmBJGZNy", "reviewer": "Reviewer_7iBa", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose to improve the unlabeled data generation for few-shot hypothesis adaptation (FHA) which transfer a source hypothesis to target domain with only a few labeled data in target domain. They first provide theoretical proof that dependency among generated data (measured by log-coefficient) affects performance of FHA, and show that when unlabeled data is weakly dependent one can still learn a good classifier. They further propose a novel diversity-enhancing generative network that generates high diversity unlabeled while adapting the classifier to the target domain, by using HSIC. Extensive experiment results are shown on image datasets where proposed method achieves SOTA when there is good amount of target data.\n\n", "summary_of_the_review": "The paper proposes a better unlabeled data generator for FHA problem with diversity enhancing loss. It is grounded with theoretical guarantees and achieves reasonably good empirical results.  ", "strength_and_weaknesses": "Strength\n\n- The authors identify sample diversity as an important factor in auxiliary unlabeled data generation for FHA and support the finding with solid theoretical analysis. \n- They further come up with practical solutions with HSIC based regularization and propose effective and non-trivial modification to TOHAN model (Chi et al). \n- The empirical results are quite comprehensive and the proposed method achieves better performance in most of the cases, and the discussion on failure cases and ablation study is clear. \n\nWeakness\n\n- The framework of unlabeled data generation is based on existing work TOHAN, and the theoretical analysis is based on prior work related to data dependency and semi-supervised learning. \n- The improvement relative to TOHAN on 6 digits FHA tasks is less significant with overlapping confidence interval.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "GGlQlyzKvfG", "reviewer": "Reviewer_TLkA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers the task of few-shot hypothesis adaptation (FHA), in which a classifier trained on a source domain is to be adapted to a target domain containing only a few labeled examples. Previous FHA approaches synthesize additional data with a generative model, but the performance boost may be limited due to lack of diversity in the generated data. The main contributions of this paper are twofold: an analysis extending previous non-i.i.d. learning theoretical results to the FHA setting thereby suggesting that the diversity of the data is important, and the proposal of a regularizer that encourages diversity in the generated examples via the Hilbert-Schmidt independence criterion.", "summary_of_the_review": "Post-rebuttal update: I would like to thank the authors for their responses. In particular, I appreciated the clarification from the authors that the novelty lies primarily in the theoretical analysis rather than a novel algorithm. From this perspective, the two remaining concerns I have are: (1) the disconnect in assumptions between the true and synthetic data distribution, and (2) the relationship between log-influence and HSIC.\n\nI did not see a response from the authors that addressed (1), namely that the theoretical analysis focuses on the extent to which dependency in true data distribution causes decreased generalization, but the idea behind this paper is to improve diversity in the synthetically generated data. The two may be related but it this is not clear based on the contents of the current version of the paper.\n\nRegarding (2), the authors state that the connection between log-influence and HSIC is \"obvious\". I can understand that both log-influence and HSIC are both measures of independence, but if the goal is to use HSIC, then I believe that either HSIC should be used in the theoretical analysis or a more detailed discussion of the relationship between HSIC and log-influence should be included in the paper.\n\nOverall, due to these concerns, and after having read the other reviews, I will maintain my original rating.\n\n---\n\nThis paper investigates an interesting problem: whether data diversity in data-generation approaches to FHA is important, and what can be done to encourage such diversity. However, there are significant issues with both novelty and quality (outlined above), and therefore requires revision.", "strength_and_weaknesses": "Strengths:\n- Investigating the effect of diversity in generated data for training few-shot classifiers is an interesting direction that should be of fairly broad interest. \n- The connection between non-i.i.d. learning theoretic results (Dagan et al., 2019) and data generation in FHA is interesting.\n\nWeaknesses:\n- Both main contributions can be viewed as relatively minor extensions of TOHAN (Chi et al., 2021). On the one hand, Theorem 1 adapts the corresponding result of Chi et al. (2021) to incorporate the log-coefficient capturing dependency of the data. On the other, the methodological contribution consists primarily of adding a regularizer to TOHAN.\n- The relationship to data-generation, i.e. \"hallucination,\" approaches in few-shot learning is not discussed. At a deeper level, I did not get a strong sense why FHA was the chosen setting as opposed to standard few-shot learning (or possibly studying both), as both can rely on generated data.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject