Transfer Learning with Pre-trained Conditional Generative Models
Authors: Shin'ya Yamaguchi
Sekitoshi Kanai
Atsutoshi Kumagai
Daiki Chijiwa
Hisashi Kashima
Venue: Submitted to ICLR 2023 (2023)
Keywords: Deep Learning
Transfer Learning
Deep Generative Models
Semi-supervised Learning

Abstract: Transfer learning is crucial in training deep neural networks on new target tasks. Current transfer learning methods always assume at least one of (i) source and target task label spaces overlap, (ii) source datasets are available, and (iii) target network architectures are consistent with source ones. However, holding these assumptions is difficult in practical settings because the target task rarely has the same labels as the source task, the source dataset access is restricted due to storage costs and privacy, and the target architecture is often specialized to each task. To transfer source knowledge without these assumptions, we propose a transfer learning method that uses deep generative models and is composed of the following two stages: pseudo pre-training (PP) and pseudo semi-supervised learning (P-SSL). PP trains a target architecture with an artificial dataset synthesized by using conditional source generative models. P-SSL applies SSL algorithms to labeled target data and unlabeled pseudo samples, which are generated by cascading the source classifier and generative models to condition them with target samples. Our experimental results indicate that our method can outperform the baselines of scratch training and knowledge distillation.

Under review as a conference paper at ICLR 2023
TRANSFER LEARNING WITH PRE-TRAINED CONDITIONAL GENERATIVE MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
Transfer learning is crucial in training deep neural networks on new target tasks. Current transfer learning methods always assume at least one of (i) source and target task label spaces overlap, (ii) source datasets are available, and (iii) target network architectures are consistent with source ones. However, holding these assumptions is difficult in practical settings because the target task rarely has the same labels as the source task, the source dataset access is restricted due to stor- age costs and privacy, and the target architecture often specialized to each task. To transfer source knowledge without these assumptions, learning method that uses deep generative models and composed of the follow- ing two stages: pseudo pre-training (PP) and pseudo semi-supervised learning (P-SSL). PP trains a target architecture with an artificial dataset synthesized by using conditional source generative models. P-SSL applies SSL algorithms la- beled target data and unlabeled pseudo samples, which are generated by cascading the source classifier and generative models to condition them with target samples. Our experimental results indicate that our method can outperform the baselines of scratch training and knowledge distillation.
propose transfer
INTRODUCTION
For training deep neural networks on new tasks, fransfer learning is essential, which leverages the knowledge of related (source) tasks to the new (target) tasks via the joint- or pre-training of source models. There are many transfer learning methods for deep models under various conditions (Pan & Yang 2010 Wang & Deng, 2018). For instance, domain adaptation leverages source knowl- edge the target task by minimizing the domain gaps (Ganin et al., 2016). and fine-tuning uses the pre-trained weights on source tasks as the initial weights of the target models YYosinkki et al.) 2014). These existing powerful transfer learning methods always assume at least one of (i) source and target label spaces have overlaps, e.g., target task composed of the same class categories as a source task, (ii) source datasets are available, and (iii) consistency of neural network architectures i.e., the archi- tectures in the target task must be the same as that in the source task. However, these assumptions are seldom satisfied real-worle settings (Chang et al.) 2019 Kenthapadi et al.) 2019 Tan et al.) 2019). For instance, suppose embedded device in an automobile company. The developers found an optimal neural architecture for the target dataset and the device by neural architecture search, but they cannot directly access the source dataset for the reason of protecting customer information. In such a situation, the exist- ing transfer learning methods requiring the above assumptions are unavailable, and the developers cannot obtain the best model.
of developing
image classifier on totally new task for an
To promote the practical application of deep models, we argue that we should reconsider the three assumptions on which the existing transfer learning methods depend. For assumption (i), new target tasks do not necessarily have the label spaces overlapping with source ones because target labels are often designed on the basis of their requisites. In the above example, if we train models on StanfordCars (Krause et al. 2013), which a fine-grained car dataset, there is no overlap with ImageNet (Russakovsky et al., 2015) even though ImageNet has 1000 classes. For (ii), the accessi- bility of source datasets is often limited due to storage costs and privacy (Liang et al. 2020 Kundu et al.] 2ang al. 20) e.g., ImageNet consumes over 100GB and contains person faces co-occurring with objects that potentially raise privacy concerns (Yang et al.) 2022). For (iii,, the consistency of the source and target architectures is broken if the new architecture is specialized for

---

Under review as a conference paper at ICLR 2023
Initiolize
Unsupervisen Loss
G5
Dur-
D
1/s ~ 3/+ - G. - (#sr No) -
Supervi sed Loss
Vac 1
Ciat
CaA.
Dr
(1)
Supervised Loss
(a) Pseudo Pre-training
(b) Pseudo Semi-supervised Learning
Figure I: Proposed transfer learning methods leveraging conditional source generative model G.. Red color represents given source models, light blue represents target models and datasets, and dark blue represents the output of the proposed methods. (a) We produce initial weights of a target architecture A by training source classifier O. with pairs of conditional sample f. (y.)) and uniformly sampled source label (b) We penalize target classifier C₂ with unsupervised loss derived from SSL method by applying a pseudo sample while supervised training on target dataset Dr. 2 is sampled from G, conditioned by pseudo source label Vse- (xt).
Table Comparison transfer learning settings
no label overlap
(ii) sounce dalaset access (iii) architecture inconsistency
Domain Adaptation Fine tuning Ours
the new tasks like the above example. Deep models are often specialized for tasks or computational resources by neural architecture search (Zoph & Le 2017 Lee et al.) 202T) in particular when de- ploying on edge devices; thus, their architectures can differ for each task and runtime environment. Since existing transfer learning methods require one of the three assumptions, practitioners must de- sign target tasks and architectures to fit those assumptions by sacrificing performance. To maximize the potential performance of deep models, new transfer learning paradigm is required.
In this paper, we shed light on an important but less studied problem setting of transfer learning, where (i) source and target task label spaces do not have overlaps, (ii) source datasets are not avail- able, and (iii) target network architectures are not consistent with source ones (Tab. I To transfer source knowledge while satisfying the above three conditions, our main idea is to leverage source pre-trained discriminative and generative models; their architectures differ from that of target tasks. We focus on applying the generated samples from source coadsssoniiiinall generative models for tar- get training. Deep conditional generative models precisely replicate complex data distributions such as ImageNet (Brock et al. 2018 Karras et al., 2020 Dhariwal Nichol, 202T), and the pre-trained models are widely used for downstream tasks (Wang al. 2018] al.) 2020aa Patashnik et al.) 2021 Ramesh et al. (022). Furthermore, deep generative models have the potential to resolve the problem of source dataset access because they compress information of large datasets into much smaller pre-trained weights (e.g., about 100MB in the case of a BigGAN generator), and safely generate informative samples without re-generating training samples by differential privacy training
techniques (Torkzadehmahan et al. 2019 Augenstein et al. 2020 Liew et al. 2022).
By using conditional generative models, propose two-stage transfer learning method composed of pseudo pre-training (PP) and pseudo semi-supervised learning (P-SSL). Figure | illustrates an overview of our method. PP pre-trains the target architectures by using the artificial dataset gen- erated from the source conditional generated samples and given labels. This simple pre-process provides effective initial weights without accessing source datasets and architecture consistency. To address the non-overlap of the label spaces without accessing source datasets, P-SSL trains target model with SSL (Chapelle et al. 2006 Van Engelen & Hoos 2020) by treating pseudo samples drawn from the conditional generative models as the unlabeled dataset. Since SSL assumes the la- beled and unlabeled datasets are drawn from the same distribution, the pseudo samples should be target-related samples, whose distribution is similar enough to the target distribution. To generate target-related samples, we cascade a classifier and conditional generative model the source do- main. Specifically, we (a) obtain pseudo source soft labels from the source classifier by applying
2

---

Under review as a conference paper at ICLR 2023
"Jaap" "Llavosiin" 0.11 "Naetegtin"
"Hummer"
Dt
C.A.
G
Dett
Vanut
Poe-t
Figure 2: Pseudo conditional sampling. We obtain pseudo soft label You-t by applying target data 2; to a source classifier Cz and then generate a target-related sample 2 from a source generative model G. conditioned by Vs-t. In this example, C output Ys-e from the input car image z of Hummer class by interpreting 24 mixture of source car classes (Jeep, Limousine, MovingVan, etc.), and then Gg generate larget-related image from Ys-t-
target data, and (b) generate conditional samples given a pseudo source soft label. By using the target-related samples, P-SSL trains target models with off-the-shelf SSL. algorithms.
In the experiments, first confirm the effectiveness of our method through motivating example scenario where the source and target labels do not overlap, the source dataset is unavailable, and the architectures are specialized by manual neural architecture search (Sec. 4.2p. Then, we show that our method can stably improve the baselines in transfer learning without three assumptions under various conditions: e.g., multiple target architectures (Sec. 4.3) and multiple target datasets (Sec. (4.4.. These indicate that our method succeeds to make the architecture and task designs free from the three assumptions. Further, we confirm that our method can achieve practical performance without the three assumptions: the performance was comparable to the methods that require one of the three assumptions (Sec. (Sec.[4.5)and) (4..).)(4.7) We also provide extensive analysis revealing the conditions for the success of method. For instance, we found that the target performance highly depends on the similarity of generated samples to the target data (Sec. 3.2. 4.2.44am and|4.4) and a general source dataset (ImageNet) is more suitable than specific source dataset (CompCars) when the target dataset is StanfordCars (Sec. 4.6).
PROBLEM SETTING
We consider a transfer learning problem where we train a neural network model for: on labeled target dataset D, {xr) E Xr of Di)) given source classifier C,A. and source conditional generative model G,; C.A. and G, are off-the-sheif i.e., we do not access source datasets to pre-train them. fel: is parameterized by 0 of target neural architecture A,. C.A. outputs class probabilities by softmax function and G3 is pre-trained on a labeled source dataset D.. We mainly consider classification problems and denote fol as the target classifier C.Al in the below.
In this setting, assume the following conditions.
no label overlap: Vd u 34- =
no source dataset access: DD not available when training C.A.
(iii) architecture inconsistency: .., + Ar
Existing methods are not available when the three conditions are satisfied simultaneously. Unsu- pervised domain adaptaion (Ganin et al.] (0166) require Vd n V/= * 01, accessing D., and A, A Source-free domain adaptation methods (Liang et al. 2020 Kundu al. 2020 Wang et al. 202 la) can adapt models without accessing D., but still depend on Vr n J @ and A, tuning YYsiinkii et al.] 2014) can be applied to the problem with the condition (i) and (ii) and only if A. A, However, since recent progress of neural architecture search Zoph & Le 2017 Wang ct al. 2021b Lee et al. 2021) enable to find specialized A, for each task, situations A, * A, common when developing models for environments requiring both accuracy and model size e.g., embedded devices. As result, the specialized O.A: currently sacrifices the accuracy that the size requirement can satisfied. Therefore, tackling this problem setting has the potential to enlarge the applicability of deep models.
A.. Fine-

---

Under review as a conference paper at ICLR 2023
3 PROPOSED METHOD
In this section, we describe our proposed method. An overview our method illustrated in Figure [ PP yields initial weights for target architecture by training it the source task with synthesized samples from a conditional source generative model. P-SSL takes into account the pseudo samples drawn from generative models as unlabeled dataset in an SSL setting. In P-SSL, we generate target-related samples by pseudo conditional sampling (PCS, Figure 2), which cascades source classifier and source conditional generative model.
3.1 PSEUDO PRE-TRAINING
Without accessing source datasets and architectures consistency, we cannot directly use the exist- ing pre-trained weights for fine-tuning C,A,. To build useful representations under these condi- tions, pre-train the weights of A, with synthesized samples from conditional source generative model. Every training iteration of PP is composed of two simple steps: (step 1) synthesizing a
batch of generating source conditional samples { ~ Gy(y)) from uniformly sampled source labels y E V., and (step 2) optimizing 0 on the source classification task with the labeled batch of { vt) B = by minimizing
B¹ Σ CE (E; 0),
1
(1)
where B", is the batch size for PP and CE is cross-entropy loss. Since PP alternately performs the sample synthesis and training in an online manner, it efficiently yields pre-trained weights without consuming massive storage. Further, offline strategy: i.e., synthesizing fixed samples in advance of training (Appendix C.4). We use the pre-tree rre-trained weights from PP as the initial weights of C.A, by replacing the final layer of the source task that of the target task.
found that this online strategy is better in accuracy than the
3.2
PSEUDO SEMI-SUPERV ISED LEARNING
3.2.1
SEMI-SUPERVISED LEARNING
Given a labeled dataset D, = [(x,, y' and an unlabeled dataset D,, = (, SSL is used to optimize the parameter 9 of a deep neural network by solving the following problem.
min 0 N
1
Σ
Cupp (z, y, 0)
1
Σ ZED.
Lunsup (x,0),
(2)
No.
(2,0)DD:
where Csup is supervised loss for labeled sample (xI, y1) (c.g., cross-entropy loss), Lunsup is an unsupervised loss for an unlabeled sample Ta. and is a hyperparameter for balancing L and Lumsup- In SSL, it is generally assumed that D, and D, shares the p(z). If there a large gap between the labeled and unlabeled data distribution, the performance of SSL algorithms degrades (Oliver et al., 2018). However, Xie al. (2020) have revealed that unlabeled samples in another dataset different from a target dataset can improve the performance of SSL algorithms by carefully selecting target-related samples from source datasets. This implies that SSL algorithms can achieve high performances as long as the unlabeled samples are related to target datasets, even when they belong to different datasets. On the basis of this implication, our P-SSL exploits pseudo samples drawn from source generative models as unlabeled data for SSL..
generative distribution
3.2.2 PSEUDO CONDITIONAL SAMPLING
To generate informative target-related samples, our method uses PCS, which generates target-related samples by cascading C.A. and G.. With PCS, we first obtain pseudo source label yerr from source classifier C.A. with a uniformly sampled 2, from D,-
== (ax))
Intuitively, Use represents the relation between source class categories and in the form of the probabilities. We then generate target-related samples Istt with yet+ as the conditional label by
Fstt (s(ysee) G, (xx))
4

---

Under review as a conference paper at ICLR 2023
Algorithm Pseudo conditional sampling
Table 2: Distribution gaps of pseudo samples on multi- ple target datasets
Require: TTrget dataset Dr source classifier CA source generator G., number of pseudo samples N2t-1, output label function y
Ensure: Pseudo unlabeled dataset D.
Distribution Gap (FID)
1: Y +
(D,,
(F(D.).
(Dat.t De)
C <- SvapFinalLayor((A1
for D: do
Caltech-256-60
31.27
87.18
19.69
4: Uo-. (z,)
CUB 200-2011
131.85
65.34
19.15
5: Add Vor-t to Yart
DTTDD
100.51
82.63
87.57
for
FGVC-Aircraft
189.16
47.29
23.68
Repeut concatenating Y.rt with itself until the
Indoor67
96.68
44.09
34.27
length reaches to Nor-t
OxfordFlower1002
190.33
137.64
118.39
D.-t f-
OxfordPets
95.16
20.59
16.94
for Vo-e Y,.: do
StunfordCars
147.27
54.76
19.92
10: Jamt Ga(y)-s)
StanfordDogs
80.94
6.09
7.34
11:
Add Tame Dami
12: end for
Although Gg trained with discrete (one-hot) class labels, can generate class-wise interpolated
samples by the continuously mixed labels of multiple class categories (Miyato & Koyama 2018 Brock et al.] 2019). By leveraging this characteristic, we aim to generate target-related samples by Yse-t constructed with interpolation of source classes.
For the training of the target task, we compose a pseudo dataset D..- by applying Algorithm | In line 2, swap the final layer of C.A. with output label function y. which is softmax function as the default. Sec. C.5.3 we empirically evaluate the effects of the choice of
3.2.3 TRAINING
By applying PCS, we obtain a target-related sample Ezt-t from Tt. In the training of CrA,, one can assign the label y/ of Tr to Ese-t since Tst-r generated from F₁. However, difficult directly (I:e-1,) Wt) in supervised learning because Ust-r harm the performance of C,Ae due to the gap between label spaces; we empirically confirm that this naïve approach fails to boost the target performance mentioned in Sec. [C5.I] To extract informative knowledge from 2st-t, we apply SSL algorithms that train target classifier O₂ by using a labeled target dataset D, and unlabeled dataset D.tt generated by PCS. On the basis of the implication discussed in Sec. 3.2.1 we can expect that the training by SSL improve C₂A: if D.+t contains target-related samples. We compute the unsupervised loss function for pseudo sample Jst-t as Lumsup (Tise Mist t. A). We can adopt arbitrary SSL algorithms for calculating Lumup- For instance, UDA (Xie et al. 2020 is defined by
Cunsup (x, =1
max V'ECAA (a,r):
/>B / CE CE (,,
(5)
where 1 is an indicator function, CE is cross entropy function, C.A (zz, r) is the target classifier replacing the final layer with the temperature softmax function with temperature hyperparameter 3 is confidence threshold, and T(.) is an input transformation function such as RandAug- ment (Cubuk et al. 2020). the experiments, we used UDA because achieves the best result; we compare and discuss applying the other SSL algorithms in Sec. C Eventually, we optimize the parameter by the following objective function based on Eq
min N
Σ ZLNEED,
Caup (o,, Vc, 0) + Note zoe iEDic
1
Σ
Lunsup (F
3.2.4 QUALITY OF PSEUDO SAMPLES
In this method, since we treat Øxe- ~ Gs (C.AAA()zz)) as the unlabeled sample of the target task, the following assumption is required:
Assumption 3.1 PD, (x) PD.... (z) N., Ex, Po,
where Pr (x) is data distribution of a dataset D. That is, if pseudo samples satisfy Assumption P-SSI. should boost the target task performance by P-SSL.
5

---

Under review as a conference paper at ICLR 2023
Table 3: List of target datasets
Table 4: Evaluations on Motivating Example
Tusk
Classes
Sixe (Trato/Tes)
Architecture
Top- I Acc. (8)
Caitech2256-0 60
CUB-200-3011
Finerranne Object Recegnition
Generi Onject Recognitinn
256
200
15, 15,46015,,10 189
5,09415, 764
Scratch
RN18 (4,4,4
73.27#043
DTT)
Testur Recegnaiinn
47
3.760V1.880 580
Fine-tuning
RN18 (1,44
87,75=035
Indoce5?
PCVC-Airemi
Finerrannal Object Recegaition
Scene Recognition
100
6,66723,333
5,36601,540 540
Seratch
Custom RN18 (2. 10,
77.01 =0.01
OnfordFlomer
Finegratned Object Recognition
2,04EV5 149
Logir Matching
Custom RN18 (2, 2,
85, 2008
StaffonCaare
OnferdPas
Finerranee Object Recegsition
Finegratned Object Recegnition
196
37
3,68003, 369
8.144/%,041
Soft Targel
Ours
Custom RN18 (2, 2,
Custom RN18 (2, 10,
89.13-903
82.67=030
To confirm that pseudo samples can satisfy Assumption 3.1 we assess the difference between the target distribution and pseudo distribution. Since it is difficult directly compute the likelihood of the pseudo samples, we leverage Fréchet Inception Distance (FII), Heusel et al. (2017), which mea- sures a distribution gap between two datasets by 2-Wasserstein distance in the closed-form (lower FID means higher similarity), We evaluate the quality of Da-t by comparing FID D,) to (ID Dz) and FID (.,,.,,) Dr), where F F(D.) is a subset of D. constructed by confidence-based filtering similar to a previous study (Xie et al.) (2020) (see Appendix Aapendix(B.4]for more detailed protocol).
That is, if D achieves lower FID than D. or Dz,, then Da can approximate D, well.
Table 2 shows the FID scores when D, is ImageNet. The experimental settings are shared with Sec. Except for DTD and StanfordDogs, Dzt+ outperformed DD and F(D,) in terms of the similarity to D,. This indicates that PCS can produce more target-related samples than the natu- ral source dataset. On the other hand, in the case of DTD (texture classes), Dee- relatively has low similarity to D.. This implies that PCS does not approximate PD, (z) well when the source and target datasets are not so relevant. Note that StanfordDogs is a subset of ImageNet, and thus (ID(F(D)). D2) was much smaller than other datasets. Nevertheless, the FID D,) is com- parable to FID (F.,), Dr). From this result, we can say that our method approximates the target distribution almost well as the actual target samples. We further discuss the relationships between the pseudo sample quality and target performance in Sec. 4.4
4 EXPERIMENTS
We evaluate our method with multiple target architectures and datasets, and compare it with base- lines including scratch training and knowledge distillation that can be applied to our problem setting with a simple modification. We further conduct detailed analyses of our pseudo pre-training (PP) and pseudo semi-superve supervuper learning (P-SSL) in terms of (a) the practicality of our method by com- paring to the transfer learning methods that require the assumptions of source dataset access and architecture consistency, (b) the effect of source dataset choices, (c) the applicability toward another target task (object detection) other than classification. We further provide more detailed experiments including the effect of varying target dataset size (Appendix C.2). the performance difference when varying source generative model (Appendix C.3), the detailed analysis of PP (Appendix CC.) and P- SSL (Appendix C.5) J and the qualitative evaluations of pseudo samples (Appendix D) We provide the detailed settings for training in Appendix B.3
4.1 SETTING
Baselines
Basically, there are no existing transfer learning methods that are available on the prob- lem setting defined in Sec. 2 Thus, we evaluated our method by comparing it with the scratch training (Seratch), which trains a model with only a target dataset, and naive applications of knowl- edge distillation methods: Logit Matching (Ba & Caruana 2014) and Soft Target (Hinton et al.) 2013). Logit Matching and Soft Target can be used for transfer learning under architecture inconsis- tency since their loss functions use only final logit outputs of models regardless of the intermediate layers. To transfer knowledge in C.A. to C,A,, C.A. on the target task, and then train C.A, with knowledge distillation penalties by treating C.A, as the teacher model. We provide more details in Appendix B.2
first yield C,A, by fine-tuning the parameters of
Datasets We used ImageNet (Russakovsky et al. 2015) as the default source datasets. In Sec. 4.6 report the case of applying CompCars (Yang et al. 2015) as the source dataset. For the target dataset, we mainly used StanfordCars (Krause et al. 2013), which is for fine- -raigrr classification of car types, the target dataset. In Sec. 4.4 used the nine classification datasets listed in

---

Under review as a conference paper at ICLR 2023
Table 5: Performance comparison of multiple target architectures on StanfordCars (Top-1 Acc.(%))
ResNet-50
WRN-50-2
MobileNeeV33-L MNASNec1.0
EfficientNet-B0
ElficientNet-BS
Scratch
71.8610.0
76.2
79.2240.0
80,9810.7
80.80
81.7343350
Logit Matching
84,36 Lo.47
86,28
06,28-1113
85.08-0.08
85.11 10.10
86,37-0.99
88.42-10.0
Soft Target
79.95
82.34=1.5
83,55
84.64
85.16
85,30#1137
Ours
90.69
91.766#m41
87.39
88.40
89.28
90.04=0034
Table 6: Performance comparison of WRN-50-2 classifiers multiple target datasets (Top-1 Acc. (%))
Caltech-256-60
CUB-200-2011
DTD
FGVC-Aireaat
Indoor67
OxfordFlower
OxfordPets
StanfordDogs
Scratch
48 18.0751.3
$2.61
45,11
74.04=00
50,77
67.91
63.03=120
57.16
Logit Matching
55, 33.28=1a
62,5242.1
49,294us
78.9 78.91
57.61
75. 75,23
70 70.56
64. 64.46=188
Soft Target
54.84=188
60.5
48.39
77.08=530
51.0g=1.2
69 69,90
65.624409
63.64-1m
Ours who PP
51.625679
$6.61
45,50
77.13-00
$1,23-101
68. 68
68 70
61.26 61.26
Ours P-SSL
70.88=121
71,78+002
61,28=*8
86.08 66.08=0144
66 66.79=922
94.02*937
86 86-31
73.30=000
Ours
71.35
74.93
57,48
87,98-10
67.72-
90.31
89,97=641
75.25 75.25 sais
Table as the target datasets. We used these datasets with the intent to include various granularities and domains. We constructed Caltech-256-60 by randomly sampling 60 images per class from the original dataset in accordance with the procedure of Cui et al. (20T8). Note that StanfordDogs is a subset of ImageNet, and thus has the overlap of the label space to ImageNet, but we added this dataset confirm the performance when the overlapping exists.
Network Architecture As a source architecture A,, we used the ResNet-50 architecture (He) et al. 2016) with the pre-trained weight distributed by torchviss in official repository For a target architecture Ar. we used five architectures publicly available on torchvisi on: WRN-50- 2 (Zagoruyko & Komodakis] 2016). MNASNet1.0 (Tan et al. 2019). MobileNetV3-LL (Howard et al.) 2019). and ElficientNettB0/B5 Tan & Le. 2019). Note that, to ensure reproducibility, we assume them as entirely new architectures for the target task in our problem setting, while they are actu- ally existing architectures. For source conditional generative model G,, used BigGAN (Brock) jet al.] (018)) generating 256 x 256 resolution images as the default architecture. We also tested the other generative models such as ADM-G (Dhariwal Nichol 2021) in Sec. C.3 We implemented
BigGAN on the basis of open source repositories including pytorch aettorch-peeraaae ppeerained--ee gigN we used the pre-trained weights distributed by the repositories.
4.2 MOTIVATING EXAMPLE: MANUAL ARCHITECTURE SEARCH
First of all, we confirm the effectiveness of our setting and method through practical scenario described in Sec. [ Here, we consider case of manually optimizing the number of layers of ResNet- 18 (RN18) for the target task to improve the performance while keeping the model size.
We evaluated our method on the above scenario by assuming StanfordCars as the target dataset and ImageNet the source dataset. We scarched the custom architecture by grid search of the layers in four blocks of RNI8 from (2, 2, 2, 2) to (2, 2, 2, 10) by varying layers in {2, 4, 6, 8, 10} for each block of ResNet while keeping the sum of layers less than equal to 18 to maintain architecture size. We found that the best architecture is one with (2, 10, 2, 2). The test accuracies on StanfordCars shown in Table 4 We can see that finding an optimal custom RNI8 architecture for the target task brings test accuracy improvements, and our method contributes to further improvements under this difficult situation. This result indicates that our method can widen the applicability of neural architecture search techniques, which have been difficult in practice in terms of accuracy.
4.3 TARGET ARCHITECTURES
We discuss the performance evaluations by varying the neural network architectures of the target classifiers to evaluate method on the condition of architecture inconsistency. Table 5 lists the results on StanfordCars with multiple target architectures. Note that we evaluated our method by fixing the source architecture to ResNet-50. Our method outperformed the baselines on all target architectures without architecture consistency. Remarkably, our method stably performs arbitrary
7

---

Under review as a conference paper at ICLR 2023
Table 7: Comparison to fine-tuning (FT) and
Table 8: Comparison of source datasets on Stan- fordCars
semi-supervised learning (SSL) on StanfordCars
Top Acc. (%)
Source Dataset
Generative Model
PP
P-SSL
ImageNet
CompCars
BigGAN (Brock al. 2019
90.95-..m
80
Real Dataset
FT
R-SSL.
PP
90.95+0.21
87.57
All ImageNet
88, 2417
73.0123:0
P-SSL
80.01 20.14
79.97#0.13
Filtered ImageNet
74.08+2.77
77.21=029
PP P-SSL
91.76++041
87.80
relationships between target and source architectures including from smaller architecture (ResNet- 50) to larger ones (WRN-50-2 and EfficientNet-B5), and from a larger one (ResNet-50) to smaller ones (MNASNetl. .0, MobileNetV3, and EfficientNet-BO) This flexibility can lead the effective- ness of the neural architecture search in Sec. 4.2
4.4 TARGET DATASETS
We show the efficacy our method on multiple target datasets
other than StanfordCars. We used WRN-50-2 as the architec-
Accuracy by P-UDA (%)
regression (- - 0.800
ture of the target classifiers. Table 6 lists the top-1 accuracy
of each classification task. Our method stably improved the
baselines across datasets. We also print the ablation results of method (Ours w/o PP and Ours w/o P-UDA) for assessing the dependencies of PP and P-SSL on datasets. PP outper- formed the baselines on all target datasets. This suggests that building basic representation by pre-training is effective for various target tasks even if the source dataset is drawn from generative models. For P-SSL, we observed that it boosted the scratch models all target datasets except for DTD and OxfordFlower. As the reasons for the degradation on DTD and OxfordFlower, we consider that the pseudo samples do not satisfy Assumption 3.1 as discussed in Sec. 3.2.4 In fact, we observe that FID Dt) is correlated to the accuracy gain from Scratch models (-0.80 of correlation coefficient, -0.97 of Spearman rank correlation) shown in Fig. These experimental
- 13
25
50
76 D.)
100
(D....
Figure 3: Correlation between FID and accuracy gain
results suggest that our method is effective on the setting without no label overlap as long as D. approximates D, well.
4.5 COMPARISON METHODS REQUIRING PARTIAL CONDITIONS
To confirm the practicality of our method, we compared it with methods requiring A., A and accessing D.. We tested Fine-tuning (FT) and R-SSL, which use a source pre-trained model and a real source dataset for SSL as reference. For FT and R-SSL, we used ImageNet and subset of ImageNet (Filtered ImageNet), which was collected by confidence-basee filtering similar to Xie let al. (2020) (see Appendix B.4). This manual filtering process corresponds to PCS in P-SSL. In Table PP and P-SSL outperformed FT and R-SSL, respectively. This suggests that the samples from G, not only preserve essential information of D, but also useful than D,, i.e., access- ing D₂ may not be necessary for transfer learning. In summary, PP and P-SSI. are practical enough compared to existing methods that require assumptions.
4.6 SOURCE DATASETS
We investigate the preferable characteristics of source datasets for PP and P-SSL by testing another source dataset, which was CompCars (Yang et al., 2015), fine-grained vehicle dataset containing 136K images (see Appendix B.5 for details). This is more similar to the target dataset (StanfordCars) than ImageNet. All training settings were the same as mentioned Sec. 4.1 Table flitss the scores for each model of our methods. The models using ImageNet were superior to those using CompCars. To seek the difference, we measured FID D₂) when using CompCars as with the protocol in 22. 12, which is inferior to 19.92 when using ImageNet. This suggests that the fidelity of pseudo samples toward target samples is important to boost the target performance and is not simply determined by the similarity between the source and target datasets. We consider
Sec. 4.5 and the score
8

---

Under review as a conference paper at ICLR 2023
that the diversity of source classes provides the usefulness of pseudo samples; thus ImageNet is
superior to CompCars as the source dataset.
4.7 APPLICATION TO OBJECT DETECTION TASK
Although we have mainly discussed the cases where the target task is classification through the paper, our method can be applied to any task for which the SSL method exists. Here, we evaluate the applicability of PP+P-SSI. toward another target task other than classification. To this end, we applied our method to object detec- tion task on PASCAL-VOR 2007 (Everingham
Table 9: Results PASCAL-VOC 2007
trainval07+12
trainval07
AP
AP50
AP7s
AP
AP5o
APrs
FT (ImageNet)
52.5
80.3
56.2
42.2
74.0
43.7
PP PP+P-SSL (UT)
52.6
80.1
57.4
41.2
70.6
43.4
57.3
83.1
63.5
52.2
80.3
56.8
eet al. 2013) by using FPN (Lin et al. (017) models with ResNet-50 backbone. As the SSL method, used Unbiased Teacher (UT, (Liu et al. (022)), which is method for object detection based on self distillation and pseudo labeling, and implemented PP+P-SSL on the code base provided by Liu al. (202T). We generated samples for PP and P-SSL in the same way as the classification periments; we used ImageNet as the source dataset. Table 9 shows the results of average precision scores calculated by following detectron2 (Wu al. 2019p. Note that the baseline of Table 9 is FT instead of Scratch because the Scratch setting of the object detection task is hard to train and too slow to converge. We confirm that PP achieved competitive results FT and P-SSL significantly boosted the PP model. This can be caused by the high similarity between D.-r and D, (19.0 in FID). This result indicates that our method has the flexibility to be applied to other tasks as well as classification and is expected to improve baselines.
5 RELATED WORK
We briefly discuss related works by focusing on training techniques applying generative models. We also provide discussions on existing transfer learning and semi-supervised learning in Appendix
Similar to our study, several studies have applied the expressive power of conditional generative models to boost the performance discriminative models; Zhu et al. 2018)) and Yamaguchi et al. 2020) have exploited the generated images from conditional GANs for data augmentation in clas- sification, and Sankaranarayanan et al. (2018) have introduced conditional GANs into the system of domain adaptation for learning joint-feature spaces of source and target domains. Moreover, [Li et al. (220b) have implemented an unsupervised domain adaptation technique with conditional GANs in setting of no accessing source datasets. These studies require label overlapping between source and target tasks or training of generative models on target datasets, which causes problems of over- fitting and mode collapse when the target datasets are small (Zhang et al., 2020 Zhao et al., 2020bb] Karras et al. (2220). Our method, however, requires no additional training in generative models because simply extracts samples from fixed pre-trained conditional generative models.
CONCLUSION AND LIMITATION
We explored a new transfer learning setting where (i) source and target task label spaces do not have overlaps, (ii) source datasets are not available, and (iii) target network architectures are not consistent with source ones. In this setting, we cannot use existing transfer learning such as do- main adaptation and fine-tuning. To transfer knowledge, we proposed simple method leveraging pre-trained conditional source generative models, which is composed of PP and P-SSL. PP yields effective initial weights of a target architecture by generated samples and P-SSL applies an SSL algorithm by taking into account the pseudo samples from the generative models as the unlabeled dataset for the target task. Our experiments showed that method can practically transfer knowl- edge from the source task to the target tasks without the assumptions of existing transfer learning settings. One of the limitations of our method is the difficulty to improve target models when the gap between source and target is too large. A future step is to modify the pseudo sampling process by optimizing generative models toward the target dataset, which was avoided in this work to keep the simplicity and stability of the method.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Pulkit Agrawal, Ross Girshick, and Jitendra Malik. Analyzing the performance of multilayer neural networks for object recognition. In European conference on computer vision, 2014.
Sean Augenstein, H Brendan McMahan, Daniel Ramage, Swaroop Ramaswamy, Peter Kairouz, Mingqing Chen, Rajiv Mathews, et al. Generative models for effective ml private, decentral- ized datasets. In International Conference on Learning Representations, 2020.
Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? In Advances in Neural Informa- tion Processing Systems, 2014.
Philip Bachman, Ouais Alsharif, and Doina Precup. Learning with pseudo-ensembles. In Advances in neural information processing systems, 2014.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2018.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019.
Xiaobin Chang, Yongxin Yang. Tao Xiang, and Timothy M Hospedales. Disjoint label space transfer learning with common factorised space. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 3288-3295, 2019.
Olivier Chapelle, Bernhard Schölkopf, and Alexander Zien. Semi-supervised Learning. MIT Press, 2006.
Xinyang Chen, Sinan Wang, Bo Fu, Mingsheng Long, and Jianmin Wang. Catastrophic forgetting meets negative transfer: Batch spectral shrinkage for safe transfer learning. In Advances in Neural Information Processing Systems, 2019.
Boris Chidlovskii, Stephane Clinchant, and Gabriela Csurka. Domain adaptation in the absence of source domain data. In ACM SIGKDD International Conference on Knowledge Diseovery and Data Mining, 2016.
M. Cimpoi, S. Maji. I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2014.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 702-703, 2020.
Yin Cui, Yang Song, Chen Sun, Andrew Howard, and Serge Belongie. Large scale fine-grained categorization and domain-specific transfer learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2018.
Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans image synthesis. In Advances in Neural Information Processing Systems, 2021.
M. Everingham, S. M. A. Eslami, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The pascal visual object classes challenge: retrospective. International Journal of Computer Vision, 111, 2015.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural net- works. The journal of machine learning research, 2016.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for ac- curate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2014.
Yves Grandvalet and Yoshua Bengio. Supi-supe learning by entropy minimization. In Ad- vances in Neural Information Processing Systems, 2005.
10

---

Under review as a conference paper at ICLR 2023
Gregory Griffin, Alex Holub, and Pietro Perona. Caltech-256 object category dataset. 2007.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In Advances in Neural Information Processing Systems, 2017.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv: 02531, 2015.
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1314-1324, 2019.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. In Advances in Neural Information Processing Systems, 2020.
Krishnaram Kenthapadi, Ilya Mironov, and Abhradeep Guha Thakurta. Privacy-preserving data mining in industry. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, 2019.
Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei. Novel dataset for fine- grained image categorization. In First Workshop Fine-Grained Visual Categorization, IEEE Conference on Computer Vision and Pattern Recognition, 2011.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition, Syd- ney, Australia, 2013.
Jogendra Nath Kundu, Naveen Venkat, Rahul M V, and R. Venkatesh Babu. Universal source-free domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In International Conference Learning Representations, 2016.
Dong-Hyun Lee al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop challenges in representation learning, ICML, 2013.
Hayeon Lee, Eunyoung Hyung, and Sung Ju Hwang. Rapid neural architecture search by learning to generate graphs from datasets. In International Conference on Learning Representations, 2021.
Hao Li, Pratik Chaudhari, Hao Yang. Michael Lam, Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto. Rethinking the hyperparameters for fine-tuning. In International Conference on Learning Representations, 2020a.
Rui Li, Qianfen Jiao, Wenming Cao, Hau-San Wong, and Wu. Model adaptation: Unsuper- vised domain adaptation without source data. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2020b.
Xingjian Li, Haoyi Xiong, Hanchao Wang, Yuxuan Rao, Liping Liu, Zeyu Chen, and Jun Huan. Delta: Deep learning transfer using feature map with attention for convolutional networks. In International Conference Learning Representations, 2019.
Xuhong Li, Yves Grandvalet, and Franck Davoine. Explicit inductive bias for transfer learning with convolutional networks, In International Conference on Machine Learning, 2018
Jian Liang, Dapeng Hu, and Jiashi Feng. Do really need to access the source data? Source hypothesis transfer for unsupervised domain adaptation. In Proceedings of the 37th International Conference Machine Learning, 2020.
11

---

Under review as a conference paper at ICLR 2023
Seng Pei Liew, Tsubasa Takahashi, and Michihiko Ueno. Pearl: Data synthesis via private embed- dings and adversarial reconstruction learning. International Conference on Learning Repre- sentations, 2022.
Tsung-Yi Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, 2017.
Yen-Cheng Liu, Chih- Yao Ma, Zijian He. Chia- Wen Kuo, Kan Chen, Peizhao Zhang, Bichen Wu, Zsolt Kira, and Peter Vajda. Unbiased teacher for semi-supervised object detection. Interna- tional Conference on Learning Representations, 2021.
S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. Technical report, 2013.
Andre Martins and Ramon Astudillo. From softmax to sparsemax: A sparse model of attention and multi-label classification, In International Conference on Machine Learning, pp. 1614-1623. PMLR, 2016.
Kang Minguk, Shim Woohyeon, Cho Minsu, and Park Jaesik. Rebooting acgan: Auxiliary classifier gans with stable training. In Conference on Neurul Information Processing Systems, 2021.
Takeru Miyato and Masanori Koyama, cGANs with projection discriminator. International Confer- ence on Learning Reprexentations, 2018.
Takeru Miyato, Shin-ichi Macda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 2017.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. International Conference on Learning Representations, 2018.
M-E. Nilsback and Zisserman. Automated flower classification over large number of classes. In Proceedings of the Indian Conference on Computer Vision, Graphics and Image Processing, 2008
Avital Oliver, Augustus Odena, Colin Raffel, Ekin Dogus Cubuk, and lan Goodfellow. Realistic evaluation of semi-supervised learning algorithms. In Advances in Neural Information Processing Systems, 2018.
S. J. Pan and Q Yang. A survey transfer learning. IEEE Transactions Knowledge and Data Engineering, 22(10), Oct 2010. ISSN 1041-4347. doi: 10. 19.1099/EEE.2000.1999
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In IEEE Conference on Computer Vision and Pattern Recognition, 2012.
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text- driven manipulation of stylegan imagery. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision (ICCV), 2021.
Ariadna Quattoni and Antonio Torralba. Recognizing indoor scenes. In IEEE Conference Com- puter Vision and Pattern Recognition, 2009.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint 20Xx:22044.06155, 2022.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3), 2015.
Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transfor- mations and perturbations for deep semi ceni-supervised learning. In Advances in neural information processing systems, 2016.
12

---

Under review as a conference paper at ICLR 2023
Swami Sankaramarayanan, Yogesh Balaji, Carlos D Castillo, and Rama Chellappa. Generate to adapt: Aligning domains using generative adversarial networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2018.
Yang Shu, Zhi Kou, Zhangjie Cao, Jianmin Wang, and Mingsheng Long. Zoo-tuning: Adaptive transfer from a 200 of models. In International Conference on Machine Learning, 2021.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In Advances in Neural Information Processing Systems, 2020.
Mingxing Tan and Quoc Le. EfficientNet: Rethinking model scaling for convolutional neural net- works. In International Conference on Machine Learning, 2019.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V Le. Mnasnet: Platform--aware neural architecture search for mobile. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
Reihaneh Torkzadehmahani, Peter Kairouz, and Benedict Paten. Dp-cgan: Differentially private synthetic data and label generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition Workshops, pp. 0-0, 2019.
Jesper E Van Engelen and Holger H Hoos. A survey semi-supervised learning. Machine Learn- ing, 2020.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. International Conference on Learning Repre- sentations, 2021a.
Dilin Wang, Chengyue Gong. Meng Li, Qiang Liu, and Vikas Chandra. Alphanet: Improved training of supernets with alpha- diverriieeeeecc. In International Conference on Machine Learning, 2021b.
Mei Wang and Weihong Deng. Deep visual domain adaptation: A survey. Neurocompating, 2018.
Yaxing Wang, Chenshen Wu, Luis Herranz, Joost van de Weijer, Abel Gonzalez-Garcia, and Bog- dan Raducanu. Transferring gans: generating images from limited data. In Proceedings of the European Conference on Computer Vision, 2018
P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. Caltech-UCSD Birds 200. Technical report, California Institute Technology, 2010.
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. 2019.
https://githut: http:/hwwb.coofaaeeeeeeeeeee http:////ccccooreeeeeeeeeetttttt
Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. In Advances in Neural Information Processing Systems, 2020.
Shin'y ya Yamaguchi, Sckitoshi Kanai, and Takcharu Eda. Effective data augmentation with multi- domain learning gans. In Proceedings of the AAAI Conference Artificial Intelligence, 2020.
Kaiyu Yang, Jacqueline H. Yau, Li Fei-Fei, Jia Deng, and Olga Russakovsky. study of face obfuscation in ImageNet. In International Conference Machine Learning, 2022.
Linjie Yang, Ping Luo, Chen Change Loy, and Xiaoou Tang. A large-scale car dataset for fine- grained categorization and verification. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2015.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Advances in Neural Information Processing Systems, 2014.
Kaichao You. Zhi Kou, Mingsheng Long, and Jianmin Wang. Co-tuning for transfer learning. Ad- vances in Neural Information Processing Systems, 2020.
13

---

Under review as a conference paper at ICLR 2023
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference 2016, 2016.
Han Zhang. Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In International conference on machine learning, 2019.
Han Zhang, Zizhao Zhang, Augustus Odena, and Honglak Lee. Consistency regularization for generative adversarial networks. In International Conference on Learning Representations, 2020.
Miaoyun Zhao, Yulai Cong, and Lawrence Carin. On leveraging pretrained gans for generation with limited data. In International Conference on Machine Learning, pp. 11340-11351. PMLR, 2020a.
Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for data-efficient gan training. In Advances in neural information processing systems, 2020b.
Yezi Zhu, Mare Aoun, Marcel Krijn, and Joaquin Vanschoren. Data augmentation using conditional generative adversarial networks for leaf counting arabidopsis plants. In British Machine Vision Conference, 2018
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. In Interna- tional Conference on Learning Representation, 2017.
14

---

Under review as a conference paper at ICLR 2023
APPENDIX
The following manuscript provides the supplementary materials of the main paper: Transfer Learn- ing with Pre-trained Conditional Generative Models. We describe ( additional related works of domain adaptation and fine-tuning, (B) details of experimental settings used in the main paper, (C) additional experiments including comparison of our method and fine-tuning, and detailed analysis of PP and P-SSL, and (D) qualitative evaluations pscudo samples by PCS.
A EXTENDED RELATED WORK
A.1 DOMAIN ADAPTATION
Domain adaptation leverages source knowledge to the target task by minimizing domain gaps be- tween source and target domains through joint-training (Ganin et al. (016). It is generally assumed that the source and target task label spaces overlap (Pan & Yang. 2010; Wang Deng 2018 and labeled source datasets are available when training target models. Several studies have attempted to solve the transfer learning problems called source-free adaptation @Chidlovskii et al.. 2016 Liang et al. 20200 Kundu et al. 2020; Wang et al. (0211a), where the model must adapt to the target domain without target labels and the source dataset. However, they still require architecture consistency and overlaps between the source and target tasks, i.e., it is not applicable to problem setting.
A.2 FINETUNING
Our method is categorized as an inductive transfer learning approach (Pan & Yang 2010), where thelabeled target datasets are available and the source and target task label spaces do not overlap. In deep learning, fine-tuning (Yosinski et al., 2014] Agrawal et al. 2014; Girshick et al.) 2014), which leverages source pre-trained weights as initial parameters of the target modesl, one of the most common approaches of inductive transfer learning because of its simplicity. Previous studies have attempted improve fine-tuning by adding a penalty of the gaps between source and target models such as adding L penalty term (Li et al. 2018) penalty using channel-wise importance of feature maps (Li et al., 2019). You et al. (2020) have introduced category relationships between source and target tasks into target-task training and penalized the target models to predict pseudo source labels that are the outputs of the source models by applying target data. [Shu et (202 1) have presented an approach leveraging multiple source models pre-trained different datasets and tasks by mixing the outputs via adaptive aggregation modules. Although these methods outperform the naive fine-tuning baselines, they require architecture consistency between source and target tasks. In contrast to the fine-tuning methods, our method can be used without architecture consistency and source dataset access since it transfers source knowledge via pseudo samples drawn from source pre-trained generative models.
A.3 SHMI-SUPERVISED LEARNING
SSL is a paradigm that trains a supervised model with labeled and unlabeled samples by minimizing supervised and unsupervised loss simultaneously. Historically, various SSL algorithms have been used or proposed for deep learning such as entropy minimization (Grandvalet & Bengio 2005). pseudo-label (Lee et al. 2013), virtual adversarial training (Miyato et al. (017)). and consistency regularization (Bachman et al., 2014; Sajjadi et al. 2016 Laine & Aila 2016). UDA (Xie et al.) 2020) and FixMatch (Sohn et al.J 2020). which combine ideas of pseudo-label and consistency regularization. have achieved remarkable performance. An assumption with these SSL algorithms that the unlabeled data are sampled from the same distribution as the labeled data. there is a large gap between the labeled and unlabeled data distribution, the performance of SSL algorithms degrades (Oliver et al. 2018). However, Xie et al. (2020) have revealed that unlabeled samples in another dataset different from a target dataset can improve the performance of SSL algorithms by carefully selecting target-related samples from source datasets. This indicates that SSL algorithms achieve high performances long as the unlabeled samples related to target datasets, when they belong to different datasets. On the basis of this implication, our P-SSL exploits pseudo samples drawn from source generative models as unlabeled data for SSL.. We tested SSL algorithms
15

---

Under review as a conference paper at ICLR 2023
for P-SSL and compared the resulting P-SSL models with SSL models using the filtered real source dataset constructed by the protocol of Xie et al. (2020) in Appendix C.3.I and Sec. 4.3
B
DETAILS OF EXPERIMENTS
B. DATASET DETAILS
ImageNet Ruussakovsyy et al.
2015): We downloaded ImageNet from the official site https:
//www. image net org/
ImageNet is released under license that allows it to be used for
non-commercial research/eduucational purposes (see StEpss//imae-nettorrg. www.a//eoontttoo///oooo/llloo
php).
Caltech-256 (Griffin et al. 2007): We downloaded Caltech-256 from the official site https://
dataa ca www.cateeeeo wew.he..ecc... www.c///eeor/////20 20087 Caltech-256 is released under CC-BY license.
CUB-200-2011 (Welinder
2010 We downloaded CUB-200-2011 from the official site http://wwy www.vvisio..ommmm www.on.callleeooo caw.ech...c.................cc.c edu edu/datasets cub_ 200 cruu__20_2011171100 CUB-200-201 is
released under license that allows it be used for non-commereial purposes (see https://
Nuthors..ibrary.o caw.hcc.aatthhc..a wtt.eechhh..a2225222111
DTD (Cimpoi et 2014): We downloaded DTD from the official site httphhttt//wwwt robots.
DTD is released under license that allows it to be used for non-commercial research purposes (see Ittpps:/ www.ww..bbbs..oo ac. wgggg//at//
ac www.ac../.vvva ww/.vuk//v/////o/ http://gg/aaaaatee/e/o/ data/dtd/
dtd/).
FGVC-Aircraft MMai et al. 2013): downloaded FGVC-AArraff from the official sitchttps:
7/www. www.oooooo.oooooooo ox. uk/ /vgg/ dataa /tta///aaaaaa/eeearrrrat//////// FGVC-Aireraft is released
under license that allows it to be used for non cow.mooommiiilleeecrerrror purposes (see http:::://
robots ox ac. http
Indoor67 Quattoni & Torralba 2009): We downloaded Indoor67 from the official sitcht
7/web. mic...ii.. www.ttrrraaaaaaa. www hww.//inoor...o htmll Indoor67 is released under license that
allows it to be used for non-commercial research purposes (see httpp:://wet meteb.mit.eeeee
Eorralba secralbaawww/indoonm html).
OxfordFlowerr (Nilsback & Zisserman 2008): We downloaded OxfordFloweer from the offi OxfordFlower
cial sitehttps: www. robots..coo. ac uk/ wulukk///vuue http://gg/aaa/lllweeeeeaeee/a///
is released under unknown license.
OxfordPets (Parkhi et al. 2012): We downloaded OxfordPets from the official sitchttps:// www www.ts..ooooo.oo. ox. ac. uk/ "vgg/ wggg/data/pettt OxfordPets is released under Creative Com-
mons SourbbutionSShar-AAle 4.0 International License.
StanfordCars (Krause et al. 2013): We downloaded StanfordCars from the official sitcht //ai.stanfoze edu/ krause/ catiss//aas//aar. dataaatddtttttttttaee hhtll StanfordCars released
under license that allows it to be used for non-commercial research purposes (see https::/aii.
1ranforddedu/-1111 / cars/ cttte//aaaarraaaatttttttt.t .ht.tasset.ttl).........
StanfordDogs (Khosla et al. 2011): We downloaded StanfordDogs from the official siteht //vision www.fooffrrrroooo edu/ adi y waeddiaa@@immmaeeeeeeeee http///ianneeetetg//oooo StanfordDogs released under
license that allows it to be used for non-commercial research/educational purposes (see https:
//image-net.ozz wttp:////ooooo...ppppooo
CompCars (Yngg et al. 2015): We downloaded CompCars from the official /ttptttp::///mll
ie. www.kukkeeee taakk/attaeeee http://ataattttemooocrccrara//// CompCars released under license that allows to be used for non-commercial research purposes (see httt:://mlla: ie. cuhk edu hk/ Batasets/comp_cart
PASCAL-VOC (Everingham al. 2015): We downloaded PASCAL-VOC from the official atthhtp::// //t::://htt.....o rowotstt...ooo ox. ac uk wk/pascca//vcCCC PASCAL-VOC is released under li-
cense of flickr (see https: www. flickr. com he httt//hell/etrreee/eeeeee p/terms).
16

---

Under review as a conference paper at ICLR 2023
B.2 KNOWLEDGE DISTILLATION BASELINES
As stated in the main paper, we evaluated our method by comparing it with naive knowledge distil- lation methods Logit Matching (Ba & Caruana 2014) and Soft Target (Hinton et al. 2013). We exploited Logit Matching and Soft Target for transfer learning under the architecture inconsistency since their loss functions use only final logit outputs of models regardless of the intermediate layers. To tranfer knowledge in C.A. to C²A,, we first fine-tune the parameters of C on the target task then train C₁ with knowledge distillation penalties by treating the trained C.A, the teacher model. We optimized the knowledge distillation models by the following objective function:
min Σ Te M..
Cup (Tt, ye, + (e(xx((l((x)))(((0))))
where Àa is hyperparameter, LKD is loss function of knowledge distillation method, ø is the parameter of a teacher model, and lo(-) and (o(-) are the output of the logit function on 9 and ø. In Logit Matching, LKD is simple mean squared loss between lo and l.- Soft Target computes -ulbback-Leibler divergence between the softmax output of lo and the temperature softmax output of la as LKD- We set the temperature parameter T to 4 by searching in (2, 6).
B.3 TRAINING SETTINGS
We selected the training configurations on the basis of the previous works (Li et al., 2020a Xie et al. 2020). In PP, we trained source classifier by Neterov momentum SGD for 1M iterations with a mini-batch size of 128, weight decay of 0.0001, momentum of 0.9, and initial learning rate of 0.1; we decayed the learning rate by 0.1 at 30, 60, 90 epochs. We trained a target classifier C.A. by Neterov momentum SGD for 300 epochs with mini-batch size of 16, weight decay of 0.0001, and momentum 0.9. We set the initial learning rate to 0.05 for the scratch models and 0.005 for the models with PP. We dropped the learning rate by 0.1 150 and 250 epochs. For each target dataset, we split the training set into 9 1, and used the former in the training and the later validating. The input samples were resized into 224 x 224 resolution. For the SSL. algorithms,
set the mini- batch size for Lussup to 112, and fixed À in Eq. (2) to 1.0. We fixed the hyperparameters of UDA as the confidence threshold 3 0.5 and the temperature parameter = 0.4 following Xie et al. (2020). For P-SSL, we generated 50,000 samples by PCS. We ran the target trainings three times with different seeds and selected the best models in terms of the validation accuracy for each epoch. We report the average top-1 test accuracies and standard deviations.
B.4 FILTERING OF REAL DATASET BY RELATION TO TARGET
We provide the details of the protocol of dataset filtering discussed in Sec. 4.5 of the main paper and list the correspondences between the target classes and selected source classes. For filtering datasets, we first calculated the confidence (the maximum probability of a class in the prediction) of the target samples by using the pre-trained source classifiers, then averaged the confidence scores for each target class, and finally selected the source classes with a confidence higher than 0.001 as the unlabeled dataset similar to (Xie et al. 2020). We list the filtered ImageNet classes for each target dataset in Table 24 and
B.5 SOURCE DATASET: COMPCARS
CompCars YYngg al. 2015) is fine-grained vehicle image dataset for classifying the vehicle manufacturers or models. It contains 163 manufacturer classes, 1,716 model classes, and 136,726 images of entire vehicles collected from the web. As the source dataset for PP and P-SSL, used 163 manufacturer classes for training classifiers and conditional generative models since the manufacturer classes do not overlap with the classes of the target dataset, i.e., StanfordCars.
17

---

Under review as a conference paper at ICLR 2023
Table 10: Comparison of our methods with fine- tuning methods
Table 11:
Comparison of methods with fine-
tuning discriminator
Top-1 1
(%)
Top-1 1 Acc. (%%
Buseline
P-SSL
Scratch (WRN-50-2)
76. 76.92 in
Pseudo Pre-training (Ours)
90,9520221
91.76
Finetuning Discriminator (BigGAN)
16. 16.09=9000
Ours (WRN-50-2)
91.76-b041
Fine-tuning
90,564607
91,4]
L2-SP Liet al.
2018
91.20=000
91,43+000
DELTA Li ct a
2019
91,52#0026
91.824029
BSS (Chen et al
2019
91.25 91.25
91.45+
Co-Tumin You et al. 2020
91.08 01.08
91.16
Table 12: Performance comparison of multiplc target architectures on StanfordCars (Top-1 Acc. (%)
ResNet-50
WRN-50-2
MNASNetL.0 0
MobileNerV3-LL
EfficiemiNet-B0
Eflicien(Net-B5
Scratch
71.86 41.86
76.21
79.22-49.6
80,98 29:37
80.80=99.
81.73
Logit Matching
84.36
86.28
85.08 20000
85.11 10.30
86.37 -..0
88.42-4000
Soft Target
79.95
82.34*1115
83.55
84, 84.64
85. 16.
85,30=
Ours w/o PP
80.14
80.01 20:14
82,22-9.16
83.26 21:21
83.22
85,8245447
Ours w/o P-SSL
90.25 19
90.95 #0.21
87. 87.14 14.005
87. $7.82 =
88.27
89.50
Ours
90.69
91.762:041
87.39 29.19
88.40 2..67
39,28-9.41
90,04:9.34
Fine-tuning (FT)
40.56
88.24
89. 19.18 20.14
87.57 29:28
90.06 2.0225
91.64-19..3
FT P-SSLL
91.41 10.15
91.95-9000
89,46 1..15
88.19 9.11
90.13 9.114
91.71 19909
R-SSL with ImageNet FT R-SSL.
77 77.48=t0.4
77,93*145
81. 74.
82, -0.75
82, 1.05
83,86 83,86**1.54
90.70-..17
=1.87
88.63 20.20
87.26 1000
89.91 29.19
91.10
C ADDITIONAL. EXPERIMENTS
C.I COMPARISON OF OUR METHOО WITH FINE-TUNING METHOMS
For assessing the practicality of our method, we additionally compare our method with the fine- tuning methods that require architecture consistency, i.e., Fine-tuning: naively training target clas- sifiers by using the source praitrdined weights the initial weights. L2-SP (Li et al. 2018): fine- tuning with the T,2 penalty term between the current training weights and the pre-trained source weights. DELTA (Li et al. 2019): fine-tuning with penalty minimizing the gaps of channel-wise outputs of feature maps between source and target models. BSS (Chen et al. 2019): fine-tuning with the penalty term enlarging eigenvalues of training features to avoid negative transfer. Co- Tuning You et al. 2020): fine-tuning on source and target task simultaneously by translating the target labels the source labels. We implemented these methods on the basis of the open source repositories provided by the authors. All of hyperparameters used in L2-SP. DELTA, BSS, and Co-Tuning are those in the respective papers: /3 for L.-SPP and DELTA was 0.01, 1j for BSS was 0.001, and À for Co-Tuning was 2.3. We also tested the models by combining our methods and the fine-tuning methods. Table 12 and 13 list the extended results of the experiments on multiple archi- tectures and target datasets in Secs. 4.3 and 4.4, respectively. Table 10 summarizes the results using the fine-tuning variants. The +P-SSL column indicates the results of the combination models of a fine-tuning method and P-SSL. We confirm that method (pseudo pre-training P-SSL) achieved competitive or superior results to the naïve fine-tuning. This means that our method can improve target models as well fine-tuning without architectures consistency. We also observed that P-SSL can outperform fine-tuning baselines by being combined with them. These results indicate that our P-SSL can be applied even if the source and target architectures are consistent.
C.I.1 COMPARSION TO FINE-TUNING DISCRIMINATOR
We compare PP with a transfer learning method applying encoders of generative models as the pre-trained encoder for target tasks. Here, we assume the discriminator of ImageNet pre-trained BigGAN as the pre-trained encoder, and fine-tune it on the target classification task of StanfordCars. Table II shows the results. The BigGAN discriminator easily overfitted the target dataset and catas- trophically degraded the test performance. It seems difficult to achieve high accuracy by naïvely applying the discriminator as simple pre-trained encoder. This suggests that the representation of
18

---

Under review as a conference paper at ICLR 2023
Table 13: Performance comparison of classifiers multiple target datasets (Top-1 Acc. (%)
Caltech-256-60 256-60
CUB-200-200
DTD
FGVC-Airvaft
Indoor67
OxfordiPoweer
OxfordPets
StanfordDogs
Sernic
48 1.00221.0
526111.3
45. 11-11
74, 14.44to
50,775100
67.91 201
63.03=12
57.16-111
Logit Matching
58. 58.28
62 62.525213
49, 20.29
78.91 78.91
57,61
75.23#121
70.56-33
64,46
Solt Target
54,84-138
60,532115
48, 3921.0
77.08
$4.08 $4,08=122
60,00-905
65.62=09
63 63164
Ours
51.62
56.61411
45,50 1917
77.13 1046
51,23
68.11
68.70
61.26 1:3
Ours w/o P-SSL
70.88 1122
71.78
61.28 tus
86. 26.08-4a4
66.79-922
94.02
86.31 1800
73.30
Ours
71.38 1k32
1939
57.48-124
87.98 18o1
67.72 0m
00.31-917
89,97 8.41
75.25
Fine tuning (FT) FT P.SSI.
75.02
76.69294)
65. 65.39
86.677t13
70,27
95.76
87.73
75.79
75.93:0a4
80.46-914
62 48-00
87.32
71,00=915
9659-
91,48-9.4
78853=
R-SSL with ImageNet FT + R-SSL.
52.22-112
55
41. 3354
75. 75.36 1o00
52.13-1.4
68.55
66.31-11m
59,93=125
76. 16
80 80.38
61. (2
87.08
61.52
96
96.20
90.33
78.275249
Table 14: Top-1 1 accuracy in various target dataset sizes
Samping Rate
25%
50%
75%
Scratch
6.14
40.69
67. 67.46
Logit Matching
10.39 1.43
63.14
79.64 1.88
Soft Target
1.50
#7.99
72.08
Ours w/o PP
11.73=0.88
45.27=0092
69.21
1 +0.8
Ours w/o P-SSI.
59.48=1.23
82.46-0036
88.31 88.31=0.10 +0.10
Ours
61.90=0.60
82.65+0..3
89.33+00119
the generative model is quite different from one of the classification tasks and that our method is well suited to absorb such difference between tasks for knowledge transfer.
C.1.2 COMPARSION OF RN18 AND CUSTOM ARCHITECTURE
We provide further comparison of RN-18 architecture and the custom architecture found by the architecture search in Sec. 4.2 Here, we tested RNI8 with our methods (PP, P-SSL, and PP+P-SSL) on the same setting as Sec. 4.2 to confirm the superiority of the custom architecture. We summarize the results in Table [6] This shows that the custom architecture is superior to the original RN18 (4, 4, 4, 4) in all settings, and also indicates that this is fair comparison among methods.
C.2 TARGET DATASET SIZE
We evaluate the performance of our method on a limited target data setting. We randomly sampled the subsets of the training dataset (StanfordCars) for each sampling rate of 25, 50, and 75%, and used them to train target models with our method. Note that the pseudo datasets were created by using the reduced datasets. Table 14 lists the results. We observed that our method can outperform the baselines in all sampling rate settings.
C.3 SOURCE GENERATIVE MODELS
Table 17 lists the results with the architectures of multiple generative models. We tested our method by using pseudo samples generated from the generative models for multiple resolutions including SNGAN MMyato et al. 2018). SAGAN (Zhang et al.] 2019), and ADM-G (Dhariwal & Nichol, We implemented these generative models the basis of open source repositories includ- ing pytorch-pretraineee wewaaainnn-iooo CCAN" Be Settorcc--tudiiooe
2021.
by Minguk et al. (2021), and qui muided-aiffusionf by Dhariwal & Nichol 2222 ; we used the pre-trained weights distributed by the repositories. We measured the top-1 test accuracy on the target task (StanfordCars) and Fréchet Inception Distance (FID, Heusel et al. (2017)). In Table 17 the generative models with better FID scores tended to achieve a higher top-1 accuracy score with PP and P-SSL. Regarding
19

---

Under review as a conference paper at ICLR 2023
Table 15: Comparison of RN18 18 and Custom RN
Table 16: Comparison of RNI8 and Custom RN
Architectures
Top-1 Acc. (%)
Top-1 Acc. (3)
RN18 (4, 4,444
Custom RNI8 (2. 10, 2)
RN18 (4, 4, 4, 4)
73.27-1043
Scratch
73.278843
77.01 $9.37
CRN
2,
76.10
Fine-tuning
87.75:40s
CRN (2,2, 10,2)
75.65=1660
P-SSL
PP
77.56++02
86. 86.091944
80.86 #9.16
88,75
CRN (2,10,
77.01 0.55
PP+P-SSL
87.61 27.61
89.13
CRN 2,
76.76 1.58
CRN (8, 2,
76.78=1.72
Table 17: Comparison multiple source generative models (StanfordCars)
Top-1 Acc. (%)
Source Generative Model
PP
P-SSL.
FID(Dst=1;
128 X 128 resolution
SNGAN Miyyao et al.
2018
87.07=0.26
72.75=260
108.31
SAGAN Zhang et al.
2019
89.10#0.19
77.66 77.66+215
58.67
BigGAN Brock et al.
2019
89.97
77. 17.54
27.17
256 x 256 resolution
BigGAN (Brock et al.
2019)
90.25+0.19
80.01 #0.14
19.92
ADM-G Dhariwal & Nichol
2021
90.23
79.44
18.72
512 512 resolution
BigGAN (Boock et al.
2019
90.1440.24
38.38
17.87
the resolution, the models of 256 x266, the generated samples of which are the nearest to the input size of Cr (224x224), were the best. From these results, we recommend using generative mod- els synthesizing high-fidelity samples a resolution close the target models when applying our method.
C.4 ANALYSIS PSEUDO PRE-TRAINING
We analyze the characteristics of PP by varying the synthesized samples.
C.4.1 SYNTHESIZING STRATEGY
We compare the four strategies using the source generative models for PP. We tested Uniform: syn- thesizing samples for source classes (default), Filtered: synthesizing samples for target-related source classes identified by the same protocol as in Sec. 4.5) PCS: synthesizing samples by PCS, Offline: synthesizing fixed samples in advance training and training C.As with them instead of sampling from G.. In PCS, optimized the pre-training models with pscudo source soft labels generated in the process PCS. Table 18 shows the target task performances. We found that the Uniform model achieved the best performance. We infer that, in PP, pre-training with various classes and samples is more important than that with only target-related classes or fixed samples.
C.5 ANALYSIS OF PSEUDO SEMI-SUPERVISED LEARNING
C.5.1 SEMI-SUPERVISED LEARNING ALGORITHM
We compare SSL algorithms in P-SSL. We used six SSL algorithms: EntMin (Grandvalet & Ben- gio 2005), Pseudo Label (Lee et al. 2013), Soft Pseudo Label. Consistency Regularization, Fix- Match (Sohn al. 2020), and UDA (Xie et al. 2020). Soft Pseudo Label is variant of Pseudo Label, which uses the sharpen soft pseudo labels instead of the one-hot (hard) pseudo labels. Consis- tency Regularization computes the unsupervised loss of UDA without the confidence thresholding.
Table 19 shows the results on StanfordCars, where Pseudo Supervised a model using a pairs of (Jr t. yt in pseudo conditional sampling for supervised training. UDA achieved the best result.
20

---

Under review as a conference paper at ICLR 2023
Table 18: Comparison of pseudo pre-training variants
Top-1 Acc. (%)
Uniform
90.95-10.1
Filtered
84.89 1:0.3
PCS
85.90=
Offline (1.3M)
89.08 -9..0
80
Table 19: Comparison of algorithms for P-SSL (Ac: WRN-50-2)
Top-1 Accuracy (%)
78
Top-1 1 (9))
Scratch
76.21=1.0
Pseudo Supervised
53.03=1.79
74
EntMin (Grundvaler & Bengio 2005
72.56
Pseudo Lahcl (Lee al. 2013
74.49*226
BigGAN (ImageNet Pre-traine (re-trained)
Label
#4
72
Consistency Regularization
79.17
FixMatch (Sohn al. 2020
74.31
10
50
100
500
UDA Xie et al. 2020
80.01=104
Pseudo Dataset Size (K images)
Figure 4: Top-1 accuracy of P-SSL when scaling
pscudo dataset size
More importantly, the methods using hard labels (Pseudo Supervised, Pseudo Label, and FixMatch) failed to outperform the scratch models, whereas the soft label based methods improved the perfor- mance. This indicates that translating the label of pseudo samples the interpolation of the target labels can improve the performance as mentioned Sec. 3.2.3
C.5.2 SAMPLE SIZE
We evaluate the effect of the sizes of pseudo datasets for P-SSL the target test accuracy. We varied the pseudo dataset sizes in {10K, 50K, 100K, 500K and tested the target performance of P-SSL on the StanfordCars dataset, as shown in Figure (right). We found that the middle range of the dataset size (50K and 100K images) achieved better results. This suggests that P-SSL. does not require generating extremely large pseudo datasets for boosting the target models.
C.5.3 OUTPUT LABEL. FUNCTION
We discuss the performance comparison of output label functions PCS. The output label function crucial for synthesizing the target-related samples from source generative models since it directly determines the attributes on the pseudo samples. We tested six labeling strategies, i.e., Random Label: attaching uniformly sampled source labels, Softmax: using softmax outputs of C.A. (de- fault), Temperature Softmax: applying temperature scaling to output logits of C.A. and using the softmax output, Argmax: using one-hot labels generated by selecting the class with the maximum probability in the softmax output of C.A., Sparsemax (Martins & Astudillo 2016): computing the Fuclidean projections of the logit of C. representing sparse distributions in the source label space, and Classwise Mean: computing the mean of softmax outputs of C.A. for each target class and us- ing it as representative pseudo source labels the target class to generate pseudo samples. Table 20 shows the comparison of the labeling strategies. Among the strategies, Softmax is the best choice for PCS in terms of the target performance (top-1 accuracy) and the relatedness toward target datasets (FID). This means that the pseudo source label youc by Softmax succeeds in representing the char- acteristics of a target sample 2, and its form of the soft label important to extract target-related information via source generative model G,-
21

---

Under review as a conference paper at ICLR 2023
Table 20: Comparison of output label functions in PCS
Table 21: Effect of in UDA
Top-1 Acc. (%)
FID
Top-1 Acc. (%)
Random Label
75. 75,55:20.35
134.37
0.2
79.59
Softmax
80.01 1::11
19.92
0.4
80.01 10.14
Temperature Softmax (r = 0.4)
19
20.68
0.6
80.43=0025
Argmax
78.89 28.8
22.35
0.8
81.81 -0.58
Sparsemax
76.08-11.14
24.28
1.0
79.16 L0.25
Classwise Mean
78.5642.83
22.14
C.5.4 SEARCHING TEMPERATURE HYPERPARAMETER FOR UDA
Here, we investigate the effect of hyperparameter r for UDA, which is mainly used in the main paper. By the definition of temperature softmax function (exp(y/77) /)/ Σj exp(y;/T)), the temperature parameter 7 controls the sharpmess of the predicted class-conditional distribution; lower temperature outputs sharper distribution. In this regard, in our P-SSL, Cx(xeet: =; 0), which determines how to represent the pseudo sample Vse-t with the soft target class labels, be changed by т. Thus, the choice of 7 can be an important factor for training in P-SSL. To evaluate the effect, we tested P-SSL by varying т on StanfordCars as shown in Table 21 The results show that the moderately higher achieved better target performances. This suggests that representing T by softer target class labels can bring a positive effect to target classifiers, which consistent with the discussions in Sec. 3.2. and 3.2.3
C.5.5 P-SSL VS. SSL USING TARGET LABELED DATA
We provide further analysis to confirm the performance of the unla- beled pseudo samples E=-+ by comparing with a SSL method using real target samples in the unsupervised loss function. We call this method as T-SSI,, and T-SSL can be another simple baseline for P-SSL because it can simply discard pseudo sample synthesis in P-SSL. The results are shown in Table 22 We can see that apply- ing supervised loss and unsupervised loss to the same sample has negative effect: the use of pseudo-labels to target labeled data might promote overfitting. This result indicates that E in P-SSL certainly performs as useful unlabeled
Table 22: P-SSI vs. T-SSL
Top-1 1 Acc. (%)
PP PP4T-SSL PP+P-SSL
90,954021
87.65 +035
91.76
data in UDA than directly using target data.
D QUALITATIVE EVALUATION OF PSEUDO SAMPLES
We discuss qualitative studies of the pseudo samples generated by PCS. To confirm the correspon- dences between the target and pseudo samples, used StanfordCars the target dataset and gener- ated samples from BigGAN with the same setting as in Sec. 4 Figure]3Jshows the visualizations of the source dataset (ImageNet), target dataset (StanfordCars), and pseudo samples generated by PCS. The samples were randomly selected from each dataset. We can see that PCS succeeded in gener- ating target-related samples from the target samples. assess the validity of using pseudo source soft labels in PCS, we analyzed the pseudo samples corresponding to each target label. Figure shows the pseudo samples generated by the target samples of Hummer and Aston Martin Convert tt i bll classes in StanfordCars. We confirm that the pseudo samples by PCS can capture the features of target classes. This also can confirm the ranking of the confidence scores for source classes listed in Table 23 the pscudo source soft labels seem represent the target samples by the interpolation of source classes.
22

---

Under review as a conference paper at ICLR 2023
ImageNet (target-related)
StanfordCars
PCS
Figure 5: Samples of source, target, and pseudo datasets (random picking).
StanfordCars
PCS
Target Label: Hummer
ve Convertible
Target Label: Aston Ma Martin
Figure 6: Pseudo samples generated by PCS (random picking)
23

---

Under review as a conference paper at ICLR 2023
Table 23: Ranking of averaged confidence scores of ImageNet classes corresponding to target classes
Target Class Label
Rank
All target classes
Hummer
Aston Martin V8 Convertible
1st
sports car (23.5%)
jeep (41.0%)
convertible (62.8%) sports car (27. 27.4%)
2nd
beach wagon (15,8%)
limousine (11.1%)
3rd
minivan (12.0%)
snowplow (6.7%)
racer (1 (1.3%)
4th
convertible (10.0%)
moving van (6.7%)
pickup truck (1.3%)
5th
pickup truck (7.0%)
tow truck (6.3%)
car wheel (1 (1.3%)
Table 24: Corresponding ImageNet classes to target datasets (1)
Target Dataset
ImageNNtt Classes
goldiish, electrie ostric, great arey owl, tree tailed Joggetheod. leatherhack turtle, common iguant. triormitpp, trfishino, scopion. spider, contipede,
drake, poose, jellytihh, nematode, cunch, snail rock crab, liddler crab. American lobser, isopad, black sturk, American egret. king ponguin, killer while, whippet
wallaht.
leopud, jaguar, chectah, brown bear, American black hese, fly, grasihoppes cockreach, mantis, monarch, starish, sea urchin, poccupine. sorrel, zebra, ihes, haricheest, Arahina camel. Ilama, shunk, gwvilla, chimpanzes, Indian elaphant, African clephunt, acoustic gultar, airliner, nirships altar. analog clock, assault rifle. bockpack. balloon, ballpoint, Band Asd. barbell. battow, buthtub, beacon, beaker.
bearskin, beer glass, bell cose, bicy scycle-built--e foe---- binder, batoculars, balo tic. bottlecap, bow, brass. opener, corpenter's is kit, wheel, emsoette cassette player coffee mug, coffeepot.
bresstplate, buckle, candle, cannon,
player, chain, chest, chime. cradie, crash helmet, croquet ball, deskinp computer, dial telephonc. doaemat drilling
combination lock, cariscrew, cowboy hat.
drum.
dumbholl. electric fan. electric guitar, envelope. face powder, engine. fire screen flagpole folding football helmet, tountain, Freach born, frying pan, gasmusk. pas pump, gobles, gelf bull, gong, grand pann. guillotine, bair slide, hamper. hand blower. hand held computer. handkerchiel. harmonica, harp, harvester, book, hors cart. hoarglass, iPod, jarvey jigan puzzle, joystick, knee pod. ladis lanpdiide liptnp. laws mower. letter opener, lighter, loulspakerr loug, lumbermill, magnetic compuss, mailbag mailbox, muraen, maiims. maze, microphone, microware, missile, modem, moged mortar, mesque, momatii bike, mountain sent, mouse, mousetrap. murzle, sail, nock beace, nipple, notchook ohelisk, ocarinn, oil filter, oscillascope oxygon mark, packer, poddle, pulace. paracbute. park bench. pay phone. pedestal. pencil sharpener. perfume. Petri dish, photocopier. pitcher, plane. pole, pool table. pot, printer, projectlic, penjactor. tolescope
Calocch2266600
laptopp
pick. pier, pill bottle. ping-pong puck. punching bug, quill, racket, sule. running shoe, safety pin, sandal, scabburd, scale, school bus, schooner, screet. screw, screwdriver. shield. ski, slot, snowmbbile. stap dispenser, soccer ball. sock sohr dish, sombrero. space bur. sputuls, spredboul. spollight, sttassecope stopwatch. stretcher. studio couch, sunglass, sussereen, suspension bridge, soing, swasch, syrings, table lamp tape player teoper, reddy conni hall. thimble. threshes, kuster tobseno shop, solldt seat. torch, tow track, tryy tricycle, trpodd sub, opewriee keybourd, umbrella, uncyycle, upright. vase. waflle irun. wall clock, wallet, warplase, unsher, water jug, whiskey jug, whisll. Windsor tic, wine bottle, wool, worm fesoe, yaul, web site, comic book, street sign, trallic light, book jacket, plate, cheeseburger, hotdog, spughetti squach. fig. carbonara red wine, cup, eggnog, cliff, gevser, lakeside. promontry, seashure, valley. volcuno. daisy, hip, cantistar ben-of-llo-wood
refrigerator,
nibher
finch, junen, indigo buncing, robea, bulbul. jay, magpie. chickadee, water ouzel. kite, hald eagle, valture, grest grey oul black gronse paammigan, nutfed grouse, prairic chicken, quail, pastritge. macoww lorikeet, concal. ber eoter. horsbill, humminghinl. jucamar. ioncan. drike, red- breasted mergasser, goese. black swan, white stock, black stork, spoenbill, little blue heron, American egret. biteen emase limpkin. European gallinule. American coer, bustee,, middy tarattoee ni- nel-backed andpiper, redshank, dowischer, ry mystorotcherr pelican. king penguin, albotros,, worm fence
ben, brumbiigg goldlinch,
CUB-33002001
electric ray, stingray, leatherhock tatie, fhunder snake, hoganse stoke. hornel viper, sidewinder, trilobite harvestman, barn spider black widno tick. jellyfish
brain cural, flatworm, nematide, conch. slug, chambered nauiilus fiddler crah, Isopod, kamonder, spron, backpuck, hikey, hilloon, Rond hunnister wwel. glass hinder, bonner, bottlecup. bow bressiplate, broom, buckle, candle, cartiignn chain, chainlink fonce, mail, clift dwelling, Slook, coil, confectionay, caste, cuirass, dishing dome, dooemas invelope, face powderr feather has, fire screen. founainn for cost, golf ball, gown, huir slide, hamper. handkerchief. honcycomb, book, hocpskirt, jean, jeney, jigsw puzole. knot, limeshade, lighter, Inudqpeskrr, mailhsg monholc cover, mask, matchatick, maze, megalith, microphone, mittnn mosquio net, zail, necilace, overskin. packet, palleck. paintbrush pojame. paper towel, pencil box, Peni dish, picket fence, pillow, pawwheel, plantc beg, pancho, pot, prayer prison,
leaf beetle,
bectle. bec,
walking enekranch,
DTD
shopping basket, quill, quilt, mddiatrr, radio, nubber sleeping bug, soluz dish, space heater, spider stole. thenter
rule, safety pin, saltchaker,
senrw, shell, shoj..
shower cap, shower
wall.
swah, sweatshirr swimming trunks, waflle wall clock, wallet, wandtube, window shade, Windsar ne, wondon spoon, weel, web alte, croswwerd puzzle, book jacket, tnfle, creams ice tolly. French laaf, pretiel, head cabbago, brosslll, cullifower. strawberry. lence fig, jackfriil, enstard apple. pomegranate hay, chocolate stuce, dough, mear kuff potpic. cap, eggnog. bubbic, clfff, cona reef, gayser, sandlar, valley, volcano. comn, buckeye, coral fungus, hen-of- the woods, cor, inilet tissue
gpring,
thimhle, reef.
try. trench vont, embeella, tass, vaull,
wig, window screen.
REVC-Aircaatt
atrrraft carrier, airlincr, sinship missile, projectile. space shuttle, specdbout, trimaran, worplane, wing
ocodemic gown, altar, hakery, balance heam, hannister, bahhell barber chair, hathershipp banel, bothing buthtuh, beer botil, bookense. bonkshop, builet train, butcher shop, comousel, carton, cash muchine, china cabinet, crutch, church. coil, confectionay, cnaiee entertiiment
computer. dining table. dishwasher. foldingg
file. fire fouztain, poster, golfcart, grand piano, greenhuuee, grucery store, guillotine, home theater. horizonaa bar, jiesew puzzic, cost, laptop, libeaay. limousine, loudspeakes, humbenmill, maramba, maze, medicine chest. microwave, minibas, momsiery, monitor, mesquito net, organ, oxygen mask, paltee, parallel bass, passenger car. patio, photocapicr, pias, ping-pong hull, planetaiumm plate mek, pole, pool table pot, getter's wheel, paayer rig printes prisun, projectir, quilt, radio, reffigenaco. restauram, rocking chair, rotiserrie scurebound, screen, shoe shop. shots shapping basker, shower curtain sldding door, sist, solar disl,, spotlight, stgg steel arch bridge, stove, streetcar, strncheer sudio couch, table limp, tope player, television, thenter cutiiin throne lobacon shop, toilet seat, toyshop. tripod, tub, tumstike upright. window sereen, window shode, wine bottle, comic baok, plate, groem
dome, drom, dumbhell, eleatiic lecameiiee
fiekliff
Indcort?
vauli, vending machine, vestment wardrube, washibusin, washer,
24

---

Under review as a conference paper at ICLR 2023
Table 25: Corresponding ImageNet classes to target datasets (2)
Tepe Doceet
ImageNet Clvoons
goblishh, cock, harvesimus, zedin spide. humninghid, seu incmone, conds, snail, slug, aca slug, clumbered suitllss Islytug leng-homed beetle, leaf bortle, fl. her, ant, greebepper. cnoket, walking stick. mantis boorwing, admiral, ringle monuth, cabbage butterfly, siiphr huttrlly lycamd, ar unhir. hechoree, honne, candic. chainlink firor, procalorres luir slice. lumper finnleenhicff beim, peper towd, per fame, picket fines pinchezl plodic hag pel, quili, shower cap, try. unteella, vase, velvet, wool, cucmbbr: srricooee pepper, cardaon. mushroom, strawberry orane, lemen. fig. piteopple.
OxfredFlweer
cablogge caulifower, zucctini, spaghent squzsh.
acom squash, custrdd appte. pomegratone, corl rapeseed, dany. yellow laty's slipper acom hip tockeye, fingus. contidarr
Maltese cog, Pekinese Shih Tou, Blentein spantel, papilion. semeer Rhodesan ndgeback. bacer, beagle Noodhound, bluenck, Walter hound, English foshound, rndrone Inalan greshound, Ihizan bound. Norwegun elkhornd Weinurater Stufondaliee bulterries America Suffuddhire temier, Iishh terfer. Noew teriie, Yudahire tenier, Lakeland teriee. caina, Australian terior. Dundie Diamoni, Boston bull, Scotch serfer. Tibetan terreer diky tenier
Chibrahro,
soft coated wheeten temer West Highlund white lembrr Lhisa, flat coated retrieves golden retiever. Labrador retriver Chespeake Bay catriver Gemun shon-hained poines Esgleh adiee Brittay spanicl, English sprnge Wicsh spring spanel, cocker spacidl kurneo, sclippeske gresendadl kelpic, Gaman Gammanshephnnn minisaune pinscher boter bull masilll Tikeua mesiili, Fresch bulldey Great Dan: Saitt Benand. Eskino day, Siberian husky, allespiscche. baseni pug, Loouberg, Sewfounllnd Great Pyretees Sanoyed, Pomeraiian chow, ketshard, Brabancos griffor Penbroke, toy poodle, mininare poodle, Mesisun hirkes, white wall, dingo Inbyy liger cat, Percer Simmoe ort Fgypiin coe lyrs. buth towel tucket, cotton plactic bag. quilt. skoppng hag quar heater. ternis ball, window serses
OxfordPess
amphihion, wagon,
moning parking conventbbee grill, jop
SuatondCars
prempgree pickep, paisor
recrotional velikk, sropplow, sponts
traller
Rhodeson ndgoback, Afphan hound. lnadt beggh, Hlodhound blstick, bld Micl-nndo cueebound Walker hoand Faglish fochound, nilhone, burmi, lrihh wullhound Inaisa peyhoudd, whippe. lhizan hound. Norwegion clthoumd stzzhmnn Saliki, Sastiish dochhound, Weinzzater. Staffondsiiee bulleerter. Amenian Stafforisiire temter, Bedlington temer Border temer Kerry bise terterr Insh vernic, Nacfolk temier Noewich temerr Yorkshire torrer, wire baired lox tenter. Lakeland temer Sealy hant temer Airedale, camm, Austalian temer Dandle Dinmore Boston bell, miniaure schasmer giant schtanzer standirt schnameer Sertch tervier Tibetn semanr silly semm soft-cnsted wheate semer. Win Highhnn while semerr Uhass, fun-crused remiever curly costed cetreverr godden ittiever, Labrader reclaves Ceaueee Bay resieves. German sbort haired pointer vizsl.. English setter, Irish setter. Goedon setter, Beimany spaniel, clumber. English springer. Welsh spriner spaniel, cocker spuniel Susses spantel Irih water sponiel, losas, schipprzz, groenzndael. malinnos brinnd kelpce, komondac. Old English sheepdog. Shelland sheppay, zullic Bands colia. Bousis des Fundirs, Ruttealle Gemun sheplicnd Dehemun, ministne pinschrs Tibeas musilf Fresch bulling, Dane: Berard, Eskimo dog mulamte, Sibesian hasks aftrppttchher bucuji, pig. Leatbers. Neofoudllndd Great Pyresees Samoyed, Pomenaiaan chow, keettnnd. Brubancon
Chibushus Jspuneso speniel, Maliese dog Pekineer, Shih- T'u, Hlenhein spanee, papillon.
Sondoollings
Gueater Mountain dog, Benese moumais dog Appeszelles. EatlBBaches buner, bell
Penbrois, Carigan ty poodit, miniatue poodle. dingn
peodic
huriing deg
25


Reviews:
[{"review_id": "w2NWXNbONtt", "reviewer": "Reviewer_kCqh", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper discussed the transfer learning in relatively relaxed conditions. Particularly, the paper assumes (1) no label overlapping space between source and target tasks, (2) no source dataset access when training the target task, and (3) the inconsistent model architectures.\n\nTo tackle the transfer learning problem in above assumptions, the authors propose the pseudo pre-training (PP) and pseudo semi-supervised learning (P-SSL). \n\nDuring PP stage, the target classifier is pre-trained on the pseudo data generation from source generative model but the final classification layer will be replaced later because the output is in source label space. \n\nDuring P-SSL, the target data will be first fed into pre-trained target classifier and output a pseudo soft label in source space. The pseudo soft label will be fed into source generative model to produce a pseudo data. This pseudo data will be used for unsupervised training. For unsupervised training, the paper adopted UDA loss rather than the cross entropy loss based on the original label. \n\nThe experiments compare the proposed method with scratch training and knowledge distillation.", "summary_of_the_review": "In general, the paper raises an important and difficult topic of transfer learning, and proposes a pseudo data based framework to tackle this problem. However, the experiments can not fully verify the proposed method. My initial review is marginally below the acceptance threshold.", "strength_and_weaknesses": "Strength:\n1. The proposed transfer learning in the three assumptions is difficult to learn but the authors make it work.\n2. The author found directly using the original label for the pseudo data in P-SSL didn't work and UDA as the unsupervised loss can achieve better performance. This observation is important and may help other related research works.\n\nWeaknesses:\n1. I'm afraid the main contribution is UDA. Because UDA loss is used, the author can directly apply UDA loss on the original target data $x_t$ rather than $x_{s\\leftarrow t}$. In this way, the pseudo data construction is basically discarded. Although the authors indicate there is less prior work in this area and compare with two simple baselines, I think this is also an important and simple baseline. \n2. I think a more rigorous experiment setup is needed or should be added. For example, in sec 4.6, both the source and target dataset are vehicle or cars related. According the proposed transfer learning with three assumptions, the source data can be vehicle related, and the target dataset could be animal related.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "kEQkgW1AuAG", "reviewer": "Reviewer_FAEQ", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper identifies the key assumptions in transfer learning, addressing an interesting problem that how to solve it when some assumptions does not hold from a systematic perspective. This paper proposes a two-way step, first by pseudo pre-training by using a powerful generative model to generate source data set, then using P-SSL, to fine tune the model combined with unsupervised learning. The extensive empirical exps shows that this approach works \"when source dataset is unaccessible\".", "summary_of_the_review": "This paper tries to solve the problem when pretraining dataset is not accessible, but instead using a powerful generative model as a replacement.", "strength_and_weaknesses": "Strength:\nThis paper is well written, easy to follow and conducts a lot of experiments from scratch, demonstrating the effectiveness of the proposed approach.\n\nWeaknesses:\nHowever, if thinking it carefully, this paper still didn't really solve the problem when condition 2/3 is missing, and condition 1 is trivial.\nfor condition 1, it is trivial by definition because nowadays most transfer learning tasks assume that source/target class labels are different, that's why pre-training and fine-tuning are so popular.\nfor condition 2, the authors claimed that source dataset is usually accessible, so they propose a solution that using synthetic data from a generative model G_s with uniform label distribution label_s will be good, however, the exp set up for source dataset is ImageNet/CompCars(in sec 4.6), but this paper is using BigGAN as G_s, which is already powerful enough to represent the missing source dataset. So the question is, what if BigGAN is not available? this paper is just playing a game that basically replacing an unaccessible source dataset with a powerful generative model, So in this regard, this paper never works with condition 2 missing. \nthen for condition 3, it is obvious that if source dataset (for pretraining) is given, then we can just pretrain it on architecture A_t, and finetune  A_t on target dataset. So in other words, condition 3 is just a byproduct of condition 2, and this paper is just following the same pre-train -> finetune workflow.\nIn addition, what is the semantic meaning of using an extra unsupervised loss based on [Xie et al., 2020], what will the performance be if without such loss? such ablation study is missing.\n\nIt will be more interesting to see that, in this exp set up, if ImageNet is not available as source dataset, can we resort to other less powerful source data set? e.g, using dogs pics to pretrain model and then finetune a task to recognize cars? Not just using a powerful generative model, which could be used to generate a powerful and representative source data set. The problem of how to find a good source data set in this experimental setup is quite trivial, e.g, like privacy concern mentioned in the introduction is interesting but never discussed.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "fT-CNC6pm4L", "reviewer": "Reviewer_JCMy", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Transfer learning presents different types of problems like non-availability of source data (privacy, costs, etc), or the difference in source and target labels or the target task is different. In this light, the paper proposes to use conditional generative models (CGM) in a two stage proposed method which involves pseudo pretraining (PP) on target architecture on synthesized datasets (CGM),  and then using pseudo semi-supervised learning (P-SSL) on the target data and pseudo samples to tackle the task.  \n\nThe results show that the approach conditioned on the CGM does provide considerable and stable improvements, specifically as compared to scratch training or knowledge distillation. \n\nAdditionally, the empirical experiments reveal that source and target datasets do not necessarily need to be close, and still achieve a good performance gain.    ", "summary_of_the_review": "The paper demonstrates, through various experiments, the validity of their claims satisfactorily. For me, the work is quite good and gives a lot of take-aways for further research. ", "strength_and_weaknesses": "**Strength**\n\n1. PP combined with P-SSL gives a strong experimental evidence that one can use standard Generative models and their corresponding classifiers in the context of transfer learning. \n\n2. The proposed method seems stable under different target architectures and datasets, as evidenced by various empirical experiments. \n\n3. The proposed method is demonstrated for classification as well as object detection tasks, which makes it more generic.  \n\n**Weakness**\n\nThe generated samples are then used to learn an encoder representation which is then used to train a P-SSL. Why not use the encoder of generative models to train P-SSL directly, instead of training it on synthetic samples?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "3JFNe9q7to", "reviewer": "Reviewer_MbsB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors utilize the pre-trained conditional generation model as a data augmentation technique to facilitate transfer learning. Extensive experience done prove the effectiveness of the proposed method.", "summary_of_the_review": "The idea is novel. But I do think some expression and explanation could be improved. ", "strength_and_weaknesses": "Strength:\nBased on my knowledge, the authors are the first to apply the pre-trained GAN to transfer learning. It's a novel. The authors carefully design two scenarios with and without source data. In the meantime, the authors conduct extensive experiments on classification and detection. \n\nWriting: \n\nThe introduction and the description of the main idea are clear, however, the symbols are a little misleading. Though the reviewer can guess the general meaning, it will be better to make it clear.\n1. In equation (2) do supervised loss and unsupervised loss share similar weight parameters?  Where is $\\theta$ on the right side of equation (5)? \n2. Does \\hat(C_t) and C_t share similar parameters? How does $\\tau$ select? Does $C_t$ also have a temperature? \n\n\nFigure:\n- Why does C_s^{A_t} has different color in Figure 1(a)(b)? It will be better to describe the color meaning in caption part. \n- Is the complete PP algorithm used in experiments is the combination of Figure 1(a) and Figure 2? My understanding is that we can train $A_s$ in Figure 1(a) and fix it in Figure 2 for transfer learning. If my understanding is correct, it will be better to merge Figure 1/2 and use dotted lines (or anything else) to differentiate different training stages.\n\n\nQuestions:\n1. Method\nThe architecture inconsistency part in Table 1 is interesting, but I have some questions about the second point: \"no source dataset access\".\nDid you fine-tune GAN using the source dataset? I understand the case that the source dataset is available. However, when the source dataset is not available and GAN never sees a similar source dataset before, does the algorithm still work? If not, the description of \"no source dataset access\" is inaccurate, under which case, we can hardly match the space of the source label and the label space (condition) of GAN, not to mention generate augmentations for transfer learning.\n\n2. Experiments\n- How to select the custom architecture in Table 4? Does each method select its own architecture independently using the validation set or are they selected by \"ours\"? What's the meaning of \"from (2,2,2,2) to (2,2,2,10)\". What's the performance with Architecture RN18(4,4,4,4)? \n- It seems that PP contributes the most in Table 6  but harms the performance in Table 9, any comments on this? \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject