One-Shot Generative Domain Adaptation
Authors: Ceyuan Yang
Yujun Shen
Zhiyi Zhang
Yinghao Xu
Jiapeng Zhu
Zhirong Wu
Bolei Zhou
Venue: ICLR 2022 Submitted (2022)
Keywords: generative domain adaptation

Abstract: This work aims at transferring a Generative Adversarial Network (GAN) pre-trained on one image domain to a new domain $\textit{referring to as few as just one target image}$. The main challenge is that, under limited supervision, it is extremely difficult to synthesize photo-realistic and highly diverse images, while acquiring representative characters of the target. Different from existing approaches that adopt the vanilla fine-tuning strategy, we import two lightweight modules to the generator and the discriminator respectively. Concretely, we introduce an $\textit{attribute adaptor}$ into the generator yet freeze its original parameters, through which it can reuse the prior knowledge to the most extent and hence maintain the synthesis quality and diversity. We then equip the well-learned discriminator backbone with an $\textit{attribute classifier}$ to ensure that the generator captures the appropriate characters from the reference. Furthermore, considering the poor diversity of the training data ($\textit{i.e.}$, as few as only one image), we propose to also constrain the diversity of the generative domain in the training process, alleviating the optimization difficulty. Our approach brings appealing results under various settings, $\textit{substantially}$ surpassing state-of-the-art alternatives, especially in terms of synthesis diversity. Noticeably, our method works well even with large domain gaps, and robustly converges $\textit{within a few minutes}$ for each experiment.

Under review as a conference paper at ICLR 2022
ONE-SHOT GENERATIVE DOMAIN ADAPTATION
A Anonymous authors Paper under double-blind review
Large-scale pre-training
One-shot adaptation
Kid
Source
Latent space
Synthesis
Generative domain adaptation
Van Gugh's
Sunglasses
Latent space
Classifier
Adaptor
D
Synthesis
Learnable
Fixed
Figure Diagram of one-shot generative domain adaptation. Left: The overall framework, where GAN model pre-trained the large-scale source data is transferred to the target domain with only one training sample. A lightweight attribute adaptor and attribute classifier are introduced to the frozen generator and discriminator respectively. Right: Realistic and highly diverse synthesis results after adapting the pre-trained model with different targets. Here, our efficient design manages to transfer both semantic attributes and artistic styles within few minutes.
ABSTRACT
This work aims at transferring a Generative Adversarial Network (GAN) pre- trained on one image domain new domain referring to as few as just one targef image. The main challenge is that, under limited supervision, it is extremely difficult synthesize photo-realistic and highly diverse images, while acquiring representative characters of the target. Different from existing approaches that adopt the vanilla fine-tuning strategy, we import two lightweight modules the generator and the discriminator respectively. Concretely, we introduce an attribute adaptor into the generator yet freeze original parameters, through which it can reuse the prior knowledge to the most extent and hence maintain the synthesis quality and diversity. We then equip the well-learned discriminator backbone with attribute classifier to ensure that the generator captures the appropriate characters from the reference. Furthermore, considering the poor diversity of the training data (i.e., few as only image), propose also constrain the diversity of the generative domain the training process, alleviating the optimization difficulty. Our approach brings appealing results under various settings, substantially surpassing state-of-the-ar alternatives, especially in terms of synthesis diversity. Noticeably, our method works well even with large domain gaps, and robustly converges within few minutes for each experiment.
INTRODUCTION
Generative Adversarial Network (GAN) (Goodfellow et al., 2014), consisting of generator and discriminator, has significantly advanced image synthesis yet relies on large number of training samples (Karras al., 2018; 2019; 2020b; Brock et al., 2018). Many attempts have been made to train GANs from scratch with limited data (Zhang & Khoreva, 2019; Zhao et al., 2020d:b; Karras al., 2020a; Yang et al., 2021a), but it still requires hundreds thousands of images get

---

Under review as a conference paper at ICLR 2022
satisfying performance. Sometimes, however, may have as few as only one single image as the reference, like the masterpiece Mona Lisa from Leonardo da Vinci. Under such a case, learning a generative model with both good quality and high diversity becomes extremely challenging.
Domain adaptation is a commonly used technique that can apply an algorithm developed on one data domain to another (Csurka, 2017). Prior works (Wang et al., 201 8; Noguchi & Harada, 2019; Wang et al., 2020; Mo et al., 2020; Zhao et al., 2020a; Li et al., 2020; Robb et al., 2020) have introduced GAN training alleviate the tough requirement on the data scale. Typically, they first train a large-scale model in the source domain with adequate data, and then transfer it to the target domain with only few samples. A common practice to fine-tune both the generator and the discriminator on the target dataset until the generator produces samples conforming to the target domain. To stabilize the fine-tuning process and improve the generation quality and diversity, existing approaches propose to tune partial parameters (Noguchi & Harada, 2019; Mo et al., 2020; Robb et al., 2020) and introduce some regularizers (Li et al., 2020; Ojha al., 2021), but the overall adaptation strategy remains. When there is only one image from the target domain, these methods would fell short of synthesis diversity, producing very similar images.
this technique
Recall that the pre-trained model can already produce highly diverse images from the source domain. makes us wonder what indeed causes the diversity drop in the adaptation process. We argue that directly tuning the model weights will result in the loss of the prior knowledge gained from the large-scale data. On the other hand, however, when adapting the model to the target domain, most variation factors (e.g., gender and pose of human faces) could be reused. These observations help raise a question: is it possible to simply focus on the most representative characters of the reference image while inheriting all the other knowledge from the source model?
In this work, we develop a method, called GenDA, for one-shot Generative Domain Adaptation. In particular, we design lightweight module connecting the latent space and the synthesis network. We call this module an attribute adaptor since it helps adapt the generator with the attributes of the target image. Unlike the conventional fine-tuning strategy, we freeze the parameters of the original generator and merely optimize the attribute adaptor during training. Thereby, we manage to reuse the prior knowledge learned by the source model and hence inherit the synthesis quality and, more importantly, diversity. Meanwhile, we employ the discriminator to compete with the generator via a domain-spectin attribute classification. In this way, the generator is forced to capture the most representative attributes from the reference, or otherwise, the discriminator would spot the discrepancy. However, instead of directly tuning the original discriminator, we freeze its entire backbone and introduce lightweight attribute classifier top of that. Similar to the generator, the discriminator has also learned rich knowledge in its pre-training. Since the synthesized images before and after adaptation share most visual concepts (e.g., face model would still produce faces after domain transfer), the discriminator can be reused as a well-learned feature extractor. Therefore, we simply train the attribute classifier help guide the generator. Furthermore, since there is only one training sample (which means no diversity in the target domain), we propose to also constrain the diversity of the generative domain by truncating the latent distribution during training. Intuitively, learning one-to-one mapping would be easier than learning many-to-one mapping. Such design mitigates the optimization difficulty and further improves the synthesis quality.
We evaluate our approach through extensive experiments on human faces and outdoor churches. Given only one training image, GenDA can convincingly adapt the source model to the target domain with sufficiently high quality and diversity. Such adaptation includes both attribute-lvee and style-level, as shown in Fig. 1. Our method outperforms the state-of-the-art competitors by a large margin both qualitatively and quantitatively. We also show that, when the number of the samples available in the target domain increases, GenDA can filter out the individual attributes and only preserve their common characters (see Fig. 4). Noticeably, GenDA works well for the extreme cases where there is a large domain gap, like transferring the characters of Mona I Lisa to churches (see Fig. 5). Besides, thanks to the lightweight design of both the attribute adaptor and the attribute classifier, GenDA can finish each adaptation experiment within a few minutes.
METHODOLOGY
The primary goal of this work is to transfer a pre-trained GAN to synthesize images conforming to new domain with as few as only one reference image. Due to the limited supervision, it is
2

---

Under review as a conference paper at ICLR 2022
challenging to ensure both high quality and large diversity of the synthesis. Intuitively, according to the rationale of GANs (i.e, adversarial training between the generator and the discriminator), the discriminator can easily memorize that only the reference image is real while all others fake. In this way, to fool the discriminator, the generator may have to learn to produce images highly alike the reference, resulting in poor synthesis diversity. To mitigate this problem, we propose new adaptation algorithm, different from the previous fine-tuning scheme. Concretely, we (i) interpose an attribute adaptor between the latent space and the generator to help acquire the most representative characters from the target image; (ii) augment the discriminator backbone with an attribute classifier to guide the generator to make appropriate adjustments; and (iii) propose a Diversity-constraie training strategy. Before going into technical details, we first give some preliminaries of GANs.
2.1 PRELIMINARIES
Generative Adversarial Network (GAN) (Goodfellow et al., 2014) formulated as a two-player game between a generator and discriminator. Given a collection of observed data {x,} with N samples, the generator G(-) aims at reproducing the real distribution X' via randomly sampling latent codes subject to a pre-defined latent distribution Z. As for the discriminator D(-). it targets at differentiating the real data x and the synthesized data G(z) as a bi-classification task. These two models are jointly optimized by competing with each other,
LG = Ezez [log( ()G())))
LD Exxx llo(( [xx()(x))] E llog(1 - D(G()))))
(2)
After the training converges, the generator is expected to produce images as realistic as the training set, so that the discriminator cannot distinguish them anymore.
In this work, we start with GAN model that is well trained on a source domain x and work on adapting
target domain de = [xdat] that has only one image. In fact, it is ambiguous to define a "domain" using one image. We hence expect the model to acquire the most representative characters from the reference image. Taking face synthesis as an example, the character can include facial attributes (e.g., age or wearing sunglasses) and artistic styles, as shown in Fig. 1.
2.2 ONE-SHOT GENERATIVE DOMAIN ADAPTATION
common practice to transfer GANs is to simultaneously tune the generator and the discriminator on the target dataset (Wang et al.. 2018). However, as discussed above, the transferring difficulty increases drastically when given only one training sample. Existing methods attempt to address this issue by reducing the number of learnable parameters (Mo et al., 2020; Robb et al., 2020) and introducing training regularizers (Ojha et al., 2021). Even so, the overall fine-tuning scheme propose
(i.e., directly tuning G(.) and D(-)) remains and the diversity is low. Differently, new adaptation pipeline to preserve the synthesis diversity, which includes an attribute adaptor, an attribute classifier, and a diiversty-cconsrrin training strategy. Details are introduced follows.
Attribute Adaptor. Prior works have found that, a well-learned generator is able to encode rich semantics to produce diverse images (Shen et al., 2020; Yang et al., 2021b). For instance, face synthesis model could capture the variation factors like gender, age, wearing glasses, lighting, etc. Ideally, this knowledge should be sufficiently reused much as possible in the target domain, and then the synthesis diversity can be preserved accordingly. In this way, the generator focuses on transferring the most distinguishable characters of the only reference, instead of learning the common variation factors repeatedly and leading to overfitting due to lack of samples. This helps improve the data efficiency significantly, which is vital to the one-shot setting.
According to Xu et al. (2021), the latent code z can be viewed as the generative feature of G(z) that determines the multi-level attributes of the output image. Motivated by this, we propose to adapt such features regarding the reference image, yet keep the convolutional kernels untouched. Concretely, before feeding the latent code z to the generator G(.). we propose to first transform through a lightweight attribute adaptor A(-), as
= A(z) = a Z b,
(3)
where c stands for the element- wise multiplication, while and b are the learnable weight and bias respectively. With such a design, the transformed latent code z' is assumed to carry the sufficient information of the reference, and therefore G(z') would conform to the target domain dst
3

---

Under review as a conference paper at ICLR 2022
Attribute Classifier. Only having the attribute adaptor cannot guarantee the generator to acquire the representative characters from the training sample. Following the formulation of GANs, we incorporate the discriminator D(-). which is also pre-trate pre-trained on the source domain, to compete with the generator. In particular, we reuse the backbone d(-) but remove the last real/fake classification head, and then equip it with lightweight atribute classifier o(-). Given image x, either the reference image x or transferred synthesis G(A(z)), the classifier outputs a probability of how likely possesses the target attribute,
x((xx)))
(4)
However, due to the limited supervision provided by one image, the discriminator can easily memorize the real data, which leads to the overlitting of discriminator as well as the collapse of the generator (Karras et al., 2020a). As discussed above, the generated images before and after domain adaptation are expected to share most variation factors (i.e., a face model remains to produce faces after adaptation). From this viewpoint, the knowledge learned by the discriminator in its pre- training could be also reused. Therefore, unlike existing approaches that fine-tune all or partial parameters of D(-) (Mo et al., 2020; Wang al., 2020; Li et al., 2020; Ojha et al., 2021), we freeze all parameters of d(-) in the entire training process and merely optimizes <(-) to guide A(-) with adequate adjustments. Regarding the one-shot target domain, the mechanism behind the classifier is very similar to Exemplar SVM (Malisiewicz et al., 2011), which also suggests that it is sufficient to obtain good decision boundary with one positive and many negative samples. Differently, our attribute classifier is learned in an adversarial manner through competing with the attribute adaptor.
Diversity-constraint Strategy. Recall that this work targets at generative domain adaptation with only one reference image, which means no diversity of real data. On the contrary, however, the latent code can be sampled randomly and the pre-trained generator can produce highly diverse images from the source domain. From this perspective, it might be challenging to match these two distributions with such a huge diversity gap. To alleviate the optimization difficulty, we propose a diversity- constraint strategy, which retains the diversity of the generator during training. Specifically, truncate the latent distribution with a strength factor B, as
A(32 + (1 - 3)z),
(5)
where z indicates the mean code. Note that, truncation is a common trick used in the inference of of-te-of-the-ar GANs, like StyleGAN (Karras et al., 2019) and BigGAN (Brock et al., 2018), to improve synthesis quality. Nevertheless, to our best knowledge, this the first time that truncation is introduced in the training process to preserve the synthesis diversity.
Full Objective Function. In summary, the adaptor A()) and the classifier o(-) are trained with
L = - EEEz [log (dd((G(2))))
(6)
L6 = ExEx [log (ø(d(x))) - Ezez F (1-o(((((2))))
(7)
3 EXPERIMENTS
We evaluate the proposed method on multiple datasets and settings. In Sec. 3. we focus on one-shot generative domain adaptation. Quantitative and qualitative results indicate that the proposed GenDA can produce much more diverse and photo-realistic images than previous alternatives. Interestingly, the shared representative attributes of multiple shots could be also captured and transferred from the source to the target domain. Sec. 3.2 presents the quantitative comparison to prior approaches, demonstrating the effectiveness of our GenDA under the general few-shot setting. Additionally, when there exists a large domain gap, GenDA can still synthesize reasonable outputs in Sec. 3.3. Noticeably, the comprehensive ablation studies of each component are available Appendix.
We choose the state-of-the--rt StyleGAN2 (Karras et al., 2020b) as our base GAN model, following Ojha et al. (2021). StyleGAN2 proposes a more disentangled W space in addition to the native latent space Z. Here, our attribute adaptor is deployed on the W space. z in Eq. (5) becomes Warg. which a statistical average in the training of the source generator. apply differentiable augmentation image and the adapted synthesis (Zhao et al., 2020b; Karras et al., 2020a). In particular, we adopt the augmentation pipeline from StyleGAN2-ADA (Karras et al., 2020a) and linearly increase the
Implementation.
To alleviate the overfitting problem,
both the reference
4

---

Under review as a conference paper at ICLR 2022
Table 1: Quantitative comparison on one-shot adaptation between FreezeD (Mo et al., 2020), Cross-Domas (Ojha et al., 2021), and our proposed GenDA, Evaluation metries include FID (lower better), precision (higher means better quality), and recall (higher means higher diversity). All results are averaged over 5 training shots.
Methods
FID
Prec.
Recall
FID
Prec.
Recall
FID
Prec.
Recall
Freezel)
147.91
0.61
0.012
87.32
0.93
0.000
102.11
0.52
0.009
146.74
0.53
0.000
91.92
0.80
0.000
90.42
0.19
0,000
GenDA (ours)
80.16
0.74
0.033
44.96
0.76
0.067
87.55
0.16
0.053
(a) Babies
(b) Sunglasses
(c) Sketches
Target
GenDA (Ours)
FreezeD
Cross-Domain
Figure 2: Qualitative comparison on one-shot adaptation between Freezel (Mo et al., 2020), Cross-Domain (Ojha al., 2021), and our proposed GenDA. The first column shows the reference images. GenDA significantly outperforms the other competitors from the diversity perspective.
augmentation strength from 0 to 0.6. We use Adam optimizer (Kingma & Ba, 2014) for training and the learning rates are set as 1. 1.25e and 2.5 2.5e-4 4 for A(- and ()-), respectively. All experiments are stopped when the discriminator has seen 200 real samples. The last "ToRGB" layer (with 1 x convolution kernels) of the generator is tuned together with the adaptor A(-) to make sure aligning the synthesis color space to the reference image. The syper-parameter 3 in Eq. (5) is set as 0.7.
Datasets. Our source models are pre-trained on large-scale datasets (e.g. FFHQ (Karras et al., 2019) and LSUN Church (Yu et al., 2015)) with resolution 256 x 256. We collect target images from UFHO-babies, PPHQ--unnlaasess face sketches (Ojha et al. (2021)), and samples from masterpieces (i.e., Mona Lisa and Van Gogh's houses) and Artistic-Face dataset.
Evaluation Metrics. Fréchet Inception Distance (FID) (Heusel et al., 2017) serves as the main metric. Akin to Ojha al. (2021), always calculate the FID between 5,000 generated images and all the validation sets. Additionally, we report the precision and recall metric (Kynkäääniemi al., 2019) to measure the quality and diversity respectively.
3.1 ONE-SHOT DOMAIN ADAPTATION
Comparison with Existing Alternatives. For one-shot generative domain adaptation, we compare against FreezeD (Mo et al., 2020) and Cross-Domain (Ojha et al., 2021). Noticeably, the implemented FreezeD is further improved by the data augmentations, which is already a challenging baseline. Considering the variance brought by different single shots, we calculate all metrics over training shots. As shown in Tab. 1, our GenDA remains surpass multiple approaches by clear gap from the perspective of FID. Besides, to further compare different methods from the views image quality and diversity, also report the precision and recall. Intuitively, higher precision means
5

---

Under review as a conference paper at ICLR 2022
Target
Source domain: FFHQ Face
Target
Source domain: LSUN Church
Figure 3: One-shot adaptation with different target samples. GenDA manages capture the representative characters from the given reference, such as sunglasses, artistic style for faces, and vegetation, pyramid material for churches.
higher image quality (more closer the real sample), while higher recall indicates higher diversity. Although FreezeD (Mo et al., 2020) and Cross-Domain (Ojha et al., 2021) plausibly achieves better synthesis quality on sunglasses and sketches, their low recalls strongly imply overfitting. Fig. confirms that there is insufficient diversity for such two methods. As contrast, GenDA results in the competitive quality and standing-on out diversity from the quantitative and qualitative perspectives. In particular, GenDA manages synthesize knowledge. Fig. 3 suggests that our GenDA works on transferring both attributes (sunglasses, gender, vegetation and material) and artistic styles for face and church model respectively.
with cap which is only available in prior
Common Attributes from Multiple References. There might be a number of representative variation factors in given face image (e.g., age, gender, smile and sunglasses). Therefore, when transferring a pre-trained generative model on one image, multiple factors could be adapted together. Fig. 4 provides an example our GenDA on sunglasses. Specifically, we train GenDA on multiple target domains which might contain single individual with different identities or pair of images. Compared with the source output (the first row), representative attributes besides sunglasses are also transferred. For instance, the second and third rows suggest that although all individuals wear sunglasses, the target's gender is also adapted to the new synthesis. The third shot could even affect the gender and hairstyles. This reveals that our GenDA could capture multiple representative attributes the target domain. When the target domain contains more than one shot like the last two of Fig. 4, the representative attributes become the common attributes of all individuals, leading to the corresponding results (i.e., sunglasses and gender). Namely, our GenDA able to capture and adapt the representative attributes matter how many images the target domain has.

---

Under review as a conference paper at ICLR 2022
Before Adaptation
Target
Sunglasses Temale
After Adaptation
Sunglasses Malo
Hair Female
Sung Sunglastes
Female
Transferring the "common" semantic of more than one reference image. On the left are the training samples, while the top are the samples synthesized before adaptation. In the remaining rows, images from the same column are produced with the same latent code. Under the settings of two-shot adaptation (i.e., the last two rows), all people are wearing eyeglasses (common attribute of the two references) but they have the same gender (divergent attribute) the original synthesis in the top row. (2) similarly, in the bottom row, all people are female yet the "eyeglasses" attribute is preserved from the original synthesis.
Figure 4:
tell that (1) in the second last row,
3.2 10-SHOT DOMAIN ADAPTATION
As GenDA could capture the representative attributes precisely with an increasing number of target shots, we also compare again prior approaches under the general few-shot adaptation setting. Following the setting of Ojha et al. (2021), choose 10-shot target domain for comparison.
Quantitative Comparison. Multiple baselines are introduced to conduct the quantitative com- parisons, including Transferring GANs (TGAN) (Wang et al., 2018), Batch Statistics Adaptation (BSA) (Noguchi & Harada, 2019), FreezeD (Mo et al., 2020), MineGAN (Wang et al., 2020), EWC (Li al., 2020) and Cross-Domain (Ojha et al., 2021). All these methods adapt pre-trained source model to a target domain (e.g.. babies, sunglasses and sketches). Akin to al., 2021), FID serves the metric for evaluation.
(Ojha
Tab. 2 presents the 10-shot quantitative comparison several target domains. Although Cross- Domain (Ojha et al., 2021) has already obtained the competitive results, our method could still significantly improve the performance by clear margin, resulting in the new state-of-the-an synthesis quality on few-shot adaptation. Specifically, for domains that differ in semantic attributes (i.e., babies and sunglasses), huge gains are obtained by GenDA. It also worth noting that when the domain gap increases (i.e. from realistic facial images to sketches), the performances of Cross- domain (Ojha et al., 2021) and FreezeD (Mo et al., 2020) become almost identical (45,67 46.54) while substantial improvements of synthesis could remain observed by our method (31.97). demonstrates the effectiveness of our method on the general few-shot settings.

---

Under review as a conference paper at ICLR 2022
Table 2:
Quantitative comparison on 10-shot adaptation. FID (lower better) serves as the evaluation metric. Numbers in blue indicate our improvements over the state-of-the-art alternative (Ojha et al., 2021).
10-shot transfer
Babies
Sunglasses
Sketches
TGAN (Wang et al., 2018)
104,79
55.61
53.41
TGAN+ADA (Karras et al., 2020a)
102.58
53.64
66.99
BSA (Noguchi & Harada, 2019)
140.34
76.12
69.32
FreezeD (Mo et al., 2020)
110.92
51.29
46.54
MineGAN (Wang al., 2020)
98.23
68.91
64.34
EWC (Li et al., 2020)
87.41
59.73
71.25
Cross Dossan (Ojha et al., 2021)
74.39
42.13
45.67
GenDA (ours)
47.05
22.34)
22.62 19.51)
31.97
13.70)
Target
Source domain: FFHQ Face
Target
Source domain: LSUN Church
Figure 5: Cross-domain adaptation, where GenDA manages to transfer the character of an out- of-domain target (first column) to the source domain.
3.3 CROSS-DOMANE ADAPTATION
In this part, study the adaptation on unrelated source and target domains. Specifically, Van Gogh's houses, Superman and Mona Lisa serve as the target for a face and church source model respectively. Considering the motivation that aim at reusing the prior knowledge (i.e., variation factors) by freezing the parameters, the synthesis after adaptation is supposed to share similar visual concepts. That is, face model would still produce faces matter what the target image is. Fig. suggests that the source models remain to produce the corresponding content. More importantly, the color scheme and painting styles are also transferred. For example, the red roof the first row renders the red glasses, the yellow sky at the second row draws the front head in yellow. The blue hair, green background and the shadows of Superman are well adapted to church. The painting style of Mona Lisa is also transferred.
Obviously, the shared attributes between face and church are quite rare. Therefore, GenDA pours more attention to the variation factors like color scheme, texture and painting styles which could be directly transferred across unrelated domains. From this perspective, our GenDA might enable alternative for neural style transfer task that aims at transferring the styles of given image. But more generally, our GenDA able to transfer more high-level attributes like gender, age, and sunglasses while the technique of style transfer might fail, which might be of benefit to art creation.
8

---

Under review as a conference paper at ICLR 2022
4 RELATED WORK AND LIMITATIONS
Training Generative Models with Limited Data. Many attempts have been taken to train a generative model on limited data. For one thing, some of the prior approaches proposed leverage the data augmentation to prevent the discriminator from overfitting. Specifically, Zhang & Khoreva (2019) introduced a type of progressive augmentations. Zhao et al. (2020d) investigated the effects of various augmentations during the training. Theoretical analysis was conducted by Tran et al. (2021) for several data augmentations. Zhao et al. (2020b) proposed to apply the augmentations to both the real and synthesized images in differentiable manner. Karras et al. (2020a) designed an adaptive discriminator augmentation that does not leak to stabilize the training process. For another, multiple regularizers were also introduced to provide extra supervision. For instance, Zhao et al. (2020c) involved the consistency regularization for GANs which shows competitive performances with limited data. Yang et al. (202 incorporated contrastive learning as an extra task improve the data efficiency. Karras et al. (2020a) also pointed out that decreasing the number of parameters of generators and introducing the dropout (Srivastava al.. 2014) could alleviate the overfitting problems. Shaham et al. (2019) and Sushko et al. (2021) designed different frameworks which learn from a single natural image or video. However, when the number of available images is less than 10, they usually lead to unsatisfying diversity. Different from these works which learn from scratch, we focus on the generative domain adaptation, a practical alternative that first pre-trains a source model on the large-scale dataset and then transfers it on target domain with the one-shot image.
Few-shot Generative Domain Adaptation. Generative domain adaptation has attracted a considerable number of interests due to its practical importance. Wang et al. (2018) proposed to use the same objective for adaptation. Noguchi & Harada (2019) fine-tuned the batch statistics merely for the few-shot adaption. Wang al. (2020) transformed the original latent space and tuned the entire parameters for the target domain. Mo et al. (2020) froze the lower-level representations of the discriminator to prevent overfitting. Zhao et al. (2020a) revealed that low-level filters of both the generator and discriminator can be transferred via a new adaptive filter modulation. Li et al. (2020) penalized certain weights identified by Fisher information. Robb et al. (2020) learned adapt the singular values of the pre-trained weights while freezing the corresponding singular vectors. Ojha al. (2021) proposed the cross-domain consistency as a regularization to maintain the diversity. Different from prior work, we aim at reusing the learned knowledge on the source domain and thus identify and adapt the representative attributes, which enables the one-shot domain adaptation and outperforms other alternatives by clear margin under the general few-shot settings.
Limitations. Despite the state-of- performances on both one-shot and few-shot generative domain adaptation, our proposed GenDA still has some limitations. For example, the rationale behind GenDA is to reuse the prior knowledge learned by the source GAN model, which hinders it from transferring model to completely different domain. As suggested in Fig. 5, when adapt a church model regarding a face image, the outputs are still churches but not faces. This implies that our method would fail when the inter-subclass variations are huge. Such property is a silver lining, depending on the practical application. A second limitation is that our current design treats all characters of the reference image as a whole. Taking the first row of Fig. 3 as an example, the sunglasses, skin color, and background are transferred simultancously. It is hard to accurately transfer some particular attributes. However, it indeed possible to use some auxiliary samples to help define a common attribute, as shown in Fig. 4. Besides, our GenDA also relies on the myeerwwise stochasticity involved in the generator structure. Concretely, in our base model, StyleGAN2 (Karras et al., 2020b), the latent code is fed into all convolutional layers instead of the first layer only. Without the layer-wise design, the supervision will hard to back-propagatt to the attribute adaptor given a deep synthesis network. Fortunately, however, such design is commonly adopted by the state-of-the GANs (Karras et al., 2019; 2020b; Brock et al., 2018).
5 CONCLUSION
In this work, we propose GenDA for one-shot generative domain adaptation. We introduce two lightweight modules, i.e., an attribute adaptor and an attribute classifier, to the fixed generator and discriminator respectively. By efficiently learning these two modules, manage to reuse the prior knowledge and hence enable one-shot transfer with impressively high diversity. Our method demonstrates substantial improvements over existing baselines in a wide range of settings.

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
Training a generative model with limited data is a fundamental and practical problem. This work significantly advances this field by using as few as only one image for domain adaptation. Such an improvement is of great importance to the community. Firstly, it pushes the data efficiency in GAN training to the limit and demonstrates an early success. believe more and more methods will be proposed to keep improving the training efficiency along this direction. Secondly, our approach based on reusing the prior knowledge a pre-trained GAN model. It provides insights on how to better utilize the pre-trained generative model for various downstream tasks (like semantic manipulation and super resolution), which is a widely studied research area. Thirdly, this work also enables many potential real-world applications, like data augmentation (to solve the long-tail problem), entertainment, creative design, artistic imitation, etc. However, every coin has two sides. Our method considerably reduces the cost of producing fake data, in terms of using less data, less time, and fewer computing resources. But we believe that its academic contribution and its positive impact on the industrial algorithm are more valuable, and that the negative impact will be lowered along with the development of deep fake detector.
REPRODUCIBILITY STATEMENT
The base model used in this work is StyleGAN2 (Karras al., 2020b), with official imple- mentation syylegan2-ada-pytoree Church. For one-shot domain adaptation, the reference image can be easily customized by the users, either from online or from other datasets, since our approach only requires one training sample. The implementation and training details are described the beginning of Sec. 3, which are shared by all experiments. Besides, all experiments are conducted with 8 GPUs. The evaluation metries, including FID (Heusel al., 2017), precision and recall (Kynkäänniemi et al., 2019). also follow their official implementation, FID and improved improved-precision-aaee precision-aan--reall! metric. To ensure the reproducibility, the code and models will be made publicly avail- able.
The large-scale source datasets we use are FFHQ and LSUN
REFERENCES
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. In Int. Conf. Learn. Represent. 2018. 1, 1,4,
Gabriela Csurka.
Domain adaptation for visual applications: A comprehensive survey. arXiv
preprint arXiv:/702.05337 2017. 2
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Neural Inform. Process. Syst., 2014. 1,3
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. In Adv. Neural Inform. Process. Syst., 2017. 5, 10
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. In Int. Conf. Learn. Represent. 2018. 1
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In IEEE Conf. Comput. Vis. Pattern Recog., 2019. 1, 9,
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. In Adv. Neural Inform. Process. Syst., 2020a. 1,4.8, 8, 9
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of StyleGAN. In IEEE Conf. Comput. Vis. Pattern Recog., 2020b. 1,4,9 9, 10
10

---

Under review as a conference paper at ICLR 2022
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. arXiv preprint arXiv: 1904, 904.06991, 2019. 5, 10
Yijun Li, Richard Zhang, Jingwan Lu, and Eli Shechtman. Few-shot image generation with elastic weight consolidation. In Adv. Neural Inform. Process. Syst., 2020. 2, 4, 7,8,
Tomasz Malisiewicz, Abhinav Gupta, and Alexei Efros. Ensemble of exemplar-svms for object detection and beyond. In 2011 International conference computer vision, pp. 89-96. IEEE, 2011. 4
Sangwoo Mo, Minsu Cho, and Jinwoo Shin. Freeze the discriminator: a simple baseline for fine- tuning gans. arXiv preprint arXiv:2002. 10964, 2020. 3, 4, 5, 6, 7, 8, 9, 12
Atsuhiro Noguchi and Tatsuya Harada. Image generation from small datasets via batch statistics adaptation. In Int. Conf. Comput. Vis., 2019. 2, 9
Utkarsh Ojha, Yijun Li, Jingwan Lu, Alexei A Efros, Yong Jae Lee, Eli Shechtman, and Richard Zhang. Few-shot image generation via cross-domain correspondence. In IEEE Conf. Comput. Vis. Pattern Recog., 2021. 2, 4, 5, 6,
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.064:: 2015. 15
Esther Robb, Wen-Sheng Chu, Abhishek Kumar, and Jia-Bin Huang. generative adversarial networks. arXiv preprint 1eXXi:20100/1943, 2020. 2, 2,3,
Few-shot adaptation of
Tamar Rott Shaham, Tali Dekel, and Tomer Michaeli. Singan: Learning a generative model from
single natural image. In Conf. Comput. Vis., 2019.
Yujun Shen, Ceyuan Yang, Xiaoou Tang, and Bolei Zhou. Interfacegan: Interpreting the disentangled face representation learned by gans. IEEE Trans. Pattern Anal. Mach. Intell., 2020.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way prevent neural networks from overfitting. The journal of machine learning research, 2014. 9
Vadim Sushko, Jurgen Gall, and Anna Khoreva. One-shot gan: Learning to generate samples from single images and videos. In JEEE Conf. Comput. Vis. Pattern Recog., 2021. 9
Ngoc-Trung Tran, Viet-Hung Tran, Ngoc-Bao Nguyen, Trung-Kien Kien Nguyen, and Ngai-Man Cheung. On data augmentation for gan training. IEEE Trans. Image Process., 2021. 9
Pei Wang, Yijun Li, and Nuno Vasconcelos. Rethinking and improving the robustness of image style transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 124-133, 2021. 14,
Yaxing Wang, Chenshen Wu, Luis Herranz, Joost van de Weijer, Abel Gonzalez-Garcia, and Bogdan Raducanu. Transferring gans: generating images from limited data. In Eur. Conf. Comput. Vis., 2018. 2, 3, 8,
Yaxing Wang, Abel Gonzalez-Garcia, David Berga, Luis Herranz, Fahad Shahbaz Khan, and Joost van de Weijer. Minegan: effective knowledge transfer from gans target domains with few images. In IEEE Conf. Comput. Vis. Pattern Recog., 2020. 2, 4, 8,
Yinghao Xu, Yujun Shen, Jiapeng Zhu, Ceyuan Yang, and Bolei Zhou. Generative hierarchical features from synthesizing images. In IEEE Conf. Comput. Vis. Pattern Recog.. 2021. 3
Ceyuan Yang, Yujun Shen, Yinghao Xu, and Bolei Zhou. Data-efficient instance generation from instance discrimination. In Ady. Neural Inform. Process. Syst., 2021a. 1,9

---

Under review as a conference paper at ICLR 2022
Ceyuan Yang, Yujun Shen, and Bolei Zhou. Semantic hierarchy emerges in deep generative representations for scene synthesis. Int. J. Comput. Vis., 2021b. 3
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint 0tti::/506.333500 2015. 5, 20
Dan Zhang and Anna Khoreva. Pa-gan: Improving gan training by progressive augmentation. In Adv. Neural Inform. Process. Syst. 2019. 1,
Miaoyun Zhao, Yulai Cong, and Lawrence Carin. On leveraging pretrained gans for generation with limited data. In Int. Conf. Mach. Learn., 2020a. 2, 9
Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for data-efficient gan training. In Adv. Neural Inform. Process. Syst., 2020b. 1,4, 4,
Zhengli Zhao, Sameer Singh, Honglak Lee, Zizhao Zhang, Augustus Odena, and Han Zhang.
Improved consistency regularization for gans. In Assoc. Adv. Artif. Intell., 2020c. 9
Zhengli Zhao, Zizhao Zhang, Ting Chen, Sameer Singh, and Han Zhang. Image augmentations for gan training. arXiv preprint 200xx22000.02555, 2020d. 1,
APPENDIX
This appendix is organized as follows. Sec. A conducts ablation study on the proposed approach, including the attribute adaptor, the attribute classifier, the diversity-constrainn strategy, and the hyper-parameter 3 in Eq. (5). Sec. B analyzes the transformation (i.e., the learnable vector a in Eq. (3)) for attribute adaptation, which provides insights on how our algorithm works. Sec. shows how the latent representations are transformed after adaptation. Sec. D presents the additional comparison with the style transfer method and investigates the visual relation between and after adaptation. Sec. E performs latent interpolation with the models adapted on the target domain, which suggests that our GenDA does not harm the learned latent structure.
Table Al: Ablation study on the components proposed in GenDA, including the attribute adaptor (AA), the attribute classifier (AC), and the diversity-constraii strategy (DC). Evaluation metrics include FID (lower is better), precision (higher means better quality), and recall (higher means higher diversity).
AC DC
FID
Prec.
Recall
FID
Prec.
Recall
FID
Prec.
Recall
109.87
0.80
0.000
53.06
0.72
0.003
39.96
0.70
0.024
53.59
0,57
0.064
28.35
0.72
0.333
37.68
0.30
0.034
/
50.37
0.62
0.106
25.29
0.71
0.349
33.29
0.39
0.089
1
/
47.05
0.71
0.065
22.62
0.78
0.204
31.97
0.57
0.076
(a) Babies
(b) Sunglasses
(c) Sketches
A ABLATION STUDY
Component Ablation. In order to further investigate the role of each proposed component (i.e., attribute adaptor, attribute classifier and truncation strategy), we conduct comprehensive ablation studies on 10-shot adaptation. In particular, we choose FreezeD (Mo et al., 2020) as our baseline model which freezes the lower features of the discriminator while fine-tunes the rest of the discriminator and the entire generator. Moreover, also apply adaptive discriminator augmentation strategy (ADA) on it since data augmentations usually help alleviate the overfitting problem.
As is shown in Tab. AI, the baseline alternative could achieve the satisfying synthesis quality but worse diversity, suggesting that might overfit the training shot significantly. By involving the attribute adaptor, the baseline is substantially improved, which already outperforms prior approaches. Mcanwhile, the boosted diversity also demonstrates our motivation that we could reuse the variation factors of the source model and hence maintain the synthesis diversity to extent.
12

---

Under review as a conference paper at ICLR 2022
Table A2: Ablation study on architecfure designs. two-layer multi-lay -laeer perceptron (MLP) is introduced as the heavy version of the attribute adaptor and the attribute classifier, respectively.
Models
Babies
Sunglasses
Sketches
Heavy attribute adaptor
290.22
262.32
114.25
Heavy attribute classifier
132.14
115.88
109.23
Lightweight design
80.16
44.96
87.55
Table A3: Ablation study on the strength of diversity-constraint strategy, which is noted as / in Eq. (5). Here, a smaller 3 denotes stronger diversity constraint. Evaluation metrics include FIDD (lower is better), precision (higher means better quality), and recall (higher means higher diversity).
FID
Precision
Recall
0.5
24.87
0.82
0.163
0.7
22.62
0.78
0.204
0.9
24.32
0.73
0.269
1.0
25.29
0.71
0.349
However, the quality (i.e., precision) on babies and sketches becomes worse. After being equipped with the attribute classifier, the overall measurement is further enhanced. Together with the diversity- constraint strategy, GenDA could facilitate the quality to the baseline level but achieve better diversity, leading to the new state-of-the-art few-shot adaptation performances.
Architectural Ablation. To further study the effect of the lightweight design of the attribute adaptor and the attribute classifier, we conduct experiments of replacing the lightweight module with two- layer multiOlayer perceptron (MLP), which has higher learning capacity. Tab. A2 presents the results. Obviously, with either a heavier attribute adaptor or a heavier attribute classifier, the synthesis performance drops significantly. This might be caused by the overfitting problem since we only have one training image under our one-shot setting. It verifies the effectiveness and necessity of our lightweight design for the challenging meeshot generative domain adaptation task.
Slyper-paramette Ablation. We also conduct the ablation of hyper-parameter | in Eq. (5). Specifically, we train our model with 10-shot sunglasses images and maintain the evaluation metrics used in Sec. 3.2. As is shown in Tab. A3, when increasing the value of 8, the synthesis quality becomes worse while the diversity enhanced. Noticeably, when the is less than 0.5, the training diverges. Therefore, we choose 0.7 in all experiments for its overall performance.
B ANALYSIS ON TRANSFORMATION VECTOR
We also visualize the learned attributes via Principal components analysis (PCA). Concretely, we first choose 15 shots from 3 domains (i..., babies, sunglasses and sketches). The corresponding models are trained on them and 15 attribute vectors in Eq. (3) are collected to perform PCA. Fig. A1 presents the PCA results. Obviously, the attribute vectors of different shots but the same domain assemble closely, while there are obvious decision boundaries across domains. Such interpretation happens to match our motivation that the attribute adaptor could capture the representative attributes and make the corresponding adjustment for source model to multiple target domains.
C ANALYSIS ON TRANSFORMED LATENT REPRESENTATION
We also visualize how the latent representations change before and after adaptation, with the help of PCA. Specifically, we sample 2K latent codes from the latent space after adaptation for each model used Fig. 4 and Fig. 5. Fig. A2 suggests that the distributions after adapting the source model to three different target domains are clearly separable. Also, when using more than one reference image, which have common attributes, from the target domain, the transformed representations tend to locate at the overlapped region between the representations learned from each reference image independently. Fig. A3 shows the latent representation change after cross- dross-domain adaptation, where the latent distribution of the source domain is successfully shifted to the target one.
13

---

Under review as a conference paper at ICLR 2022
Figure A1: Analysis the transformation vector for domain adaptation, which is noted in Eq. (3). We launch 15 one-shot adaptation experiments with 5 kids, 5 sunglasses, and sketches as the target, respectively. After the model converges, we collect the transformation vector learned by the attribute adaptor, and then perform PCA on all 15 vectors. We can tell that our GenDA tends to learn similar transformation regarding the reference with similar characters. The 15 training samples are visualized together with the PCA results.
Sunglasses Female
Sunglasses Female
Sunglasses
Female
Sunglasses Male
Hair Female
Sunglasses Male
Hair Female
Visualization of transformed latent spaces after domain adaptation. Each cluster corresponds particular training set. Left: The latent spaces learned with different reference images are clearly separable. Right: When we use two reference images, which have common attributes, as a combined training set, the transformed representations tend locate at the overlapped region between the representations learned from each reference image independently.
Figure A2:
D ADDITIONAL RESULTS
Comparison with the style transfer task. As discussed in Sec. 3, the proposed approach might be an alternative for style transfer. Here, we qualitatively compare our approach with the state-of- the-art method, SWAG (Wang et al., 2021) regarding the style transfer task. The comparison results are included in Fig. A4. From the perspective of the color transfer, SWAG (Wang et al., 2021) does better since it could successfully transfer the representative color from the target image to the synthesis. For example, the green of the vegetation and the pink of the skin appears in the transferred samples. However, also reveal its weakness that the transfer is inconsistent. Specifically, the wall the sky in church synthesis is in pink while our method pictures them in the same color. This
14

---

Under review as a conference paper at ICLR 2022
Source Domain
Target Domain
Synthesis After Adaptation
Representation Change
Figure A3: Latent representation comparison before and after adaptation. The latent space is clearly pushed from the red cluster to the blue cluster when there exists an obvious gap between the source and the target domains.
implies that the style transfer based methods could have no semantic concepts regarding to content, leading to such inconsistency and obvious artifacts. Therefore, it naturally cannot transfer higher-level attributes like sunglasses while ours could.
The visual relation before and after adaptation. A byproduct of proposed GenDA is that manage to inherit some characteristics from the source model the process domain adaptation. Here, we take the images synthesized by the source generator the references, and feed the adapted generator with the same latent codes. In this way, we can check how the synthesis varies before and after adaptation. Fig. A5 and Fig. A6 correspond the settings of Fig. and Fig. 5, respectively. We can tell that many attributes are maintained in the adapting process, verifying our motivation, which that the most of variation factors could be reused in the task of generative domain adaptation.
INTERPOLATION OF LATENT VECTORS
Here, we perform the semantic interpolation evaluate the tuned generator. Typically, the interpolation at the latent space (Radford et al., 2015) could interpolate the semantics of two images, which is widely used measure the diversity of generative models. Fig. A7 and Fig. A8 present the semantic interpolation of face and church model respectively. In particular, all models are fine-tuned with one shot image. Obviously, satisfying interpolation i.e., smoothing change clearly observed, even if the model is fine-tuned with one shot image.
15

---

Under review as a conference paper at ICLR 2022
Source domain: FFHQ Face
Target
Adaptation By Ours
Style Transfer by SWAG
Source domain: LSUN Church
Target
Adaptation By Ours
Style Transfer by SWAG
Figure A4: Comparison with the style transfer task. GenDA could transfer the characteristic of the target image to the source model in more harmonious way, while SWAG (Wang et al., 2021) transfers the color inconsistently, leading obvious artifacts.
16

---

Under review as a conference paper at ICLR 2022
Source domain: FFHQ Face
Target
After Adaptation
Source domain: LSUN Church
Target
After Adaptation
Figure A5: Visual relation of one-shot adaptation with different target samples. Here, for each model (i.e., faces and churches), each column generated with the same latent code. Our GenDA is able to transfer the most distinguishable attributes of the target domain, but almost maintains other semantics (e.g., the face pose and the church shape).
17

---

Under review as a conference paper at ICLR 2022
Source domain: FFHQ Face
Target
After Adaptation
Source domain: LSUN Church
Target
After Adaptation
Figure A6: Visual relation of cross-domain adaptation. Here, for each model (Le. faces and churches), each column generated with the same latent code. We can tell that, under the cross- domain setting, our GenDA can still inherit some characteristics from the source model (e.g. the face pose and the church shape).
18

---

Under review as a conference paper at ICLR 2022
Figure A7: Semantic interpolation with the model pre-trained on FFHQ human faces (Karras et al., 2019). The first and last columns are the synthesized images with two different latent codes after one-shot domain adaption, while the remaining columns are the outputs by linearly interpolating the two codes. All intermediate results are still high-quality faces, with gradually varying semantics (e.g., the gender and the face pose).
19

---

Under review as a conference paper at ICLR 2022
Figure A8: Semantic interpolation with the model pre- pre-trained LSUN Churches (Yu et al., 2015). The first and last columns are the synthesized images with two different latent codes after one-shot domain adaption, while the remaining columns are the outputs by linearly interpolating the two codes. All intermediate results are still high-quality churches, with gradually varying semantics (e.g., the church shape and the cloud in the sky).
20


Reviews:
[{"review_id": "vJUSofAoFj", "reviewer": "Reviewer_Zh5e", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is generally well written and understandable. The paper conveys the main ideas of the work to the reader and I believe I got a good understanding from what the authors did by reading the paper. It seems very doable to reproduce the results on the paper alone.\n\nThe two technical main ideas are simple, but somewhat novel and can be considered a contribution to the field. Most, I like the idea of reusing a pre-trained discriminator. I am not aware of similar work and this seems to be a creative idea.\n\nOn the downside, I do not like the idea of the adaptor layer. The adaptor does not seem powerful enough to match the complexity of the task. There are multiple concerns here. The layer is too early in the network and seems to manipulate z-space rather than w-space (or some of the other available latent spaces later in the network). The layer is not powerful enough to do a more complex computation. Also, naturally, domain transfer generally means that the generator does not have the capabilities to generate images in the new domain. What is this network really doing by only manipulating a z-vector? If the z-vector is not split, it is hard to imagine the generator has the range to produce the desired results. If the z-vector is split, this could be a bit better, but also carries more risks. As it is, we can only observe a limited form of domain transfer.\n\nOne concern regarding the results is Figure 4. Visually all faces look really nice. But the task is not that clear and it's really difficult to see a relationship between input and output. This is not a result that should be considered domain adaptation. I would strongly suggest that all results transferring faces to other faces (but both input and output are in the same domain) should be removed from the paper. While this uses the same software system, it is essentially a different task that has been addressed better by many other papers. For example, attribute-based GAN editing papers can do that as well and there are multiple of them. I am pretty sure the authors do not want to go down that route.\nThe challenging domain transfer results, e.g. Figure 5, are indeed what I was hoping to see more of. Unfortunately, these results do not look particularly good. We can mainly observe a slight adaptation of all results shifting to a particular color scheme. It would also suggest to synchronize the examples. I think this is would be more insightful if we can see the same inputs transferred to different domains. I also feel the results are hard to interpret in multiple figures (e.g. 3), because I do not understand where the corresponding images in the original domain are. Maybe the input changes a lot for the challenging examples. I am afraid that this is a considerable omission, because the images in the input domain are such an essential component of the results. How can anybody judge these results without seeing the input images? In addition, there should be corresponding results shown. E.g., the last two rows in Figure 5 should correspond to each other. I do like the result in Figure 2. In a future version of the paper it would be good to see more of these interesting results.\n\nThe evaluation of this type of work is primarily visual. The metrics are not especially revealing if the results are good. It's good to have them, but I feel the authors still need quite a bit of work to make the system function as desired. My recommendation would be to revisit the adaptor design and find something better.\n", "summary_of_the_paper": "The paper proposes to do one-shot domain adaption by fine tuning a state of the art network StyleGAN. The paper introduces a few techniques to do so. First, the paper proposes to add a lightweight attribute adaption layer in the beginning of StyleGAN to modify the latent code. An existing latent code can be modified to give a new latent code that is in the new domain. Second, the paper proposes a novel type of discriminator reuse. A discriminator has its features frozen, but a new classification head is added. The paper shows various visual results.", "summary_of_the_review": "I am very positive about the potential of this work, but it needs a substantial improvement.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "UDWt3rkNOzJ", "reviewer": "Reviewer_NUUH", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n* This paper tackles a challenging domain adaptation problem which is very interesting.\n* This paper demonstrates convincing qualitative comparisons (e.g., realism and diversity) to the existing efforts including Mo et al., 2020 and Ojha et al. 2021.\n* This paper also shows comparable and improved quantitative results to the state-of-the-art methods in the case of a one-shot and ten-shot adaptation, respectively.\n\nWeaknesses:\n* This paper discovered an interesting phenomenon that a lightweight attribute adaptor could generate surprisingly-good domain adaptation results. Although it has been explained in the paper, the reviewer would appreciate it if some sort of ablation study (both qualitative and quantitative results) can be included in the paper (at least in the supplementary material of the final version). For example, what happens if we use different architecture or operators for the attribute adaptor.  The same rule applies to the attribute classifier (though it might be overfitting).\n* At a higher level, this paper reminds me of the Exemplar SVM published 10 years ago. The conclusion in the paper is that it is often sufficient to obtain a good decision boundary if you have one positive example and many negative examples. This is very similar to the attribute classifier proposed in the paper. The reviewer would suggest incorporating such discussions in the final version of the paper.\n* The limitation of the paper is not clearly stated. For example, the reviewer would like to know if the one-shot learning method applies to the “car -> abandoned car” application studied in the Ojha et al., 2021. It seems to the reviewer that the proposed method could fail at “car -> truck” application when the inter-subclass variations are huge. Please comment on this in the rebuttal.\n* Furthermore, the cross-domain experiment (see Figure 5) seems to be a special case of style transfer. In this sense, this paper should also include comparisons to the style-transfer methods.\n\nReferences:\nEnsemble of Exemplar-SVMs for Object Detection and Beyond, Malisiewicz et al., In ICCV 2011.", "summary_of_the_paper": "This paper studies the challenging problem of transferring a pre-trained GAN from the source domain to a target domain with only one example available. To achieve the source-to-target domain adaptation while being able to synthesize diverse samples, this paper proposed a novel method called GenDA. The key idea is to freeze the generator and discriminator backbones while fine-tuning the linear layer added on top. Convincing experiment results have been demonstrated on the standard face and building benchmarks.", "summary_of_the_review": "Overall, this is a very interesting paper with convincing results. I believe the key idea from this paper can have a broader impact on the community (e.g., 3D and video synthesis) in the future. I recommend accepting this paper but would kindly ask the authors to incorporate the suggested discussions in the final version.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "TEMHNNCDG0B", "reviewer": "Reviewer_CemL", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. The one-shot image generation is a novel and interesting task.\n\n2. The overall idea and the proposed pipeline towards addressing it are easy to follow.\n\n3. In the experimental, the claim about one-shot attribute-related results is being met.  \n\nCons:\n\nWhile the authors aim to address a challenging and novel task, I believe some parts need more clarification (even after considering the supplementary material):\n\n1. The biggest weakness is that the proposed method has limited novelty. While the authors propose a stacked pipeline to address the quality and diversity, the key contribution they made is unclear. \n\na. The z+/w/w+/s space analysis and adaption has been widely conducted in the latest works [r1, r2, r3]. What are the differences between the proposed adaptor and these prior works? Why the proposed adaptor would like to perform better?\n\nb. Related to the above, the attribute classifier has been used in StyleFlow [r2]. Why the proposed one is better? In addition, if I understand correctly, the attribute classifier only judges the output is real or fake, instead of predicting attribute labels, because some examples in Figures 2 and 3 should not have corresponding labels. If this classifier just outputs real or fake labels, why not just fine-tuning the final layer of the original discriminator?\n\nc. I cannot buy the novelty of reusing truncation trick for diversity-constraint strategy. As mentioned by the authors, this trick is a normal one in the current generation code. The authors did not provide a new direction to sell this strategy. \n\n2. The impressive results in Figures 4 and 5 are interesting, but the analysis is not clear enough. It would like to be better to visualize the learning representation after the adaptor as in AgileGAN [r3].\n\na.  The results in the final two rows of Figure 4 are quite interesting. I guess the learned w space has been pushed to a narrow space (the final one will be more narrow). If the authors can visualize the learning representation distribution, it will be clearer to demonstrate such mapping in the representation domain. \n\nb. A similar situation is in Figure 5.  It is interesting to use the one-shot pipeline for style translation. As the pre-trained G and D have not been updated, I guess the learned w space has been pushed into a different distribution to the original w space. How to visualize such change and then demonstrate the swap of learning representation would be significant. \n\n[r1] Wu, Z., Lischinski, D., & Shechtman, E. (2021). Stylespace analysis: Disentangled controls for stylegan image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 12863-12872).\n[r2] Abdal, R., Zhu, P., Mitra, N. J., & Wonka, P. (2021). Styleflow: Attribute-conditioned exploration of stylegan-generated images using conditional continuous normalizing flows. ACM Transactions on Graphics (TOG), 40(3), 1-21.\n[r3] Song, G., Luo, L., Liu, J., Ma, W. C., Lai, C., Zheng, C., & Cham, T. J. (2021). AgileGAN: stylizing portraits by inversion-consistent transfer learning. ACM Transactions on Graphics (TOG), 40(4), 1-13.", "summary_of_the_paper": "In this work, the authors propose a framework for one-shot generation, which aims to tame a new GAN using a pre-trained GAN model as well as one target image. Here, the authors present three strategies to ensure the quality and diversity of generated images: introducing an attribute adaptor to map the latent space to the target attribute; presenting an attribute classifier in the discriminator, and applying the truncation trick for the training. Experiments are conducted on the face and building datasets. ", "summary_of_the_review": "My preliminary rating for this submission is marginal reject. While I like this novel task that aims to generate diverse images with high quality, the key novelty and contribution are unclear explained to the prior related works. In addition, interesting experimental results are provided, but they are not well analyzed in the representation domain. The realistically visual results cannot demonstrate the contribution perfectly. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work was the subject of significant back and forth (between authors and reviewers, but also between reviewers & myself) due to the wide range of opinions. Two of the reviewers have found this work below the bar: they have provided multiple reasonings that I would rather not repeat here. The third reviewer found this work more compelling and argued for its acceptance. My attempts at reaching a consensus have yielding the following conclusions:

  * There's agreement that one-shot generation is indeed a challenging task
  * Some of the results are indeed impressive, but many results are not compelling.
  * The rebuttal addressed some of the concerns (e.g. visualization of latents), but some issues are unaddressed (e.g. more motivation, explanation of why the proposed method works better)
  * One of the reviewers has argued rather forcefully that the work doesn't quite do domain adaptation in the typically understood sense. Moving beyond definitions of domain adaptation, the same reviewer was not very convinced by the quality of the results themselves.
  * The reviewer most positive about this work agrees that this work only explores a limited form of domain transfer. They argued that some of the potential applications of this work do make the submission interesting. 

Fundamentally, the discussion did not necessarily resolve the differences in opinion one way or another. Ultimately, all 3 reviewers believe that it would fine if this work was not accepted to ICLR at this time, despite some of the interesting results and promise. Given the discussion and this mildest consensus, I am inclined to recommend rejection too. I do think there's a substantial amount of constructive feedback in the reviews that would make a subsequent revision of this work quite a bit better.
Decision: Reject