One-Shot Generative Domain Adaptation

1 INTRODUCTION . Generative Adversarial Network ( GAN ) ( Goodfellow et al. , 2014 ) , consisting of a generator and a discriminator , has significantly advanced image synthesis yet relies on a large number of training samples ( Karras et al. , 2018 ; 2019 ; 2020b ; Brock et al. , 2018 ) . Many attempts have been made to train GANs from scratch with limited data ( Zhang & Khoreva , 2019 ; Zhao et al. , 2020d ; b ; Karras et al. , 2020a ; Yang et al. , 2021a ) , but it still requires hundreds or thousands of images to get a satisfying performance . Sometimes , however , we may have as few as only one single image as the reference , like the masterpiece Mona Lisa from Leonardo da Vinci . Under such a case , learning a generative model with both good quality and high diversity becomes extremely challenging . Domain adaptation is a commonly used technique that can apply an algorithm developed on one data domain to another ( Csurka , 2017 ) . Prior works ( Wang et al. , 2018 ; Noguchi & Harada , 2019 ; Wang et al. , 2020 ; Mo et al. , 2020 ; Zhao et al. , 2020a ; Li et al. , 2020 ; Robb et al. , 2020 ) have introduced this technique to GAN training to alleviate the tough requirement on the data scale . Typically , they first train a large-scale model in the source domain with adequate data , and then transfer it to the target domain with only a few samples . A common practice is to fine-tune both the generator and the discriminator on the target dataset until the generator produces samples conforming to the target domain . To stabilize the fine-tuning process and improve the generation quality and diversity , existing approaches propose to tune partial parameters ( Noguchi & Harada , 2019 ; Mo et al. , 2020 ; Robb et al. , 2020 ) and introduce some regularizers ( Li et al. , 2020 ; Ojha et al. , 2021 ) , but the overall adaptation strategy remains . When there is only one image from the target domain , these methods would fell short of synthesis diversity , producing very similar images . Recall that the pre-trained model can already produce highly diverse images from the source domain . It makes us wonder what indeed causes the diversity drop in the adaptation process . We argue that directly tuning the model weights will result in the loss of the prior knowledge gained from the large-scale data . On the other hand , however , when adapting the model to the target domain , most variation factors ( e.g. , gender and pose of human faces ) could be reused . These observations help raise a question : is it possible to simply focus on the most representative characters of the reference image while inheriting all the other knowledge from the source model ? In this work , we develop a method , called GenDA , for one-shot Generative Domain Adaptation . In particular , we design a lightweight module connecting the latent space and the synthesis network . We call this module an attribute adaptor since it helps adapt the generator with the attributes of the target image . Unlike the conventional fine-tuning strategy , we freeze the parameters of the original generator and merely optimize the attribute adaptor during training . Thereby , we manage to reuse the prior knowledge learned by the source model and hence inherit the synthesis quality and , more importantly , diversity . Meanwhile , we employ the discriminator to compete with the generator via a domain-specific attribute classification . In this way , the generator is forced to capture the most representative attributes from the reference , or otherwise , the discriminator would spot the discrepancy . However , instead of directly tuning the original discriminator , we freeze its entire backbone and introduce a lightweight attribute classifier on top of that . Similar to the generator , the discriminator has also learned rich knowledge in its pre-training . Since the synthesized images before and after adaptation share most visual concepts ( e.g. , a face model would still produce faces after domain transfer ) , the discriminator can be reused as a well-learned feature extractor . Therefore , we simply train the attribute classifier to help guide the generator . Furthermore , since there is only one training sample ( which means no diversity in the target domain ) , we propose to also constrain the diversity of the generative domain by truncating the latent distribution during training . Intuitively , learning a one-to-one mapping would be easier than learning a many-to-one mapping . Such a design mitigates the optimization difficulty and further improves the synthesis quality . We evaluate our approach through extensive experiments on human faces and outdoor churches . Given only one training image , GenDA can convincingly adapt the source model to the target domain with sufficiently high quality and diversity . Such an adaptation includes both attribute-level and style-level , as shown in Fig . 1 . Our method outperforms the state-of-the-art competitors by a large margin both qualitatively and quantitatively . We also show that , when the number of the samples available in the target domain increases , GenDA can filter out the individual attributes and only preserve their common characters ( see Fig . 4 ) . Noticeably , GenDA works well for the extreme cases where there is a large domain gap , like transferring the characters of Mona Lisa to churches ( see Fig . 5 ) . Besides , thanks to the lightweight design of both the attribute adaptor and the attribute classifier , GenDA can finish each adaptation experiment within a few minutes . 2 METHODOLOGY . The primary goal of this work is to transfer a pre-trained GAN to synthesize images conforming to a new domain with as few as only one reference image . Due to the limited supervision , it is challenging to ensure both high quality and large diversity of the synthesis . Intuitively , according to the rationale of GANs ( i.e. , adversarial training between the generator and the discriminator ) , the discriminator can easily memorize that only the reference image is real while all others are fake . In this way , to fool the discriminator , the generator may have to learn to produce images highly alike the reference , resulting in a poor synthesis diversity . To mitigate this problem , we propose a new adaptation algorithm , different from the previous fine-tuning scheme . Concretely , we ( i ) interpose an attribute adaptor between the latent space and the generator to help acquire the most representative characters from the target image ; ( ii ) augment the discriminator backbone with an attribute classifier to guide the generator to make appropriate adjustments ; and ( iii ) propose a diversity-constraint training strategy . Before going into technical details , we first give some preliminaries of GANs . 2.1 PRELIMINARIES . Generative Adversarial Network ( GAN ) ( Goodfellow et al. , 2014 ) is formulated as a two-player game between a generator and a discriminator . Given a collection of observed data { xi } Ni=1 with N samples , the generator G ( · ) aims at reproducing the real distribution X via randomly sampling latent codes z subject to a pre-defined latent distribution Z . As for the discriminator D ( · ) , it targets at differentiating the real data x and the synthesized data G ( z ) as a bi-classification task . These two models are jointly optimized by competing with each other , as LG = −Ez∈Z [ log ( D ( G ( z ) ) ) ] , ( 1 ) LD = −Ex∈X [ log ( D ( x ) ) ] − Ez∈Z [ log ( 1−D ( G ( z ) ) ) ] . ( 2 ) After the training converges , the generator is expected to produce images as realistic as the training set , so that the discriminator can not distinguish them anymore . In this work , we start with a GAN model that is well trained on a source domain X src , and work on adapting it to a target domain X dst = { xdst } that has only one image . In fact , it is ambiguous to define a “ domain ” using one image . We hence expect the model to acquire the most representative characters from the reference image . Taking face synthesis as an example , the character can include facial attributes ( e.g. , age or wearing sunglasses ) and artistic styles , as shown in Fig . 1 . 2.2 ONE-SHOT GENERATIVE DOMAIN ADAPTATION . A common practice to transfer GANs is to simultaneously tune the generator and the discriminator on the target dataset ( Wang et al. , 2018 ) . However , as discussed above , the transferring difficulty increases drastically when given only one training sample . Existing methods attempt to address this issue by reducing the number of learnable parameters ( Mo et al. , 2020 ; Robb et al. , 2020 ) and introducing training regularizers ( Ojha et al. , 2021 ) . Even so , the overall fine-tuning scheme ( i.e. , directly tuning G ( · ) and D ( · ) ) remains and the diversity is low . Differently , we propose a new adaptation pipeline to preserve the synthesis diversity , which includes an attribute adaptor , an attribute classifier , and a diversity-constraint training strategy . Details are introduced as follows . Attribute Adaptor . Prior works have found that , a well-learned generator is able to encode rich semantics to produce diverse images ( Shen et al. , 2020 ; Yang et al. , 2021b ) . For instance , a face synthesis model could capture the variation factors like gender , age , wearing glasses , lighting , etc . Ideally , this knowledge should be sufficiently reused as much as possible in the target domain , and then the synthesis diversity can be preserved accordingly . In this way , the generator focuses on transferring the most distinguishable characters of the only reference , instead of learning the common variation factors repeatedly and leading to overfitting due to lack of samples . This helps improve the data efficiency significantly , which is vital to the one-shot setting . According to Xu et al . ( 2021 ) , the latent code z can be viewed as the generative feature of G ( z ) that determines the multi-level attributes of the output image . Motivated by this , we propose to adapt such features regarding the reference image , yet keep the convolutional kernels untouched . Concretely , before feeding the latent code z to the generator G ( · ) . we propose to first transform it through a lightweight attribute adaptor A ( · ) , as z′ = A ( z ) = a z + b , ( 3 ) where stands for the element-wise multiplication , while a and b are the learnable weight and bias respectively . With such a design , the transformed latent code z′ is assumed to carry the sufficient information of the reference , and therefore G ( z′ ) would conform to the target domain X dst . Attribute Classifier . Only having the attribute adaptor can not guarantee the generator to acquire the representative characters from the training sample . Following the formulation of GANs , we incorporate the discriminator D ( · ) , which is also pre-trained on the source domain , to compete with the generator . In particular , we reuse the backbone d ( · ) but remove the last real/fake classification head , and then equip it with a lightweight attribute classifier φ ( · ) . Given an image x , either the reference image xdst or a transferred synthesis G ( A ( z ) ) , the classifier outputs a probability of how likely it possesses the target attribute , as p = φ ( d ( x ) ) . ( 4 ) However , due to the limited supervision provided by one image , the discriminator can easily memorize the real data , which leads to the overfitting of discriminator as well as the collapse of the generator ( Karras et al. , 2020a ) . As discussed above , the generated images before and after domain adaptation are expected to share most variation factors ( i.e. , a face model remains to produce faces after adaptation ) . From this viewpoint , the knowledge learned by the discriminator in its pretraining could be also reused . Therefore , unlike existing approaches that fine-tune all or partial parameters of D ( · ) ( Mo et al. , 2020 ; Wang et al. , 2020 ; Li et al. , 2020 ; Ojha et al. , 2021 ) , we freeze all parameters of d ( · ) in the entire training process and merely optimizes φ ( · ) to guide A ( · ) with adequate adjustments . Regarding the one-shot target domain , the mechanism behind the classifier is very similar to Exemplar SVM ( Malisiewicz et al. , 2011 ) , which also suggests that it is sufficient to obtain a good decision boundary with one positive and many negative samples . Differently , our attribute classifier is learned in an adversarial manner through competing with the attribute adaptor . Diversity-constraint Strategy . Recall that this work targets at generative domain adaptation with only one reference image , which means no diversity of real data . On the contrary , however , the latent code can be sampled randomly and the pre-trained generator can produce highly diverse images from the source domain . From this perspective , it might be challenging to match these two distributions with such a huge diversity gap . To alleviate the optimization difficulty , we propose a diversityconstraint strategy , which retains the diversity of the generator during training . Specifically , we truncate the latent distribution with a strength factor β , as z′ = A ( βz + ( 1− β ) z̄ ) , ( 5 ) where z̄ indicates the mean code . Note that , truncation is a common trick used in the inference of state-of-the-art GANs , like StyleGAN ( Karras et al. , 2019 ) and BigGAN ( Brock et al. , 2018 ) , to improve synthesis quality . Nevertheless , to our best knowledge , this is the first time that truncation is introduced in the training process to preserve the synthesis diversity . Full Objective Function . In summary , the adaptor A ( · ) and the classifier φ ( · ) are trained with LA = −Ez∈Z [ log ( φ ( d ( G ( z′ ) ) ) ) ] , ( 6 ) Lφ = −Ex∈X src [ log ( φ ( d ( x ) ) ) ] − Ez∈Z [ log ( 1− φ ( d ( G ( z′ ) ) ) ) ] . ( 7 )

Summary: The paper proposes to do one-shot domain adaption by fine tuning a state of the art network StyleGAN. The paper introduces a few techniques to do so. First, the paper proposes to add a lightweight attribute adaption layer in the beginning of StyleGAN to modify the latent code. An existing latent code can be modified to give a new latent code that is in the new domain. Second, the paper proposes a novel type of discriminator reuse. A discriminator has its features frozen, but a new classification head is added. The paper shows various visual results.