Title: Domain Re-Modulation for Few-Shot Generative Domain Adaptation
Venue: NeurIPS 2023
Accepted: yes
Keywords: StyleGAN

Few-Shot Generative Domain Adaptation
Abstract: In this study, we delve into the task of few-shot Generative Domain Adaptation (GDA), which involves transferring a pre-trained generator from one domain to a new domain using only a few reference images. Inspired by the way human brains acquire knowledge in new domains, we present an innovative generator structure called $\textbf{Domain Re-Modulation (DoRM)}$. DoRM not only meets the criteria of $\textit{high quality}$, $\textit{large synthesis diversity}$, and $\textit{cross-domain consistency}$, which were achieved by previous research in GDA, but also incorporates $\textit{memory}$ and $\textit{domain association}$, akin to how human brains operate. Specifically, DoRM freezes the source generator and introduces new mapping and affine modules (M\&A modules) to capture the attributes of the target domain during GDA. This process resembles the formation of new synapses in human brains. Consequently, a linearly combinable domain shift occurs in the style space. By incorporating multiple new M\&A modules, the generator gains the capability to perform high-fidelity multi-domain and hybrid-domain generation. Moreover, to maintain cross-domain consistency more effectively, we introduce a similarity-based structure loss. This loss aligns the auto-correlation map of the target image with its corresponding auto-correlation map of the source image during training. Through extensive experiments, we demonstrate the superior performance of our DoRM and similarity-based structure loss in few-shot GDA, both quantitatively and qualitatively. Code will be available at https://github.com/wuyi2020/DoRM.

Domain Re-Modulation for Few-Shot Generative Domain Adaptation

Abstract

In this study, we delve into the task of few-shot Generative Domain Adaptation (GDA), which involves transferring a pre-trained generator from one domain to a new domain using only a few reference images. Inspired by the way human brains acquire knowledge in new domains, we present an innovative generator structure called Domain Re-Modulation (DoRM) . DoRM not only meets the criteria of high quality , large synthesis diversity , and cross-domain consistency , which were achieved by previous research in GDA, but also incorporates memory and domain association , akin to how human brains operate. Specifically, DoRM freezes the source generator and introduces new mapping and affine modules (M&A modules) to capture the attributes of the target domain during GDA. This process resembles the formation of new synapses in human brains. Consequently, a linearly combinable domain shift occurs in the style space. By incorporating multiple new M&A modules, the generator gains the capability to perform high-fidelity multi-domain and hybrid-domain generation. Moreover, to maintain cross-domain consistency more effectively, we introduce a similarity-based structure loss. This loss aligns the auto-correlation map of the target image with its corresponding auto-correlation map of the source image during training. Through extensive experiments, we demonstrate the superior performance of our DoRM and similaritybased structure loss in few-shot GDA, both quantitatively and qualitatively. Code will be available at https://github.com/wuyi2020/DoRM.

1 Introduction

Domain adaptation aims to bridge the domain gap and transfer knowledge to mitigate the limitations imposed by a lack of extensive labeled data [8; 40; 51]. Recent research has explored the field of few-shot Generative Domain Adaptation (GDA), which aims to achieve realistic and diverse generation with only a few training images [29; 32; 34; 43; 47; 59; 31; 42; 23; 62]. Particularly, the

JD Explore Academy

Latent

Source

Mapping

Source

Baby

Mapping

Baby

Sunglasses

Mapping

Sunglasses

Sketch

Mapping

Sketch

Multi-domain Generation

Hybrid-domain Generation

+

Synthesis

Network

=

Fixed

Colorful

=

Trainable

purpose of few-shot GDA 4 is to transfer a generator pre-trained on the source domain to the target domain using a few reference images.

Existing few-shot GDA works primarily focus on three properties: (i) High quality and (ii) Large diversity , where the adapted generator can synthesize high quality and diverse images in the target domain and (iii) Cross-domain consistency , where the adapted images and their corresponding source images should be consistent in terms of domain-sharing attributes, such as pose and identity. A majority of them update the entire generator with regularization techniques including GAN inversion [55; 47; 63], Contrastive-Language-Image-Pretraining (CLIP) [10; 55; 63], and consistency loss [32; 47; 59] to achieve large diversity and cross-domain consistency of adaptation. However, updating the entire generator imposes limitations on its ability to synthesize multiple target domains. In contrast, the human brain employs a more efficient approach to learn from new domains by forming new proteins[4], allowing for the retention and integration of knowledge from multiple domains. This creative capacity enables humans to explore previously unseen domains.

Drawing inspiration from human learning processes, we propose a novel generator structure called Domain Re-Modulation (DoRM) based on StyleGAN2 [19]. DoRM not only fulfills three essential properties of few-shot GDA but also introduces two new capabilities: (iv) Memory , enabling the generator to retain knowledge from previously learned domains when generating images in new domains, and (v) Domain Association , allowing the generator to integrate multiple learned domains and synthesize hybrid domains not encountered during training. As depicted in Figure 1, DoRM freezes the pre-trained source generator and introduces new mapping and affine modules (M&A modules) to achieve the necessary domain shift in the style space. This approach enables DoRM to perform high-fidelity generation across multiple domains by selectively activating different M&A modules, which significantly reduces the required storage space. Moreover, as StyleGAN's style space is a linear subspace [36; 24; 45], the domain shift obtained through DoRM is linearly combinable. Consequently, DoRM can synthesize images in hybrid domains not present in the training dataset by simultaneously activating multiple M&A modules. To further enhance cross-domain consistency, we introduce a similarity-based structure loss denoted as L ss . Specifically, we leverage the CLIP image

encoder [33] to extract intermediate tokens from the target image and its corresponding source image. We then enforce consistency between the auto-correlation maps of these tokens, ensuring improved alignment between target and source images.

We conducted extensive experiments on few-shot GDA involving various source and target domains. Our experiments, both quantitative and qualitative, highlight the competitive performance of our method compared to state-of-the-art approaches in 10-shot GDA. We achieved remarkable results in terms of quality, diversity, and cross-domain consistency. Notably, our proposed DoRM stands out by demonstrating exceptional performance in hybrid-domain generation, a hard task that previous methods have not accomplished well. The key contributions of this work can be summarized as follows:

We introduce DoRM, a novel generator structure for few-shot generative domain adaptation, inspired by the learning mechanism of human brains. DoRM not only excels in synthesizing high-quality, diverse, and cross-domain consistent images, but also integrating memory and domain association capabilities that remain relatively unexplored in the field. Notably, our approach is one of the very few that encompasses all five desired properties of GDA.

Additionally, we propose a novel consistency loss called similarity-based structure loss, which further enhances cross-domain consistency in our approach.

Through comprehensive evaluations, our proposed method outperforms existing competitors across various settings, showcasing its superior performance.

2 Related Work

2.1 Generative Adversarial Networks

Deep learning has made remarkable achievements in various fields [46; 44; 22]. Among them, generative adversarial networks [11; 37; 28] play a two-player adversarial game, where the generator aims to synthesize realistic images to fool the discriminator, while the discriminator learns how to distinguish the synthesized images from real ones. Previous works [16; 2; 53; 18; 19] have significantly improved the synthesis capability on high-resolution datasets. Style-based methods [18; 19; 35] remain the state-of-the-art unconditional GANs owing to their unique architecture. However, GANs training requires a large number of training images, or severe discriminator overfitting can occur [26; 27]. To alleviate this issue and improve training stability, various data augmentation techniques [57; 15; 17; 61; 60; 25] and regularization technologies [26; 48; 39; 14] have been introduced. Among these, Adaptive Discriminator Augmentation (ADA) is an adaptive strategy that controls the strength of augmentations and has shown remarkable performance, becoming the default operation in data-efficient GANs. Tseng et al. [39] proposed a regularization scheme to modulate the discriminator's prediction, mitigating discriminator overfitting. However, most of these techniques fail in the extremely small (e.g., 10) training data regime.

2.2 Few-shot Generative Domain Adaptation

Previous works in few-shot GDA mainly have primarily focused on three essential properties: High quality , Large diversity , and Cross-domain consistency . Some studies have fine-tuned the entire generator using regularization techniques. For example, [32; 47] introduced a consistency loss based on Kullback-Leibler (KL) divergence to preserve the relative similarities and differences between instances, inheriting diversity and maintaining consistency during adaptation. The contrastive term [12; 5] has also been employed to construct consistency loss in [59]. In addition, leveraging the semantic power of large Contrastive-Language-Image-Pretraining (CLIP) [33] models, [10; 55; 63] proposed to define the domain-gap direction in CLIP embedding space, guiding the optimization of the generator. Furthermore, GAN inversion [7; 38] technique has been widely used in few-shot GDA to meet different purposes, such as exploring the domain-sharing attributes [55; 63], and compressing the latent space to a subspace to relax the cross-domain alignment [47]. Compared to optimizing the entire generator, some studies [31] only update the crucial part of the generator. Moreover, some work [49] adds a lightweight attribute adaptor and attribute classifier before the frozen generator and after the frozen discriminator, respectively. The proposed method achieves remarkable performance in synthesis quality and diversity but lacks cross-domain consistency.

Latent

𝑧 ∈ 𝒵

Normalize

FC

FC

FC

FC

FC

FC

FC

FC

FC

FC

FC

FC

FC

FC

FC

FC

𝒲𝑡

𝒲𝑠

Target mapping

𝑓 𝑡

Source mapping

𝑓 𝑠

Mod

Conv 3x3

𝑐1

𝑤2

𝐴s

𝐴𝑡

+

1 - 𝛼

𝛼

+

b2

B

Mod

Upsample

Conv 3x3

+

b3

B

𝑤3

Mod

Conv 3x3

𝑤4

+

𝐴s

𝐴𝑡

1 - 𝛼

𝛼

+

b4

B

+

𝐴s

𝐴𝑡

1 - 𝛼

𝛼

…

noise

= fixed

= trainable

Synthesis network

𝑔

Source affine

Target affine

Source affine

Target affine

Source affine

Target affine

Furthermore, [20; 1] share the similar idea with us of incorporating multi-domain generation capabilities into a single generator. However, these methods employ multiplicative modulation, which is indeed different with ours. Specifically, HyperDomainNet [1] adopts a solitary modulation parameter, primarily a scale ( δ ), to adjust the weights of the convolutional layer, influencing the "s" space of StyleGAN. In this setup, the scale parameter remains constant for all images within a target domain. Formally, HyperDomainNet's architecture is represented as w · s i · δ , where w and s i denote the convolutional weight and style code of the source StyleGAN2, respectively. However, the introduction of this all-encompassing scale parameter can potentially restrict the generator's learning capacity. Especially in scenarios with significant domain gaps between target and source domains, HyperDomainNet's performance may decline. Furthermore, the non-linear combinability of the scale modulation parameter impedes its ability to achieve robust domain association. Among the approaches closest to our DoRM, DynaGAN [20] introduces two modulation parameters-shift ( ∆ s ) and scale ( δ )-to the convolutional weights affecting StyleGAN's "s" space. DynaGAN's architecture can be defined as w · ( s i +∆ s ) · δ . Similar to HyperDomainNet, the non-linear combinability of the scale modulation parameter impedes its ability to achieve robust domain association. In contrast, our DoRM solely embraces a sample-specific domain shift denoted as ∆ s i , formalized as w · ( s i +∆ s i ) . This distinct design substantially enhances the generator's learning potential, enabling adaptation to a diverse spectrum of target domains, even in the presence of considerable domain gaps. Notably, the sample-specific domain shift proves sufficient for both few-shot and one-shot GDA, eliminating the necessity for an additional domain scale parameter. This streamlined approach not only fosters domain association but has been effectively demonstrated. Additionally, although [30] defines and addresses a related task called Domain Expansion and domain composition (referred to as memory and domain association in our paper), it inadvertently curtails the source domain's generative capacity, amplifying the intricacies and temporal requisites of domain adaptation. As depicted in the experimental results, [30] encounters challenges when faced with substantial domain gaps between source and target domains. Evidently, its performance is compromised, as evidenced in instances such as adapting to the 10-shot generation context of the Sketch dataset and the one-shot generation context of the "elsa" dataset.

3 Approach

In this section, we present our generator structure, DoRM (Section 3.1), which takes inspiration from the way human brains learn new knowledge in different domains. DoRM excels in generating high-quality, diverse images while maintaining strong cross-domain consistency in few-shot GDA. Additionally, it possesses the unique ability to integrate multiple learned domains, allowing for the synthesis of images in hybrid domains that were not present in the training dataset. To address the issue of discriminator overfitting, we freeze the backbone of the source discriminator and introduce a target domain classifier (Section 3.2). Furthermore, we propose a novel similarity-based structure loss, denoted as L ss (Section 3.3), to enhance the maintenance of cross-domain consistency. Finally, we introduce the overall training loss in Section 3.4.

3.1 Domain Re-modulation Structure of Generator

StyleGAN2. Our method is applied to a pre-trained StyleGAN2 [19]. Unlike general GANs that directly feed latent code to the generator, StyleGAN2 [19] employs a non-linear mapping network f ( · ) to transform the latent space Z into an intermediate latent space W . The latent code ( w ∈ W ) is transformed into the style code ( s ∈ S ) through a learned affine transformation A . Finally, the style code is inserted into the synthesis network g ( · ) through the modulation component at each convolution layer.

Domain Re-modulation of Generator. To equip the generator with memory and inherit the semantic information ( e.g. , glasses, gender, ages) from the source domain, we propose DoRM, illustrated in Figure 2. We freeze the given StyleGAN2 model pre-trained on the source domain and employ a new mapping network f t and new affine transformation A t (M&A module) to build W t space and S t space, respectively. Through the new M&A module, we obtain domain shift as well as the target style codes. The added f t and A t have the same architecture as f s and A s , respectively, and are initialized by f s and A s . Accordingly, each layer of the synthesis network is controlled by source and target style codes. Given a latent code z sampled from Normal distribution Z ( z ∈ Z ),

where W t and W s contain information from the target domain and source domain, respectively. Transformed by their own learned affine layers (Eq. 2), the information in these two domains represents their respective styles and is combined into general styles s (Eq. 3) to control the synthesis network together:

where α is a hyper-parameter that controls the strength of the domain shift. In GDA, it tends to reuse various factors like pose, content, structure, etc. from the source domain and learns the most distinguishable characteristics of the target domain. To preserve variation factors as much as possible in the source domain, α is set to a relatively small value (More analyses can be found in Section A.1). The combined style code s modulates the convolution weights through w ′ i = s · w i , where w ′ i is the modulated weights. Then, demodulation is employed to eliminate the influence of the style code s from the statistics of the convolution's output feature maps, which has been formalized as w ′′ i = w ′ i ∥ w ′ ∥ 2 , where ∥ w ′ ∥ 2 represents the L2-Norm function.

3.2 Target Domain Classifier

In GANs training, the discriminator typically distinguishes the training images from the generated images. However, in few-shot GDA, where there are extremely few training images, the discriminator can easily overfit to the training images, leading to severe model collapse in the generator. In this work, we treat the discriminator as a target domain classifier that measures the probability of the images belonging to the target domain. Defining the target domain by only a few training images is ambitious, so we desire the discriminator to extract the most representative feature of training images that portray the target domain. Inspired by [29], we reuse and fix the feature extractor d ( · ) of the pre-trained source discriminator to inherit its strong feature extraction capability. Additionally, we introduce a target domain classifier ϕ ( · ) on the top of the feature extractor d ( · ) . We use a two-layer multi-layer perceptron (MLP) as the target domain classifier ϕ ( · ) and the target domain classifier

ϕ ( · ) , and it is updated from scratch, which outperforms directly fine-tuning the final layer of the original discriminator, especially for large domain-gap GDA. Given an image x , the discriminator measures the probability that the image belongs to the target domain using p = ϕ ( d ( x )) .

3.3 Similarity-based Structure Loss

To explicitly model the cross-domain consistency in generative domain adaptation, we propose a novel similarity-based structure loss called L ss . Our intuition is that the auto-correlation maps of the source image and its corresponding target image should be consistent during generative domain adaptation. To achieve this, we extract the intermediate tokens F A and F B of the source image I A and its corresponding target image I B from the k-th layer pf the CLIP image encoder. These tokens are denoted by F A = F 1 A , · · · , F n A and F B = F 1 B , · · · , F n B , respectively. We define the auto-correlation maps as M A = F T A | F T A | × F A | F A | and M B = F T B | F T B | × F B | F B | , where M i,j A = F i A · F j A | F i A || F j A | and

M i,j B = F i B · F j B | F i B || F j B | . The L ss is then defined as the L2 norm of the difference between M A and M B :

where || · || is the L2 norm function.

3.4 Overall Training Loss

Our overall training loss includes original adversarial training loss in StyleGAN2 and the similaritybased structure loss L ss :

where X t is the training dataset. In our experiments, we set λ ss = 10 .

4 Experiments

We begin by introducing the experimental settings of our method, which encompass implementation, datasets, and metrics (Section 4.1). Next, we apply our method to various 10-shot datasets, showcasing its performance in Section 4.2. Section 4.3 explores the capabilities of domain association among multiple domains. Furthermore, we conduct ablation studies (Section 4.4 and Section A.1) to assess the impact of both the similarity-based structure loss and the generator structure. Additionally, we provide the user study in Section A.4 and demonstrate the applicability of our DoRM to one-shot GDA in Section A.3.

4.1 Experiments Settings

Implementation. We adopt the pre-trained StyleGAN2 [19] on FFHQ [18] as our source model, and our training parameters and settings follow StyleGAN2-ADA [17]. Due to the small amount of training data, we set the batch size to 4, and we terminate the training process after the discriminator has processed 100K real samples. Our implementation is based on the official implementation of StyleGAN2-ADA.

Datasets. Following previous literature [32; 59], we use FFHQ [18] with resolution 256 × 256 as the source domain. In 10-shot GDA, we evaluate our method on multiple target datasets, including Sketches [41], FFHQ-Babies [18], FFHQ-Sunglasses [18], Face-Caricatures, Face paintings by Amedeo Modigliani, Face paintings by Raphael, and Face paintings by Otto Dix [50]. All the training datasets are shown in Section A.2.

Metrics. Following [59; 32], we use Fréchet Inception Distance (FID) [13] as the metric to evaluate the synthesis quality and diversity simultaneously. Additionally, we adopt intra-cluster LPIPS (intraLPIPS) [59; 32] based on LPIPS [54] to measure the synthesis diversity. Specifically, we synthesize 1000 images and then assign each of the synthesized images to the k training images with the lowest

LPIPS distance, forming k clusters. We calculate the average LPIPS distance within each cluster and then average over all the clusters. Furthermore, we report the identity (ID) similarity [56] predicted by the Arcface [9] to measure the preservation of identity information, which is a metric to measure the cross-domain consistency.

4.2 10-shot Generative Domain Adaptation

Qualitative comparison. In 10-shot GDA, following the previous studies [32], we sample 10 training images from the target domain to transfer the pre-trained generator. Figure 3 shows the results of 10-shot GDA with different methods. We observe severe generator overfitting in the FreezeD [29]. GenDA [49] shows high quality and diverse synthesis but fails to maintain cross-domain consistency. CDC [32] and RSSA [47] improve the cross-domain consistency, but the synthesis quality is unsatisfactory. AdAM [58] also increases the synthesis diversity but fails in synthesis quality. By contrast, our method preserves all the domain-sharing attributes from the source domain by freezing the original generator and acquires the domain shift through new concurrent network components. Our method shows appealing synthesis quality and diversity while maintaining better cross-domain consistency than previous methods. More qualitative results can be found in Section A.2, Section A.5, and Section A.6.

Quantitative comparison. We use Fréchet Inception Distance (FID) to evaluate synthesis quality and diversity (lower is better). In 10-shot GDA, to better reflect the synthesis diversity , we use the Intra-LPIPS[32] to measure the synthesis diversity (higher means better synthesis diversity) . Additionally, we measure cross-domain consistency using Identify similarity (ID) [56] (higher is better). Table 1 summarizes the three evaluation metrics for 10-shot GDA over different target domains. FreezeD [29] and GenDA [49] struggle to maintain cross-domain consistency. CDC [32], RSSA [47], DCL [59] and AdAM [58] improve the synthesis diversity and cross-domain consistency but fails in the synthesis quality (as seen in Figure 3), leading to unsatisfactory FID scores. Our method outperforms all these methods on the different target domains, achieving not only better synthesis diversity and cross-domain consistency (higher Identity score and Intra-LPIPS) but also better synthesis quality (lower FID score).

4.3 Multi-domain and Hybrid-domain Generation

Multi-domain generation. Our proposed DoRM utilizes a novel approach by learning and retaining knowledge of new domains through the formation and update of new M&A modules, instead of updating the entire generator. This unique characteristic enables our DoRM, with a single generator, to generate images across multiple domains (as depicted in the top part of Figure 4). In contrast to previous methods [32; 47; 59; 58] that require updating the entire generator for few-shot GDA, limiting its capability into single domain generation, our DoRM offers significant storage space savings in multi-domain generation. As shown in Table 2, our method's models are about 3 × smaller than previous methods [32] on 10-domain generation.

Hybrid-domain generation. In contrast to previous methods [32; 59; 55] that fine-tune the entire generator, our DoRM achieves an effective linear domain shift through the update of new M&A

Real

babies

Source

AdAM

GenDA

FreezeD

CDC

RSSA

DoRM

Source

Real

sketches

AdAM

RSSA

CDC

FreezeD

GenDA

DoRM

modules. By learning multiple M&A modules that store the knowledge of various training domains, DoRM not only enables multi-domain generation but also facilitates the synthesis of images in hybrid domains not present in the training data (as depicted in the bottom left part of Figure 4). For comparison, we conduct hybrid-domain generation experiments using CDC [32]. In CDC, the domain shift between the source and target domain generators is obtained by subtracting their corresponding parameters. Similar to our DoRM, CDC achieves hybrid-domain generation by combining multiple domain shifts, adding the parameters of each shift to the source domain generator. However, as illustrated in Figure 4, CDC exhibits unsatisfactory synthesis quality in hybrid-domain generation. In contrast, the images synthesized by DoRM not only inherit domain-sharing attributes such as pose, gender, and identity, but also seamlessly blend domain-specific attributes. Furthermore, we extensively explore domain association among multiple domains using our proposed DoRM in Figure 5. The results demonstrate that DoRM efficiently integrates diverse domains to generate high-quality

Multi-domain generation (Ours)

images in hybrid domains not present in the training data. In this case, our proposed DoRM has the ability to generate creative outputs in previously unseen domains by incorporating knowledge learned from other domains, similar to how humans can draw on past experiences to generate novel ideas. To the best of our knowledge, our proposed DoRM is the first method to achieve efficient domain association in this manner . Additional visualizations of hybrid domain generation can be found in Section A.6.

4.4 Ablation Study

We conduct ablation studies to evaluate the impact of our proposed DoRM generator structure and similarity-based structure loss. As shown in Figure 6, the images synthesized by baseline (which simply fine-tunes the StyleGAN2 generator with adversarial loss [18]) exhibit severe overfitting to the training images. By contrast, fine-tuning our DoRM structure generator with adversarial loss yields better synthesis diversity than the baseline. Moreover, when we optimize our DoRM generator structure with both adversarial loss and similarity-based structure loss, the synthesis images exhibit appealing quality and diversity, while maintain high cross-domain consistency, which indicates the proposed loss effectively preserves cross-domain consistency in few-shot GDA. More ablation studies on the network component of our DoRM are presented in Section A.1.

5 Limitation and Conclusion

Limitation. i) Although our method can achieve appealing results on the few-shot GDA across different target domains, however, the strength of domain shift α currently needs to be manually adjusted. Furthermore, since the various depth layers capture different semantic attributes, equipping each layer with an appropriate α may help to further improve synthesis quality. ii) The current

Baby

Sunglasses

Sketch

Caricature

Baby-Sketch

Baby-Caricature

Baby-Otto Dix

Sunglasses-Sketch

Sunglasses-Caricature Sunglasses-Otto Dix

Source domain

DoRM

DoRM +

Baseline

manuscripts only simply combines the M&A modules of different target domains and activate them at the same time to realize the domain association. To further improve the performance of the domain association, not only combining the trained target M&A modules but also employing a new M&A module and additional consistency loss is a better method to blend the target domains.

Conclusion. In this paper, we present DoRM, a novel generator structure inspired by the learning and storage mechanisms of the human brain, specifically designed for few-shot GDA. Our DoRM is characterized by its simplicity and efficiency. Additionally, we introduce a novel similarity-based structure loss to ensure cross-domain consistency in the few-shot GDA. Through extensive qualitative and quantitative evaluations, we demonstrate the superiority of our method over existing approaches in terms of synthesis quality, diversity, and cross-domain consistency. Importantly, akin to the human brain, our DoRM exhibits memory and the ability to integrate knowledge from different domains, enabling the generation of images in novel hybrid domains not encountered during training.

6 Acknowledgments

The work was supported in part by the National Natural Science Foundation of China under Grands U19B2044 and 61836011.

References

Aibek Alanov, Vadim Titov, and Dmitry Vetrov. Hyperdomainnet: Universal domain adaptation for generative adversarial networks. arXiv preprint arXiv:2210.08884 , 2022.

Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096 , 2018.

Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometryaware 3d generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16123-16133, 2022.

Chun-Chao Chen, Jie-Kai Wu, Hsuan-Wen Lin, Tsung-Pin Pai, Tsai-Feng Fu, Chia-Lin Wu, Tim Tully, and Ann-Shyn Chiang. Visualizing long-term memory formation in two neurons of the drosophila brain. science , 335(6069):678-685, 2012.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 1597-1607. PMLR, 2020.

Min Jin Chong and David Forsyth. Jojogan: One shot face stylization. In Computer VisionECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XVI , pages 128-152. Springer, 2022.

Antonia Creswell and Anil Anthony Bharath. Inverting the generator of a generative adversarial network. IEEE Transactions on Neural Networks and Learning Systems , 30(7):1967-1974, 2019.

Hal Daumé III. Frustratingly easy domain adaptation. arXiv preprint arXiv:0907.1815 , 2009.

Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4690-4699, 2019.

Rinon Gal, Or Patashnik, Haggai Maron, Gal Chechik, and Daniel Cohen-Or. Stylegan-nada: Clip-guided domain adaptation of image generators. arXiv preprint arXiv:2108.00946 , 2021.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems , 27, 2014.

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9729-9738, 2020.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems , 30, 2017.

Jongheon Jeong and Jinwoo Shin. Training gans with stronger augmentations via contrastive discriminator. arXiv preprint arXiv:2103.09742 , 2021.

Liming Jiang, Bo Dai, Wayne Wu, and Chen Change Loy. Deceive d: Adaptive pseudo augmentation for gan training with limited data. In Thirty-Fifth Conference on Neural Information Processing Systems , 2021.

Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196 , 2017.

Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. Advances in Neural Information Processing Systems , 33:12104-12114, 2020.

Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4401-4410, 2019.

Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 8110-8119, 2020.

Seongtae Kim, Kyoungkook Kang, Geonung Kim, Seung-Hwan Baek, and Sunghyun Cho. Dynagan: Dynamic few-shot adaptation of gans to multiple domains. In SIGGRAPH Asia 2022 Conference Papers , pages 1-8, 2022.

Bonan Li, Zicheng Zhang, Xuecheng Nie, Congying Han, Yinhan Hu, and Tiande Guo. Styo: Stylize your face in only one-shot. arXiv preprint arXiv:2303.03231 , 2023.

Gang Li, Heliang Zheng, Daqing Liu, Chaoyue Wang, Bing Su, and Changwen Zheng. Semmae: Semantic-guided masking for learning masked autoencoders. Advances in Neural Information Processing Systems , 35:14290-14302, 2022.

Yijun Li, Richard Zhang, Jingwan (Cynthia) Lu, and Eli Shechtman. Few-shot image generation with elastic weight consolidation. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 15885-15896. Curran Associates, Inc., 2020.

Ziqiang Li, Rentuo Tao, Jie Wang, Fu Li, Hongjing Niu, Mingdao Yue, and Bin Li. Interpreting the latent space of gans via measuring decoupling. IEEE transactions on artificial intelligence , 2(1):58-70, 2021.

Ziqiang Li, Muhammad Usman, Rentuo Tao, Pengfei Xia, Chaoyue Wang, Huanhuan Chen, and Bin Li. A systematic survey of regularization and normalization in gans. ACM Comput. Surv. , nov 2022. Just Accepted.

Ziqiang Li, Chaoyue Wang, Heliang Zheng, Jing Zhang, and Bin Li. Fakeclr: Exploring contrastive learning for solving latent discontinuity in data-efficient gans. In European Conference on Computer Vision , pages 598-615. Springer, 2022.

Ziqiang Li, Xintian Wu, Beihao Xia, Jing Zhang, Chaoyue Wang, and Bin Li. A comprehensive survey on data-efficient gans in image generation. arXiv preprint arXiv:2204.08329 , 2022.

Ziqiang Li, Pengfei Xia, Rentuo Tao, Hongjing Niu, and Bin Li. A new perspective on stabilizing gans training: Direct adversarial training. IEEE Transactions on Emerging Topics in Computational Intelligence , 7(1):178-189, 2022.

Sangwoo Mo, Minsu Cho, and Jinwoo Shin. Freeze the discriminator: a simple baseline for fine-tuning gans. arXiv preprint arXiv:2002.10964 , 2020.

Yotam Nitzan, Michaël Gharbi, Richard Zhang, Taesung Park, Jun-Yan Zhu, Daniel Cohen-Or, and Eli Shechtman. Domain expansion of image generators. arXiv preprint arXiv:2301.05225 , 2023.

Atsuhiro Noguchi and Tatsuya Harada. Image generation from small datasets via batch statistics adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , October 2019.

Utkarsh Ojha, Yijun Li, Jingwan Lu, Alexei A Efros, Yong Jae Lee, Eli Shechtman, and Richard Zhang. Few-shot image generation via cross-domain correspondence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10743-10752, 2021.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning , pages 8748-8763. PMLR, 2021.

Esther Robb, Wen-Sheng Chu, Abhishek Kumar, and Jia-Bin Huang. Few-shot adaptation of generative adversarial networks. arXiv preprint arXiv:2010.11943 , 2020.

Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. In ACM SIGGRAPH 2022 Conference Proceedings , pages 1-10, 2022.

Yujun Shen, Ceyuan Yang, Xiaoou Tang, and Bolei Zhou. Interfacegan: Interpreting the disentangled face representation learned by gans. IEEE transactions on pattern analysis and machine intelligence , 44(4):2004-2018, 2020.

Rentuo Tao, Ziqiang Li, Renshuai Tao, and Bin Li. Resattr-gan: Unpaired deep residual attributes learning for multi-domain face image translation. IEEE Access , 7:132594-132608, 2019.

Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or. Designing an encoder for stylegan image manipulation. ACM Transactions on Graphics (TOG) , 40(4):1-14, 2021.

Hung-Yu Tseng, Lu Jiang, Ce Liu, Ming-Hsuan Yang, and Weilong Yang. Regularizing generative adversarial networks under limited data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7921-7931, 2021.

Mei Wang and Weihong Deng. Deep visual domain adaptation: A survey. Neurocomputing , 312:135-153, 2018.

Xiaogang Wang and Xiaoou Tang. Face photo-sketch synthesis and recognition. IEEE transactions on pattern analysis and machine intelligence , 31(11):1955-1967, 2008.

Yaxing Wang, Abel Gonzalez-Garcia, David Berga, Luis Herranz, Fahad Shahbaz Khan, and Joost van de Weijer. Minegan: Effective knowledge transfer from gans to target domains with few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern

Recognition (CVPR) , June 2020.

Yaxing Wang, Chenshen Wu, Luis Herranz, Joost van de Weijer, Abel Gonzalez-Garcia, and Bogdan Raducanu. Transferring gans: generating images from limited data. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 218-234, 2018.

Conghao Wong, Beihao Xia, Ziming Hong, Qinmu Peng, Wei Yuan, Qiong Cao, Yibo Yang, and Xinge You. View vertically: A hierarchical network for trajectory prediction via fourier spectrums. In European Conference on Computer Vision , pages 682-700. Springer, 2022.

Zongze Wu, Dani Lischinski, and Eli Shechtman. Stylespace analysis: Disentangled controls for stylegan image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12863-12872, 2021.

Beihao Xia, Conghao Wong, Qinmu Peng, Wei Yuan, and Xinge You. Cscnet: Contextual semantic consistency network for trajectory prediction in crowded spaces. Pattern Recognition , 126:108552, 2022.

Jiayu Xiao, Liang Li, Chaofei Wang, Zheng-Jun Zha, and Qingming Huang. Few shot generative model adaption via relaxed spatial structural alignment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11204-11213, 2022.

Ceyuan Yang, Yujun Shen, Yinghao Xu, and Bolei Zhou. Data-efficient instance generation from instance discrimination. Advances in Neural Information Processing Systems , 34:9378-9390, 2021.

Ceyuan Yang, Yujun Shen, Zhiyi Zhang, Yinghao Xu, Jiapeng Zhu, Zhirong Wu, and Bolei Zhou. One-shot generative domain adaptation. arXiv preprint arXiv:2111.09876 , 2021.

Jordan Yaniv, Yael Newman, and Ariel Shamir. The face of art: landmark detection and geometric style in portraits. ACM Transactions on graphics (TOG) , 38(4):1-15, 2019.

Kaichao You, Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I Jordan. Universal domain adaptation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2720-2729, 2019.

Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365 , 2015.

Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In International conference on machine learning , pages 7354-7363. PMLR, 2019.

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 586-595, 2018.

Yabo Zhang, mingshuai Yao, Yuxiang Wei, Zhilong Ji, Jinfeng Bai, and Wangmeng Zuo. Towards diverse and faithful one-shot adaption of generative adversarial networks. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

Zicheng Zhang, Yinglu Liu, Congying Han, Tiande Guo, Ting Yao, and Tao Mei. Generalized one-shot domain adaption of generative adversarial networks. arXiv preprint arXiv:2209.03665 , 2022.

Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for data-efficient gan training. Advances in Neural Information Processing Systems , 33:7559-7570, 2020.

Yunqing Zhao, Keshigeyan Chandrasegaran, Milad Abdollahzadeh, and Ngai-Man Man Cheung. Few-shot image generation via adaptation-aware kernel modulation. Advances in Neural Information Processing Systems , 35:19427-19440, 2022.

Yunqing Zhao, Henghui Ding, Houjing Huang, and Ngai-Man Cheung. A closer look at fewshot image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9140-9150, 2022.

Zhengli Zhao, Sameer Singh, Honglak Lee, Zizhao Zhang, Augustus Odena, and Han Zhang. Improved consistency regularization for gans. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 11033-11041, 2021.

Zhengli Zhao, Zizhao Zhang, Ting Chen, Sameer Singh, and Han Zhang. Image augmentations for gan training. arXiv preprint arXiv:2006.02595 , 2020.

Jingyuan Zhu, Huimin Ma, Jiansheng Chen, and Jian Yuan. Few-shot image generation via masked discrimination. arXiv preprint arXiv:2210.15194 , 2022.

Peihao Zhu, Rameen Abdal, John Femiani, and Peter Wonka. Mind the gap: Domain gap control for single shot domain adaptation for generative adversarial networks. arXiv preprint arXiv:2110.08398 , 2021.

A Supplementary Materials

A.1 Ablation Study

Effect of Target Mapping. In this section, we first investigate the significance of the target mapping module in our proposed DoRM. To evaluate the significance of the target mapping, we conduct an ablation study using the source mapping as a substitute for the target mapping, which remains frozen during the entire 10-shot generative domain adaptation training. As shown in Table 3, our results indicate a significant deterioration in the FID score without the target mapping, implying a considerable drop in the quality and diversity of the generated samples. Furthermore, Figure 7 illustrates that the generative domain adaptation barely occurs during training without the target mapping. This is because the target mapping plays a crucial role in capturing the representative attributes of the target domain and assisting in the acquisition of the domain shift during the adaptation process.

Source

Sketch

Baby

Sunglasses

w/ TG

Effect of Re-Modulation Layers. Another crucial component of our proposed DoRM approach is the target affine module. To investigate the roles of the different target affines in DoRM, we perform experiments where we drop the target affines in both the low-resolution and high-resolution feature maps. Specifically, we conduct 10-shot generative domain adaptation experiments to evaluate the Fréchet Inception Distance (FID) of the generated samples. For an image with a resolution of 256 × 256 , the low-resolution feature maps include resolutions of 4 × 4 , 8 × 8 , 16 × 16 , and 32 × 32 . As presented in Table 4, our results demonstrate that all target affines are crucial for the performance of our DoRM approach and their removal leads to a significant drop in the quality and diversity of the generated samples.

Effect of Re-Modulation Weight. The re-modulation weight is a crucial parameter that controls the strength of the acquired domain shift in our proposed DoRM approach. A small re-modulation weight leads to a lower strength of the domain shift, resulting in more attributes of the source domain being preserved during generative domain adaptation. To investigate the impact of the re-modulation weight, we conduct 10-shot generative domain adaptation experiments using different re-modulation weights. The results are presented in Table 5. Our results demonstrate that different domain gaps have different optimal re-modulation weights, indicating that the selection of the re-modulation weight should be tailored to the specific target domain.

Effect of Target domain classifier. We investigate the effect of the target domain classifier on the performance of our proposed DoRM approach. Specifically, we experiment with different depths

( e.g. , number of MLP layers) and initialization methods for the target domain classifier in 10-shot generative domain adaptation. As presented in Table 6, our results indicate that the two-layer MLP target domain classifier achieves the best performance. This is because the one-layer MLP lacks the ability to classify the target domain effectively, while the three-layer MLP is prone to overfitting due to the limited number of training images.

Ablation of Individual Components of Previous State-of-the-art Methods. To highlight the specific strengths of the proposed method that outperform other works, we consider the individual components and techniques utilized in previous state-of-the-art approach [55]. Specifically, we systematically analysis and compare individual components and techniques utilized in our DoRM and DiFa [55] to showcase how our method surpasses these works. According to the Introduction of the manuscript, GDA needs three fundamental properties. To resolve it, DiFa [55] proposed two CLIP-based loss: global loss L global and local loss L local for realizing large diversity/cross-domain consistency and high quality, respectively. Differently, we adopt Similarity-based Structure Loss ( L ss ) and adversarial loss ( L adv ) for realizing large diversity/cross-domain consistency and high quality, respectively. As shown in the Figure 8, the L local in DiFa mainly focuses on textual features and failes to capture the complete features (e.g. the white background in sketches) in generative domain adaptation. In our DoRM, we employ adversarial loss which can fully capture the features of the target images.

A.2 More Synthesis Results in 10-shot GDA

Qualitative results on 10-shot generative domain adaptation. We present qualitative results of our proposed DoRM approach on more target datasets, including face caricatures, face paintings by Raphael, face paintings by Amedeo Modigliani, and face paintings by Otto Dix [50]. All training images are shown in Figure 9, and the qualitative results are presented in Figure 10. Our results

Source(FFHQ)

𝐿𝑎𝑑𝑣 +𝐿𝑠𝑠

𝐿local

+𝐿𝑔𝑙𝑜𝑏𝑎𝑙

𝐿local +𝐿𝑠𝑠

demonstrate that our proposed DoRM approach achieves appealing synthesis quality in various target domains.

Results of latent interpolation. We perform latent space interpolation to demonstrate that our DoRM is not harmful to the learned latent space. In Figure 11, the first and last columns show the generated images with two latent codes after 10-shot generative domain adaptation, while the remaining columns show the results obtained by linearly interpolating the two latent codes. Our results demonstrate that all intermediate synthesized images have high target-domain consistency and high cross-domain consistency. Moreover, the semantics of the generated images, such as gender, haircut, and pose, vary gradually throughout the interpolation, indicating that our proposed DoRM approach preserves the underlying semantic structure of the learned latent space.

Results of latent editability. We have performed editing on a real image adapted into a new target domain using StyleGAN-CLIP to discover editing directions in the source domain. The results, as illustrated in Fig 12, indicate that the adapted generator maintains similar latent-based editing capabilities to the original generator. This demonstrates the preservation of editability in the adapted generator.

Results on 3D generator. We conducted some initial studies using the popular 3D-aware image generation method, EG3D [3], for one-shot GDA with FFHQ as the source domain and Sketch as the target domain. The results, as shown in the Figure 13, reveal that directly migrating the proposed DoRM to 3D one-shot domain adaptation poses challenges and might not be straightforward. Some potential challenges in the 3D-GAN domain include:

(1) Overfitting: 3D GANs often face more severe overfitting issues when dealing with limited training data, requiring more rigorous regularization techniques and training strategies for one-shot GDA.

(2) Volumetric Data Representation: Handling volumetric data representation in 3D-GANs necessitates specialized techniques for data manipulation, augmentation, and visualization.

(3) Spatial Artifacts: Generating high-quality 3D objects may encounter spatial artifacts, such as geometric distortions or inconsistent shapes, which need to be addressed.

(4) Computational Complexity: The computational demands of 3D-GANs can be significantly higher than their 2D counterparts, presenting challenges in both training and inference.

Results on three-domain hybrid domain generation. As depicted in the Figure 14, we have showcased an example of applying our method to create hybrid-domain images by activating the trained M&A modules of Baby, Sunglasses, and Sketch domains. This illustration demonstrates how easily our approach can be adapted to generate hybrids involving more than two domains, showcasing the versatility and potential of our approach.

A.3 Experiments on one-shot Generative Domain Adaptation

Although our DoRM is mainly for few-shot generative domain adaptation, DoRM is also can be employed for one-shot generative domain adaptation [21; 55; 6]. In the one-shot GDA, the training dataset is a single image, which is difficult for the backbone of discriminator to extract the main characters of the target domain because of the overfitting issue. In this case, we introduce a clip-based local-level adaptation loss L local from [55] to help to acquire the local-level characters and styles of the target domain. Concretely, we extract the intermediate tokens of the adapted image I B synthesized by DoRM and the single target image I tar from the k -th layer of CLIP image encoder. And align each of adapted token F B with its closest target token from F tar , where F B = F 1 B , ..., F n B and

F tar = F 1 tar , ...F m tar are the extracted tokens. The clip-based local-level adaptation loss is defined as:

where C is calculated as:

Furthermore, to better identify and maintain the domain-sharing attributes in one-shot generative domain adaptation, we also employ the inversion-based selective cross-modal consistency loss L scc from [55]. Specifically, this loss function aims to identify and preserve domain-sharing attributes in the W + space. The underlying assumption is that attributes that are similar in W + space between the source and target domains during adaptation are more likely to be domain-sharing attributes. To achieve this, L scc dynamically analyzes and retains these attributes. First, it inverts the source and corresponding target images into W + latent codes, w A and w B , respectively, using a pre-trained inversion model such as pSp pr e4e, for each iteration. Next, it computes the difference ∆ w , between the centers of a source queue of W + latent codes, X A and the target queue of W + latent codes, X B , where X A and X B are dynamically updated with w A and w B during training. The loss function then encourages w A and w B to be consistent in channels with less difference, thereby facilitating the preservation of domain-sharing attributes. The inversion-based selective cross-modal consistency loss L scc is defined as follows:

where α represents the proportion of preserved attributes, and mask (∆ w,α ) determines which channels to retain. Specifically, let | ∆ w s αN | be the αN -th largest element of ∆ w . Then, each dimension of mask (∆ w,α ) is calculated as follows:

We compare our DoRM++ approach which denotes introducing the two new loss terms into training with state-of-the-art one-shot generative domain adaptation (GDA) methods, including JoJoGAN [6], Generalized One-shot Domain Adaptation [56], DynaGAN[20] and DiFa [55]. Figure 16 shows the comparison results. Our results indicate that JoJoGAN, DynaGAN and Generalized One-shot Domain Adaptation fail to achieve GDA when the target image is FFHQ-Baby and FFHQ-Sunglasses, and the synthesis quality of DynaGAN is limited. Similarly, DiFa also fails to achieve GDA when the target image is FFHQ-Sunglasses, and the synthesis diversity is unsatisfactory when the target image is FFHQ-Baby.

In contrast, our DoRM++ approach achieves one-shot GDA among all the reference images, resulting in high-quality and diverse synthesis, while maintaining appealing cross-domain consistency. Moreover, our DoRM++ generator has memory to realize multiple target domains' generation, which saves a significant amount of storage space. Our DoRM++ generator also has the ability to integrate the learned knowledge of multiple target domains to synthesize images in hybrid domains that are unseen in the target domains. As shown in Figure 17, the DoRM++ generator can synthesize high-quality and diverse images in hybrid domains while maintaining the domain-sharing attributes (e.g, pose, identity).

A.4 User Study in One-shot GDA

In this section, we have planned to conduct a user study in one-shot GDA to enhance our evaluation process. Specifically, our user study involves presenting users with a reference image, a source image, and four adapted images from different methods (DORM++, Generalized One-shot Domain Adaption [56], DynaGAN [20], and DiFa [55]). We will ask users to choose the best adapted image for each of three measurements: (i) image quality, (ii) style similarity with the reference, and (iii) attribute consistency with the source image. To ensure statistical significance, we will generate 500 samples for each method and involve 50 users in our study. Each user will be randomly assigned 50 samples from the 500, and they will have unlimited time to complete the evaluation. As shown in Table 7, preliminary results indicate that users strongly favor our DoRM in all three aspects, reflecting the effectiveness of our approach compared to the alternative methods.

A.5 Experiments of DoRM and DoRM++ on the one-shot and 10-shot GDA

Figure 15 thoroughly explore the performance of DoRM++ in a 10-shot GDA scenario and that of DoRM in a one-shot GDA context. The results of these experiments illustrate that both DoRM++ and DoRM exhibit strong performance in both few-shot and one-shot GDA scenarios. Notably, DoRM++ showcases enhanced cross-domain consistency compared to DoRM in the context of one-shot GDA.

A.6 Hybrid-domain Generation

In Figure 17, we present the results of generating hybrid domains using our proposed model. Our DoRM has a unique generator structure that is similar to the mechanism of the human brain. This structure endows the DoRM with two novel capabilities: memory and domain association. These capabilities enable the DoRM to not only retain knowledge from previously learned domains when generating images in new domains, but also integrate multiple learned domains and synthesize images in hybrid domains that were not encountered during training.

Additionally, we proceed to provide a comparative analysis of three methods within this section: Our DoRM/DORM++, DynaGAN [20], and Domain Expansion [30]. As illustrated in Figure 18, we strive to alleviate any confusion by conducting comprehensive experiments on the memory and domain association capabilities of these three methods under one-shot and 10-shot settings. These results effectively demonstrate that the baseline methods exhibit poorer performance across various settings compared to our DoRM/DoRM++. Notably, both DynaGAN and Domain Expansion experience difficulties in achieving successful hybrid domain generation, highlighting the distinctive advantages of our approach.

Finally, we meticulously scrutinized the quantitative experiments conducted within the Domain Expansion framework, leading us to adopt a cosine similarity evaluation metric termed "domain similarity" ( Sim ) based on CLIP image encoder ( E I ). This metric is employed to provide a quantitative assessment of the fidelity exhibited by hybrid domains. In detail, for given generative images in the hybrid domain "sketch-baby" ( I SB ), we extract image features from the provided images ( E I ( I SB ) ) as well as its corresponding target images ( E I ( I S ) and E I ( I B ) ). Therefore, the "domain similarity" to "sktech" and "baby" domain are defined as Sim 1 = cos ( E I ( I SB ) , E I ( I S )) and Sim 2 = cos ( E I ( I SB ) , E I ( I B )) , respectively.

Subsequently, we compute the cosine similarity for each case. The quantification of results from both the one-shot and 10-shot experiments is meticulously presented in Table 8 and Table 9, respectively. These outcomes distinctly illustrate the remarkable performance of our proposed DoRM in both single-domain and hybrid-domain generation. It is imperative to highlight a notable observation amidst these findings: a certain outlier exists. Specifically, the images generated by DynaGAN and HyperdomainNet within the "elsa-sunglasses" domain exhibit a remarkably high domain similarity with the "sunglasses" domain, while conversely displaying significantly lower domain similarity with the "elsa" domain. This phenomenon can be rationalized by the generated images closely resembling the "FFHQ" domain, as depicted in Figure 6 of the attached PDF in the rebuttal. Consequently, the domain similarity to the "FFHQ-sunglasses" domain also emerges as notably high. Drawing upon this observation, we emphasize the necessity for a comprehensive evaluation of hybrid-domain generation that seamlessly integrates both qualitative and quantitative results.

A.7 Experiments on Other Source Domains

In addition to the experiments on FFHQ, we conduct other 10-shot generative domain adaptation experiments to qualitatively evaluate the effectiveness of our proposed DoRM approach. Specifically,

we pre-trained a StyleGAN2 on the LSUN-church [52] dataset and adapted the pre-trained GAN to generate haunted house images. The results of our experiments are presented in Figure 19.

Sketches

Sunglasses

Caricature

Amedeo

Baby

Source

Sunglasses

Amedeo

Caricature

Rapheal

Original

Man

Man+Pose

Look up

Source

S

B

G

S-B-G

DoRM++

DoRM

10-shot

one-shot

DynaGAN

Domain expansion

Haunted House

Target images

Source images

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies generative domain adaptation (GDA), on the StyleGAN2 architecture. Methods in this setting are commonly evaluated in terms of quality, diversity, and cross-domain consistency. This work claims to also be the first to explore the abilities of memory (“retain knowledge from previously learned domains”) and domain association (“integrate multiple learned domains and synthesize hybrid domains”).  To tackle all these aspects, a method called DoRM is proposed. DoRM learns a new mapping network and affine transformations – M&A - (components of StyleGAN2) for each new domain. Hybrid domains are generated by interpolating the style codes of the original mapping and affine layers with the new ones.\n\nSTRENGTHS:\n1.\tThe paper is clearly written and easy to follow.\n2.\tThe tasks studied in this paper are interesting and relevant.\n3.\tThe proposed method is intuitive and novel, and as demonstrated by some experiments it is effective in performing domain adaptation.\n\nWEAKNESSES:\nThe claimed contributions are incorrect, and the evaluation is severely lacking. Both issues stem from outright ignoring the role and contributions of the most relevant previous works – HyperDomainNet [1], DynaGAN [18], Domain Expansion [24]. Below I list several examples of severe problems with claims and evaluations.\n\nAll three works consider settings that cover the supposably new aspects of domain adaptation that are discussed in this paper. Both DynaGAN and HyperDomainNet rely on similar modulation techniques and achieve *”memory”* the same way. In Domain Expansion, the original domain is preserved on a dedicated subspace. Also, both DynaGAN and Domain Expansion discuss *”association”* in length (using the terms “domain interpolation” and “domain composition”). So, clearly, this work’s claims (lines 64, 247) of being the first to consider this setting are false. \n\nThe previous fact is even somewhat acknowledged in the Related Work section (lines 99-105). The three works are said to “fall short in integrating multiple domains”. However, this claim is not supported by any experiment. \nNot only this one claim is not supported, but the authors choose to not compare their method with the three most relevant works - all of which have code on Github and all have been published well before the submission deadline (October 2022-January 2023). Instead, the method is compared to the more conventional setting of “single-domain, entire generator” adaptation, which is negligent at best. Comparing the generation of “hybrid” domains to CDC [25] which was not designed for this purpose (and published in 2021) and not to DynaGAN and Domain Expansion is unreasonable. Similarly, claiming superiority over CDC in terms of storage required is irrelevant. DoRM trains a mapping network (~6M params) for each new domain, while HyperDomainNet trains ~6K params, and Domain Expansion requires no additional weights whatsoever!\n\nThe inaccurate contribution claims and lack of comparison to relevant previous works is fatal as it prevents the community and practitioners from understanding the landscape of works in this field. Perhaps the paper was written around the end of 2022 and is simply not up to date. I urge the authors to consider significantly rewriting this paper before resubmitting or arXiving a preprint.\n\nAdditional issues:\n1. Measuring consistency between domains using ArcFace – a face recognition network trained on faces -- makes little sense to me. For example, why would it output anything meaningful on sketches? \n2. Missing citation to StyleSpace [Wu et al.] who introduced the $\\mathcal{S}$ used in this paper (line 120).\n\nQUESTIONS:\n1.\tPlease clarify in what way is the setting that is supposably suggested was not already covered by HyperDomainNet, DynaGAN and Domain Expansion?\n2.\tWhy did you not include comparisons to these methods despite acknowledging that they are highly similar and relevant?\n\nLIMITATIONS:\nOne limitation is mentioned. However, several questions remain:\n1. Can hybrids be produced by combining more than two domains?\n2. Are the produced hybrids less faithful to each individual domain and to what extent?", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the concept of few-shot generative domain adaptation. Drawing inspiration from the human memory mechanism, the authors introduce a novel approach called DoRM (Domain-Adaptive Mapping and Affine Modules) to adapt the generator to a new domain. By incorporating new mapping networks and affine modules into the frozen source generator, they aim to enhance its adaptability. Additionally, the paper introduces a consistency loss based on CLIP image features to ensure the preservation of domain-sharing attributes during adaptation. Notably, the authors explore the intriguing concept of domain association in generative models, which involves merging the generative abilities of multiple domain-specific generators to enable generation in a new domain. Through a combination of mapping networks and affine modules within a single generator, the proposed DoRM generator successfully achieves domain association and multi-domain generation concurrently. To validate the effectiveness of their approach, the authors conduct extensive qualitative and quantitative experiments.\n\nSTRENGTHS:\n(1) This paper presents a novel task: domain association, which involves combining the generative abilities of different domain-specific generators to enable generation in a new domain. This topic represents an exciting avenue for further exploration in the field of generative models. Notably, Figure 4 demonstrates impressive synthesis quality in the hybrid domain, particularly in the baby-sketch domain.\n\n(2) The proposed generator network structure, DoRM, is skillfully designed to facilitate domain association in a straightforward yet effective manner. By incorporating lightweight modules, the method achieves domain adaptation while preserving the generative abilities in the learned domains. This approach is both efficient in terms of storage and yields promising results.\n\n(3) The paper is excellently written and presents its concepts in a clear and understandable manner, making it easy for readers to follow along with the proposed methodology.\n\n(4) Through extensive qualitative and quantitative experiments, the paper demonstrates the superior performance of the proposed method compared to state-of-the-art techniques. Particularly impressive are the synthesis results achieved through domain association using the proposed method.\n\nWEAKNESSES:\nPlease view the Questions below.\n\nQUESTIONS:\n(1) It is worth noting that the proposed generator structure bears similarities to the one introduced in [1]. This overlap may somewhat diminish the novelty of the proposed method. To enhance the clarity of differentiation, it would be beneficial if the authors provide a more comprehensive explanation highlighting the distinct features and advancements of their approach in comparison to [1].\n\n(2) In addition to the proposed loss regularization, the authors have included supplementary regularizations, referred to as DoRM++, based on [2], in the supplementary material's one-shot experiments. Although the synthesis results of DoRM++ outperform those of [2], it is important to note that the proposed DoRM approach already addresses overfitting concerns by freezing the discriminator's backbone. Consequently, the inclusion of these supplementary loss components may be considered redundant. Therefore, it would be beneficial to conduct further qualitative and quantitative comparisons between DORM and DORM++ to gain a comprehensive understanding.\n\n(3) Section 4.3 would greatly benefit from further elaboration on the training details and specific datasets employed in the domain association experiments. Providing more specific information in these areas would enhance the reproducibility and understanding of the experimental setup for readers.\n\n(4) In recent times, the diffusion model has made remarkable strides in image generation, particularly in the area of few-shot image generation. Therefore, there is a need for further discussion regarding the application of the diffusion model in the context of few-shot Generative Domain Adaptation and domain associations. \n\n[1] HyperDomainNet: Universal Domain Adaptation for Training Generative Adversarial Networks. NeurIPS’22\n[2] Towards Diverse and Faithful One-shot Adaption of Generative Adversarial Networks. NIPS 2022\n\nLIMITATIONS:\nLimitations are discussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes two advanced criteria for few-shot Generalized Domain Adaptation (GDA) inspired by the way human brains acquire knowledge in new domains: memory and domain association.\n\n To fully realize the potential of few-shot GDA, an innovative generator structure called Domain Re-Modulation (DoRM) is introduced.\n\n This structure freezes the source generator and incorporates new lightweight mapping and affine modules (M&A modules) to capture the attributes of the target domain during GDA, resulting in a linearly combinable domain shift in the style space.\n\n By incorporating multiple M&A modules, the generator gains the ability to perform high-fidelity multi-domain and hybrid-domain generation.\n\nSTRENGTHS:\n1. This paper presents a novel exploration into the use of memory and domain association in few-shot GDA, which has not been previously explored. These advanced properties greatly reduce memory usage and simplify deployment, while also producing impressive results.\n\n2. The proposed DoRM structure and similarity-based structure loss are innovative and advanced.\n\n3. The figures in this paper effectively demonstrate the major contributions and the overall organization is clear and easy to follow.\n\n4. The experiments conducted in this paper are robust and include both one-shot and 10-shot settings. Extensive testing demonstrates that our proposed method outperforms the previous state-of-the-art method in all five properties: quality, synthesis diversity, cross-domain consistency, memory, and domain association.\n\nWEAKNESSES:\nThere are a few areas where this paper could be improved.\n\n1. It would benefit from including some recent studies on one-shot image generation [1].\n\n2. The paper should include experiments of DoRM++ in 10-shot GDA and DoRM in one-shot GDA, as this ablation experiment would help to demonstrate the advancement of the mentioned losses and the impact of data size on GDA tasks.\n\n3. The paper lacks a future outlook, and it would be helpful to explore more possible improvement directions, especially as the effect of domain association on some datasets is not always satisfactory.\n\n[1] StyO: Stylize Your Face in Only One-Shot\n\nQUESTIONS:\nSee weakness\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel approach for domain adaptation of StyleGAN2 called Domain Re-Modulation, which is a few-shot technique. The authors argue that this method draws inspiration from the workings of the human brain. To achieve the desired domain shift, the paper utilizes the stylespace of StyleGAN along with adversarial and consistency losses. Additionally, a structural similarity loss is incorporated, which is based on the auto-correlation map extracted from the CLIP encoder. The effectiveness of the proposed method is evaluated through both quantitative and qualitative analyses. Furthermore, the paper includes comparisons with other state-of-the-art StyleGAN-based methods, highlighting its advancements. The authors also conduct an ablation study to analyze the individual components employed in their approach.\n\nSTRENGTHS:\n* The paper introduces the concept of hybrid domain generation, which involves combining the affine layers of StyleGAN to generate results from multiple domains using a single generator. The authors demonstrate that this approach yields consistent and generalizable outcomes across multiple domains.\n\n* Notably, the method proves effective even in scenarios with limited training data, such as 1-shot training. By leveraging the StyleGAN2 backbone, the proposed technique achieves reliable and coherent generation outcomes.\n\n* In terms of evaluation, the paper compares its approach with various StyleGAN-based methods. It provides both qualitative and quantitative results to assess the performance of the proposed method. Furthermore, the paper includes comparisons with related methods to highlight the advantages and advancements of the proposed domain generation approach.\n\nWEAKNESSES:\n1. The paper lacks substantial novelty as there exist other works such as StyleCLIP and 3DAvatarGAN that demonstrate the capability of performing image editing and few-shot domain adaptation using the \"s\" space of StyleGAN. Although the paper incorporates a CLIP-based loss, it fails to provide a comparative analysis with papers falling under the same category, such as StyleGAN-NADA and Mind the GAP HyperStyle. The absence of such comparisons and explanations regarding the differences between the methods makes it challenging to determine the uniqueness of the paper. To assess how the current method surpasses these works, it is essential to consider the individual components and techniques utilized in each approach. By conducting a thorough comparison, the paper could highlight the specific strengths of the proposed method that outperform other works. This would provide a clearer understanding of the paper's novelty and identify the components that excel in comparison to existing methods. Hence, it is imperative for the paper to address these concerns by including a detailed comparison that elucidates how the current method improves upon or surpasses existing approaches, specifically identifying the components that outperform these methods. How is the current method better than these works? Which component of the method outperforms these methods? \n\n\n2. Why did the authors use 2D generators especially when 3D StyleGAN based generators are available trained on the same data. Besides the concerns above, it would be interesting to see how the method will perform in the 3D -GAN domain. What are some additional challenges in that domain?\n3. How is the editability of the generator after the domain adaptation. Since there are no edits performed, it is difficult to assess if the latent space properties of the generator are preserved or it just overfits the given Styles.\nThere are other datasets like AFHQ Cats, Dogs, Cars that are tested for such few shot domain adaptation tasks in the StyleGAN domain. The method seems specific to the face domain. What about the results on these domains? How generalizable is the method to these domains?\n4. The artistic domains are quite subjective. It is not fair to just evaluate quantitative metrics for such tasks. I would suggest conducting a user study. It would be better to embed real images in the generators and ask users about the similarity, consistency and identity preservation of the target images.\n\nQUESTIONS:\nRefer to Weakness section\n\nLIMITATIONS:\nThe authors discuss the ethics and limitations.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel approach for few-shot generation using a lightweight GAN architecture and a new loss function. The method is capable of handling multi-domain and hybrid domain tasks with a single model. The experiments demonstrate the superior performance of the proposed method in terms of both qualitative and quantitative results.\n\nSTRENGTHS:\n1. The paper addresses a novel task and presents a unique approach to handle it.\n2. The experiments are comprehensive and well-organized, providing strong evidence for the effectiveness of the proposed method.\n3. The proposed method outperforms previous methods in terms of both qualitative and quantitative results.\n4. The exploration of the generative ability to merge two or three domains for few-shot generation is interesting and innovative.\n\nWEAKNESSES:\n1. The paper should provide more analysis or improvement on the potential issue of unrealistic results due to domain association.\n2. The related works should be discussed more thoroughly, especially with respect to HyperDomainNet [1] and DynaGAN [2], which share similar ideas with the proposed method.\n3. The proposed method is designed specifically for StyleGAN2 architecture, limiting its generalization to other architectures. \n4. It would be beneficial to expand the list of baselines in Table 1 to include additional approaches, such as MineGAN [3] and EWC [4].\n5. This paper utilized adversarial training while freezing the discriminator's backbone. However, in my understanding, the discriminator may still overfit the training data due to the adversarial loss. This issue has also been observed in comparable works, like CDC, where the adversarial loss weight is only 1 while the consistent loss weight is 1000, but the overfitting problem persists as training progresses. How about of the proposed approach?\n\n\n[1] Hyperdomainnet: Universal domain adaptation for generative adversarial networks. NIPS 2022\n\n[2] Dynagan: Dynamic few-shot adaptation of gans to multiple domains. SIGGRAPH Asia 2022\n\n[3] Minegan: effective knowledge transfer from gans to target domains with few images. CVPR 2020\n\n[4] Few-shot image generation with elastic weight consolidation. NIPS 2020\n\nQUESTIONS:\n1. The paper should provide more analysis or improvement on the potential issue of unrealistic results due to domain association.\n2. The related works should be discussed more thoroughly, especially with respect to HyperDomainNet [1] and DynaGAN [2], which share similar ideas with the proposed method.\n3. The generalization of the proposed method should be discussesd.\n4. It would be beneficial to expand the list of baselines in Table 1 to include additional approaches, such as MineGAN [3] and EWC [4].\n\nLIMITATIONS:\nAuthors have addressed some limitation of this paper. More Limitation should be further addressed: The proposed method is designed specifically for StyleGAN2 architecture, limiting its generalization to other architectures.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-17"}