Domain Re-Modulation for Few-Shot Generative Domain Adaptation
Authors: Yi Wu
Ziqiang Li
Chaoyue Wang
Heliang Zheng
Shanshan Zhao
Bin Li
Dacheng Tao
Venue: NeurIPS 2023 poster (2023)
Keywords: StyleGAN
Few-Shot Generative Domain Adaptation
TL;DR: The proposed method can realize high fidelity multi-domain and hybrid-domain generation by integrating multiple M&A modules in a single generator.

Abstract: In this study, we delve into the task of few-shot Generative Domain Adaptation (GDA), which involves transferring a pre-trained generator from one domain to a new domain using only a few reference images. Inspired by the way human brains acquire knowledge in new domains, we present an innovative generator structure called $\textbf{Domain Re-Modulation (DoRM)}$. DoRM not only meets the criteria of $\textit{high quality}$, $\textit{large synthesis diversity}$, and $\textit{cross-domain consistency}$, which were achieved by previous research in GDA, but also incorporates $\textit{memory}$ and $\textit{domain association}$, akin to how human brains operate. Specifically, DoRM freezes the source generator and introduces new mapping and affine modules (M\&A modules) to capture the attributes of the target domain during GDA. This process resembles the formation of new synapses in human brains. Consequently, a linearly combinable domain shift occurs in the style space. By incorporating multiple new M\&A modules, the generator gains the capability to perform high-fidelity multi-domain and hybrid-domain generation. Moreover, to maintain cross-domain consistency more effectively, we introduce a similarity-based structure loss. This loss aligns the auto-correlation map of the target image with its corresponding auto-correlation map of the source image during training. Through extensive experiments, we demonstrate the superior performance of our DoRM and similarity-based structure loss in few-shot GDA, both quantitatively and qualitatively. Code will be available at https://github.com/wuyi2020/DoRM.

of few-shot Generative Domain Adaptation (GDA), which aims to achieve realistic and diverse Particularly, the Domain Re-Modulation for Few-Shot Generative Domain Adaptation
Yi Wu,, Ziqiang Li University of Science and Technology of China
Chaoyue Wang!
Heliang Zheng, JD Explore Academy
Shanshan Zhao
Bin Li University of Science and Technology of China
Dacheng Tao JD Explore Academy
Abstract
In this study, we delve into the task of few-shot Generative Domain Adaptation (GDA), which involves transferring pre-trained generator from one domain to new domain using only few reference images. Inspired by the way human brains acquire knowledge in new domains, we present an innovative generator structure called Domaín Re-Modulation (DoRM). DoRM not only meets the criteria of high quality, large synthesis diversity, and cross-domain consistency, which were achieved by previous research GDA, but also incorporates memory and domain association, akin to how human brains operate. Specifically, DoRM freezes source generator and introduces new mapping and affine modules (M&A modules) to capture the attributes of the target domain during GDA. This process resembles the formation new synapses in human brains. Consequently, a lincarly combinable domain shift occurs in the style space. By incorporating multiple new M&A modules, the generator gains the capability to perform high-fidelity multi-domain and domrid-domain generation. Moreover, maintain cross-domain consistency more effectively, we introduce a similarity- bbased structure loss. This loss aligns the cotrecoorel map of the target image with its corresponding auto-correlation map of the source image during training. Through extensive experiments, demonstrate the superior performance of our DoRM and similarity- based structure loss in few-shot GDA, both quantitatively and qualitatively. Code will be available at
Introduction
Domain adaptation aims to bridge the domain gap and transfer knowledge to mitigate the limitations imposed by lack of extensive labeled data [8] 40 [5I]. Recent research has explored the field
generation with only few training images 29 32
"Equal Contribution.
This work performed when Ziqiang Li was visiting JD Explore Academy research intern. Coorespondiig Author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Sourre Mapsing
Source Affice
aby Mapping
Raby Affine
Latent
Synthesis Networks
Sunglasses Mapping
Sungllasses fflmn
Mulüi Generatun Tfyrril dimaii Gervinbonn
Skeech Mapping
Skeech Affie
Rixed
Colorfal
Trainable
Figure I: Left: Framework of our DoRM. Given generator of StyleGAN2 structure pre-trained on source domain (components in the white dotted blocks), we can realize multi-domain generation and hybrid-domain generation by activating the corresponding trained M&A modules (components in the colourful solid blocks). Right: Multi-domain generation (right top) and hybrid-domain generation (right bottom). For instance, by activating the trained M&A modules of Baby and Sketch domains, hybrid-domain (Sketch-Baby domain) generation has been realized easily.
purpose of few-shot GDA is to transfer generator pre-trained on the source domain to the target domain using few reference images.
Existing few-shot GDA works primarily focus on three properties: (i) High quality and (ii) Large diversity, where the adapted generator can synthesize high quality and diverse images in the target domain and (iii) Cross-domain consistency, where the adapted images and their corresponding source images should be consistent in terms of domain-sharing attributes, such as pose and identity. A majority of them update the entire generator with regularization techniques including GAN inversion (CLIP) [1O: [55] 63 J. and consistency loss to achieve large diversity and cross dossin consistency of adaptation. However, updating the entire generator imposes limitations on its ability to synthesize multiple target domains. In contrast, the human brain employs a more efficient approach to learn from new domains by forming new proteins 44. allowing for the retention and integration of knowledge from multiple domains. This creative capacity enables humans to explore previously unseen domains.
55
47
63
Languugg--maae-PPee
32
47
59
Drawing inspiration from human learning processes, we propose a novel generator structure called Domain Re-Modulation (DoRM) based on StyleGAN2 [9] DoRM not only fulfills three essential properties of few-shot GDA but also introduces two new capabilities: (iv) Memory, enabling the generator to retain knowledge from previously learned domains when generating images in new domains, and (v) Domain Association, allowing the generator to integrate multiple learned domains and synthesize hybrid domains not encountered during training. As depicted in Figure D DoRM freezes the pre-trained source generator and introduces new mapping and affine modules (M&A modules) achieve the necessary domain shift in the style space. This approach enables DoRM to perform high-fidelity generation across multiple domains by selectively activating different M&A modules, which significantly reduces the required storage space. Moreover, StyleGAN's 's style space is a linear subspace [36] 24) 45), the domain shift obtained through DoRM is linearly combinable. Consequently, DoRM can synthesize images in hybrid domains not present in the training dataset by simultaneously activating multiple M&& modules. To further enhance cross-domain consistency, we introduce a similarity- -based structure loss denoted as L.s. Specifically. we leverage the CLIP image
*It is important to note that the objective sho-shot Generative Domain Adaptation (GDA) differs from that of Few-Shot Image Generation (FSIG). FSIG focuses on generating high-quality and diverse images in the target domain using limited number of training samples, whereas few-shot GDA involves additional considerations beyond FSIG.

---

encoder [33 to extract intermediate tokens from the target image and its corresponding source image. We then enforce consistency between the auto-correlation maps of these tokens, ensuring improved alignment between target and source images.
We conducted extensive experiments on few-shot GDA involving various source and target domains. Our experiments, both quantitative and qualitative, highlight the competitive performance of our method compared state-of-the-are approaches in 10-shot GDA. We achieved remarkable results in terms of quality, diversity, and cross-domain consistency. Notably, our proposed DoRM stands out by demonstrating exceptional performance in hybrid-doman generation, a hard task that previous methods have not accomplished well. The key contributions of this work can be summarized as follows:
We introduce DoRM, a novel generator structure for few-shot generative domain adaptation, inspired by the learning mechanism of human brains. DoRM not only excels in synthesizing high-quality, diverse, and cross-domain consistent images, but also integrating memory and domain association capabilities that remain relatively unexplored in the field. Notably, our approach is one of the very few that encompasses all five desired properties of GDA.
Additionally, we propose a novel consistency loss called similarity -based structure loss, which further enhances cross-domain consistency in our approach.
Through comprehensive evaluations, our proposed method outperforms existing competitors across various settings, showcasing its superior performance.
2 Related Work
2.1 Generative Adversarial Networks
Deep learning has made remarkable achievements in various fields 46
44
Among them,
generative adversarial networks [1] 37 28 play two-player adversarial game, where the generator aims to synthesize realistic images to fool the discriminator, while the discriminator learns how to distinguish the synthesized images from real ones. Previous works []6 2
have significantly improved the synthesis capability on high-resolution datasets. Style-based methods remain the state-of-the-ar unconditional GANs owing their unique architecture. However, GANs training requires a large number of training images, or severe discriminator overfitting can occur (26 27]. To alleviate this issue and improve training stability, various data augmentation 61
53
18 [9
[8
19 35)
60; 25 and regularization technologies (26) [8) 39 [4] have been introduced. Among these, Adaptive Discriminator Augmentation (ADA) an adaptive strategy that controls the strength of augmentations and has shown remarkable performance, becoming the default operation in data-efficient GANs. Tseng et al. (39 proposed a regularization scheme to modulate the discriminator's 's prediction, mitigating discriminator overfitting. However, most of these techniques fail in the extremely small (e.g., 10) training data regime.
techniques [57 15 17
2.2 Few-shot Generative Domain Adaptation
Previous works in few-shot GDA mainly have primarily focused on three essential properties: High quality, Large diversity, and Cross-domain consistency. Some studies have fine-tuned the entire generator using regularization techniques. For example, 33 47
introduced a consistency loss based on Kullback-Leibler (KI.) divergence preserve the relative similarities and differences between instances, inheriting diversity and maintaining consistency during adaptation. The contrastive term has also been employed construct consistency loss in [59] In addition, leveraging the
12, 5 semantic power of large Laatrrasiive-Lnuuggg--lee proposed define the domain-gap direction CLIP embedding space, guiding the optimization of
(CLIP) (33) models, [10 55 63
the generator. Furthermore, GAN inversion 7 38 technique has been widely used in few-shot GDA
to meet different purposes, such as exploring the domain- -hariin attributes [55] 63), and compressing the latent space to a subspace to relax the cross- cross-domain alignment [4]] Compared to optimizing the entire generator, some studies 31 only update the crucial part of the generator. Moreover, some work [49 adds a lightweight attribute adaptor and attribute classifier before the frozen generator and after the frozen discriminator, respectively. The proposed method achieves remarkable performance in synthesis quality and diversity but lacks dross-domain consistency.
3

---

noise
fixed
Lateat E 2
Source adfine
Synthesss netwerk
F
trainable
Normalize
A
Target mepping
Source mapping
Target affine
Mod
FC
eemod
Conv 3x33
FC
b2
Source affine
FC
FC
Target affine
Mod
Upsample
FC
A
D Demod
Conv 3x3
FC
FC
FC
Sourre affine
W4
Target affine
Mod
We
Ws
A
D emmd
Conv 3x3
Our framework (DoRM) is based on StyleGAN2. In the Synthesis network g(-) (gray blocks), the source affines A, and the source mapping fs constitute the pre-trained source generator on a source domain. To achieve generative domain Adaptation, we incorporate a new target mapping ft and target affines A, During training, we only optimize the parameters of the solid yellow blocks.
Figure 2:
share the similar idea with us of incorporating multi-domain generation capabil- ities into single generator. However, these methods employ multiplicative modulation, which is indeed different with ours. Specifically, HyperDDomiiNee [1] adopts solitary modulation parameter, primarily scale (6), to adjust the weights of the convolutional layer, influencing the "s" space of StyleGAN. In this setup, the scale parameter remains constant for all images within a target domain. Formally, HyperDomainNert' architecture is represented as w 8; 6, where w and 8; denote the convolutional weight and style code of the source StyleGAN2, respectively. However, the introduction of this all all-encompassing scale parameter can potentially restrict the generator's learning capacity. Especially in scenarios with significant domain gaps between target and source domains, HyperDomainNet's performance may decline. Furthermore, the non-linear combinability of the scale modulation parameter impedes its ability to achieve robust domain association. Among the approaches closest to our DoRM, DynaGAN 200 introduces two modulation parameters- shift (As) and scale (ō)-to to the convolutional weights affecting StyleGAN's "s" space. DynaGAN's architecture can be defined as w (s; + As)- 5. Similar to HyperDDmainNNet, the non-linear combinability of the scale modulation parameter impedes its ability achieve robust domain association. In contrast, our DoRM solely embraces a sample-specific domain shift denoted as As; formalized as w (st + As;). This distinct design substantially enhances the generator's learning potential, enabling adaptation a diverse spectrum of target domains, even in the presence of considerable domain gaps. Notably, the sample-specific domain shift proves sufficient for both few-shot and one-shot GDA, eliminating the necessity for an additional domain scale parameter. This streamlined approach not only fosters domain association but has been effectively demonstrated. Additionally, although (30] defines and addresses related task called Domain Expansion and domain compasition (referred to memory and domain association our paper), inadvertently curtails the source domain's 's generative ca- pacity, amplifying the intricacies and temporal requisites of domain adaptation. As depicted in the experimental results, 30 encounters challenges when faced with substantial domain gaps between source and target domains. Evidently, its performance is compromised, as evidenced in instances such as adapting to the 10-shot generation context of the Sketch dataset and the one-shot generation context of the "elsa" dataset.
Furthermore, 20

---

3 Approach
In this section, we present our generator structure, DoRM (Section 3.1 which takes inspiration from the way human brains learn new knowledge in different domains. DoRM excels in generating high-qe nigh-quality, diverse images while maintaining strong cross-domain consistency in few-shot GDA. Additionally, it possesses the unique ability to integrate multiple learned domains, allowing for the synthesis of images in hybrid domains that were not present in the training dataset. To address the issue of discriminator overfitting, we freeze the backbone of the source discriminator and introduce target domain classifier (Section 3.2) Furthermore, we propose a novel similarity- aamilarity-based structure loss, denoted L.z (Section 3.3 to enhance the maintenance of cross-domain consistency. Finally, we introduce the overall training loss in Section 3.4
3.1 Domain Re-modulation Structure of Generator
StyleGAN2. Our method is applied to a pre-trained StyleGAN2 [9]. Unlike general GANs that directly feed latent code to the generator, StyleGAN2 [19] employs a non-linear mapping network f(+) to transform the latent space Z into an intermediate latent space W. The latent code (w E W) is transformed into the style code (s E S) through a learned affine transformation A. Finally, the style code is inserted into the synthesis network g(-) through the modulation component at each convolution layer.
Domain Re-modulation of Generator. To equip the generator with memory and inherit the semantic information (e.g., glasses, gender, ages) from the source domain, we propose DoRM, illustrated in Figure 2 We freeze the given StyleGAN2 model pre-trained on the source domain and employ new mapping network fi and new affine transformation A, (M&A module) to build Wi space and S, space, respectively. Through the new M&A module, we obtain domain shift well as the target style codes. The added fe and Ag have the same architecture as fa and Ag. respectively, and are initialized by fa and Ag. Accordingly, each layer of the synthesis network is controlled by source and target style codes. Given a latent code 2 sampled from Normal distribution 2 (z E 2),
Wt = f((z)= E We, W/ = fz( E Ws.
(1)
where W. and W. contain information from the target domain and source domain, respectively. Transformed by their own learned affine layers (Eq. 2 the information in these two domains represents their respective styles and is combined into general styles s (Eq. 3 to control the synthesis network together:
S; Aw,), 8, = A.,w.,,
$ = as + (1- a))2.
(3)
where a a hyper-parameter that controls the strength of the domain shift. In GDA, it tends to reuse various factors like pose, content, structure, etc. from the source domain and learns the most distinguishable characteristies of the target domain. To preserve variation factors as much as possible in the source domain, a is set to a relatively small value (More analyses can be found in Section A.I). The combined style code s modulates the convolution weights through w = - W's, where w/ the modulated weights. Then, demodulation is employed to eliminate the influence of the style code 8 from the statistics of the convolution's output feature maps, which has been formalized as my
Tar where ||'") lw'lla represents the L2-Norm function.
3.2 Target Domain Classifier
In GANs training. the discriminator typically distinguishes the training images from the generated images. However, in few-shot GDA, where there are extremely few training images, the discriminator can easily overfit to the training images, leading to severe model collapse in the generator. In this work, we treat the discriminator as a target domain classifier that measures the probability of the images belonging to the target domain. Defining the target domain by only few training images is ambitious, so we desire the discriminator to extract the most representative feature of training images that portray the target domain. Inspired by 22), we reuse and fix the feature extractor d(.) of the pre-trained source discriminator to inherit its strong feature extraction capability. Additionally, we introduce a target domain classifier Φf+) on the top of the feature extractor d(.). We use two-layer multi-layer perceptron (MLP) as the target domain classifier <6-) and the target domain classifier
5

---

(()) and it is updated from scratch, which outperforms directly fine-tuning the final layer of the original discriminator, especially for large domain-gay GDA. Given an image z, the discriminator measures the probability that the image belongs to the target domain using p = o ((d(x)).
3.3 Similarity-based Structure Loss
To explicitly model the cross-domain consistency in generative domain adaptation, we propose novel similarity- structure loss called L.s. Our intuition is that the auto-correlation maps of the source image and its corresponding target image should be consistent during generative domain adaptation. To achieve this, we extract the intermediate tokens FA and F3 of the source image IA and its corresponding target image Lg from the k-th layer pf the CLIP image encoder. These FB. respectively. We define the F F and F
tokens are denoted by FA FA.
FA and FB FB:
auto-correlation maps MA
[PAT
F
X Fa and MB
[Pal
FF
Fg
where M [F
Fg.
F
M
F The Las is then defined as the L. norm of the difference between MA and MB: Fb Fbl
Las
I -M lb
where ||
is the L2 norm function.
3.4
Overall Training Loss
Our overall training loss includes original adversarial training loss in StyleGAN2 and the similarity- based structure loss Las:
AgsCss. LD = - -=xeptt))lo D(G(=))) - Ex [Log D(x)))
CG - -=:n-tttllog
(5)
where X is the training dataset. In our experiments, we set Ags = 10.
Experiments
We begin by introducing the experimental settings of method, which encompass implementation, datasets, and metrics (Section[4.1) Next, we apply our method to various 10-shot datasets, showcasing its performance in Section 4.2, Section 4.3 explores the capabilities of domain association among multiple domains. Furthermore, we conduct ablation studies (Section 4.4 and Section AI to assess the impact of both the similarity-based structure loss and the generator structure. Additionally, provide the user study in Section A.4 and demonstrate the applicability of our DoRM to one-shot GDA Section A.3
4.1 Experiments Settings
Implementation. We adopt the pre-trained StyleGAN2 19] on FFHQ 18 and our training parameters and settings follow StyleGAN2-ADA [17] Due to the small amount of training data, we set the batch size to 4, and we terminate the training process after the discriminator has processed 100K real samples. Our implementation is based the official implementation of StyleGAN2-ADA
as our source model,
Datasets. Following previous literature 32 59],
use FFHQ [18 with resolution 256 x 256 as the source domain. In 10-shot GDA, we evaluate our method on multiple target datasets, including
Sketches AT. FFHQ-Babies (18), TFHQQ-Snglasser [18. Face-Caricatures. Face paintings by
Amedeo Modigliani, Face paintings by Raphael, and Face paintings by Otto Dix [0] All the training datasets shown in Section
we use Fréchet Inception Distance (FID) 13 as the metric to evaluate the synthesis quality and diversity simultaneously. Additionally, we adopt intra-cluster LPIPS (intra- LPIPS) [59] (32 based on I.PIPS 54 measure the synthesis diversity. Specifically, we synthesize 1000 images and then assign each of the synthesized images to the & training images with the lowest
Metrics. Following 559 32

---

LPIPS distance, forming k clusters. We calculate the average LPIPS distance within each cluster and then average over all the clusters. Furthermore, we report the identity (ID) similarity [56] predicted by the Areface | to measure the preservation of identity information, which is a metric to measure the cross-domain consistency.
4.2 10-shot Generative Domain Adaptation
Qualitative comparison. In 10-shot GDA, following the previous studies [32 we sample 10 training images from the target domain to transfer the pre-trained generator. Figure shows the results of 10-shot GDA with different methods. We observe severe generator overfitting in the FreezeD 29). GenDA [49 shows high quality and diverse synthesis but fails to maintain cross-domain consistency. CDC (32) and RSSA (47 improve the cross-domain consistency, but the synthesis quality is unsatisfactory. AdAM [58] also increases the synthesis diversity but fails in synthesis quality. By contrast, method preserves all the domain-sharinn attributes from the source domain by freezing the original generator and acquires the domain shift through new concurrent network components. Our method shows appealing synthesis quality and diversity while maintaining better cross-domain consistency than previous methods. More qualitative results can be found Section A.2 Section A.5 and Section A.6
Quantitative comparison. We use Fréchet Inception Distance (FID) to evaluate synthesis quality and diversity (lower is better). In 10-shot GDA, to better reflect the synthesis diversity use the Intra-LPIPS(32) to measure the synthesis diversity (higher means better synthesis diversity) Additionally, we measure cross-domain consistency using Identify similarity (ID) [56] (higher better). Table T summarizes the three evaluation metrics for 10-shot GDA over different target domains. FreezeD (29 and GenDA [9] struggle to maintain cross cross-domain consistency. CDC (32),
RSSA 47). DCL [59] and AdAM (58) improve the synthesis diversity and cross dross-domain consistency but fails in the synthesis quality (as in Figure 3), leading to unsatisfactory FID scores. Our method outperforms all these methods on the different target domains, achieving not only better synthesis diversity and cross-domain consistency (higher Identity score and Intra-I.PIPS) but also better synthesis quality (lower FID score).
Table 1: Quantitative evaluation on 10-shot GDA. All the source generators are pre-trained on FFHQ The target domains include Sketches, FFHQ-Bale and metrics include FID, Intra-LPIPS (I-LPIPS). and Identify similarity (ID). Noting that Sketches dataset
Evaluation
only contains about 300 images, the large synthesis diversity will harm to the FID score.
Datasets
FFHO-Babies
TFHO-Sunglassed
Sketches I-LPIPS
Method
FID
I-LPIPS
ID
FID
I-LPIPS
ID
FID
ID
minegan
98.23
0.514
0.132
68.91
0.42
0.171
64.34
0.40
0.092
FreezeD
110.92
0.346
0.037
51.29
0.337
0.030
46.54
0.325
0,010
GenDA
47.05
0.556
0.029
22.62
0.548
0.004
31.97
0.407
0.011
CDC
74.39
0.573
0.326
42.13
0.562
0.318
45.67
0.453
0.214
RSSA
77.77
0.576
0.314
70.41
0.563
0.307
63.44
0.480
0.296
DCL
52.56
0.582
38.01
37.90
0.486
AdAM
48.83
0.590
0.249
28.03
0.592
0.306
55.74
0.495
0.198
DoRM
30.31
0.623
0.445
17.31
0.644
0.389
40.05
0.502
0.365
4.3
Multi-domain and Hybrid-domain Generation
Multi-domain generation. Our proposed DoRM utilizes a novel approach by learning and retaining knowledge of new domains through the formation and update of new M&A modules, instead of updating the entire generator. This unique characteristic enables our DoRM, with a single generator, to generate images across multiple domains (as depicted in the top part of Figure 4. In contrast that require updating the entire generator for few-shot GDA, limiting its capability into single domain generation, our DoRM offers significant storage space savings in than previous methods [32 on 10-domain generation.
to previous methods [32 47
59 58)
generation. As shown Table our method's models are about 3x smaller
Hybrid-domain generation. In contrast to previous methods 32 generator, our DoRM achieves effective linear domain shift through the update of new M&A
55 that fine-tune the entire
7

---

Frezzel
GenDA
CDC
RSSA
AdAM
Real babies
DoRM
Source
Frezzel
GenDA
CDC
RSSA
AdAM
DoRM
Real
sketches
Source
Figure 3: Qualitative comparison 10-shot GDA. The source domain is FFHQ, and the target domains include Sketches and FFHQ-Babies. We compare our method with FreezeD[29) GenDA (4), CDC RSSA[47] and AdAM[58). Feeding the same latent code into the source generator and the target generator, we obtain the corresponding images in the source and target domains. Our method shows better cross-domain consistency and synthesis quality than the other methods.
modules. By learning multiple M&A modules that store the knowledge of various training domains, DoRM not only enables multi-domain generation but also facilitates the synthesis of images in hybrid domains not present in the training data (as depicted in the bottom left part of Figure 4 For comparison, we conduct hybrid-domale generation experiments using CDC [32]. In CDC, the domain shift between the source and target domain generators is obtained by subtracting their corresponding parameters. Similar to our DoRM, CDC achieves hybrid-domain generation by combining multiple domain shifts, adding the parameters of each shift the source domain generator. However, as illustrated in Figure CDC exhibits unsatisfactory synthesis quality in hybrid-doman generation. In contrast, the images synthesized by DoRM not only inherit domain-sharing attributes such as pose, gender, and identity, but also seamlessly blend domain-specific attributes. Furthermore, we extensively explore domain association among multiple domains using our proposed DoRM in Figure
5 The results demonstrate that DoRM efficiently integrates diverse domains to generate high-quality

---

Multi-domain generation (Ours)
Source domain
Sketch domain
Baby domain Hybrid-omman G Generation (CDC)
Sunglasses domain
Hybrid-domain domain G Generation (Ours)
Sketch-Bai Babb domain
Sketch- Sunohh--snglsssss domain
Sketch- Baby domain
Sketch-Sunglast domain
Figure 4: Qualitative results on multi-domain and hybrid-domane generation. The top part shows the multi-domain generation of our methods on three target domains. The bottom part illustrates the comparison on hybrid-domain generation between our proposed DoRM (bottom left) and CDC [32] (bottom right). The source generator is pre-trained on FFHQ 18] The adopted M&A modules in our DoRM are trained in Section 4.2
images in hybrid domains not present in the training data. In this case, our proposed DoRM has the ability to generate creative outputs in previously unseen domains by incorporating knowledge learned from other domains, similar to how humans can draw past experiences to generate novel ideas. To the best of our knowledge, our proposed DoRM the first method to achieve efficient domain association in this manner. Additional visualizations of hybrid domain generation can be found in Section A.6
Table 2: Storage Comparison. The number of generator parameters required to realize different multi-domain generations.
Model Size
2-domain
5-domain
10-domain
CDC B32
48M
120M
240M
Ours
30M
54M
84M
4.4 Ablation Study
We conduct ablation studies to evaluate the impact of our proposed DoRM generator structure and similarity-based structure loss. As shown in Figure 6 the images synthesized by baseline (which simply fine-tunes the StyleGAN2 generator with adversarial loss [18) exhibit severe overfitting to the training images. By contrast, fine-tuning our DoRM structure generator with adversarial loss yields better synthesis diversity than the baseline. Moreover, when we optimize our DoRM generator structure with both adversarial loss and similarity-based structure loss, the synthesis images exhibit appealing quality and diversity, while maintain high cross-domain consistency, which indicates the proposed loss effectively preserves cross cross-domas domain consistency in few-shot GDA. More ablation studies on the network component of our DoRM are presented in Section A.
Limitation and Conclusion
Limitation. i) Although our method can achieve appealing results on the few-shot GDA across different target domains, however, the strength of domain shift currently needs to be manually adjusted. Furthermore, since the various depth layers capture different semantic attributes, equipping each layer with appropriate may help to further improve synthesis quality. ii) The current

---

Source
Sketch
Caricature
Otto Dix painting
Baby
Sketch
Baby-Caricatun
Baby- Dix
Sunglasses
Sunglasses- Sketch
Sunglasee-Caricahuure Caricalure Sunglasses Otto Dix
Figure 5: Domain association between different domains by our DoRM. The first row and the first column show the 10-shot GDA results on different target domains. By simply combining the corresponding pretrained M&A modules in 10-shot GDA, DoRM can integrate the domains and synthesize the high-quality images in the hybrid domains.
Source domain
Baseline
DoRM
DoRM + +
Figure 6: Ablation study on DoRM and Les. The training images are 10-shot sketches.
manuscripts only simply combines the M&A modules of different target domains and activate them at the same time to realize the domain association. To further improve the performance of the domain association, not only combining the trained target M&A modules but also employing a new M&A module and additional consistency loss is better method to blend the target domains.
Conclusion. In this paper, we present DoRM, novel generator structure inspired by the learning and storage mechanisms of the human brain, specifically designed for few-shot GDA. Our DoRM is characterized by its simplicity and efficiency. Additionally, we introduce novel similarity-bart based structure loss to ensure cross-domain consistency in the few-shot GDA. Through extensive qualitative superiority of our method over existing approaches
and quantitative evaluations, we demonstrate in terms of synthesis quality, diversity, and cross-domaii consistency. Importantly, akin to the human brain, our DoRM exhibits memory and the ability to integrate knowledge from different domains, enabling the generation of images in novel hybrid domains not encountered during training.
10

---

6 Acknowledgments
The work was supported in part by the National Natural Science Foundation of China under Grands U19B2044 and 61836011.
References
[1] Aibek Alanov, Vadim Titov, and Dmitry Vetrov. Hyperdomainnet: Universal domain adaptation for generative adversarial networks, arXiv preprint www.v:2210.00880000 2022.
[2]
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1809.
1ee.1889.119900 11096, 2018.
[3] Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay. Sameh Khamis, et al. Efficient geometry- aware 3d generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages (6123-16133, 2022.
[4] Chun-Chao Chen, Jie-Kai Wu, Hsuan- Wen Lin, Tsung-Pin Piig Pai, Tsai-Feng Fu, Chia-Lin Wu, Tim Tully, and Ann-Shyn Chiang. Visualizing long-term memory formation in two neurons of the drosophila brain. science, 335(6069))678-683) 2012.
[5] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[6] Min Jin Chong and David Forsyth. Jojogan: One shot face stylization. In Computer Vision- ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings,
Part XVI, pages 128-152. Springer, 2022.
[7]
Antonia Creswell and Anil Anthony Bharath. Inverting the generator of generative adversarial
network. IEEE Transactions on Neural Networks and Learning Systems, 30(7):9667-1974, 2019.
domain adaptation. arXiv preprint arXiv:0907.1815 2009. [9] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4690-4699, 2019.
[8] Hal Daumé III. Frustratingly
[10] Rinon Gal, Or Patashnik, Haggai Maron, Gal Chechik, and Daniel Cohen-Or. Stylegan Clip-guided domain adaptation of image generators. arXiv preprint arXiv:2108.00000 2021. [11] Ian Goodfellow, Jean Pouget-Abatee Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems. 27, 2014.
[12) Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729-9733 2020.
[13] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[14] Jongheon Jeong and Jinwoo Shin. Training gans with stronger augmentations via contrastive discriminator. arXiv preprint arXiv:2203.. 09742, 2021.
[15] Liming Jiang, Bo Dai, Wayne Wu, and Chen Change Loy. Deceive d: Adaptive pseudo aug- mentation for gan training with limited data. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021.
[16] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint
10196, 2017. [17] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. Advances in Neural Information Processing Systems,
14, 2020.
[18] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 4401-44 2019.
[19] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEF/CVF conference on computer vision and patiern recognition, pages 8110-8119 19, 2020.
[20] Seongtae Kim, Kyoungkook Kang, Geonung Kim, Seung-Hwan Baek, and Sunghyun Cho. Dynagan: Dynamic few-shot adaptation of gans to multiple domains. SIGGRAPH Asia 2022 Conference Papers, pages 1-8, 2022.
11

---

[21] Bonan Li, Zicheng Zhang, Xuecheng Nie, Congying Han, Yinhan Hu, and Tiande Guo. Styo: Stylize your face in only shot. arXiv preprint arXiv:2303. (3Xi::2333.0333,, 2023.
[22] Gang Li, Heliang Zheng, Daqing Liu, Chaoyue Wang, Bing Su, and Changwen Zheng. Semmae: Semantic-guider masking for learning masked autoencoders. Advances in Neural Information Processing Systems, 15:14290-144
2022.
[23] Yijun Li, Richard Zhang. Jingwan (Cynthia) Lu, and Eli Shechtman. Few-shot image generation with elastic weight consolidation. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 15885-15896. Curran Associates, Inc., 2020.
[24] Ziqiang Li, Rentuo Tao, Jie Wang, Fu Li, Hongjing Niu, Mingdao Yue, and Bin Li. Interpreting the latent space of gans via measuring decoupling. IEEE transactions on artificial intelligence,
2(1):58-70, 2021.
[25] Ziqiang Li, Muhammad Usman, Rentuo Tao, Pengfei Xia, Chaoyue Wang, Huanhuan Chen, and Bin Li. A systematic survey of regularization and normalization in gans. ACM Comput.
Surv., nov 2022. Just Accepted.
[26] Ziqiang Li, Chaoyue Wang, Heliang Zheng, Jing Zhang, and Bin Li. Fakeclr: Exploring con- trastive learning for solving latent discontinuity in data-efficient gans. In European Conference
on Computer Vision, pages 598-615. Springer, 2022.
[27] Ziqiang Li, Xintian Wu, Beihao Xia, Jing Zhang, Chaoyue Wang. and Bin Li. comprehensive survey on data-efficient gans in image generation. arXiv preprint (rxXi:2204.08322 2022.
[28] Ziqiang Li, Pengfei Xia, Rentuo Tao, Hongjing Niu, and Bin Li. A new perspective on stabilizing gans training: Direct adversarial training. IEEE Transactions on Emerging Topics in Computational Intelligence, 7(1):178 189, 2022.
[29] Sangwoo Mo, Minsu Cho, and Jinwoo Shin. Freeze the discriminator: a simple baseline for fine-tuning gans, arXiv preprint arXiv:2002. 10964, 2020.
[30] Yotam Nitzan, Michaël Gharbi, Richard Zhang, Taesung Park, Jun Yan Zhu, Daniel Cohen-Or Or, and Eli Shechtman. Domain expansion of image generators. arXiv preprint 2:31:22301.05225,
2023.
[31] Atsuhiro Noguchi and Tatsuya Harada. Image generation from small datasets via batch statisties adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
[32] Utkarsh Ojha, Yijun Li, Jingwan Lu, Alexei A Efros, Yong Jae Lee, Eli Shechtman, and Richard Zhang. Few-shot image generation via cross-domain correspondence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10743-10752, 2021. [33] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning.
pages 8748-8766 PMLR, 2021.
[34] Esther Robb, Wen-Sheng Chu, Abhishek Kumar, and Jia-Bin Bin Huang. Few-shot adaptation of generative adversarial networks. arXiv preprint arXiv:2010. / 2020.
[35] Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xI: xl: Scaling stylegan to large diverse datasets. In ACM SIGGRAPH 2022 Conference Proceedings, pages 1-10, 2022.
[36] Yujun Shen, Ceyuan Yang, Xiaoou Tang, and Bolei Zhou. Interfacegan: Interpreting the disentangled face representation learned by gans. IEEE transactions on pattern analysis and machine intelligence, 44(4):2004-200 2020.
[37] Rentuo Tao, Ziqiang Li, Renshuai Tao, and Bin Li. Resattr-gan gas: Unpaired deep residual attributes learning for multi-doman face image translation. IEEE Access, 11132594-13266
2019.
[38] Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or. Designing an encoder for stylegan image manipulation. ACM Transactions Graphics (TOG), -044)
2021.
[39] Hung-Yu Yu Tseng, Lu Jiang, Ce Liu, Hiug- Yang. and Weilong Yang. Regularizing generative adversarial networks under limited data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7921-7931, 2021.
[40] Mei Wang and Weihong Deng. Deep visual domain adaptation: A survey. Neurocomputing, 312:135-153, 2018
[41] Xiaogang Wang and Xiaoou Tang. Face photo-sketch synthesis and recognition. IEEE transac- tions on pattern analysis and machine intelligence. 31(11): 2008 David Berga, Luis Herranz, Fahad Shahbaz Khan, and Joost van de Weijer. Minegan: Effective knowledge transfer from gans to target domains with few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
[42] Yaxing Wang, Abel Gonzalez-Garee
12

---

Recognition (CVPR), June 2020.
[43] Yaxing Wang, Chenshen Wu, Luis Herranz, Joost de Weijer, Abel Gonzalez-Garcia, and Bogdan Raducanu. Transferring gans: generating images from limited data. In Proceedings of the European Conference on Computer Vision (ECCV), pages 218-234, 2018.
[44] Conghao Wong, Beihao Xia, Ziming Hong, Qinmu Peng, Wei Yuan, Qiong Cao, Yibo Yang. and Xinge You. View vertically: A hierarchical network for trajectory prediction via fourier spectrums, In European Conference on Computer Vision. pages 682-700. Springer, 2022.
[45] Zongze Wu, Dani Lischinski, and Eli Shechtman. Stylespace analysis: Disentangled controls for stylegan image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12863-12872 2021.
[46] Beihao Xia. Conghao Wong, Qinmu Peng, Wei Yuan, and Xinge You. Cscnet: Contextual semantic consistency network for trajectory prediction in crowded spaces. Pattern Recognition, 108:52 2022.
[47] Jiayu Xiao, Liang Li, Chaofei Wang, Zheng-Jun Zha, and Qingming Huang. Few shot generative
model adaption via relaxed spatial structural alignment. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 11204 1120-11213, 2022.
[48] Ceyuan Yang, Yujun Shen, Yinghao Xu, and Bolei Zhou. Data-efficient instance generation from instance discrimination. Advances in Neural Information Processing Systems,
2021.
[49] Ceyuan Yang, Yujun Shen, Zhiyi Zhang, Yinghao Xu, Jiapeng Zhu, Zhirong Wu, and Bolei Zhou. One-shot generative domain adaptation. arXiv preprint arXiv:211 1009876, 2021.
[50] Jordan Yaniv, Yael Newman, and Ariel Shamir. The face of art: landmark detection and geometric style in portraits. ACM Transactions on graphics (TOG), 38(4):1-15, 2019.
[51] Kaichao You. Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I Jordan. Universal domain adaptation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
2019.
[52] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a Sarge-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 1506. 03365, 2015.
[53] Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In International conference on machine learning, pages 7354-7363.
PMLR, 2019.
[54] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
[55] Yabo Zhang, mingshuai Yao. Yuxiang Wei, Zhilong Ji, Jinfeng Bai. and Wangmeng Zuo. Towards diverse and faithful one-shot adaption of generative adversarial networks. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[56] Zicheng Zhang, Yinglu Liu, Congying Han, Tiande Guo, Ting Yao, and Tao Mei. Generalized one-shot domain adaption of generative adversarial networks. arXiv preprint Xix:2209.003665 2022.
[57] Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for data-efficient gan training. Advances in Neural Information Processing Systems, 33:7559-7570,
2020.
[58] Yunqing Zhao, Keshigeyan Chandrasegaran, Milad Abdollahzadeh, and Ngai-Man Man Cheung. Few-shot image generation via udaptation-awant kernel modulation. Advances in Neural Information Processing Systems, 35: 19427-199440 2022.
[59] Yunqing Zhao, Henghui Ding, Houjing Huang, and Ngai-Man Cheung. A closer look at few- shot image generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9140-91 950,, 2022.
[60] Zhengli Zhao, Sameer Singh, Honglak Lee, Zizhao Zhang, Augustus Odena, and Han Zhang Improved consistency regularization for gans. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 11033-11041, 2021.
[61] Zhengli Zhao, Zizhao Zhang, Ting Chen, Sameer Singh. and Han Zhang. Image augmentations for gan training. arXiv preprint arXiv:2006.00 02595, 2020.
[62] Jingyuan Zhu. Huimin Ma. Jiansheng Chen, and Jian Yuan. Few-shot image generation via masked discrimination. arXiv preprint arXiv:2210.1572
2022.
[63] Peihao Zhu, Rameen Abdal, John Femiani, and Peter Wonka. Mind the gap: Domain gap control for single shot domain adaptation for generative adversarial networks. arXiv preprint arXiv:2110.08398, 2021.
13

---

A Supplementary Materials
A.1 Ablation Study
Effect of Target Mapping, In this section, we first investigate the significance of the target mapping module in our proposed DoRM. To evaluate the significance of the target mapping, we conduct an ablation study using the source mapping as a substitute for the target mapping, which remains frozen during the entire 10-shot generative domain adaptation training. As shown in Table [3] our results indicate a significant deterioration in the FID score without the target mapping, implying a considerable drop in the quality and diversity of the generated samples. Furthermore, Figure illustrates that the generative domain adaptation barely occurs during training without the target mapping. This because the target mapping plays crucial role capturing the representative attributes of the target domain and assisting in the acquisition of the domain shift during the adaptation process.
Souace
Seatch
Boby
Sundl
WTGG
we TC
Figure 7: Qualitative ablation study of the target mapping. We compare the performance of our DoRM method, shown in the first row, with a variant of the method where the frozen source mapping is used substitute for the target mapping, depicted in the second row. We evaluate the performance of both methods on three different target domains: Sketches, FFHQ-Babies and UFHO--Snglassee
Table 3: Quantitative ablation study of the target mapping. The evaluation metric is FID (lower is better). We compare the performance of our DoRM method, shown in the first row, with variant of the method where the frozen source mapping is used as a substitute for the target mapping, depicted in the second row. The source generator is pre-trained on FFHQ[T8), and the target domains include FFHQ-Babies and FFHQ-Sunglasses.
Babies
Sunglasses
DoRM
30.31
17.31
DoRM w/o Target Mapping
86.52
74.71
Effect of Re-Modulation Layers. Another crucial component of our proposed DoRM approach is the target affine module. To investigate the roles of the different target affines in DoRM, we perform experiments where drop the target affines in both the low-resolution and high-resolution feature maps. Specifically, we conduct 10-shot generative domain adaptation experiments to evaluate the Fréchet Inception Distance (FID) of the generated samples. For image with resolution of 256 x 256, the low cww-resolution feature maps include resolutions of 4 x 4, 8 x 8, 16 x 16, and 32 x 32. As presented in Table 4 our results demonstrate that all target affines are crucial for the performance of our DoRM approach and their removal leads to a significant drop in the quality and diversity of the generated samples.
Effect of Re-Modulation Weight. The re- re-modulation weight is a crucial parameter that controls the strength of the acquired domain shift in proposed DoRM approach. A small re-modulation weight leads to a lower strength of the domain shift, resulting in more attributes of the source domain being preserved during generative domain adaptation. To investigate the impact of the re-modulation weight, we conduct 10-shot generative domain adaptation experiments using different re- re-mouulation weights. The results presented in Table Our results demonstrate that different domain gaps have different optimal re-modulation weights, indicating that the selection of the re- re-modulation weight should be tailored the specific target domain.
Effect of Target domain classifier. We investigate the effect of the target domain classifier on the performance of our proposed DoRM approach. Specifically, experiment with different depths
14

---

Table 4: Quantitative ablation study of the target affine layers. The evaluation metric is FID (lower better). We compare the FID score of our proposed DoRM method, shown in the first row, with two variants: one where the target affines are removed from the low-resolution feature maps, shown in the second row, and another where the target affines are removed from the high-resolution feature maps, shown in the third row. We use source generator pre-trained on FFHQ 08 and evaluate all three methods on two different target domains: FFHQ-Babies and UFHQ-Sunglassed
babies
sunglasses
DoRM
30.31
17.31
DoRM w/o target affines in low resolution
93.28
92.42
DoRM w/o target affines in high resolution
37.16
20.81
Table 5: Quantitative ablation study of the re-modulation weight. The evaluation metric is FID
(lower is better). We conduct 10-shot generative domain adaptation experiments using our DoRM approach with varying re- re-modulation weights. We use source generator pre- pre-trained on FFHQ and evaluate our method on two different target domains: FFHQ-Baie -aby and SSngo--slssssss...
[8
Re-modulation weight a
0.5
0.2
0.05
0.005
0.001
FFHQ-Bate -Bayy FFnQ--Sunllassss
37.9
36.1
34.0
30.3
32.3
18.7
17.3
17.9
18.5
19.4
(e.g, number of MLP layers) and initialization methods for the target domain classifier 10-shot generative domain adaptation. As presented in Table our results indicate that the two-layer MLP target domain classifier achieves the best performance. This is because the one-layer MLP lacks the ability to classify the target domain effectively, while the three-layer MLP is prone to overfitting due to the limited number of training images.
Ablation of Individual Components of Previous State-of-the-aat Methods. To highlight the specific strengths of the proposed method that outperform other works, we consider the individual components and techniques utilized in previous state-of-the-art approach [55] Specifically, we systematically analysis and compare individual components and techniques utilized in our DoRM and DiFa [55 to showcase how our method surpasses these works. According to the Introduction of the manuscript, GDA needs three fundamental properties. To resolve it, DiFa (55) proposed two CLIP-based loss: global loss Lgiobul and local loss Liocal for realizing large diversity/croos-domee domann consistency and high quality, respectively. Differently, we adopt Similarity- -bssed Structure Loss (Tegs) and adversarial loss (Tuavv) for realizing large diversity/cross-donal domaiin consistency and high quality, respectively. As shown in the Figure &8 the I-tocal in DiFa mainly focuses on textual features and failes capture the complete features (e.g. the white background in sketches) in generative domain adaptation. In our DoRM, we employ adversarial loss which can fully capture the features of the target images.
Table 6: Quantitative ablation study of the target domain classifier. The evaluation metric is FID (lower is better). We experiment with different depths of the target domain classifier, using various initialization methods. We use source generator pre-trained on FFHQ 18] and evaluate our proposed DoRM approach on 10-shot generative domain adaptation tasks.
FFHQ Baby
FFHQ-Sume Sunglasses
MLP Depth source initial
random initial
source initial random initial
one layer
38.03
37.15
24.63
22.15
three layers 33.32
two layers
30.31
20.40
17.31
33.25
20.42
A.2 More Synthesis Results in 10-shot GDA
Qualitative results on 10-shot generative domain adaptation. We present qualitative results of our proposed DoRM approach on more target datasets, including face caricatures, face paintings by
Raphael, face paintings by Amedeo Modigliani, and face paintings by Otto Dix [50]. All training
images are shown in Figure 9 and the qualitative results presented in Figure I0 Our results
15

---

Souroo(FFH))
Ladv + Lis
Lioca + Lgtobat
Llocal + bss
Figure 8: Ablation of individual components of previous state-of-the-art methods.
demonstrate that our proposed DoRM approach achieves appealing synthesis quality in various target domains.
Results of latent interpolation. We perform latent space interpolation to demonstrate that our DoRM is not harmful to the learned latent space. Figure [T] the first and last columns show the generated images with two latent codes after 10-shot generative domain adaptation, while the remaining columns show the results obtained by linearly interpolating the two latent codes. Our results demonstrate that all intermediate synthesized images have high target-domain consistency and high cross-domain consistency. Moreover, the semantics of the generated images, such as gender, haireut, and pose, vary gradually throughout the interpolation, indicating that our proposed DoRM approach preserves the underlying semantic structure of the learned latent space.
Results of latent editability. We have performed editing on a real image adapted into new target domain using StyleGAN-CLIP to discover editing directions in the source domain. The results, as illustrated in Fig [2] indicate that the adapted generator maintains similar ftent-bbsed editing capabilities to the original generator. This demonstrates the preservation of editability in the adapted generator.
Results on 3D generator. We conducted some initial studies using the popular 3D-aware image generation method, EG3D [. for one-shot GDA with FFHQ as the source domain and Sketch as the target domain. The results, as shown in the Figure 13] reveal that directly migrating the proposed DoRM to 3D one-shot domain adaptation poses challenges and might not be straightforward. Some potential challenges in the 3D-GAN domain include:
(1) Overfitting: 3D GANS often face more severe overfitting issues when dealing with limited training data, requiring more rigorous regularization techniques and training strategies for one-shot GDA.
(2) Volumetric Data Representation: Handling volumetric data representation in 3D-GANs necessi- tates specialized techniques for data manipulation, augmentation, and visualization.
(3) Spatial Artifacts: Generating high-quality 3D objects may encounter spatial artifacts, such as geometric distortions or inconsistent shapes, which need to be addressed.
(4) Computational Complexity: The computational demands of 3D-GANs can be significantly higher than their 2D) counterparts, presenting challenges in both training and inference.
Results on three-domain hybrid domain generation. As depicted in the Figure 14 we have showcased an example of applying our method to create hybrid-domain images by activating the trained M&A modules of Baby, Sunglasses, and Sketch domains. This illustration demonstrates how easily our approach can be adapted to generate hybrids involving more than two domains, showcasing the versatility and potential of our approach.
A.3 Experiments on one-shot Generative Domain Adaptation
Although our DoRM is mainly for few-shot generative domain adaptation, DoRM also can be employed for one-shot generative domain adaptation [21 6]. In the one-shot GDA, the training dataset single image, which is difficult for the backbone of discriminator to extract the main characters of the target domain because of the overfitting issue. In this case, we introduce a clip-based local-level adaptation loss Lioent from (55) to help to acquire the local-level characters and styles of the target domain. Concretely, we extract the intermediate tokens of the adapted image In synthesized by DoRM and the single target image her from the & layer of CLIP image encoder. And align each of adapted token F3 with its closest target token from Fror, where FB Fl...., F...FB and
16

---

Flar Flar:... Far are the extracted tokens. The clip- hipp-baeed local-level adaptation loss is defined as:
1
Aocat max
Σ
min Ct.j.
Σ
min Ci.z)
(6)
n
j
where C' is calculated as:
FB Flar [Fi Faar
C2. =
(7)
Furthermore, to better identify and maintain the Nomain--haring attributes in one-shot generative domain adaptation, we also employ the inversion-base from [55]. Specifically, this loss function aims to identify and preserve gomain-sharing attributes in the W + space. The underlying assumption is that attributes that are similar in W+ space between the source and target domains during adaptation are more likely to be domain-sharing attributes. To achieve this, Lsce dynamically analyzes and retains these attributes. First, it inverts the source and corresponding target images into W+ latent codes, WA and wa, respectively, using pre-trained inversion model such as pSp pr e4e, for each iteration. Next, it computes the difference Awc, between the centers of source queue of W+ latent codes, X A and the target queue of W + latent codes, XB, where X and X B are dynamically updated with WA and wy during training. The loss function then encourages a and We to be consistent in channels with less difference, thereby facilitating the preservation of Nomain-sharing attributes. The inversion- based selective cross-modal consistency loss Lec is defined as follows:
selective cross-modal consistency loss Lsco
(8) where represents the proportion of preserved attributes, and mask(Aw. a) determines which channels to retain. Specifically, let Awa.x be the a - th largest element of Aw. Then, each dimension of mask(Aur a) is calculated as follows:
Loce = ||mask( a) - (wB wA)lla
mask( a): {
A < AWs..
Aw 2 Aw...
We compare our DoRM++ approach which denotes introducing the two new loss terms into training with otate-of-the-art one-shot generative domain adaptation (GDA) methods. including JoJoGAN 6). Generalized One-shot Domain Adaptation (56), DynaGAN and DiFa (55). Figure [6 shows the comparison results. Our results indicate that JoJoGAN, DynaGAN and Generalized One-shot Domain Adaptation fail to achieve GDA when the target image FFHQ-Bate and FFHQ-Sune S and the synthesis quality of DynaGAN is limited. Similarly, DiFa also fails to achieve GDA when the target image is HFFH--Sunglasser and the synthesis diversity is unsatisfactory when the target image is FFHQ-Baby.
In contrast, our DoRM++ approach achieves one-shot GDA among all the reference images, resulting in high-quality and diverse synthesis, while maintaining appealing cross-domain consistency. More- over, our DoRM++ generator has memory realize multiple target domains' generation, which saves significant amount of storage space. Our DoRM++ generator also has the ability to integrate the learned knowledge of multiple target domains to synthesize images in hybrid domains that are unseen in the target domains. As shown in Figure 17 the DoRM++ generator can synthesize high-quality and diverse images in hybrid domains while maintaining the domain-sharing attributes (e.g. pose, identity).
A.4 User Study in One-shot GDA
In this section, we have planned conduct user study in one-shot GDA to enhance our evaluation process. Specifically, our user study involves presenting users with a reference image, a source image, and four adapted images from different methods (DORM++, Generalized One-shot Domain Adaption [56. DynaGAN 20 and DiFa (55)). We will ask users to choose the best adapted image for each of three measurements: (i) image quality, (ii) style similarity with the reference, and (iii) attribute consistency with the source image. To ensure statistical significance, we will generate 500 samples for each method and involve 50 users our study. Each user will be randomly assigned 50 samples from the 500, and they will have unlimited time to complete the evaluation. As shown in Table 7 preliminary results indicate that users strongly favor our DoRM in all three aspects, reflecting the effectiveness of our approach compared to the alternative methods.
17

---

Table 7: User study on one-shot GDA. The numbers represent the percentage of users who favor the images synthesized corresponding method among the all four methods.
Model Comparison
image quality
style similarity
attribute consistency
DoRM++ (Ours)
59.02%
69.35%
67.16%
Generalized One-shot Domain Adaption [66
7.38%
16.33%
11.79%
DynaGAN 20
1 1,44%
3.73%
3.41 %
DiFa [55
32.16%
10.59%
18.24%
A.5 Experiments of DoRM and DoRM++ on the one-shot and 10-shot GDA
thoroughly explore the performance of DoRM++ in a 10-shot GDA scenario and that of DoRM in a one-shot GDA context. The results of these experiments illustrate that both DoRM++ and DoRM exhibit strong performance in both few-shot and one-shot GDA scenarios. Notably, DoRM++ showcases enhanced cross-domain consistency compared to DoRM in the context of one-shot GDA.
Figure 15
A.6 Hybrid-domain Generation
In Figure 17 we present the results of generating hybrid domains using our proposed model. Our DoRM has a unique generator structure that is similar to the mechanism of the human brain. This structure endows the DoRM with two novel capabilities: memory and domain association. These capabilities enable the DoRM to not only retain knowledge from previously learned domains when generating images in new domains, but also integrate multiple learned domains and synthesize images in hybrid domains that were not encountered during training.
Additionally, we proceed to provide comparative analysis of three methods within this section: Our DORM/DORM++, DynaGAN 20). and Domain Expansion (30. As illustrated Figure [8] we strive to alleviate any confusion by conducting comprehensive experiments on the memory and domain association capabilities of these three methods under one-shot and 10-shot settings. These results effectively demonstrate that the baseline methods cxhibit poorer performance across various settings compared to our DoRM/DoRM++. Notably, both DynaGAN and Domain Expansion experience difficulties in achieving successful hybrid domain generation, highlighting the distinctive advantages of our approach.
Finally, we meticulously scrutinized the quantitative experiments conducted within the Domain Expansion framework, leading us to adopt cosine similarity evaluation metric termed "domain similarity" (Sim) based on CLIP image encoder (E₁). This metric is employed to provide quantita- tive assessment of the fidelity exhibited by hybrid domains. In detail, for given generative images in the hybrid domain "sketch-baby" (Isn), we extract image features from the provided images (E)()s)) as well as its corresponding target images (Fy and (J)) Therefore, the "do- main similarity" to "sktech" and "baby" domain are defined as Siml = cos (IsB), (Ts)s) and Sim2 = cos( (Isn), E,(n)) respectively.
Subsequently, compute the cosine similarity for each case. The quantification of results from both the one-shot and 10-shot experiments meticulously presented in Table and Table respectively. These outcomes distinctly illustrate the remarkable performance of our proposed DoRM in both single-domain and hybrid-domain generation. It is imperative to highlight a notable observation amidst these findings: certain outlier exists. Specifically, the images generated by DynaGAN and HyperdomainNet within the "elsa-sunglasse" domain exhibit a remarkably high domain similarity with the "sunglasses" domain, while conversely displaying significantly lower domain similarity with the "elsa" domain. This phenomenon can be rationalized by the generated images closely resembling the "FFHQ" domain, as depicted in Figure 6 of the attached PDF in the rebuttal. Consequently, the domain similarity to the "FFHQ-sunglassed domain also emerges notably high. Drawing upon this observation, we emphasize the necessity for comprehensive evaluation of hybrid-domain generation that seamlessly integrates both qualitative and quantitative results.
A.7 Experiments on Other Source Domains
In addition to the experiments on FFHQ, we conduct other 10-shot generative domain adaptation experiments to qualitatively evaluate the effectiveness of our proposed DoRM approach. Specifically,
18

---

we pre-trained StyleGAN2 on the LSUN-church [22 dataset and adapted the pre-trained GAN to generate haunted house images. The results of our experiments are presented Figure I9
Table 8: Cosine similarity (higher is better) of CLIP feature between generated images and corre- sponding target images on one-shot experiments.
Method
elsa
baby
sunglasses
elsa-baby
elsa-sunglasses
DynaGAN DynaaAAN-ineerpllttion
0.9165
0.7866
0.8882
0.6196/0.7832
0.6245/0.8467
0.6201/0.783
0.6330/0.888 88546
HyperdomainNer
0.8740
0.7739
0.8589
0.7007/0.7041
0.6554/0.7882
Domain expansion
0.9075
0.9614
0.9339
0.7022/0.7762
0.767 1/0.6177
Ours
0.9309
0.9814
0.9370
0.7173/0.7842
0.7734/0.6377
Table 9: Cosine similarity (higher is better) of CLIP feature between generated images and corre- sponding target images 10-shot experiments.
Method
Sketches
Babies
Sunglasses
Sketches-Babies
Sketches-Sungllases
Domain expansion
0.8958
0.9546
0.9094
0.7480/0.7111 0.8809/0.7271
0.7720/0.6735 0.8057/0.6973
Ours
0.9492
0.9780
0.9136
Sketches
Baby
Sunglasses
Raphael
Caricabure
A medeo
OdD Dix
Figure 9: Training images 10-shot generative domain adaptation experiments.
19

---

Source
Sunglasses
Amedoo
Caricature
Otto Dix
Rapheal
Figure 10: 10-shot generative domain Adaptation on FFHQ. We use source generator pre-trained on the FFHQ (18] dataset and evaluate our proposed DoRM approach various target domains, including FFHO-Sunglasses, face caricatures, face paintings by Raphael, face paintings by Amedeo Modigliani, and face paintings by Otto Dix (50). The training images shown in Figure Our results demonstrate that our proposed DoRM approach can maintain cross-domain consistency between the source domain and different target domains.
Figure 11: Latent interpolation using the generators adapted to different target domains in 10-shot generative domain adaptation (the first line is the source images). The first and last columns are the generated images with two latent codes after 10-shot generative domain adaptation. The remaining columns are the results by linearly interpolating the two latent codes. According to the figure, all the semantics (e.g., the gender, the haircut and the pose) vary gradually.
20

---

Original
Man
Bobcut
Man+Pose
Look up
Figure 12: Latent edit using the generators adapted to different target domains in 10-shot generative domain adaptation (the first line is the source images). The second and third lines are two popular target domain images, where the first column is the example of 10-shot target images and the other columns are adaptation results.
Figure 13: One-shot GDA in 3D GANs. DoRM for one-shot GDA in EG3D. Left: source image in FFHQ. Right: the generative target domain (Sketch) images.
Source
B
G
S
S-B-G
Figure 14: Three-domain hybrids generation on 10-shot Babies (B), Sunglasses (G), and Sketch (S) datasets. This illustration demonstrates how easily our approach can be adapted to generate hybrids involving more than two domains.
21

---

one-shot
10-shot
DoRM+
DoRM
Figure 15: Results of DoRM and DoRM++ on the one-shot and 10-shot GDA.
22

---

source
reference
DoRM++ (ours)
JoJoGAN
Generalized One-shot Domain daption
source
reference
DoRM++ (ours)
DynaGAN
DiFa
Figure 16: Qualitative comparison on one-shot GDA. The source domain is FFHQ, and the target domains include different reference images, as shown in the first row of the figure. We compare our method with JoJoGAN (6], Generalized One-shot Domain Adaption (56). DynaGAN 20 and (55. Our method not only achieves better synthesis quality and diversity but also maintains higher cross-domain consistency than other methods.
23

---

reference
Learned Domains
references
Hybrid Domains
reference
Learned Domains
references
Hybrid Domains
Figure 17: One-shot generative domain adaptation and domain association on FFHQ. The source domain is FFHQ, and the target domains include different reference images, shown in the first column of the figure. Once our DoRM++ generator learns to synthesize images in multiple target domains, it integrate the knowledge from the learned multiple domains and synthesize images in hybrid domains which are unseen in the target domains.

---

Source(FFHO)
FFHQ-Baby
FFHQ--Snnglaser
Source(FFHa)
FFHQ Baby
FFHQ-Sunglasee
Sketch
Sketch-Baby
Sketch-Suus Sunglasses
Sketch
Sketch-Baby
Sketch- Sungla-
Ours
10-shot
Domain expansion
one-shot
DynaGAN
Domain expansion
Source (FFHQ)
Ours
Reference images
Figure 18: Comparison with baselines on I-shot and 10-shot GDA. The outcomes underscore our DoRM's superiority, revealing inferior performance by baseline methods across various scenarios. Remarkably, other methods face challenges in achieving successful hybrid domain generation.
25

---

Haunted House
Source images
Target images
Figure 19: 10-shot generative domain adaptation on LSUN-Church 52) The source generator is pretrained on LSUN-Church 52). The target domain is haunted house (10 training images are shown on the left side). The result shows that our method can maintain the cross-domain consistency between the source domain and the target domain.
26


Reviews:
[{"review_id": "VCRxPKjtVV", "reviewer": "Reviewer_Jqfj", "summary": "This paper proposes a novel approach for few-shot generation using a lightweight GAN architecture and a new loss function. The method is capable of handling multi-domain and hybrid domain tasks with a single model. The experiments demonstrate the superior performance of the proposed method in terms of both qualitative and quantitative results.", "questions": "1. The paper should provide more analysis or improvement on the potential issue of unrealistic results due to domain association.\n2. The related works should be discussed more thoroughly, especially with respect to HyperDomainNet [1] and DynaGAN [2], which share similar ideas with the proposed method.\n3. The generalization of the proposed method should be discussesd.\n4. It would be beneficial to expand the list of baselines in Table 1 to include additional approaches, such as MineGAN [3] and EWC [4].\n\n", "limitations": "Authors have addressed some limitation of this paper. More Limitation should be further addressed: The proposed method is designed specifically for StyleGAN2 architecture, limiting its generalization to other architectures.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The paper addresses a novel task and presents a unique approach to handle it.\n2. The experiments are comprehensive and well-organized, providing strong evidence for the effectiveness of the proposed method.\n3. The proposed method outperforms previous methods in terms of both qualitative and quantitative results.\n4. The exploration of the generative ability to merge two or three domains for few-shot generation is interesting and innovative.\n", "weaknesses": "1. The paper should provide more analysis or improvement on the potential issue of unrealistic results due to domain association.\n2. The related works should be discussed more thoroughly, especially with respect to HyperDomainNet [1] and DynaGAN [2], which share similar ideas with the proposed method.\n3. The proposed method is designed specifically for StyleGAN2 architecture, limiting its generalization to other architectures. \n4. It would be beneficial to expand the list of baselines in Table 1 to include additional approaches, such as MineGAN [3] and EWC [4].\n5. This paper utilized adversarial training while freezing the discriminator's backbone. However, in my understanding, the discriminator may still overfit the training data due to the adversarial loss. This issue has also been observed in comparable works, like CDC, where the adversarial loss weight is only 1 while the consistent loss weight is 1000, but the overfitting problem persists as training progresses. How about of the proposed approach?\n\n\n[1] Hyperdomainnet: Universal domain adaptation for generative adversarial networks. NIPS 2022\n\n[2] Dynagan: Dynamic few-shot adaptation of gans to multiple domains. SIGGRAPH Asia 2022\n\n[3] Minegan: effective knowledge transfer from gans to target domains with few images. CVPR 2020\n\n[4] Few-shot image generation with elastic weight consolidation. NIPS 2020 \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. The paper should provide more analysis or improvement on the potential issue of unrealistic results due to domain association.**\n\n As highlighted, our paper represents the first systematic attempt at domain association in few-shot Generative Domain Adaptation (GDA). We recognize the significance of refining our method to produce more realistic results. In our primary paper, we achieve hybrid-domain generation by combining the M&A (Modulation and Activation) modules of different target domains and activating them simultaneously. This approach is simple and easy to implement. However, we acknowledge that further enhancements can be made to improve the performance of domain association. To address this, we propose an additional strategy where we not only combine the trained target M&A modules but also introduce a new M&A module to better blend the target domains. Specifically, we utilize a directional loss based on Contrastive-Language-Image-Pretraining (CLIP) to train these new M&A modules, as depicted in Fig. 1 of the [PDF](https://openreview.net/attachment?id=xNZlQlaQNI&name=pdf). By incorporating this new M&A module and the directional loss, we aim to enhance the realism and fidelity of the associated domains, thus mitigating the issue of unrealistic results.\n\n**Q2. The related works should be discussed more thoroughly.**\n\nThank you for your valuable feedback regarding the discussion of related works. We fully understand the importance of a comprehensive comparison and analysis of existing methods, especially those that share similarities with our proposed approach. The thoroughly discuss of related works have been demonstrated in the [General Response: Comprehensive Comparison and Discernment from Pertinent Works [1] [2] [3]](https://openreview.net/forum?id=jown9RvYn7&noteId=xNZlQlaQNI)\n\n**Q3. The proposed method is designed specifically for StyleGAN2 architecture, limiting its generalization to other architectures.**\n\nThank you for your valuable suggestion. StyleGAN2 has gained widespread recognition as one of the most popular architectures in few-shot image generation, serving as the foundation for numerous previous methods. Hence, we have adopted StyleGAN2 in our approach, following the common practice in the field. However, it is essential to emphasize that our method is not confined to StyleGAN2 alone; rather, it exhibits adaptability to any layer-wise generator. This flexibility enables our approach to be applied to various state-of-the-art GAN architectures that employ layer-wise structures, as seen in works such as [4][5][6][7][8]. The prevalence of layer-wise designs in current GAN research makes our method highly versatile and opens up opportunities for its potential application and integration into different generative models. By highlighting this adaptability to diverse layer-wise generators, we aim to underscore the broader scope and applicability of our approach, which can contribute to various GAN architectures and research domains.\n\n**Q4. It would be beneficial to expand the list of baselines in Table 1 to include additional approaches.**\n\nThanks for your  constructive suggestion. We have added the results of MineGAN and EWC in Table 1 as follows:\n\n|Datasets|Babies|Babies|Babies|Sunglasses|Sunglasses|Sunglasses|Sketches|Sketches|Sketches|\n| :-----:| :----: | :----: |:-----:| :----: | :----: |:-----:| :----: | :----: |:-----:|\n| Methods | FID |I-LPIPS  | ID|FID |I-LPIPS  | ID|FID |I-LPIPS  | ID|\n| MineGAN | 98.23|0.514|0.132 | 68.91|0.42|0.171 |64.34 |0.40 |0.092 |\n| EWC |87.41|0.523|0.145  | 59.73|0.431|0.156 |71.25|0.42|0.103 |\n| DoRM |30.31  |0.623|0.445  | 17.31 |0.644 |0.389|40.05|0.502|0.365|\n\n**Q5. How about of the proposed approach avoid overfitting?**\n\nThank you for your valuable review of our work. We acknowledge the crucial issue of discriminator overfitting during adversarial training, which can diminish the synthesis diversity of the generator, leading to outputs closely resembling training data. In response, our DoRM introduces the similarity-based structure loss $L_{ss}$ to preserve the generator's diversity in synthesis. An important distinction is that our DoRM has a lighter parameter load compared to CDC, which involves training the entire generator. Additionally, we integrate data augmentation and early stopping techniques into our DoRM to further enhance its performance.\n\n[1] HyperDomainNet: Universal Domain Adaptation for Generative Adversarial Networks. NIPS 2022\n\n[2] DynaGAN: Dynamic Few-shot Adaptation of GANs to Multiple Domains. SIGGRAPH Asia 2022\n\n[3] Domain Expansion of Image Generators. CVPR 2023\n\n[4] Large scale gan training for high fidelity natural image synthesis. \n\n[5] A style-based generator architecture for generative adversarial networks.\n\n[6]  Analyzing and improving the image quality of stylegan.\n\n[7] Alias-free generative adversarial networks\n\n[8] Stylegan-xl: Scaling stylegan to large diverse datasets. "}, {"review_id": "dr5hGpxHoL", "reviewer": "Reviewer_cgji", "summary": "The paper introduces a novel approach for domain adaptation of StyleGAN2 called Domain Re-Modulation, which is a few-shot technique. The authors argue that this method draws inspiration from the workings of the human brain. To achieve the desired domain shift, the paper utilizes the stylespace of StyleGAN along with adversarial and consistency losses. Additionally, a structural similarity loss is incorporated, which is based on the auto-correlation map extracted from the CLIP encoder. The effectiveness of the proposed method is evaluated through both quantitative and qualitative analyses. Furthermore, the paper includes comparisons with other state-of-the-art StyleGAN-based methods, highlighting its advancements. The authors also conduct an ablation study to analyze the individual components employed in their approach.", "questions": "Refer to Weakness section", "limitations": "The authors discuss the ethics and limitations.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* The paper introduces the concept of hybrid domain generation, which involves combining the affine layers of StyleGAN to generate results from multiple domains using a single generator. The authors demonstrate that this approach yields consistent and generalizable outcomes across multiple domains.\n\n* Notably, the method proves effective even in scenarios with limited training data, such as 1-shot training. By leveraging the StyleGAN2 backbone, the proposed technique achieves reliable and coherent generation outcomes.\n\n* In terms of evaluation, the paper compares its approach with various StyleGAN-based methods. It provides both qualitative and quantitative results to assess the performance of the proposed method. Furthermore, the paper includes comparisons with related methods to highlight the advantages and advancements of the proposed domain generation approach.\n", "weaknesses": "1. The paper lacks substantial novelty as there exist other works such as StyleCLIP and 3DAvatarGAN that demonstrate the capability of performing image editing and few-shot domain adaptation using the \"s\" space of StyleGAN. Although the paper incorporates a CLIP-based loss, it fails to provide a comparative analysis with papers falling under the same category, such as StyleGAN-NADA and Mind the GAP HyperStyle. The absence of such comparisons and explanations regarding the differences between the methods makes it challenging to determine the uniqueness of the paper. To assess how the current method surpasses these works, it is essential to consider the individual components and techniques utilized in each approach. By conducting a thorough comparison, the paper could highlight the specific strengths of the proposed method that outperform other works. This would provide a clearer understanding of the paper's novelty and identify the components that excel in comparison to existing methods. Hence, it is imperative for the paper to address these concerns by including a detailed comparison that elucidates how the current method improves upon or surpasses existing approaches, specifically identifying the components that outperform these methods. How is the current method better than these works? Which component of the method outperforms these methods? \n\n\n2. Why did the authors use 2D generators especially when 3D StyleGAN based generators are available trained on the same data. Besides the concerns above, it would be interesting to see how the method will perform in the 3D -GAN domain. What are some additional challenges in that domain?\n3. How is the editability of the generator after the domain adaptation. Since there are no edits performed, it is difficult to assess if the latent space properties of the generator are preserved or it just overfits the given Styles.\nThere are other datasets like AFHQ Cats, Dogs, Cars that are tested for such few shot domain adaptation tasks in the StyleGAN domain. The method seems specific to the face domain. What about the results on these domains? How generalizable is the method to these domains?\n4. The artistic domains are quite subjective. It is not fair to just evaluate quantitative metrics for such tasks. I would suggest conducting a user study. It would be better to embed real images in the generators and ask users about the similarity, consistency and identity preservation of the target images. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. The paper lacks substantial novelty**\n\nWe are committed to conducting a detailed and comprehensive comparative analysis that elucidates the specific advantages of our current method over existing works. \n1. StyleCLIP only demonstrates the capability of performing image editing using the \"s\" space of StyleGAN, which is obvious and has been explored many times by previous approaches. 3DAvatarGAN is the first study of domain adaptation in 3D-GANs. However, 3DAvatarGAN adopts a pre-trained 2D target-domain generator to achieve the domain adaptation for 3D-GANs. The pre-trained 2D generator can generate various target domain images, thereby, it only demonstrate that using the \"s\" space of StyleGAN can acquire the domain adaptation, but not few-shot domain adaptation. We provide a novel insight to \"s\" space of stylegan and demonstrate that only one image is enough for domain adaptation through using the \"s\" space of StyleGAN.  \n\n2. Compared with StyleGAN-NADA and Mind the GAP, [1] and [3] are the latest to come up with more advanced methods. These methods [1][3] also use clip-based losses, and sufficient experiments have been done to show that their methods are superior to the mentioned baseline. Therefore, our manuscript only compares the proposed method with advanced baseline [1][3] to demonstrate the advancement. Furthermore, we systematically analysis and compare individual components and techniques utilized in our DoRM and DiFa [3] to showcase how our method surpasses these works. According to the Introduction of the manuscript, GDA needs three fundamental properties. To resolve it, DiFa [3] proposed two CLIP-based loss: global loss ($L_\\text{global}$) and local loss ($L_\\text{local}$)  for realizing large diversity/cross-domain consistency and high quality, respectively. Differently, we adopt Similarity-based Structure Loss ($L_\\text{ss}$)  and adversarial loss ($L_\\text{adv}$)  for realizing large diversity/cross-domain consistency and high quality, respectively. As shown in Fig. 2 of the [PDF](https://openreview.net/attachment?id=xNZlQlaQNI&name=pdf), the $L_\\text{local}$ in DiFa mainly focuses on textual features and failes to capture the complete features (e.g. the white background in sketches) in generative domain adaptation. In our DoRM, we employ adversarial loss which can fully capture the features of the target images. \n\n**Q2. Why did the authors use 2D generators especially.**\n\nOur decision to use 2D generators was primarily influenced by the maturity and development of the 2D few-shot GDA field, which has gained significant attention since 2021. While we acknowledge the potential benefits and advancements that 3D StyleGAN-based generators can offer, we opted to validate our approach in a more established and widely studied domain. Furthermore, we conducted some initial studies using the popular 3D-aware image generation method, EG3D, for one-shot GDA with FFHQ as the source domain and Sketch as the target domain. The results, as shown in the Fig. 3 of [PDF](https://openreview.net/attachment?id=xNZlQlaQNI&name=pdf), reveal that directly migrating the proposed EG3D to 3D one-shot domain adaptation poses challenges and might not be straightforward. Some potential challenges in the 3D-GAN domain include: Overfitting, Volumetric Data Representation, Spatial Artifacts, and Computational Complexity.\n\n**Q3. How is the editability of the generator after the domain adaptation. The method seems specific to the face domain. How generalizable is the method to other domains?**\n\nTo address this concern, we have planned additional experiments and evaluations to investigate the editability and latent interpolation of the generator post-domain adaptation. Regarding editability, we have performed editing on a real image adapted into a new target domain using StyleGAN-CLIP to discover editing directions in the source domain. The results, as illustrated in the left part of Fig. 4  in the [PDF](https://openreview.net/attachment?id=xNZlQlaQNI&name=pdf), indicate that the adapted generator maintains similar latent-based editing capabilities to the original generator. This demonstrates the preservation of editability in the adapted generator. For latent interpolation, we have presented various results in Sec A.2 of the Supplementary Materials. In conclusion, our experiments confirm that the latent space properties of the generator are well preserved, and our method goes beyond mere overfitting of given styles. \n\nFurthermore, we emphasize that the proposed DoRM approach is not inherently limited to the face domain. The method can be readily extended to these datasets, and we apply our DoRM to the LSUN-church dataset and adapted the pre-trained GAN to generate haunted house image in the Sec. A.5 of the Supplementary Materials. The result shows that our method can maintain the cross-domain consistency between the non-face domain.\n\n**Q4 It would be better to do user study.**\n\nWe have planned to conduct a user study in one-shot GDA to enhance our evaluation process. By embedding real images in the generators and collecting feedback from users, we aim to gain qualitative insights that complement the quantitative metrics, offering a well-rounded evaluation of our system's performance. As shown in Table 1 of the [PDF](https://openreview.net/attachment?id=xNZlQlaQNI&name=pdf), preliminary results indicate that users strongly favor our DoRM in all three aspects, reflecting the effectiveness of our approach compared to the alternative methods.\n\n[1] Generalized One-shot domain adaption of generative adversarial networks. NIPS 2022 \n\n[2] Dynagan: Dynamic few-shot adaptation of gans to multiple domains. SIGGRAPH Asia 2022\n\n[3] Towards Diverse and Faithful One-shot Adaption of Generative Adversarial Networks. NIPS 2022"}, {"review_id": "8QScBlaK9X", "reviewer": "Reviewer_eDDg", "summary": "This paper proposes two advanced criteria for few-shot Generalized Domain Adaptation (GDA) inspired by the way human brains acquire knowledge in new domains: memory and domain association.\n\n To fully realize the potential of few-shot GDA, an innovative generator structure called Domain Re-Modulation (DoRM) is introduced.\n\n This structure freezes the source generator and incorporates new lightweight mapping and affine modules (M&A modules) to capture the attributes of the target domain during GDA, resulting in a linearly combinable domain shift in the style space.\n\n By incorporating multiple M&A modules, the generator gains the ability to perform high-fidelity multi-domain and hybrid-domain generation.", "questions": "See weakness", "limitations": "NA", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. This paper presents a novel exploration into the use of memory and domain association in few-shot GDA, which has not been previously explored. These advanced properties greatly reduce memory usage and simplify deployment, while also producing impressive results.\n\n2. The proposed DoRM structure and similarity-based structure loss are innovative and advanced.\n\n3. The figures in this paper effectively demonstrate the major contributions and the overall organization is clear and easy to follow.\n\n4. The experiments conducted in this paper are robust and include both one-shot and 10-shot settings. Extensive testing demonstrates that our proposed method outperforms the previous state-of-the-art method in all five properties: quality, synthesis diversity, cross-domain consistency, memory, and domain association.", "weaknesses": " \nThere are a few areas where this paper could be improved.\n\n1. It would benefit from including some recent studies on one-shot image generation [1].\n\n2. The paper should include experiments of DoRM++ in 10-shot GDA and DoRM in one-shot GDA, as this ablation experiment would help to demonstrate the advancement of the mentioned losses and the impact of data size on GDA tasks.\n\n3. The paper lacks a future outlook, and it would be helpful to explore more possible improvement directions, especially as the effect of domain association on some datasets is not always satisfactory.\n\n[1] StyO: Stylize Your Face in Only One-Shot", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. It would benefit from including some recent studies on one-shot image generation [1].**\n\nThank you for providing valuable feedback on our work.  We sincerely appreciate your suggestion to include recent studies on the one-shot image generation.  Staying up-to-date with the latest research in the field is crucial to ensuring the comprehensiveness of our work. We have thoroughly reviewed the referenced paper, and it is highly relevant to our research topic.  The study introduces the diffusion model into one-shot GDA, offering valuable insights that can enrich our understanding of the domain. In light of this, we are committed to revising our paper to incorporate the relevant information and acknowledge the contributions made by the diffusion model in the context of one-shot GDA.\n\n\n**Q2. The paper should include experiments of DoRM++ in 10-shot GDA and DoRM in one-shot GDA, as this ablation experiment would help to demonstrate the advancement of the mentioned losses and the impact of data size on GDA tasks.**\n\nThank you for your insightful suggestion regarding the inclusion of ablation experiments involving DoRM++ in 10-shot GDA and DoRM in one-shot GDA. We truly value your recognition of the potential insights that these experiments could offer in terms of assessing the efficacy of our proposed losses and comprehending the influence of data size on GDA tasks. In direct response to your valuable feedback, we have diligently carried out additional experiments, as demonstrated in the right part of Figure 4 in the [PDF](https://openreview.net/attachment?id=xNZlQlaQNI&name=pdf), to thoroughly explore the performance of DoRM++ in a 10-shot GDA scenario and that of DoRM in a one-shot GDA context. The results of these experiments have been thoughtfully analyzed and presented in our revised manuscript. From our findings, it is evident that both DoRM++ and DoRM exhibit strong performance in both few-shot and one-shot GDA scenarios. Notably, DoRM++ showcases enhanced cross-domain consistency compared to DoRM in the context of one-shot GDA. We greatly appreciate your guidance in enriching the empirical validation of our approach, and we are confident that these additional experiments further enhance the comprehensiveness of our research.\n\n\n\n**Q3. The paper lacks a future outlook, and it would be helpful to explore more possible improvement directions, especially as the effect of domain association on some datasets is not always satisfactory.**\n\nThank you for your valuable feedback on our paper. We appreciate your suggestion to include a future outlook and explore potential improvement directions for our research. We also acknowledge the importance of addressing cases where the effect of domain association on certain datasets might not be entirely satisfactory. In response to your insightful comment, we plan to revise the paper to incorporate a dedicated section that outlines possible avenues for future research and improvements. This section will explore the limitations and challenges we encountered during the domain association process and propose potential solutions and directions for further investigation.\n\nSome of the areas we intend to focus on include:\n\n1. Advanced Domain Adaptation Techniques: We will investigate state-of-the-art domain adaptation methods and explore how integrating these techniques into our approach might enhance the performance of domain association, particularly on datasets where the effect is less satisfactory.\n\n2. Data Augmentation Strategies: We will explore the effectiveness of different data augmentation approaches to improve the robustness of our model against variations in domain-specific characteristics.\n\n3. Analyzing the application of the proposed method to 3D few-shot GDA and extending the DoRM to 3D few-shot GDA.\n\n4. The current manuscripts only simply combines the M&A modules of different target domains and activate them at the same time to realize the domain association. To further improve the performance of the  domain association, not only combining the trained target M&A modules but also employig a new M&A module and additional consistency loss is a better method to blend the target domains."}, {"review_id": "cKbiLw1OpM", "reviewer": "Reviewer_18N9", "summary": "This paper focuses on the concept of few-shot generative domain adaptation. Drawing inspiration from the human memory mechanism, the authors introduce a novel approach called DoRM (Domain-Adaptive Mapping and Affine Modules) to adapt the generator to a new domain. By incorporating new mapping networks and affine modules into the frozen source generator, they aim to enhance its adaptability. Additionally, the paper introduces a consistency loss based on CLIP image features to ensure the preservation of domain-sharing attributes during adaptation. Notably, the authors explore the intriguing concept of domain association in generative models, which involves merging the generative abilities of multiple domain-specific generators to enable generation in a new domain. Through a combination of mapping networks and affine modules within a single generator, the proposed DoRM generator successfully achieves domain association and multi-domain generation concurrently. To validate the effectiveness of their approach, the authors conduct extensive qualitative and quantitative experiments.", "questions": "(1) It is worth noting that the proposed generator structure bears similarities to the one introduced in [1]. This overlap may somewhat diminish the novelty of the proposed method. To enhance the clarity of differentiation, it would be beneficial if the authors provide a more comprehensive explanation highlighting the distinct features and advancements of their approach in comparison to [1].\n\n(2) In addition to the proposed loss regularization, the authors have included supplementary regularizations, referred to as DoRM++, based on [2], in the supplementary material's one-shot experiments. Although the synthesis results of DoRM++ outperform those of [2], it is important to note that the proposed DoRM approach already addresses overfitting concerns by freezing the discriminator's backbone. Consequently, the inclusion of these supplementary loss components may be considered redundant. Therefore, it would be beneficial to conduct further qualitative and quantitative comparisons between DORM and DORM++ to gain a comprehensive understanding.\n\n(3) Section 4.3 would greatly benefit from further elaboration on the training details and specific datasets employed in the domain association experiments. Providing more specific information in these areas would enhance the reproducibility and understanding of the experimental setup for readers.\n\n(4) In recent times, the diffusion model has made remarkable strides in image generation, particularly in the area of few-shot image generation. Therefore, there is a need for further discussion regarding the application of the diffusion model in the context of few-shot Generative Domain Adaptation and domain associations. \n\n[1] HyperDomainNet: Universal Domain Adaptation for Training Generative Adversarial Networks. NeurIPS’22\n[2] Towards Diverse and Faithful One-shot Adaption of Generative Adversarial Networks. NIPS 2022", "limitations": "Limitations are discussed in the paper.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "(1) This paper presents a novel task: domain association, which involves combining the generative abilities of different domain-specific generators to enable generation in a new domain. This topic represents an exciting avenue for further exploration in the field of generative models. Notably, Figure 4 demonstrates impressive synthesis quality in the hybrid domain, particularly in the baby-sketch domain.\n\n(2) The proposed generator network structure, DoRM, is skillfully designed to facilitate domain association in a straightforward yet effective manner. By incorporating lightweight modules, the method achieves domain adaptation while preserving the generative abilities in the learned domains. This approach is both efficient in terms of storage and yields promising results.\n\n(3) The paper is excellently written and presents its concepts in a clear and understandable manner, making it easy for readers to follow along with the proposed methodology.\n\n(4) Through extensive qualitative and quantitative experiments, the paper demonstrates the superior performance of the proposed method compared to state-of-the-art techniques. Particularly impressive are the synthesis results achieved through domain association using the proposed method.\n", "weaknesses": "Please view the Questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. It would be beneficial if the authors provide a more comprehensive explanation highlighting the distinct features and advancements of their approach in comparison to [1].**\n\nThank you for your valuable feedback regarding the discussion of related works. We fully understand the importance of a comprehensive comparison and analysis of existing methods, especially those that share similarities with our proposed approach. The thoroughly discuss of related works have been demonstrated in the [General Response: Comprehensive Comparison and Discernment from Pertinent Works [1] [2] [3]](https://openreview.net/forum?id=jown9RvYn7&noteId=xNZlQlaQNI)\n\n**Q2.  The paper should include experiments of DoRM++ in 10-shot GDA and DoRM in one-shot GDA, as this ablation experiment would help to demonstrate the advancement of the mentioned losses and the impact of data size on GDA tasks.**\n\nWe have diligently carried out additional experiments, as demonstrated in the Right part of Figure 4 on the [PDF](https://openreview.net/attachment?id=xNZlQlaQNI&name=pdf), to thoroughly explore the performance of DoRM++ in a 10-shot GDA scenario and that of DoRM in a one-shot GDA context. The results of these experiments have been thoughtfully analyzed and presented in our revised manuscript. From our findings, it is evident that both DoRM++ and DoRM exhibit strong performance in both few-shot and one-shot GDA scenarios. Notably, DoRM++ showcases enhanced cross-domain consistency compared to DoRM in the context of one-shot GDA. \n\n**Q3. Section 4.3 would greatly benefit from further elaboration on the training details and specific datasets employed in the domain association experiments.**\n\nWithin the framework of our DoRM, a key element lies in the utilization of distinct re-modulation weights, as elaborated in Section 3.1, which are specifically tailored to individual target domains.  To illustrate, let us consider the source domain as FFHQ.  When steering towards relatively proximate target domains such as FFHQ-Baby or FFHQ-Sunglasses, which bear closer semblance to the source domain, we strategically set the re-modulation weight within the range of 0.004 to 0.05.  This meticulous calibration has proven instrumental in attaining more optimal synthesis outcomes for these domains. Conversely, when embarking upon more disparate domains such as the sketch domain or other artistic domains like the works of Amedeo and Monet, which exhibit substantial gaps when compared to the source domain, we judiciously adjust the re-modulation weight to fall within the span of 0.05 to 0.2.  This discerning adjustment has significantly contributed to the achievement of improved synthesis results for domains characterized by pronounced dissimilarity.\n\n**Q4. There is a need for further discussion regarding the application of the diffusion model in the context of few-shot Generative Domain Adaptation and domain associations.**\n\nWe sincerely appreciate your insightful review of our work. Your observation regarding the advancements of the diffusion model in image generation, particularly in the domain of few-shot image generation, is highly valuable. We agree with your point on the significance of discussing the application of the diffusion model in the context of few-shot Generative Domain Adaptation (GDA) and domain associations. Indeed, the diffusion model has shown promising capabilities in acquiring the concept (style or content) of the target domain through few-shot images. Remarkable examples like Dreambooth or Text Inversion demonstrate how the diffusion model can discern the identity or style of the target domain from just one image, achieving impressive few-shot image generation. However, we recognize that the issue of cross-domain consistency remains a challenge in this context. Addressing this problem effectively holds great potential for the diffusion model in few-shot GDA. Moreover, the disentanglement of the text space in the diffusion model presents significant opportunities for domain association. Despite its potential, we agree with your observation that this aspect has not been thoroughly explored by the community. Further investigation into utilizing the diffusion model's disentangled text space for domain association can open new avenues and contribute to advancing the field.\n\nWe genuinely appreciate your valuable insights, which have shed light on important aspects for future research. Your feedback has inspired us to delve deeper into the potential applications and capabilities of the diffusion model in few-shot GDA and domain associations.\n\n[1] HyperDomainNet: Universal Domain Adaptation for Generative Adversarial Networks. NIPS 2022\n\n[2] DynaGAN: Dynamic Few-shot Adaptation of GANs to Multiple Domains. SIGGRAPH Asia 2022\n\n[3] Domain Expansion of Image Generators. CVPR 2023"}, {"review_id": "1JNISRMn2X", "reviewer": "Reviewer_URf4", "summary": "This work studies generative domain adaptation (GDA), on the StyleGAN2 architecture. Methods in this setting are commonly evaluated in terms of quality, diversity, and cross-domain consistency. This work claims to also be the first to explore the abilities of memory (“retain knowledge from previously learned domains”) and domain association (“integrate multiple learned domains and synthesize hybrid domains”).  To tackle all these aspects, a method called DoRM is proposed. DoRM learns a new mapping network and affine transformations – M&A - (components of StyleGAN2) for each new domain. Hybrid domains are generated by interpolating the style codes of the original mapping and affine layers with the new ones. ", "questions": "1.\tPlease clarify in what way is the setting that is supposably suggested was not already covered by HyperDomainNet, DynaGAN and Domain Expansion?\n2.\tWhy did you not include comparisons to these methods despite acknowledging that they are highly similar and relevant?\n", "limitations": "One limitation is mentioned. However, several questions remain:\n1. Can hybrids be produced by combining more than two domains?\n2. Are the produced hybrids less faithful to each individual domain and to what extent?\n", "rating": 3, "confidence": 5, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "1.\tThe paper is clearly written and easy to follow.\n2.\tThe tasks studied in this paper are interesting and relevant.\n3.\tThe proposed method is intuitive and novel, and as demonstrated by some experiments it is effective in performing domain adaptation.\n", "weaknesses": "The claimed contributions are incorrect, and the evaluation is severely lacking. Both issues stem from outright ignoring the role and contributions of the most relevant previous works – HyperDomainNet [1], DynaGAN [18], Domain Expansion [24]. Below I list several examples of severe problems with claims and evaluations.\n\nAll three works consider settings that cover the supposably new aspects of domain adaptation that are discussed in this paper. Both DynaGAN and HyperDomainNet rely on similar modulation techniques and achieve *”memory”* the same way. In Domain Expansion, the original domain is preserved on a dedicated subspace. Also, both DynaGAN and Domain Expansion discuss *”association”* in length (using the terms “domain interpolation” and “domain composition”). So, clearly, this work’s claims (lines 64, 247) of being the first to consider this setting are false. \n\nThe previous fact is even somewhat acknowledged in the Related Work section (lines 99-105). The three works are said to “fall short in integrating multiple domains”. However, this claim is not supported by any experiment. \nNot only this one claim is not supported, but the authors choose to not compare their method with the three most relevant works - all of which have code on Github and all have been published well before the submission deadline (October 2022-January 2023). Instead, the method is compared to the more conventional setting of “single-domain, entire generator” adaptation, which is negligent at best. Comparing the generation of “hybrid” domains to CDC [25] which was not designed for this purpose (and published in 2021) and not to DynaGAN and Domain Expansion is unreasonable. Similarly, claiming superiority over CDC in terms of storage required is irrelevant. DoRM trains a mapping network (~6M params) for each new domain, while HyperDomainNet trains ~6K params, and Domain Expansion requires no additional weights whatsoever!\n\nThe inaccurate contribution claims and lack of comparison to relevant previous works is fatal as it prevents the community and practitioners from understanding the landscape of works in this field. Perhaps the paper was written around the end of 2022 and is simply not up to date. I urge the authors to consider significantly rewriting this paper before resubmitting or arXiving a preprint.\n\nAdditional issues:\n1. Measuring consistency between domains using ArcFace – a face recognition network trained on faces -- makes little sense to me. For example, why would it output anything meaningful on sketches? \n2. Missing citation to StyleSpace [Wu et al.] who introduced the $\\mathcal{S}$ used in this paper (line 120).\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. The claimed contributions are incorrect, and the evaluation is severely lacking**\n\nWe apologize for any oversight and plan to enhance the quality of our paper through the following steps:\n\n**1. Thorough Comparison and Differentiation from Prior Works**. It has demonstrated in the [General Response: Comprehensive Comparison and Discernment from Pertinent Works [1] [2] [3]](https://openreview.net/forum?id=jown9RvYn7&noteId=xNZlQlaQNI)\n\n**2. Clarification of First-to-Consider Claim.** We acknowledge that our manuscript has inaccurately represented our major contributions. In Section 2.1 of the manuscript, we have indeed recognized that the three mentioned  methods [1][2][3] achieve comparable domain memory capabilities to our approach. However, we find that both HyperDomainNet [1] and DynaGAN [2] lack the capacity to realize \"Domain Association\" due to their reliance on scale modulation parameters in the StyleGAN style space. Additionally, these methods did not thoroughly explore the potential applications of \"Domain Association\" in their respective papers. It is essential to clarify that the \"domain interpolation\" technique in DynaGAN [2] significantly differs from our concept of \"Domain Association.\" While \"domain interpolation\" facilitates a seamless transition between two target domains through vector interpolation, it falls short in creating a hybrid domain that simultaneously embodies the properties of multiple fundamental domains, as seen in our \"sketch\"+\"Sunglasses\" domain example. We acknowledge the \"domain composition\" aspect discussed in [3] aligns with our proposed \"Domain Association.\" Regrettably, our initial review of the references was incomplete, and we overlooked the significance of [3]'s contribution to the domain association. To rectify this, we will revise the description of our main contributions and provide a more comprehensive analysis of related works.\n\n**3. Comprehensive Evaluation of Methods.** In light of our paper's focus on \"few-shot GDA\", we conducted qualitative and quantitative comparisons using established criteria, that is using FID and Intra-LPIPS metrics on Sketch, Babies, and Sunglasses datasets. However, we found that none of [1,2,3] evaluated their approaches using this popular setting. Consequently, we categorize them as one-shot GDA methods and refrain from comparing them in 10-shot experiments. In our main paper, we utilize CDC as a baseline for hybrid domain generation and storage, showcasing the superiority of our remodulation technique across all five anticipated attributes of few-shot GDA. It's worth noting that the mentioned methods [1][2][3] lack consistent improvement compared to CDC-based approaches, particularly in terms of quality, diversity, and consistency. This raises the possibility that these methods may have traded core attributes to emphasize new ones, a conjecture supported by our additional rebuttal experiments (Fig. 6 of the [PDF](https://openreview.net/attachment?id=xNZlQlaQNI&name=pdf)). Additionally, we conduct experiments of one-shot GDA in Section A.4 of the Supplementary Materials, comparing our DoRM++ with [2] and demonstrating significant enhancements across anticipated GDA attributes. We abstain from using [1] as a baseline due to its use of the unofficial StyleGAN2 implementation, differing from the official one. [3] is also excluded as it only supports text-driven domain adaptation on their Github. \nFurthermore, we offer a comprehensive evaluation of three methods: Our DoRM, [2], and  [3] in this response. We didn't consider [1] as a baseline due to its implementation constraints. To address any confusion, we present detailed experiments on memory and domain association capabilities of these methods under 1-shot and 10-shot GDA (Fig 6 of the [PDF](https://openreview.net/attachment?id=xNZlQlaQNI&name=pdf)). The outcomes underscore our DoRM's superiority. \n\n**Q2. Measuring consistency between domains using ArcFace**\n\nIndeed, accurately measuring the performance of generation models, particularly those with few-shot training data, is a notorious challenge. ArcFace is **commonly** used in domain adaptation and has been employed to measure the consistency between different domains in previous works [1], as well as to design identity loss for ensuring consistency between synthesized and source-domain faces [2]. To provide a more comprehensive evaluation, we have conducted a user study in one-shot GDA. As indicated in Table 1 of the [PDF](https://openreview.net/attachment?id=xNZlQlaQNI&name=pdf), the users overwhelmingly favored our DoRM in all three aspects.\n\n**Q3. Missing citation to StyleSpace**\n\nThe introduction of the $s$ used in our paper is attributed to StyleSpace and we will promptly include the citation of StyleSpace in revision.\n\n**Q4. Can hybrids be produced by combining more than two domains?**\n\nAs depicted in Fig. 5 of the [PDF](https://openreview.net/attachment?id=xNZlQlaQNI&name=pdf), we have showcased an example of applying our method to create hybrid-domain images by activating the trained M&A modules of Baby, Sunglasses, and Sketch domains.  This illustration demonstrates how easily our approach can be adapted to generate hybrids involving more than two domains, showcasing the versatility and potential of our approach.\n\n**Q5. Are the produced hybrids less faithful to each individual domain and to what extent?**\n\nWhile our proposed method has showcased remarkable advancements in hybrid domain generation, it is true that some minimal compromise to the fidelity of each individual domain can occur.  For instance, in the hybrid domain \"elsa\"+\"Sunglasses\" of the Fig 6 in [PDF](https://openreview.net/attachment?id=xNZlQlaQNI&name=pdf),  there is a subtle reduction in the features of the sunglasses domain. Regrettably, quantifying this distortion is complex. We will resolve it in the future."}]

Author rebuttal:
We extend our heartfelt gratitude to all reviewers for their dedicated efforts and invaluable suggestions. We have meticulously addressed the specific concerns raised by each reviewer. For a more detailed breakdown of our responses, including supporting tables and figures, please refer to the attached [PDF](https://openreview.net/attachment?id=xNZlQlaQNI&name=pdf).

Furthermore, we are committed to enhancing the coherence and impact of our manuscript by incorporating these supplementary materials into the revised version. The reviews' insightful recommendations have significantly contributed to refining the clarity and rigor of our research. Notably, we have recognized a recurring interest among the reviewers regarding the contextualization of relevant work, prompting us to provide a General Response. Morever, we also provide some systematic and comprehensive comparisons with related work [2][3] in the Figure 6 of the [PDF](https://openreview.net/attachment?id=xNZlQlaQNI&name=pdf). The results illustrate that our propsoed Dorm is consistently superior to the previous studies.  

**General Response: Comprehensive Comparison and Discernment from Pertinent Works [1] [2] [3].**

We are poised to conduct a comprehensive and systematic analysis of pertinent prior research, underscoring the distinguishing facets that set them apart from our own endeavor:

HyperDomainNet [1]: HyperDomainNet employs a sole modulation parameter, mainly a scale ($\delta$), to manipulate convolutional layer weights, impacting the StyleGAN "s" space. This configuration maintains a constant scale across all images within a target domain. Specifically, HyperDomainNet's architecture is formalized as $w \cdot s_i \cdot \delta$, with $w$ and $s_i$ denoting source StyleGAN2's convolutional weight and style code respectively. We recognize that the overarching scale parameter's introduction may inadvertently constrain the generator's learning capacity. Notably, in scenarios where considerable domain gaps exist between the source and target domains, HyperDomainNet's efficacy could diminish. Moreover, the non-linear fusionability of the scale modulation parameter might hinder its efficacy in achieving robust domain association.

DynaGAN [2]: Among methods closely aligned with our DoRM, DynaGAN introduces two modulation parameters—shift ($\Delta s$) and scale ($\delta$)—to convolutional weights, influencing StyleGAN's "s" space. DynaGAN's structural configuration can be expressed as $w \cdot (s_i + \Delta s) \cdot \delta$. Analogous to HyperDomainNet, the non-linear blending of the scale modulation parameter may impede its capacity to establish robust domain associations (as exemplified in the Figure 6 of the [PDF](https://openreview.net/attachment?id=xNZlQlaQNI&name=pdf)). In contrast, our DoRM exclusively adopts a sample-specific domain shift denoted as $\Delta s_i$, which is mathematically formulated as $w \cdot (s_i + \Delta s_i)$. This innovative approach significantly amplifies the generator's aptitude for learning, facilitating seamless adaptation to a diverse array of target domains, even in the presence of substantial domain disparities (as demonstrated in Figure 6 of the [PDF](https://openreview.net/attachment?id=xNZlQlaQNI&name=pdf)). Impressively, this sample-specific domain shift effectively caters to both few-shot and one-shot Generalized Domain Adaptation (GDA), obviating the need for an additional domain scale parameter. Our streamlined methodology not only fosters robust domain association but substantiates its efficacy through empirical validation.

Domain Expansion [3]: Drawing inspiration from SeFA, Domain Expansion constructs a semantic and orthogonal basis V from right singular vectors obtained through SVD of the initial generator layer, impacting the latent space Z. Focusing on a subset of basis vectors, Domain Expansion aptly models source generator variability, repurposing these unexplored subsets to encapsulate desired behaviors for diverse target domains. Unlike alternative approaches, including HyperDomainNet, DynaGAN, and our DoRM, which expand the style space of the source domain to encompass target domains, Domain Expansion instead narrows the style space of the source domain. While this technique repurposes latent space, it inadvertently curtails the source domain's generative capacity, amplifying the intricacies and temporal requisites of domain adaptation. As depicted in the Figure 6 of the [PDF](https://openreview.net/attachment?id=xNZlQlaQNI&name=pdf), Domain Expansion encounters challenges when faced with substantial domain gaps between source and target domains. Evidently, its performance is compromised, as evidenced in instances such as adapting to the 10-shot generation context of the Sketch dataset and the one-shot generation context of the "elsa" dataset.

[1] HyperDomainNet: Universal Domain Adaptation for Generative Adversarial Networks. NIPS 2022


[2] DynaGAN: Dynamic Few-shot Adaptation of GANs to Multiple Domains. SIGGRAPH Asia 2022


[3] Domain Expansion of Image Generators. CVPR 2023


Meta-review: The paper received mixed ratings. All reviewers agreed that the tasks (multi-domain and hybrid domain generations) studied in this paper are interesting and the proposed method is intuitive and novel. Nonetheless, there were concerns raised about certain missing experimental results and the absence of comparisons and discussions with related works. The authors attempted to address these concerns during the rebuttal phase by providing additional insights through a user study, 3D results, ablation study, latent edit, and latent interpolation. Meanwhile, the authors conducted a comprehensive discussion and comparison with previously established state-of-the-art methods such as DynaGAN, HyperDomainNet, and Domain Expansion.

After considering the reviews and the author's responses, AC believes that the newly proposed generator demonstrates exceptional performance in domain association, a relatively unexplored facet within the realm of GANs. Overall, AC believes the value outweighs the issues in the paper and recommends acceptance.
Decision: Accept (poster)