SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Quantitative comparison. We use Fréchet Inception Distance (FID) to evaluate synthesis quality and diversity (lower is better). In 10-shot GDA, to better reflect the synthesis diversity, we use the Intra-LPIPS[32] to measure the synthesis diversity (higher means better synthesis diversity). Additionally, we measure cross-domain consistency using Identify similarity (ID) [56] (higher is better). Table 1 summarizes the three evaluation metrics for 10-shot GDA over different target domains. FreezeD [29] and GenDA [49] struggle to maintain cross-domain consistency. CDC [32], RSSA [47], DCL [59] and AdAM [58] improve the synthesis diversity and cross-domain consistency but fails in the synthesis quality (as seen in Figure 3), leading to unsatisfactory FID scores. Our method outperforms all these methods on the different target domains, achieving not only better synthesis diversity and cross-domain consistency (higher Identity score and Intra-LPIPS) but also better synthesis quality (lower FID score).
Table 1: Quantitative evaluation on 10-shot GDA. All the source generators are pre-trained on FFHQ[18]. The target domains include Sketches, FFHQ-Baby and FFHQ-Sunglasses. Evaluation metrics include FID, Intra-LPIPS (I-LPIPS), and Identify similarity (ID). Noting that Sketches dataset only contains about 300 images, the large synthesis diversity will harm to the FID score.
Datasets	FFHQ-Babies	FFHQ-Sunglasses	Sketches
Method	FID	I-LPIPS	ID	FID	I-LPIPS	ID	FID	I-LPIPS	ID
minegan	98.23	0.514	0.132	68.91	0.42	0.171	64.34	0.40	0.092
FreezeD	110.92	0.346	0.037	51.29	0.337	0.030	46.54	0.325	0.010
GenDA	47.05	0.556	0.029	22.62	0.548	0.004	31.97	0.407	0.011
CDC	74.39	0.573	0.326	42.13	0.562	0.318	45.67	0.453	0.214
RSSA	77.77	0.576	0.314	70.41	0.563	0.307	63.44	0.480	0.296
DCL	52.56	0.582	-	38.01	-	-	37.90	0.486	-
AdAM	48.83	0.590	0.249	28.03	0.592	0.306	55.74	0.495	0.198
DoRM	30.31	0.623	0.445	17.31	0.644	0.389	40.05	0.502	0.365
4.3 Multi-domain and Hybrid-domain Generation
Multi-domain generation. Our proposed DoRM utilizes a novel approach by learning and retaining knowledge of new domains through the formation and update of new M&A modules, instead of updating the entire generator. This unique characteristic enables our DoRM, with a single generator, to generate images across multiple domains (as depicted in the top part of Figure 4). In contrast to previous methods [32; 47; 59; 58] that require updating the entire generator for few-shot GDA, limiting its capability into single domain generation, our DoRM offers significant storage space savings in multi-domain generation. As shown in Table 2, our method's models are about  $3 \times$  smaller than previous methods [32] on 10-domain generation.
Hybrid-domain generation. In contrast to previous methods [32; 59; 55] that fine-tune the entire generator, our DoRM achieves an effective linear domain shift through the update of new M&A
Real babies
FreezeD
GenDA
CDC
RSSA
AdAM
DoRM
Source
Real sketches
FreezeD
GenDA
CDC
RSSA
AdAM
DoRM
Source
Figure 3: Qualitative comparison on 10-shot GDA. The source domain is FFHQ, and the target domains include Sketches and FFHQ-Babies. We compare our method with FreezeD[29], GenDA[49], CDC[32], RSSA[47] and AdAM[58]. Feeding the same latent code  $z$  into the source generator and the target generator, we obtain the corresponding images in the source and target domains. Our method shows better cross-domain consistency and synthesis quality than the other methods.
modules. By learning multiple M&A modules that store the knowledge of various training domains, DoRM not only enables multi-domain generation but also facilitates the synthesis of images in hybrid domains not present in the training data (as depicted in the bottom left part of Figure 4). For comparison, we conduct hybrid-domain generation experiments using CDC [32]. In CDC, the domain shift between the source and target domain generators is obtained by subtracting their corresponding parameters. Similar to our DoRM, CDC achieves hybrid-domain generation by combining multiple domain shifts, adding the parameters of each shift to the source domain generator. However, as illustrated in Figure 4, CDC exhibits unsatisfactory synthesis quality in hybrid-domain generation. In contrast, the images synthesized by DoRM not only inherit domain-sharing attributes such as pose, gender, and identity, but also seamlessly blend domain-specific attributes. Furthermore, we extensively explore domain association among multiple domains using our proposed DoRM in Figure 5. The results demonstrate that DoRM efficiently integrates diverse domains to generate high-quality
Source domain
Sketch domain
Multi-domain generation (Ours)
Baby domain
Sunglasses domain
Hybrid-domain Generation (Ours)
Sketch-Baby domain
Figure 4: Qualitative results on multi-domain and hybrid-domain generation. The top part shows the multi-domain generation of our methods on three target domains. The bottom part illustrates the comparison on hybrid-domain generation between our proposed DoRM (bottom left) and CDC [32] (bottom right). The source generator is pre-trained on FFHQ [18]. The adopted M&A modules in our DoRM are trained in Section 4.2.
Sketch-Sunglasses domain
Hybrid-domain Generation (CDC)
Sketch-Baby domain
Sketch-Sunglasseses domain
images in hybrid domains not present in the training data. In this case, our proposed DoRM has the ability to generate creative outputs in previously unseen domains by incorporating knowledge learned from other domains, similar to how humans can draw on past experiences to generate novel ideas. To the best of our knowledge, our proposed DoRM is the first method to achieve efficient domain association in this manner. Additional visualizations of hybrid domain generation can be found in Section A.6.
Table 2: Storage Comparison. The number of generator parameters required to realize different multi-domain generations.
Model Size	2-domain	5-domain	10-domain
CDC [32]	48M	120M	240M
Ours	30M	54M	84M
We conduct ablation studies to evaluate the impact of our proposed DoRM generator structure and similarity-based structure loss. As shown in Figure 6, the images synthesized by baseline (which simply fine-tunes the StyleGAN2 generator with adversarial loss [18]) exhibit severe overfitting to the training images. By contrast, fine-tuning our DoRM structure generator with adversarial loss yields better synthesis diversity than the baseline. Moreover, when we optimize our DoRM generator structure with both adversarial loss and similarity-based structure loss, the synthesis images exhibit appealing quality and diversity, while maintain high cross-domain consistency, which indicates the proposed loss effectively preserves cross-domain consistency in few-shot GDA. More ablation studies on the network component of our DoRM are presented in Section A.1.
5 Limitation and Conclusion
Limitation. i) Although our method can achieve appealing results on the few-shot GDA across different target domains, however, the strength of domain shift  $\alpha$  currently needs to be manually adjusted. Furthermore, since the various depth layers capture different semantic attributes, equipping each layer with an appropriate  $\alpha$  may help to further improve synthesis quality. ii) The current
Otto Dix painting
Baby
Sunglasses
Baby-Sketch
Sunglasses-Sketch
Baby-Caricature
Sunglasses-Caricature
Baby-Otto Dix
Sunglasses-Otto Dix
Figure 5: Domain association between different domains by our DoRM. The first row and the first column show the 10-shot GDA results on different target domains. By simply combining the corresponding pretrained M&A modules in 10-shot GDA, DoRM can integrate the domains and synthesize the high-quality images in the hybrid domains.
Source domain
Baseline
Figure 6: Ablation study on DoRM and  $L_{ss}$ . The training images are 10-shot sketches.
DoRM
DoRM  $+L_{SS}$
manuscripts only simply combines the M&A modules of different target domains and activate them at the same time to realize the domain association. To further improve the performance of the domain association, not only combining the trained target M&A modules but also employing a new M&A module and additional consistency loss is a better method to blend the target domains.
Conclusion. In this paper, we present DoRM, a novel generator structure inspired by the learning and storage mechanisms of the human brain, specifically designed for few-shot GDA. Our DoRM is characterized by its simplicity and efficiency. Additionally, we introduce a novel similarity-based structure loss to ensure cross-domain consistency in the few-shot GDA. Through extensive qualitative and quantitative evaluations, we demonstrate the superiority of our method over existing approaches in terms of synthesis quality, diversity, and cross-domain consistency. Importantly, akin to the human brain, our DoRM exhibits memory and the ability to integrate knowledge from different domains, enabling the generation of images in novel hybrid domains not encountered during training.
The work was supported in part by the National Natural Science Foundation of China under Grands U19B2044 and 61836011.
A Supplementary Materials
Effect of Target Mapping. In this section, we first investigate the significance of the target mapping module in our proposed DoRM. To evaluate the significance of the target mapping, we conduct an ablation study using the source mapping as a substitute for the target mapping, which remains frozen during the entire 10-shot generative domain adaptation training. As shown in Table 3, our results indicate a significant deterioration in the FID score without the target mapping, implying a considerable drop in the quality and diversity of the generated samples. Furthermore, Figure 7 illustrates that the generative domain adaptation barely occurs during training without the target mapping. This is because the target mapping plays a crucial role in capturing the representative attributes of the target domain and assisting in the acquisition of the domain shift during the adaptation process.
Figure 7: Qualitative ablation study of the target mapping. We compare the performance of our DoRM method, shown in the first row, with a variant of the method where the frozen source mapping is used as a substitute for the target mapping, depicted in the second row. We evaluate the performance of both methods on three different target domains: Sketches, FFHQ-Babies, and FFHQ-Sunglasses.
Table 3: Quantitative ablation study of the target mapping. The evaluation metric is FID (lower is better). We compare the performance of our DoRM method, shown in the first row, with a variant of the method where the frozen source mapping is used as a substitute for the target mapping, depicted in the second row. The source generator is pre-trained on FFHQ[18], and the target domains include FFHQ-Babies and FFHQ-Sunglasses.
Babies	Sunglasses
DoRM	30.31	17.31
DoRM w/o Target Mapping	86.52	74.71
Effect of Re-Modulation Layers. Another crucial component of our proposed DoRM approach is the target affine module. To investigate the roles of the different target affines in DoRM, we perform experiments where we drop the target affines in both the low-resolution and high-resolution feature maps. Specifically, we conduct 10-shot generative domain adaptation experiments to evaluate the Fréchet Inception Distance (FID) of the generated samples. For an image with a resolution of  $256 \times 256$ , the low-resolution feature maps include resolutions of  $4 \times 4$ ,  $8 \times 8$ ,  $16 \times 16$ , and  $32 \times 32$ . As presented in Table 4, our results demonstrate that all target affines are crucial for the performance of our DoRM approach and their removal leads to a significant drop in the quality and diversity of the generated samples.
Effect of Re-Modulation Weight. The re-modulation weight is a crucial parameter that controls the strength of the acquired domain shift in our proposed DoRM approach. A small re-modulation weight leads to a lower strength of the domain shift, resulting in more attributes of the source domain being preserved during generative domain adaptation. To investigate the impact of the re-modulation weight, we conduct 10-shot generative domain adaptation experiments using different re-modulation weights. The results are presented in Table 5. Our results demonstrate that different domain gaps have different optimal re-modulation weights, indicating that the selection of the re-modulation weight should be tailored to the specific target domain.
Effect of Target domain classifier. We investigate the effect of the target domain classifier on the performance of our proposed DoRM approach. Specifically, we experiment with different depths
Table 4: Quantitative ablation study of the target affine layers. The evaluation metric is FID (lower is better). We compare the FID score of our proposed DoRM method, shown in the first row, with two variants: one where the target affines are removed from the low-resolution feature maps, shown in the second row, and another where the target affines are removed from the high-resolution feature maps, shown in the third row. We use a source generator pre-trained on FFHQ [18] and evaluate all three methods on two different target domains: FFHQ-Babies and FFHQ-Sunglasses.
babies	sunglasses
DoRM	30.31	17.31
DoRM w/o target affines in low resolution	93.28	92.42
DoRM w/o target affines in high resolution	37.16	20.81
Table 5: Quantitative ablation study of the re-modulation weight. The evaluation metric is FID (lower is better). We conduct 10-shot generative domain adaptation experiments using our DoRM approach with varying re-modulation weights. We use a source generator pre-trained on FFHQ [18] and evaluate our method on two different target domains: FFHQ-Baby and FFHQ-Sunglasses.
Re-modulation weight α	0.5	0.2	0.05	0.005	0.001
FFHQ-Baby	37.9	36.1	34.0	30.3	32.3
FFHQ-Sunglasses	18.7	17.3	17.9	18.5	19.4
(e.g., number of MLP layers) and initialization methods for the target domain classifier in 10-shot generative domain adaptation. As presented in Table 6, our results indicate that the two-layer MLP target domain classifier achieves the best performance. This is because the one-layer MLP lacks the ability to classify the target domain effectively, while the three-layer MLP is prone to overfitting due to the limited number of training images.
Ablation of Individual Components of Previous State-of-the-art Methods. To highlight the specific strengths of the proposed method that outperform other works, we consider the individual components and techniques utilized in previous state-of-the-art approach [55]. Specifically, we systematically analysis and compare individual components and techniques utilized in our DoRM and DiFa [55] to showcase how our method surpasses these works. According to the Introduction of the manuscript, GDA needs three fundamental properties. To resolve it, DiFa [55] proposed two CLIP-based loss: global loss  $L_{global}$  and local loss  $L_{local}$  for realizing large diversity/cross-domain consistency and high quality, respectively. Differently, we adopt Similarity-based Structure Loss ( $L_{ss}$ ) and adversarial loss ( $L_{adv}$ ) for realizing large diversity/cross-domain consistency and high quality, respectively. As shown in the Figure 8, the  $L_{local}$  in DiFa mainly focuses on textual features and faibles to capture the complete features (e.g. the white background in sketches) in generative domain adaptation. In our DoRM, we employ adversarial loss which can fully capture the features of the target images.
Table 6: Quantitative ablation study of the target domain classifier. The evaluation metric is FID (lower is better). We experiment with different depths of the target domain classifier, using various initialization methods. We use a source generator pre-trained on FFHQ [18] and evaluate our proposed DoRM approach on 10-shot generative domain adaptation tasks.
FFHQ-Baby	FFHQ-Sunglasses
MLP Depth	source initial	random initial	source initial	random initial
one layer	38.03	37.15	24.63	22.15
two layers	33.32	30.31	20.40	17.31
three layers	—	33.25	—	20.42
A.2 More Synthesis Results in 10-shot GDA
Qualitative results on 10-shot generative domain adaptation. We present qualitative results of our proposed DoRM approach on more target datasets, including face caricatures, face paintings by Raphael, face paintings by Amedeo Modigliani, and face paintings by Otto Dix [50]. All training images are shown in Figure 9, and the qualitative results are presented in Figure 10. Our results
Source(FFHQ)
$L_{adv} + L_{ss}$
Figure 8: Ablation of individual components of previous state-of-the-art methods.
$L_{\mathrm{local}} + L_{global}$
$L_{\mathrm{local}} + L_{ss}$
demonstrate that our proposed DoRM approach achieves appealing synthesis quality in various target domains.
Results of latent interpolation. We perform latent space interpolation to demonstrate that our DoRM is not harmful to the learned latent space. In Figure 11, the first and last columns show the generated images with two latent codes after 10-shot generative domain adaptation, while the remaining columns show the results obtained by linearly interpolating the two latent codes. Our results demonstrate that all intermediate synthesized images have high target-domain consistency and high cross-domain consistency. Moreover, the semantics of the generated images, such as gender, haircut, and pose, vary gradually throughout the interpolation, indicating that our proposed DoRM approach preserves the underlying semantic structure of the learned latent space.
Results of latent editability. We have performed editing on a real image adapted into a new target domain using StyleGAN-CLIP to discover editing directions in the source domain. The results, as illustrated in Fig 12, indicate that the adapted generator maintains similar latent-based editing capabilities to the original generator. This demonstrates the preservation of editability in the adapted generator.
Results on 3D generator. We conducted some initial studies using the popular 3D-aware image generation method, EG3D [3], for one-shot GDA with FFHQ as the source domain and Sketch as the target domain. The results, as shown in the Figure 13, reveal that directly migrating the proposed DoRM to 3D one-shot domain adaptation poses challenges and might not be straightforward. Some potential challenges in the 3D-GAN domain include:
(1) Overfitting: 3D GANs often face more severe overfitting issues when dealing with limited training data, requiring more rigorous regularization techniques and training strategies for one-shot GDA.
(2) Volumetric Data Representation: Handling volumetric data representation in 3D-GANs necessitates specialized techniques for data manipulation, augmentation, and visualization.
(3) Spatial Artifacts: Generating high-quality 3D objects may encounter spatial artifacts, such as geometric distortions or inconsistent shapes, which need to be addressed.
(4) Computational Complexity: The computational demands of 3D-GANs can be significantly higher than their 2D counterparts, presenting challenges in both training and inference.
Results on three-domain hybrid domain generation. As depicted in the Figure 14, we have showcased an example of applying our method to create hybrid-domain images by activating the trained M&A modules of Baby, Sunglasses, and Sketch domains. This illustration demonstrates how easily our approach can be adapted to generate hybrids involving more than two domains, showcasing the versatility and potential of our approach.
A.3 Experiments on one-shot Generative Domain Adaptation
Although our DoRM is mainly for few-shot generative domain adaptation, DoRM is also can be employed for one-shot generative domain adaptation [21; 55; 6]. In the one-shot GDA, the training dataset is a single image, which is difficult for the backbone of discriminator to extract the main characters of the target domain because of the overfitting issue. In this case, we introduce a clip-based local-level adaptation loss  $L_{local}$  from [55] to help to acquire the local-level characters and styles of the target domain. Concretely, we extract the intermediate tokens of the adapted image  $I_B$  synthesized by DoRM and the single target image  $I_{tar}$  from the  $k - th$  layer of CLIP image encoder. And align each of adapted token  $F_B$  with its closest target token from  $F_{tar}$ , where  $F_B = F_B^1, \dots, F_B^n$  and
$F_{tar} = F_{tar}^{1},\ldots F_{tar}^{m}$  are the extracted tokens. The clip-based local-level adaptation loss is defined as:
$$
L _ {l o c a l} = \max  \left(\frac {1}{n} \sum_ {i} \min  _ {j} C _ {i, j}, \frac {1}{m} \sum_ {j} \min  _ {i} C _ {i, j}\right) \tag {6}
$$
where  $C$  is calculated as:
$$
C _ {i, j} = 1 - \frac {F _ {B} ^ {i} \cdot F _ {t a r} ^ {j}}{| F _ {B} ^ {i} | | F _ {t a r} ^ {j} |} \tag {7}
$$
Furthermore, to better identify and maintain the domain-sharing attributes in one-shot generative domain adaptation, we also employ the inversion-based selective cross-modal consistency loss  $L_{scc}$  from [55]. Specifically, this loss function aims to identify and preserve domain-sharing attributes in the  $W+$  space. The underlying assumption is that attributes that are similar in  $W+$  space between the source and target domains during adaptation are more likely to be domain-sharing attributes. To achieve this,  $L_{scc}$  dynamically analyzes and retains these attributes. First, it inverts the source and corresponding target images into  $W+$  latent codes,  $w_A$  and  $w_B$ , respectively, using a pre-trained inversion model such as pSp or e4e, for each iteration. Next, it computes the difference  $\Delta w$ , between the centers of a source queue of  $W+$  latent codes,  $X_A$  and the target queue of  $W+$  latent codes,  $X_B$ , where  $X_A$  and  $X_B$  are dynamically updated with  $w_A$  and  $w_B$  during training. The loss function then encourages  $w_A$  and  $w_B$  to be consistent in channels with less difference, thereby facilitating the preservation of domain-sharing attributes. The inversion-based selective cross-modal consistency loss  $L_{scc}$  is defined as follows:
$$
L _ {s c c} = \left\| m a s k \left(\Delta w, \alpha\right) \cdot \left(w _ {B} - w _ {A}\right) \right\| _ {1} \tag {8}
$$
where  $\alpha$  represents the proportion of preserved attributes, and  $mask(\Delta w, \alpha)$  determines which channels to retain. Specifically, let  $|\Delta w_{s_{\alpha N}}|$  be the  $\alpha N - th$  largest element of  $\Delta w$ . Then, each dimension of  $mask(\Delta w, \alpha)$  is calculated as follows:
$$
m a s k (\Delta w, \alpha) _ {i} = \left\{ \begin{array}{l l} 1 & | \Delta w _ {i} | \leq | \Delta w _ {s _ {\alpha N}} | \\ 0 & | \Delta w _ {i} | \geq | \Delta w _ {s _ {\alpha N}} | \end{array} \right. \tag {9}
$$
We compare our DoRM++ approach which denotes introducing the two new loss terms into training with state-of-the-art one-shot generative domain adaptation (GDA) methods, including JoJoGAN [6], Generalized One-shot Domain Adaptation [56], DynaGAN[20] and DiFa [55]. Figure 16 shows the comparison results. Our results indicate that JoJoGAN, DynaGAN and Generalized One-shot Domain Adaptation fail to achieve GDA when the target image is FFHQ-Baby and FFHQ-Sunglasses, and the synthesis quality of DynaGAN is limited. Similarly, DiFa also fails to achieve GDA when the target image is FFHQ-Sunglasses, and the synthesis diversity is unsatisfactory when the target image is FFHQ-Baby.
In contrast, our DoRM++ approach achieves one-shot GDA among all the reference images, resulting in high-quality and diverse synthesis, while maintaining appealing cross-domain consistency. Moreover, our DoRM++ generator has memory to realize multiple target domains' generation, which saves a significant amount of storage space. Our DoRM++ generator also has the ability to integrate the learned knowledge of multiple target domains to synthesize images in hybrid domains that are unseen in the target domains. As shown in Figure 17, the DoRM++ generator can synthesize high-quality and diverse images in hybrid domains while maintaining the domain-sharing attributes (e.g., pose, identity).
A.4 User Study in One-shot GDA