SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yi Wu*, Ziqiang Li*†
Chaoyue Wang;
Heliang Zheng,
Shanshan Zhao
JD Explore Academy
Bin Li†
Dacheng Tao
In this study, we delve into the task of few-shot Generative Domain Adaptation (GDA), which involves transferring a pre-trained generator from one domain to a new domain using only a few reference images. Inspired by the way human brains acquire knowledge in new domains, we present an innovative generator structure called Domain Re-Modulation (DoRM). DoRM not only meets the criteria of high quality, large synthesis diversity, and cross-domain consistency, which were achieved by previous research in GDA, but also incorporates memory and domain association, akin to how human brains operate. Specifically, DoRM freezes the source generator and introduces new mapping and affine modules (M&A modules) to capture the attributes of the target domain during GDA. This process resembles the formation of new synapses in human brains. Consequently, a linearly combinable domain shift occurs in the style space. By incorporating multiple new M&A modules, the generator gains the capability to perform high-fidelity multi-domain and hybrid-domain generation. Moreover, to maintain cross-domain consistency more effectively, we introduce a similarity-based structure loss. This loss aligns the auto-correlation map of the target image with its corresponding auto-correlation map of the source image during training. Through extensive experiments, we demonstrate the superior performance of our DoRM and similarity-based structure loss in few-shot GDA, both quantitatively and qualitatively. Code will be available at https://github.com/wuyi2020/DoRM.
Domain adaptation aims to bridge the domain gap and transfer knowledge to mitigate the limitations imposed by a lack of extensive labeled data [8; 40; 51]. Recent research has explored the field of few-shot Generative Domain Adaptation (GDA), which aims to achieve realistic and diverse generation with only a few training images [29; 32; 34; 43; 47; 59; 31; 42; 23; 62]. Particularly, the
This work was performed when Ziqiang Li was visiting JD Explore Academy as a research intern.
$\ddagger$  Corresponding Author.
Figure 1: Left: Framework of our DoRM. Given a generator of StyleGAN2 structure pre-trained on a source domain (components in the white dotted blocks), we can realize multi-domain generation and hybrid-domain generation by activating the corresponding trained M&A modules (components in the colourful solid blocks). Right: Multi-domain generation (right top) and hybrid-domain generation (right bottom). For instance, by activating the trained M&A modules of Baby and Sketch domains, hybrid-domain (Sketch-Baby domain) generation has been realized easily.
purpose of few-shot  $\mathrm{GDA}^4$  is to transfer a generator pre-trained on the source domain to the target domain using a few reference images.
Existing few-shot GDA works primarily focus on three properties: (i) High quality and (ii) Large diversity, where the adapted generator can synthesize high quality and diverse images in the target domain and (iii) Cross-domain consistency, where the adapted images and their corresponding source images should be consistent in terms of domain-sharing attributes, such as pose and identity. A majority of them update the entire generator with regularization techniques including GAN inversion [55; 47; 63], Contrastive-Language-Image-Pretraining (CLIP) [10; 55; 63], and consistency loss [32; 47; 59] to achieve large diversity and cross-domain consistency of adaptation. However, updating the entire generator imposes limitations on its ability to synthesize multiple target domains. In contrast, the human brain employs a more efficient approach to learn from new domains by forming new proteins[4], allowing for the retention and integration of knowledge from multiple domains. This creative capacity enables humans to explore previously unseen domains.
Drawing inspiration from human learning processes, we propose a novel generator structure called Domain Re-Modulation (DoRM) based on StyleGAN2 [19]. DoRM not only fulfills three essential properties of few-shot GDA but also introduces two new capabilities: (iv) Memory, enabling the generator to retain knowledge from previously learned domains when generating images in new domains, and (v) Domain Association, allowing the generator to integrate multiple learned domains and synthesize hybrid domains not encountered during training. As depicted in Figure 1, DoRM freezes the pre-trained source generator and introduces new mapping and affine modules (M&A modules) to achieve the necessary domain shift in the style space. This approach enables DoRM to perform high-fidelity generation across multiple domains by selectively activating different M&A modules, which significantly reduces the required storage space. Moreover, as StyleGAN's style space is a linear subspace [36; 24; 45], the domain shift obtained through DoRM is linearly combinable. Consequently, DoRM can synthesize images in hybrid domains not present in the training dataset by simultaneously activating multiple M&A modules. To further enhance cross-domain consistency, we introduce a similarity-based structure loss denoted as  $L_{ss}$ . Specifically, we leverage the CLIP image
<sup>4</sup>It is important to note that the objective of few-shot Generative Domain Adaptation (GDA) differs from that of Few-Shot Image Generation (FSIG). FSIG focuses on generating high-quality and diverse images in the target domain using a limited number of training samples, whereas few-shot GDA involves additional considerations beyond FSIG.
encoder [33] to extract intermediate tokens from the target image and its corresponding source image. We then enforce consistency between the auto-correlation maps of these tokens, ensuring improved alignment between target and source images.
We conducted extensive experiments on few-shot GDA involving various source and target domains. Our experiments, both quantitative and qualitative, highlight the competitive performance of our method compared to state-of-the-art approaches in 10-shot GDA. We achieved remarkable results in terms of quality, diversity, and cross-domain consistency. Notably, our proposed DoRM stands out by demonstrating exceptional performance in hybrid-domain generation, a hard task that previous methods have not accomplished well. The key contributions of this work can be summarized as follows:
- We introduce DoRM, a novel generator structure for few-shot generative domain adaptation, inspired by the learning mechanism of human brains. DoRM not only excels in synthesizing high-quality, diverse, and cross-domain consistent images, but also integrating memory and domain association capabilities that remain relatively unexplored in the field. Notably, our approach is one of the very few that encompasses all five desired properties of GDA.
- Additionally, we propose a novel consistency loss called similarity-based structure loss, which further enhances cross-domain consistency in our approach.
- Through comprehensive evaluations, our proposed method outperforms existing competitors across various settings, showcasing its superior performance.
2.1 Generative Adversarial Networks
Deep learning has made remarkable achievements in various fields [46; 44; 22]. Among them, generative adversarial networks [11; 37; 28] play a two-player adversarial game, where the generator aims to synthesize realistic images to fool the discriminator, while the discriminator learns how to distinguish the synthesized images from real ones. Previous works [16; 2; 53; 18; 19] have significantly improved the synthesis capability on high-resolution datasets. Style-based methods [18; 19; 35] remain the state-of-the-art unconditional GANs owing to their unique architecture. However, GANs training requires a large number of training images, or severe discriminator overfitting can occur [26; 27]. To alleviate this issue and improve training stability, various data augmentation techniques [57; 15; 17; 61; 60; 25] and regularization technologies [26; 48; 39; 14] have been introduced. Among these, Adaptive Discriminator Augmentation (ADA) is an adaptive strategy that controls the strength of augmentations and has shown remarkable performance, becoming the default operation in data-efficient GANs. Tseng et al. [39] proposed a regularization scheme to modulate the discriminator's prediction, mitigating discriminator overfitting. However, most of these techniques fail in the extremely small (e.g., 10) training data regime.
2.2 Few-shot Generative Domain Adaptation
Previous works in few-shot GDA mainly have primarily focused on three essential properties: High quality, Large diversity, and Cross-domain consistency. Some studies have fine-tuned the entire generator using regularization techniques. For example, [32; 47] introduced a consistency loss based on Kullback-Leibler (KL) divergence to preserve the relative similarities and differences between instances, inheriting diversity and maintaining consistency during adaptation. The contrastive term [12; 5] has also been employed to construct consistency loss in [59]. In addition, leveraging the semantic power of large Contrastive-Language-Image-Pretraining (CLIP) [33] models, [10; 55; 63] proposed to define the domain-gap direction in CLIP embedding space, guiding the optimization of the generator. Furthermore, GAN inversion [7; 38] technique has been widely used in few-shot GDA to meet different purposes, such as exploring the domain-sharing attributes [55; 63], and compressing the latent space to a subspace to relax the cross-domain alignment [47]. Compared to optimizing the entire generator, some studies [31] only update the crucial part of the generator. Moreover, some work [49] adds a lightweight attribute adaptor and attribute classifier before the frozen generator and after the frozen discriminator, respectively. The proposed method achieves remarkable performance in synthesis quality and diversity but lacks cross-domain consistency.
Figure 2: Our framework (DoRM) is based on StyleGAN2. In the Synthesis network  $g(\cdot)$  (gray blocks), the source affines  $A_{s}$  and the source mapping  $f_{s}$  constitute the pre-trained source generator on a source domain. To achieve generative domain Adaptation, we incorporate a new target mapping  $f_{t}$  and target affines  $A_{t}$ . During training, we only optimize the parameters of the solid yellow blocks.
Furthermore, [20; 1] share the similar idea with us of incorporating multi-domain generation capabilities into a single generator. However, these methods employ multiplicative modulation, which is indeed different with ours. Specifically, HyperDomainNet [1] adopts a solitary modulation parameter, primarily a scale  $(\delta)$ , to adjust the weights of the convolutional layer, influencing the "s" space of StyleGAN. In this setup, the scale parameter remains constant for all images within a target domain. Formally, HyperDomainNet's architecture is represented as  $w \cdot s_i \cdot \delta$ , where  $w$  and  $s_i$  denote the convolutional weight and style code of the source StyleGAN2, respectively. However, the introduction of this all-encompassing scale parameter can potentially restrict the generator's learning capacity. Especially in scenarios with significant domain gaps between target and source domains, HyperDomainNet's performance may decline. Furthermore, the non-linear combinability of the scale modulation parameter impedes its ability to achieve robust domain association. Among the approaches closest to our DoRM, DynaGAN [20] introduces two modulation parameters—shift  $(\Delta s)$  and scale  $(\delta)$ —to the convolutional weights affecting StyleGAN's "s" space. DynaGAN's architecture can be defined as  $w \cdot (s_i + \Delta s) \cdot \delta$ . Similar to HyperDomainNet, the non-linear combinability of the scale modulation parameter impedes its ability to achieve robust domain association. In contrast, our DoRM solely embraces a sample-specific domain shift denoted as  $\Delta s_i$ , formalized as  $w \cdot (s_i + \Delta s_i)$ . This distinct design substantially enhances the generator's learning potential, enabling adaptation to a diverse spectrum of target domains, even in the presence of considerable domain gaps. Notably, the sample-specific domain shift proves sufficient for both few-shot and one-shot GDA, eliminating the necessity for an additional domain scale parameter. This streamlined approach not only fosters domain association but has been effectively demonstrated. Additionally, although [30] defines and addresses a related task called Domain Expansion and domain composition (referred to as memory and domain association in our paper), it inadvertently curtails the source domain's generative capacity, amplifying the intricacies and temporal requisites of domain adaptation. As depicted in the experimental results, [30] encounters challenges when faced with substantial domain gaps between source and target domains. Evidently, its performance is compromised, as evidenced in instances such as adapting to the 10-shot generation context of the Sketch dataset and the one-shot generation context of the "elsa" dataset.
In this section, we present our generator structure, DoRM (Section 3.1), which takes inspiration from the way human brains learn new knowledge in different domains. DoRM excels in generating high-quality, diverse images while maintaining strong cross-domain consistency in few-shot GDA. Additionally, it possesses the unique ability to integrate multiple learned domains, allowing for the synthesis of images in hybrid domains that were not present in the training dataset. To address the issue of discriminator overfitting, we freeze the backbone of the source discriminator and introduce a target domain classifier (Section 3.2). Furthermore, we propose a novel similarity-based structure loss, denoted as  $L_{ss}$  (Section 3.3), to enhance the maintenance of cross-domain consistency. Finally, we introduce the overall training loss in Section 3.4.
3.1 Domain Re-modulation Structure of Generator
StyleGAN2. Our method is applied to a pre-trained StyleGAN2 [19]. Unlike general GANs that directly feed latent code to the generator, StyleGAN2 [19] employs a non-linear mapping network  $f(\cdot)$  to transform the latent space  $\mathcal{Z}$  into an intermediate latent space  $\mathcal{W}$ . The latent code  $(w \in \mathcal{W})$  is transformed into the style code  $(s \in S)$  through a learned affine transformation  $A$ . Finally, the style code is inserted into the synthesis network  $g(\cdot)$  through the modulation component at each convolution layer.
Domain Re-modulation of Generator. To equip the generator with memory and inherit the semantic information (e.g., glasses, gender, ages) from the source domain, we propose DoRM, illustrated in Figure 2. We freeze the given StyleGAN2 model pre-trained on the source domain and employ a new mapping network  $f_{t}$  and new affine transformation  $A_{t}$  (M&A module) to build  $\mathcal{W}_t$  space and  $S_{t}$  space, respectively. Through the new M&A module, we obtain domain shift as well as the target style codes. The added  $f_{t}$  and  $A_{t}$  have the same architecture as  $f_{s}$  and  $A_{s}$ , respectively, and are initialized by  $f_{s}$  and  $A_{s}$ . Accordingly, each layer of the synthesis network is controlled by source and target style codes. Given a latent code  $z$  sampled from Normal distribution  $\mathcal{Z}$  ( $z \in \mathcal{Z}$ ),
$$
w _ {t} = f _ {t} (z) \in \mathcal {W} _ {t}, \quad w _ {s} = f _ {s} (z) \in \mathcal {W} _ {s}, \tag {1}
$$
where  $\mathcal{W}_t$  and  $\mathcal{W}_s$  contain information from the target domain and source domain, respectively. Transformed by their own learned affine layers (Eq. 2), the information in these two domains represents their respective styles and is combined into general styles  $s$  (Eq. 3) to control the synthesis network together:
$$
s _ {t} = A _ {t} \left(w _ {t}\right), \quad s _ {s} = A _ {s} \left(w _ {s}\right), \tag {2}
$$
$$
s = \alpha s _ {t} + (1 - \alpha) s _ {s}, \tag {3}
$$
where  $\alpha$  is a hyper-parameter that controls the strength of the domain shift. In GDA, it tends to reuse various factors like pose, content, structure, etc. from the source domain and learns the most distinguishable characteristics of the target domain. To preserve variation factors as much as possible in the source domain,  $\alpha$  is set to a relatively small value (More analyses can be found in Section A.1). The combined style code  $s$  modulates the convolution weights through  $w_{i}^{\prime} = s\cdot w_{i}$ , where  $w_{i}^{\prime}$  is the modulated weights. Then, demodulation is employed to eliminate the influence of the style code  $s$  from the statistics of the convolution's output feature maps, which has been formalized as  $w_{i}^{\prime \prime} = \frac{w_{i}^{\prime}}{\|w^{\prime}\|_{2}}$ , where  $\| w^{\prime}\|_{2}$  represents the L2-Norm function.
3.2 Target Domain Classifier
In GANs training, the discriminator typically distinguishes the training images from the generated images. However, in few-shot GDA, where there are extremely few training images, the discriminator can easily overfit to the training images, leading to severe model collapse in the generator. In this work, we treat the discriminator as a target domain classifier that measures the probability of the images belonging to the target domain. Defining the target domain by only a few training images is ambitious, so we desire the discriminator to extract the most representative feature of training images that portray the target domain. Inspired by [29], we reuse and fix the feature extractor  $d(\cdot)$  of the pre-trained source discriminator to inherit its strong feature extraction capability. Additionally, we introduce a target domain classifier  $\phi(\cdot)$  on the top of the feature extractor  $d(\cdot)$ . We use a two-layer multi-layer perceptron (MLP) as the target domain classifier  $\phi(\cdot)$  and the target domain classifier
$\phi(\cdot)$ , and it is updated from scratch, which outperforms directly fine-tuning the final layer of the original discriminator, especially for large domain-gap GDA. Given an image  $x$ , the discriminator measures the probability that the image belongs to the target domain using  $p = \phi(d(x))$ .
3.3 Similarity-based Structure Loss
To explicitly model the cross-domain consistency in generative domain adaptation, we propose a novel similarity-based structure loss called  $L_{ss}$ . Our intuition is that the auto-correlation maps of the source image and its corresponding target image should be consistent during generative domain adaptation. To achieve this, we extract the intermediate tokens  $F_{A}$  and  $F_{B}$  of the source image  $I_{A}$  and its corresponding target image  $I_{B}$  from the k-th layer of the CLIP image encoder. These tokens are denoted by  $F_{A} = F_{A}^{1},\dots ,F_{A}^{n}$  and  $F_{B} = F_{B}^{1},\dots ,F_{B}^{n}$ , respectively. We define the auto-correlation maps as  $M_A = \frac{F_A^T}{|F_A^T|}\times \frac{F_A}{|F_A|}$  and  $M_B = \frac{F_B^T}{|F_B^T|}\times \frac{F_B}{|F_B|}$ , where  $M_A^{i,j} = \frac{\boldsymbol{F}_A^i\cdot\boldsymbol{F}_A^j}{|\boldsymbol{F}_A^i||\boldsymbol{F}_A^j|}$  and  $M_B^{i,j} = \frac{\boldsymbol{F}_B^i\cdot\boldsymbol{F}_B^j}{|\boldsymbol{F}_B^i||\boldsymbol{F}_B^j|}$ . The  $L_{ss}$  is then defined as the L2 norm of the difference between  $M_A$  and  $M_B$ :
$$
L _ {s s} = \frac {1}{n ^ {2}} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {n} \left| \left| M _ {A} ^ {i, j} - M _ {B} ^ {i, j} \right| \right|, \tag {4}
$$
where  $||\cdot ||$  is the L2 norm function.
3.4 Overall Training Loss
Our overall training loss includes original adversarial training loss in StyleGAN2 and the similarity-based structure loss  $L_{ss}$ :
$$
\mathcal {L} _ {G} = - E _ {z \sim p (z)} [ \log (D (G (z))) ] + \lambda_ {s s} \mathcal {L} _ {s s}, \tag {5}
$$
$$
\mathcal {L} _ {D} = - E _ {z \sim p (z)} [ l o g (1 - D (G (z))) ] - E _ {x \sim \mathcal {X} _ {t}} [ l o g (D (x)) ],
$$
where  $\mathcal{X}_t$  is the training dataset. In our experiments, we set  $\lambda_{ss} = 10$ .
We begin by introducing the experimental settings of our method, which encompass implementation, datasets, and metrics (Section 4.1). Next, we apply our method to various 10-shot datasets, showcasing its performance in Section 4.2. Section 4.3 explores the capabilities of domain association among multiple domains. Furthermore, we conduct ablation studies (Section 4.4 and Section A.1) to assess the impact of both the similarity-based structure loss and the generator structure. Additionally, we provide the user study in Section A.4 and demonstrate the applicability of our DoRM to one-shot GDA in Section A.3.
4.1 Experiments Settings
Implementation. We adopt the pre-trained StyleGAN2 [19] on FFHQ [18] as our source model, and our training parameters and settings follow StyleGAN2-ADA [17]. Due to the small amount of training data, we set the batch size to 4, and we terminate the training process after the discriminator has processed 100K real samples. Our implementation is based on the official implementation of StyleGAN2-ADA.
Datasets. Following previous literature [32; 59], we use FFHQ [18] with resolution  $256 \times 256$  as the source domain. In 10-shot GDA, we evaluate our method on multiple target datasets, including Sketches [41], FFHQ-Babies [18], FFHQ-Sunglasses [18], Face-Caricatures, Face paintings by Amedeo Modigliani, Face paintings by Raphael, and Face paintings by Otto Dix [50]. All the training datasets are shown in Section A.2.
Metrics. Following [59; 32], we use Fréchet Inception Distance (FID) [13] as the metric to evaluate the synthesis quality and diversity simultaneously. Additionally, we adopt intra-cluster LPIPS (intraLPIPS) [59; 32] based on LPIPS [54] to measure the synthesis diversity. Specifically, we synthesize 1000 images and then assign each of the synthesized images to the  $k$  training images with the lowest
LPIPS distance, forming  $k$  clusters. We calculate the average LPIPS distance within each cluster and then average over all the clusters. Furthermore, we report the identity (ID) similarity [56] predicted by the Arcface [9] to measure the preservation of identity information, which is a metric to measure the cross-domain consistency.
4.2 10-shot Generative Domain Adaptation
Qualitative comparison. In 10-shot GDA, following the previous studies [32], we sample 10 training images from the target domain to transfer the pre-trained generator. Figure 3 shows the results of 10-shot GDA with different methods. We observe severe generator overfitting in the FreezeD [29]. GenDA [49] shows high quality and diverse synthesis but fails to maintain cross-domain consistency. CDC [32] and RSSA [47] improve the cross-domain consistency, but the synthesis quality is unsatisfactory. AdAM [58] also increases the synthesis diversity but fails in synthesis quality. By contrast, our method preserves all the domain-sharing attributes from the source domain by freezing the original generator and acquires the domain shift through new concurrent network components. Our method shows appealing synthesis quality and diversity while maintaining better cross-domain consistency than previous methods. More qualitative results can be found in Section A.2, Section A.5, and Section A.6.