Towards Diverse and Faithful One-shot Adaption of Generative Adversarial Networks
Authors: Yabo Zhang
mingshuai Yao
Yuxiang Wei
Zhilong Ji
Jinfeng Bai
Wangmeng Zuo
Venue: NeurIPS 2022 Accept (2022)
Keywords: StyleGAN
Domain Adaption
One-shot
CLIP

Abstract: One-shot generative domain adaption aims to transfer a pre-trained generator on one domain to a new domain using one reference image only. However, it remains very challenging for the adapted generator (i) to generate diverse images inherited from the pre-trained generator while (ii) faithfully acquiring the domain-specific attributes and styles of the reference image. In this paper, we present a novel one-shot generative domain adaption method, i.e., DiFa, for diverse generation and faithful adaptation. For global-level adaptation, we leverage the difference between the CLIP embedding of the reference image and the mean embedding of source images to constrain the target generator. For local-level adaptation, we introduce an attentive style loss which aligns each intermediate token of an adapted image with its corresponding token of the reference image. To facilitate diverse generation, selective cross-domain consistency is introduced to select and retain domain-sharing attributes in the editing latent $\mathcal{W}+$ space to inherit the diversity of the pre-trained generator. Extensive experiments show that our method outperforms the state-of-the-arts both quantitatively and qualitatively, especially for the cases of large domain gap. Moreover, our DiFa can easily be extended to zero-shot generative domain adaption with appealing results.

Towards Diverse and Faithful One-shot Adaption of Generative Adversarial Networks
Yabo Zhang
Mingshuai Yao'
Yuxiang Wei
Zhilong Ji
Jinfeng Bai
Wangmeng Zuo (
Harbin Institute of Technology
Tomorrow Advancing Life
*Peng Cheng Laboratory
Abstract
One-shot generative domain adaption aims to transfer pre-trained generator on one domain to a new domain using one reference image only. However, it remains very challenging for the adapted generator (i) to generate diverse images inherited from the pre-trained generator while (ii) faithfully acquiring the domain-specific attributes and styles of the reference image. In this paper, present novel one-shot generative domain adaption method, i.e., DiFa, for diverse generation and faithful adaptation. For global-level adaptation, we leverage the difference between the CLIP embedding reference image and the mean embedding of source images constrain the target generator. For local-level adaptation, introduce an attentive style loss which aligns each intermediate token adapted image with its corresponding token of the reference image. To facilitate diverse generation, selective cross-domain consistency is introduced to select and retain the domain-sharing attributes in the editing latent W+ space inherit the diversity of pre-trained generator. Extensive experiments show that our method outperforms the state-of-the-ang both quantitatively and qualitatively, especially for the cases of large domain gaps. Moreover, our DiFa can easily be extended to zero-shot generative domain adaption with appealing results. Code available https: // //github. com/ YBYBZhang/DDFF
Introduction
Generative adversarial networks (GANs) [8] have achieved remarkable progress in generating photo- realistic and highly-diverse images [11, 12, 18]. However, GANs usually require a large number of samples for stable training, and suffer from severe mode collapse when trained with insufficient data (e.g., one image). Recently, several works [2, 10,27, 27, 29, 34,33,7,3 have been proposed to train GAN from scratch with only one or few images, but are limited in generating high quality and diverse images. In this paper, we resort to one-shot or few-shot generative domain adaption (GDA), ie., transferring pre-trained generator on one domain to a new domain using one or few reference images (as shown in Fig. 1). Thus, GDA can provide new perspective to address the above issues by inheriting the generation ability and diversity of the pre-trained generator.
Many methods [16, 19, 21, 3,,31,33, 23,,3,3330 35,38] have been proposed for one-shot GDA. Nonetheless, domain-speeific attributes and styles usually can be described by language, and thus can be well depicted by
Laaissiiee-Lnnuuge-mmaaaa Paaange-eerriiinng (CLIP) [22]. Hence, CLIP-based one-shot GDA methods [7, 11, 32, 41] have been proposed to adapt pre-trained generator, e.g., StyleGAN2 [12], to the target domain. In particular, the domain-gen direction between source and target domains is first calculated the CLIP embedding space. Then, the pre-trale pre-trained generator is transferred by aligning the CLIP direction between the source and adapted images with the domain-gap direction.
'The first author performed this work as an intern at Tomorrow Advancing Life.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

GA
Dom.
Dom. A
Adaption
Reference
Reference
G3
Dom. B
Dom. A
Figure 1: Diverse and faithful one-shot generative domain adaption. Left: One-shot generative domain adaption aims to transfer a pre-trained GAN from domain A (e.g.. FFHQ) to domain B (e.g., sculptures) by providing one reference image only. Right: Synthesised images by our DiFa. Notably, our DiFa is effective inheriting the diverse generation ability of GAN from domain A, while faithfully acquiring the representative characteristics of the reference image in domain B.
However, it remains very challenging for the adapted generator (i) to generate images as diverse as the pre-trained generator while (ii) faithfully acquiring the domain-specific attributes and styles of the reference image. Firstly, although the domain-gap direction extracts the pronounced characteristics of the reference image, the detailed local styles and attributes are usually ignored in CLIP embedding. Without considering these local styles and attributes, the adapted generator cannot faithfully acquire the domain-specifii characteristics of the reference image. Secondly, the domain-gap direction the difference between the CLIP embedding of the reference image and source domain, which contains both domain-specific attribute shifts (e.g., thick eyebrows in second row of Fig. 1) and comain-sharing attribute shifts (e.g., gender). Directly aligning the training sample-shit direction with the domain-gap direction introduces the unnecessary domain-sharing attribute changes to the adapted images, thereby being harmful to inheriting the diversity from the pre-trained generator. Although [15,41] proposed to use the style mixing and editing direction preservation to address these issues, only limited improvements are achieved.
In this work, we present novel one-shot GDA method, i.e., DiFa, for diverse generation and faithful adaption. In terms of faithful adaption, we consider both attributes and styles. For global-level adaptation, we define the domain-gan gap direction as the difference between the CLIP embedding of reference image and the mean embedding of source images. As for local-level adaptation, we introduce an attentive style (AS) loss on the intermediate layer of the CLIP image encoder. For each intermediate token of an adapted image, it first finds the nearest token of the reference image, and then minimizes their difference to make GDA adapt to the target style. In terms of diverse generation, selective cross-domain consistency (SCC) introduced to select and retain domain-sharine attributes in the editing latent W+ space to inherit the diversity of pre-trained generator, In particular, we use styleGAN inversion models [24, [0] to invert the images from source and target domains into the W+ space. Then, we compute the direction Aw between the two domains, where smaller values in Aww indicate that the corresponding latent variables in W+ space are domain- -harinn attributes. Selective cross-domain consistency encourages adapted image and its corresponding source image to be similar in domain- -hhaiig attributes, and can be different in other attributes. SCC allows the adapted generator to inherit from the pre- pre-trained generator selectively. Thus DiFa can guarantee the diversity of adapted images without the sacrifice of decreasing domain adaption ability.
Quantitative and qualitative experiments are conducted on wide range of source and target domains. Evaluation results highlight the superiority of our DiFa compared against the state-of-the-an methods, especially for the cases of large domain gaps (e.g., Cat - Tiger). To illustrate the editing capabilities of the adapted latent space, we employ InterFaceGAN [28] to edit the real images in target domain.
Overall, our contributions are summarized as follows:
We introduce a novel method namely DiFa, along with selective cross-domain consistency and attentive style losses, for diverse generation and faithful adaption.

---

Extensive experiments show the effectiveness of our DiFa in acquiring the representative domain characteristics from the reference image. and inheriting the ability of pre-trained generator to produce diverse images.
Our DiFa can be easily extended to zero-shot generative domain adaption with appealing results.
Related Work
Few-shot Domain Adaption of GANs. Few-shot generative domain adaption aims to transfer a generator pre-trained on a source domain to a new target domain with very limited reference images. Earlier studies [16, 25,21,25,11,, 33,,3,,,,38 utilized the adversarial loss [8] to capture domain-specific information from given reference images. To reduce mode collapse, these methods usually adopted fewer learnable parameters (19,25,31,35) or introduce regularization terms [16,21,33, 38), but still produce images with insufficient diversity. With the success of CLIP [22], recent works [7, 15,32,, 41] leveraged the difference between the CLIP embeddings of the source and target domains to guide the attribute-level adaption, beating the methods with adversarial loss [8]. To better capture domain- specific styles, several methods [15,41] adopted the style mixing trick during inference time, however, it may bring undesired semantic artifacts when there is significant shape discrepancy. [15,32,41] attempted to generate diverse images by preserving the editing distance of input pairs, before and after adaption. Nonetheless, they indistinguishably retain both domain-sharei -haring and attributes, which is conflicted with faithful adaption.
GAN Inversion. GAN inversion aims to invert an image into its corresponding latent codes, which can be grouped into optimization-baser and encoder-based methods. Optimization-base inversion [5]40] directly updates the latent code by minimizing the reconstruction error. Albeit high-quality and accurate reconstruction can be obtained, it usually costs few minutes for an image. In contrast, encoder-based algorithms [1,24,30] directly embed given image into latent codes, so that the inference can be completed in real-time, and the gradients of input images could also be passed backward. Moreover, encoder-based algorithms also achieve considerable performance when handling out-of-domain images, and thus it is feasible to project adapted images into W codes during training.
3 Proposed Method
In this work, we focus on one-shot generative domain adaption task, which aims to transfer a generator G pre-trained on domain A to new domain B using one reference image Turr only. Specifically, we present a novel method termed DiFa to generate diverse images inherited from the pre-trained generator while faithfully acquiring the domain-specific attributes and styles of the reference image. The overview of our DiFa is illustrated in Fig. 2. In this section, we first introduce the global-level adaption loss with estimated domain-gap direction. The attentive style loss and selective cross-domain consistency loss are then proposed for local-level adaptation and diverse generation, respectively. Finally, we introduce the overall learning objective for training.
3.1
Global-level Adaption
Recent studies [7, (7,15,41) have demonstrated the superiority of CLIP in transferring a generator in one domain to a new domain under the one-shot setting. In comparison to the methods based on adversarial loss [16, 19,21, 25,31,,,,,,,,,,,, 33, 35,3), CLIP-bbsed methods are effective in describing domain characteristics and resulting in photo-realistic images. Given a generator G pre-trained on domain A and target reference image Itar from domain B, CL.IP-based methods first calculate the domain-gan direction between domain B and A:
Avdom
Utar
Usrcs
(1)
where Vaar
Ey (Tarr) denotes the embedding of target domain B and E, is the CLIP image encoder. Usre represents the CLIP embedding of source domain A. To transfer G' to domain B, they copy a new generator GB from G and finetune it by aligning the sample-shift direction AUsomp with the domain-gap direction Av
! Uaamp
Um UA,
A U Deamp A v dom
Lytobal
(3)
U Usampl dom
3

---

Source Element wise Subtract
Freeze Weights
tar
Target
Flement-wise Product
local
CLIP Image Enc.
VD0
Cost matric
Vtar
CLIP Image Enc.
G
Ig
Av Axamp
global
AVdam
La
VA
(z)x(2)
mask(Aw, (X)
1
WB
GAN Inver.
GA
sec
Adapttive Mask
Samples
WA
from
Domain
Figure 2: Overview of our DiFa. The adapted generator GB is initialized by pre-trained generator GA. With the aid of CLIP image encoder, the global-level adaption loss Lgtobel and attentive style loss Liocal encourage G to faithfully acquire both global and local representative domain-specift characteristics from the reference har- To facilitate diverse generation inherited from GA, the selective cross-domaie consistency loss Lece selects and retains -ommiin-hharngg attributes.
where B3- (G())(z) and VA = (z)) denote the CLIP embeddings of domain and domain A samples. (0, denotes the input noise. After finetuning with the global-level adaption loss Lytobat, the adapted generator GB can generate high-quality images of domain B.
Note that the embedding of source domain Uare can be calculated in different ways. Two-stage methods [15,41] find the image corresponding to Iarr in domain A and treat its CLIP-space embedding as V Vsrc. Nonetheless, the corresponding image in domain A inevitably contains Nomain-speciffi attributes of tar- leading Au ignoring these domain-specifie attributes. One-stage methods [7] utilize the mean embedding of source images as Usre- Intuitively, the mean embedding usually represents the common attributes of source domain, and does not affect domain-speciff attribute shifts in AWdom Thus, in our experiments, we use the mean embedding of source images as the source domain embedding, i.e., Urrc = UA
3.2 Local-level Adaption
Albeit the domain-gas gap direction AVdom captures the global-level representative domain charac- teristics of reference image Jarr, the local attributes and visual styles are usually ignored in CLIP embedding. Therefore, training Gg with Lylobal only cannot faithfully capture the local-level domain- specific characteristics of I For example, images generated by StyleGAN-NADA [7] fail to acquire the mane and stripes of tigers during the Cat - Tiger adaption (the first row in Fig. 6(c)). [15,41] tried to inherit the detailed visual styles from Jear through the style mixing. However, when there is significant shape discrepancy (e.g., pose cross-category) between lar and the original adapted image J.. the mismatch of content and style in 1 will lead to visible artifacts (the last row in Fig. 5(b) and the first row in Fig. 6(b)).
To mitigate the above issue, we further present an attentive style loss Clocal to help GB faithfully acquire the local-level representative attributes and styles of Iaa-- Inspired by content-style alignment in style transfer [14], Llocal is designed to encourage each part of IB to attentively align with its first extract the intermediate tokens of In and har from the k-th layer of CLIP image encoder (shown in Fig. 4), and then align each of adapted tokens FR with its closest target token from Frar, where FB [F...... FR) and Furr [F1 fiars- ... Fm are the extracted tokens. The final attentive style loss defined as,
corresponding styles from Itar Specifically,
Liocal max L Σ
Emin min Cis).
min Cz.j+ (.ji
(4)
where C' is the cost matrix to measure the bisen-wise distances from FB to Ftar, and each element of C' is computed as:
F F. tar
Cij =
(5)
F
F tar

---

C local
Domain
Target Image
Jaaee
4 JaMes
Zi Jalee
Domatie
Adapted Image
Image Encoder (VIT)
Ey
Figure 3:
The process of computing Aw be- tween domain A and domain B. Given two clusters of W+ codes from domain A and do- main B, Aw is defined the difference of their cluster centers.
Illustration of the Attentive Style loss. We extract the intermediate adapted and target tokens from the Layer_4 of CLIP image en- coder, following by aligning each adapted token with its corresponding target token attentively.
Figure 4:
3.3 Selectively Diverse Generation
The ability generate diverse target domain images is also critical for one-shot generative domain adaption. Recall that the domain-gap direction is the difference between the embedding of har and source domain, which contains both domain-specific and domain-sharing attribute shifts. Training GB with Llocat also introduces the unnecessary domain nomain-sharing attribute changes to the adapted images, which hinders GB from inheriting the diversity of the pre-trained generator Ga. To facilitate diverse generation, we propose a selective cross-domain consistency loss to select and retain the domain-sharing attributes in W+ space. Intuitively, if attribute is similar between domains A and B during adaption, it is more likely to be a Nomain-sharing attribute. According to this assumption, we can dynamically analyze and preserve the somainn-hhring attributes. Specifically, first invert Ga(z) and Gz(z) into W+ latent codes
and wg with an pre-trained inversion model (e.g., pSp [24] or e4e [30]) for each iteration. Then, as shown in Fig. 3, we compute the difference Aur between the centers of queue of W+ latent codes X'A and queue of W++ latent codes XB. where XA and X'B are dynamically updated with w, A and wy during training. According to Aw, we encourage A and w to be consistent in channels with less difference,
Lsec = ||mask a.) (WB wa)llı.
(6)
where a represents the proportion of preserved attributes and mask(Au. a) determines which channels to be retained. Let Aww,,x be the a-Ntth largest element of Aurl and each dimension of mask( w. a) is calculated as:
mask Aw, a); = ={0
| < | WWaaN Aws 2 Aw AWS.N
Note that fine layers of StyleGAN [11, 12] usually control color information, and constraining them may have detrimental effect obtaining styles of 4ar. Hence, only latent codes corresponding to coarse spatial resolutions (42-82 and middle resolutions (162-322) in Lsee-
3.4 Overall Training Loss
Our overall training loss consists of three terms, i.e., the global-level adaption loss Lytobal the attentive style loss Llocal for acquiring detailed style information and the selective cross cross-domain consistency loss Lace for inheriting the diversity:
Loverall = Lglobal + Alocal + Asceh
(8)
-ite--aa)) where Naer and Niter In our experiments, we use Atooal = 2 and Asce = max(O, Neter- denote the total number of training iterations and the nster-th iteration of training, respectively. That is, Asee increases linearly as the training proceeds.
4 Experiments
In this section, we first introduce the experimental settings of our DiFa, including implementation details, datasets, and metrics (Sec. 4.1). Both qualitative and quantitative experiments conducted
5

---

Barference
(a) Ours
(b) Mind The Gup
(c) StyleGAN-NADA
(d) Few-Shot Adaption
Figure 5: Qualitative comparisons using the generator pre-trained on FFHQ [11] between
DiFa, Mind The Gap [41], StyleGAN-NADA [7] and Few-Shot Adaption [21]. The first row and first column show source images in domain A and reference images in domain B. Our DiFa not only inherits the ability from the pre-trained generator to produce highly diverse and photo-realistic images, but also faithfully acquires the representative characteristics from the reference images, significantly outperforming the competing methods. Results best seen at 500 % zoom.
Dom.
Ref.
(a) Ours
(b) Mind The Gap
(c) StyleGAN-NADA
Figure 6: Qualitative comparisons using the generator pre-trained on AFHQ-Cat [4]. The first row shows source images in domain while the first column presents reference images in domain B. Our DiFa captures both the representative attributes and styles from various categories of reference images, and exhibits better performance comparison to the competing methods. In contrast. StyleGAN-NADA [7] misses some domain-specine styles while Mind The Gap [41] fails to obtain the essential attributes of animals in domain Results best seen at 500 % zoom.
on a wide range of domains to demonstrate the superiority of our DiFa in generating diverse images and faithful adaption (Sec. 4.2 and Sec. 4.3). Besides, ablation studies are considered to evaluate the effects of our proposed two losses (Sec. 4.4). Finally, we also investigate the editing ability of the adapted generator and extend our DiFa zero-shot generative domain adaption (Sec. 4.5).
4.1 Experimental Settings
Implementation Details. In our experiments, we use StyleGAN2 pre-trained on FFHQ [11] and StyleGAN-ADA [10] pre-trained on AFHQ-Cat [4], and employ e4e [30] and pSp [24] as their inversion models, respectively. Following StYleGAN-NADA [7]. we utilize both ViT-B/16 and ViT-B/32 [6] models for CLIP-base losses. For training, we use ADAM optimizer [13] with learning rate 0.02 and set the batch size We finetune the generator for 300~400 iterations, which takes about 3-4 minutes on an RTX 2080Ti GPU.

---

Table KID (L) comparisons between different one-shot domain adaption methods. Each result is averaged over training shots and in the form of {mean + standard error). FSA, NADA and MTG denote Few-Shot Adaption [21], StyleGAN-NADA [7] and Mind The Gap [41], respectively.
FFHQ
Cat
Models
Amedeo.
Fernand.
Raphael
Tiger
Fox
Wolf
FSA [21]
180.10+ 1.12
187.26+ 13.10
165.25= 66.31
NADA [7]
131,03- 28.14
169.83+ 31.52
149.19- 55.91
13.83+ 2.75
73.17+ 39.30
47.96+ 20.37
MTG [41]
146.84 46.24
192.191 42.73
125.58 18.63
48.27 13.87
69.19.1 26.23
51.111 11.32
Ours
121.21 + 24.62
159.93 31.39
112.72 17.61 13.13 2.09
54.20 + 31.42
33.52 9.21
Table 2: FID (L) comparisons between different one-shot domain adaption methods. Each result is averaged over 5 training shots and in the form of [mean standard error).
FFIQQ
Cal
Models
Amedeo.
Fernand.
Raphael
Tiger
Fox
Wolf
FSA [21]
171.56 + 33.68
236.61+ 25.03
177.47= 32.21
NADA [7]
188.44- 19.15
257.27+ 19.39
186.20- 28.60
16.74 + 1.53
82.59 + 25.31
54.28+ 13.34
MTG [41]
215.88 34.14
278.46 48.27
193.76 7.07
46.72+ 13.34
82.30 15.09
58.65 6.60
Ours
187.28 + 24.45
254.68 17.73
172.34 10.15 16.26 = 1.08 71.57 + 18.18
44.39 5.96
Datasets. For FFHQ adaption, the target images are collected from three datasets: (i) Artstation- Artestic-face-H (AAHQ) [17], (ii) MetFaces [10], and (iii) face paintings by Amedeo Modigliani, Fernand Leger and Raphael [36]. Each of them contains 10 images. For Cat adaption, we collect target images from the AFHQ- AFHQ-Wild Wild validation dataset and divide them into Tiger, Fox, and Wolf datasets, which include 103, 53, and 46 images, respectively. In particular, Amedeo Modigliani, Fernand Leger, Raphael, Tiger, Fox and Wolf are used in quantitative experiments.
Metries. Following StyleGAN-ADA [10], we use Fréchet Inception Distance (FID) [9] and Kernel Inception Distance (KID) [3] to evaluate our DiFa quantitatively. Both metrics measure the quality and diversity of the images, while KID is more suitable for the few-shot setting (only a few images in validation sets). In all our experiments, both FID and KID calculated between 5,000 synthesized images and each validation sets.
4.2 Qualitative and Quantitative Evaluation
Qualitative Results. Fig. 5 shows the qualitative comparisons adapted from FFHQ [11]. As shown in the figure. Few-Shot Adaption [21] suffers from severe model collapse and generates similar images. Due to StyleGAN-NADA [7] is trained by aligning the sample-shift direction Ausamp with domain-gar -gap direction Ausom, which contains the domain-sharing attributes (e.g.. gender) shift, also cannot inherit the sufficient diversity from the pre-trained generator. For example, the gender of adapted images is changed to female 4--6-th rows in Fig. 5(c). Mind The Gap [41] retains the local styles of reference image via style mixing. However, it produces undesired semantic artifacts when there is a significant shape discrepancy between domains, e.g., redundant noses and eyes in 3rd and last row of Fig. 5(b). In contrast, with the proposed SCC and AS losses, our DiFa not only faithfully acquires the representative domain-specific attributes and styles from the reference image, but also produces images with high diversity inherited from the pre-trained generator. Additionally, we also illustrate the qualitative results adapted from AFHQ-Cat [4] in Fig. 6. Our DiFa also captures sufficient domain- speciffc characteristics from the reference image in comparison to the competing methods (e.g., the mane and stripes of the tiger in first row), further demonstrating the superiority of our method. More visualizations adapted from other domains are shown in Suppi.
Quantitative Results. We also quantitatively compare our DiFa with competing methods [7,21, 41] under six settings, i.e., FFHQ > (Amedeo Modigliani, Fernand Leger, Raphael) and Cat -> (Tiger, Fox, Wolf}. For each setting, we randomly sample an image from a target dataset to perform adaption, and report both Kernel Inception Distance (KID) [3] and Fréchet Inception Distance (FID) [9] metrics. To reduce random sampling error, we repeat it five times and use the mean value as final score. The results are listed in Table and Table 2. One can see that our DiFa clearly outperforms the competing methods, which are consistent with qualitative results in Fig. 5 and Fig, 6. We observe
7

---

Table 3: User preference study. The numbers represent the percentage of users who favor the images
synthesized by our DiFa over the other competitor.
Model
Image
Style Similarity
Attribute Consistency
Comparison
Quality
Ours vs. FSA [21]
87. 87.90%
36.00%
96.10%
Ours vs. NADA [7]
76.76%
77.33%
78.95%
Ours vs. MTG [41]
81.43%
73.62%
64.05%
Reference
Dom. A
Baseline
Baseline SCC
Baseline SCC AS (Ours)
Figure 7: Ablation studies on Selective Cross-modal Consistency (SCC) loss and Attentive Style (AS) loss. Compared to the baseline, SCC enhances keeping the consistency (e.g.. hairstyles) between source and adapted images, vastly boosting the diversity of generation. Moreover, AS encourages the adapted generator further acquire the representative styles (e.g., darker hue and purple hair) from the reference.
that FSA [21] obtains better FID scores in Amedeo and Fernand datasets, which is inconsistent with above qualitative results (see Fig. 5(d)). Note that FID cannot reflect the overfitting problem very well when target dataset extremely small and biased [10]. Specifically, these two small datasets have different data biases with FFHQ, e.g., gender bias. 8/10 images in the Amedeo Modigliani dataset and 9/10 images in the Fernand Leger dataset are female. Due to our DiFa acquiring the diversity from the original generator which is trained on FFHQ it generates male and female adapted images with similar probability. In contrast, for FSA, the adapted images are all similar to the reference image. When comparing on the above two datasets, FSA tends to generate images that have similar gender distribution to the validation dataset, thus achieving better FID results. For the Raphael dataset, which has 5/10 images that are female, our DiFa achieves better FID results.
4.3 User Study
We further perform user study to compare our DiFa with the competing methods. Specifically, we provide users a reference image, source image, and two adapted images from different methods, and ask them to choose the better adapted image for each of three measurements: (i) image quality, (ii) style similarity with the reference and (iii) attribute consistency with the source image. We randomly generate 1,050 samples for each comparison (3, 150 in total). There are 30 users. We assign 105 samples for each of them, and give them unlimited time to complete the evaluation. From Table 3, the users strongly favor our DiFa in all three aspects, especially from the perspective of image quality and attribute consistency. Note that FSA [21] suffers from severe mode collapse and simply copies from the reference, hence, it is favored on style similarity but performs worse on the other aspects.
4.4 Ablation Study
Ablation studies are conducted to evaluate the effects of two critical components of our DiFa, i.e the selective cross-domain consistency (SCC) loss and the attentive style (AS) loss. As shown in Fig. 7, the images from the baseline have very limited diversity and lack some representative characteristies of the reference image, e.g., darker hue. Benefited from SCC, the adapted generator begins to retrain the Nomain-sharing attributes (e.g., hair length and gender), thereby inheriting the diverse generation ability from the pre-trained generator. When further adding AS, we observe that the adapted generator faithfully captures the Nomain-speciff styles and local-level representative attributes from the reference image, e.g., darker hue and purple hair. More ablation studies about hyper-parametert are provided in the Suppl.
8

---

Dom,
Reference
Dom, B
Eyes
Smile
Gender
Pose
Figure 8: Editing a real image in domain B. The first three columns show real image in domain A, reference image, and an adapted real image in domain B, respectively. The other columns present the editing operations and their corresponding results in domain B. All editing directions are discovered by InterfaceGAN [28] in domain A.
"A painting in the style of Edvard Munch"
sketch black pencils"
Dom,
Ours
StyleGAN-NADA
Figure 9: Qualitative comparisons on zero-shot generative domain adaption between our DiFa and StyleGAN-NADA [7]. Given the text description in first column, our DiFa adapts source images in domain A (last row) to the described target domain, significantly surpassing StyleGAN-NADA [7] from the consistency perspective.
4.5 Extensions
Latent Space Editing. In Fig. 8, we illustrate the editing results performed on a real image adapted into a new domain. Coneretely, we employ InterfaceGAN [28] to discover some editing directions in domain A, and then leverage these directions to edit the adapted real image. As can be seen, the directions from domain A still manage to control real images in domain B, indicating that the adapted generator maintains similar ability in latent-based editing with the original generator.
Zero-shot Domain Adaption of GANs. With minor modifications (e.g., removing the AS loss), our DiFa can be easily extended to zero-shot GDA, ie., adapting to a target domain described by text only. Fig. 9 shows the comparison between our DiFa and StyleGAN-NADA [7]. One can see that adapted images from our DiFa are more consistent with their corresponding source images, thereby inheriting more diversity from the pre-trained generator. For example, when performing adaption from FFHQ to target domain described by "A sketch with black pencils", all eyes in StyleGAN-NADA results look to the left, which is inconsistent with the original eyes in source domain. More visualizations and the implementation details are given in the Suppl.
5 Discussion
In this paper, we presented DiFa address the diverse generation and faithful adaptation issues for one-shot generative domain adaption. In particular, DiFa leverages the difference between the CLIP embedding of the reference image and the embedding of source domain guide the global-level adaption. To faithfully acquire local-level domain-specific characteristics, we introduce the attentive style loss align each intermediate token of adapted images with its closest token of the reference image. For highly diverse generation, the selective cross- cross-domain consistency loss is proposed to select and retain the sohaiin attributes in W+ space. Both qualitative and quantitative experiments show the superiority of our DiFa against state-of-the-arts under a wide range of settings, especially for the cases of large domain gap. Furthermore, our DiFa can easily extended zero-shot generative domain adaption with compelling results.
When there are few domain-sharing attributes between source and target domains, e.g., Church Tiger, DiFa cannot produce highly diverse images. Fortunatcly, this issue may be
Limitations.

---

largely alleviated by adaptively inheriting the prior knowledge from large-scale generators [20,23,26], which are pre-trained on sufficient variety of source domains.
Broader Impact. Transferring pre-trained generator with very limited data plays crucial role in academia and industry. More specifically, our DiFa provides insights on tasks in computer vision, e.g.. data augmentation and few-shot adaption. Meanwhile, our DiFa also makes Al more accessible to the public. On the one hand, users could leverage our method to create the artworks with any desired styles, even without adequate computing and data resources. On the other hand, our work may bring potential concerns on the probability of producing fake images. For example, someone may use our DiFa to spoof other people's 's portraits, to synthesize deceptive interactions, or even to impersonate public figures to influence political processes, Albeit there are a few potential negative impacts, we believe that they could be well addressed with the development of DeepFake detection and proper protocols. In particular, we could verify the authenticity, integrality, and source of images by adding digital watermarks or signatures. Also, may employ DeepFake detection technique to analyze the images without digital signatures. Furthermore, our community should help the government to improve corresponding laws and regulations avoid the abuse of image generation.
Acknowledgments and Disclosure of Funding
This work was supported in part by the National Key R&D Program of China under Grant No.s waspsp 2020AAA0104500, 1021100112000, and the National Natural Science Foundation of China (NSFC) under Grant No. 62006064.
References
[1] Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? In JCCV, 2019.
[2] Raphael Bensadoun, Shir Gur, Tomer Galanti, and Lior Wolf. Meta internal learning. NeurlPS, 2021.
[3] Mikolaj Binkowski, Danica Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmnd gans.
arXiv:1801.01400 2018. 7
[4] Yunjey Choi. Youngjung Uh. Jaejun Yoo. and Jung-Wop Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In CVPR, 2020. 6,
[5] Antonia Creswell and Anil Anthony Bharath. Inverting the generator of generative adversarial network.
TNNL. 2018.
[6] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai. Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv:2010. /929, 2020.
[7] Rinon Gal, Or Patashmik, Haggai Maron, Gal Chechik, and Daniel Cohen-Or. Stylegan-nada: Clip-guided domain adaptation of image generators. arXiv:2 108.008.044 00946, 2021. 1, 3, 4, 6, 7, 8,
[8] Ian Goodfellow, Jean Pouget-Abadic, Mehdi Mirza, Bing Xu, David Warde-Farlcy, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NewrIPS, 2014. 1,3
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. NeurlPS, 2017. 7 [10] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. NeurlPS, 2020. 1, 7,8
[11] Tero Karras, Samuli Laine, and Timo Aila. A stylc-hased generator architecture for generative adversarial networks. In CVPR, 2019. 1, 5, 6,
[12] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In CVPR, 2020. 1,5
[13] Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In ICLR, 2015.
[14] Nicholas Kolkin, Jason Salavon, and Gregory Shakhnarovich. Style transfer by relaxed optimal transport and self-similarity. In CVPR, 2019. 4
[15] Gihyun Kwon and Jong Chul Ye. One-shot adaptation of gan in just one clip. arXiv:2003.09301, 2022. 1,
2.
3, 4
[16] Yijun Li, Richard Zhang, Jingwan Cynthia Lu, and Eli Shechtman. Few-shot image generation with elastic weight consolidation. In NIPS, 2020. 1.3
[17] Mingcong Liu, Qiang Li. Zekui Qin, Guoxin Zhang, Pengfei Wan, and Wen Zheng. Blendgan: Implicitly gan blending for arbitrary stylized face generation. NeurlPS, 2021. 7
[18] Ming Liu, Yuxiang Wei, Xiaohe Wu. Wangmeng Zuo, and Lei Zhang. A survey on leveraging pre- pre-trained generative adversarial networks for image editing and restoration. arXiv preprint arXiv:2207. 200.0099.00 2022.
10

---

[19] Sangwoo Mo, Minsu Cho, and Jinwoo Shin. Freeze the discriminator: a simple baseline for fine-tuning gans. arXiv:2002. 10964 2020. 1.3
[20] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, llya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. (XXv:2//2.1004// 2021. 10
[21] Utkarsh Ojha, Yijun Li, Jingwan Lu, Alexei A Efros, Yong Jac Lee, Eli Shechtman, and Richard Zhang. Few-shot image generation via cross-domain correspondence. In CVPR, 2021. 3, 6, 7,
[22] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In JCML, 2021. 1.3
[23] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-lo-image generation. ICML, 2021. 10
[24] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-
Or. Encoding in style: stylegan encoder for image-to-image translation. In CVPR, 2021. 2, 3, 5,
[25] Esther Robb, Wen-Sheng Chu, Abhishek Kumar, and Jia-Bin Huang. Few-shot adaptation of generative adversarial networks. vrXiv:2010./194, 2020. 1,
[26] Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. 2022, 10
[27] Tamar Rott Shaham, Tali Dekel, and Tomer Michacli. Singan: Learning generative model from a single natural image. In ICCV, 2019. 1
[28] Yujun Shen, Ceyuan Yang. Xiaoou Tang, and Bolci Zhou. Interfacegan: Interpreting the disentangled face representation learned by gans. TPAMI, 2020. 2,9
[29] Assaf Shocher, Shai Bagon, Phillip Isola, and Michal Irani. Ingan: Capturing and retargeting the" dna" of
natural image. In ICCV, 2019. 1
[30] Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or. Designing an encoder for stylegan image manipulation. TOG, 2021. 2, 3,
[31] Yaxing Wang, Chenshen Wu, Luis Herranz. Joost de Weijer, Abel Gonzalez-Garcia. and Bogdan Raducanu. Transferinng gans: generating images from limited data. In ECCV, 2018. 1,
[32] Yue Wang, Ran Yi. Ying Tai, Chengjie Wang. and Lizhuang Ma. Cilgan: Few-shot artistic portraits
generation with contrastive transfer learning. 3xXiv:2203.086/1 2022. 1,3
[33] Jiayu Xiao, Liang Li, Chaofei Wang, Zheng-Jun Zha, and Qingming Huang. Few shot generative model
adaption via relaxed spatial structural alignment. arXiv:2203.04 0412,, 2022. 1,
[34] Ceyuan Yang, Yujun Shen, Yinghao Xu, and Bolei Zhou. Data-efficient instance generation from instance discrimination. NeurlPS, 2021.
[35] Ceyuan Yang, Yujun Shen, Zhiyi Zhang, Yinghao Xu, Jiapeng Zhu, Zhirong Wu, and Bolci Zhou. One-shot generative domain adaptation. arXiv:2111.0987,. 2021. 1,3
[36] Jordan Yaniv, Yacl Newman, and Ariel Shamir. The face of art: landmark detection and geometric style in
portraits. TOG. 2019. 7
[37] Dan Zhang and Anna Khoreva. Pa-gan: Improving gan training by progressive augmentation. 2018. [38] Yunqing Zhao. Henghui Ding. Houjing Huang, and Ngai-Man Cheung. A closer look at few-shot image generation.
Ttt.:2205503003 2022. 1.3
[39] Zhengli Zhao, Zizhao Zhang. Ting Chen, Sameer Singh, and Han Zhang. Image augmentations for gan training. arXiv:2006. 2it.:20000002955... 2020.
[40] Jun Yan Zhu. Philipp Krähenbühl. Eli Shechtman, and Alexei Efros. Generative visual manipulation on the natural image manifold. In ECCV, 2016. 3
[41] Peihao Zhu, Rameen Abdal, John Femiani, and Peter Wonka. Mind the gap: Domain gap control for single shot domain adaptation for generative adversarial networks. arXiv: 10.08398, 2021. 1, 2, 3,4,, 4, 6, 7,
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] The main contributions can be summarized as: (1) We introduce a novel method namely DiFa, along with Selective Cross-domain Consistency Loss (SCC) and Attentive Style Loss (AS). for diverse generation and faithful adaption. (2) Extensive experiments highlight the effectiveness of our DiFa acquiring the representative characteristics from the reference image, and inheriting the capabillity to produce high-diversity images from the pre-traines generator. (3) Our DiFa can be easily extended to zero-shot generative domain adaption with appealing results.
(b) Did you describe the limitations of your work? [Yes] See Sec. 5
11

---

(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Sec. 5
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We carefully read the ethics review and comply with them.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments...
(a) Did you include the code, data, and instructions needed reproduce the main ex- perimental results (either the supplemental material or as a URL)? [Yes] See comm/BYY2hang/DD2F.
https://gi
(b) Did you specify all the training details (e.g., data splits, paraeers,,,,,eess how they were chosen)? [Yes] See Sec. 3.4, Sec. 4.1 and https://gitiiii www/coo//////// /atm//BBBBhag///// DiFa
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] See Table and Table 2
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Sec.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] We cite papers that we used in Sec. 4
(b) Did you mention the license of the assets? [Yes] We use AFHQ, MetFaces, and Artistic-Faces datasets during training, and their license are included our repository https://g thub. com/ BBBBZhang/BDDDD
(c) Did you include any assets either in the supplemental material or as URL? [Yes] We provide the URL of our source code in the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] The AFHQ, MetFaces, and Artistic-Faces datasets are widely used in the community. To our best knowledge, there is no inappropriate information in them.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [Yes] The datasets used in this paper are popular benchmarks in computer vision and do not include any harmful information.
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [Yes] See Sec. 4.3
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [Yes] Each user is paid 10 dollars per hour and we spend about 300 dollars in total.
12


Reviews:
[{"review_id": "mEhuCodoVGl", "reviewer": "Reviewer_xCjf", "summary": "To tackle the one/few-shot generative domain adaptation task, this paper proposes a novel method called DiFa, in which three different losses are introduced to achieve faithful adaptation. Among these three losses, the CLIP loss is computed between the embedding of the reference image and the embedding of the source domain for global-level adaptation. Attentive Style loss is calculated on the intermediate layer of the CLIP image encoder for local-level adaptation. Selective cross-domain consistency loss is introduced to select and retain domain-sharing attributes in the editing latent W+ space to inherit the diversity of the pre-trained generator. Extensive quantitative and qualitative comparisons show the superiority of the proposed method.", "questions": "See Weakness, can the authors provide domain adaptation results using aligned target images.", "limitations": "Adequate", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is well written and easy to follow. The framework itself is clear.\n- The visualization results demonstrate the proposed method can generate high-quality results on diverse target images, which seems promising to me. The ablation study clearly shows the effect of each loss in the pipeline.\n\nWeaknesses:\n- The results for Row 3, 5, and 7 in Figure 5 look weird. Please clarify if an alignment step is used on the target image before domain adaptation. This is somewhat unfair to competing methods if the alignment step is not done.\n- The faces generated by DiFa look almost the same except for the face shape. The proposed method appears to have failed to preserve facial identities.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "an03vx5K69e", "reviewer": "Reviewer_kovL", "summary": "The paper proposes a CLIP-based method for generative domain adaptation. The method starts with a (StyleGAN2) generator that is pretrained on a source domain and uses it to train a new version of the generator that can produce images from a target domain, from which only a single reference image is available. During training, a global loss is used to reduce the distance between the CLIP embeddings from the two domains. Furthermore, an attentive style loss is used to capture the local style of the target domain, by aligning intermediate tokens of the CLIP encoder for the generated images and the reference image. Finally, a selective cross-domain consistency loss, that operates in the latent space obtained from an inversion model, is used to preserve the diversity of attributes shared by both domains. The proposed method can also be adapted for use in a zero-shot setting and is compared to other GAN-based domain adaptation methods", "questions": "- Since the α hyperparameter is proportional to the number of shared attributes assumed between domains, how does its choice vary when the two domains are quite dissimilar.\n- The paper claims that FSA has better scores yet suffers from extreme mode collapse. Indeed from visual inspection based on the provided samples. However, FID in theory should correlate not only with perceptual quality but also with diversity. Do the authors have any hypothesis about why the FID of FSA is better in two experiments? \n- The paper mentions that for dissimilar domains (church -> tiger) the end result lacks diversity.  What do such results look like? Is it close to the reference image?  ", "limitations": "Regarding the limitations of the method, the authors mention that for dissimilar domains (church -> tiger) the end result lacks diversity but this would be better understood by the reader if some examples were included in the paper or supplementary material. \n\nThe authors have included a discussion to raise awareness about the potential negative social impact of their work.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths:\n- The method is well motivated and the paper is clearly written.\n- The additional ablation study in figure 10 of the supplementary material gives the reader a good insight into the effect of the α parameter.\n- Based on visual assessment and the quantitative evaluation of the method it seems to outperform other similar GAN-based domain adaptation methods.\n\n## Weaknesses:\n- The contributions of the paper are not immediately clear from the introduction. The authors should make clear that the novelty in the proposed approach is in the attentive style and the selective cross-domain consistency losses.\n- The ablation study is based entirely on a visual assessment of one example of style transfer. The authors should ideally also provide the FID for each version of the model in the ablation study.\n- Some points made in the paper require clarification (see Questions section of the review)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "VdXnhXHc48W", "reviewer": "Reviewer_4nF5", "summary": "This paper presents an idea for a one-shot generative domain adaptation method.  To this end, this paper proposed to minimize both global and local loss. For global loss, this paper exploits the difference between the CLIP embedding of a reference image and the mean embedding of source. Whereas, for local-level constraining, the paper employs attentive style loss.\nTo validate the idea, the experiments are conducted on multiple pre-trained GAN such as StyleGAN2 trained on FFHQ, StyleGAN-ADA trained AFHQ-Cat. Both the qualitative and quantitative experiments are performed with the baselines.\n\n\n", "questions": "how diverse were the members of the user study group? On average, how much did they take to complete the assessment? \n ", "limitations": "One limitation is that this model relies on CLIP embedding, which is trained on a very large corpus.  ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is generally well written. The paper motivates sufficiently the need of zero-shot generative model and argures the drawbacks of the existing methods.  Also, the related works are discussed adequately and the methods are presented by the equations and diagram. Both the equations and diagram are easy to follow. The experimental results are also convincing to justify their claims.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Fpu6CxoupQy", "reviewer": "Reviewer_Pk6r", "summary": "The paper studies how to transfer a pre-trained generator from source domain to target domain with only one image available. The main challenges of one-shot adaptation of GANs are (i) generating diverse images and (ii) acquiring domain-specific attributes and styles for target domain. This paper proposes DiFa for diverse generation and faithful adaptation. The key ideas are: (i) aligning the CLIP embedding of target and source images at the global-level and aligning the intermediate token of adapted and reference image with an attentive style loss at the local-level; (ii) introducing selective cross-domain consistency loss to facilitate diverse generation. Experiments results demonstrate the efficiency of the proposed techniques.", "questions": "- The authors claim that the local attributes and visual styles are usually ignored in global-level CLIP adaptation, then why the local level adaptation, which extract the intermediate tokens of reference and adapted image from CLIP image encoder, can faithfully capture the local-level domain-specific characteristics? Is it possible that the domain-specific characteristics obtained from local-level adaptation are not precise?\n- Since both of style-mixing trick and the proposed attentive style loss can capture domain-specific styles. What is the main difference between them? Style-mixing methods may lead to visible artifacts when there is a significant shape discrepancy, is there any quantitative evaluation of such shape discrepancy? As the eyeglasses of the third image in the third-to-last row of Figure 5 (a) disappeared, does the eyeglasses attribute belongs to shape discrepancy? \n- The authors claim the two-stage methods find the image corresponding to target image in source domain and treat its CLIP-space embedding as source domain embedding, which may cause the domain gap direction to ignore domain-specific attributes. Is there any experimental or theoretical evidence? In my opinion, such source domain embedding may contain different attributes with adapted images, which may facilitate the diversity.\n- How does the proposed method perform when compared to adversarial loss based methods [1] [2] [3]?\n- Line 155, “as shown in Fig.4”, should Fig.4 here be Fig.2?\n\n[1] \"Few-shot adaptation of generative adversarial networks.\" arXiv preprint arXiv:2010.11943 (2020).\n[2] \"One-shot generative domain adaptation.\" arXiv preprint arXiv:2111.09876 (2021).\n[3] \"A Closer Look at Few-shot Image Generation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.", "limitations": "yes", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- The technical ideas of this paper are simple and easy-to-follow. It is doable to reproduce the experiments results;\n- Strong empirical performance shows the significant advantage of the proposed method on one-shot adaptation of GANs, as well as compelling results when extended to latent space editing and zero-shot domain adaptation of GANs;\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept