Generalized One-shot Domain Adaptation of Generative Adversarial Networks
Authors: Zicheng Zhang
Yinglu Liu
Congying Han
Tiande Guo
Ting Yao
Tao Mei
Venue: NeurIPS 2022 Accept (2022)
Keywords: Generative Adversarial Network
Computer vision
domain adaptation

Abstract: The adaptation of a Generative Adversarial Network (GAN) aims to transfer a pre-trained GAN to a target domain with limited training data. In this paper, we focus on the one-shot case, which is more challenging and rarely explored in previous works. We consider that the adaptation from a source domain to a target domain can be decoupled into two parts: the transfer of global style like texture and color, and the emergence of new entities that do not belong to the source domain. While previous works mainly focus on style transfer, we propose a novel and concise framework to address the \textit{generalized one-shot adaptation} task for both style and entity transfer, in which a reference image and its binary entity mask are provided. Our core idea is to constrain the gap between the internal distributions of the reference and syntheses by sliced Wasserstein distance. To better achieve it, style fixation is used at first to roughly obtain the exemplary style, and an auxiliary network is introduced to the generator to disentangle entity and style transfer. Besides, to realize cross-domain correspondence, we propose the variational Laplacian regularization to constrain the smoothness of the adapted generator. Both quantitative and qualitative experiments demonstrate the effectiveness of our method in various scenarios. Code is available at \url{https://github.com/zhangzc21/Generalized-One-shot-GAN-adaptation}.

Generalized One-shot Domain Adaptation of Generative Adversarial Networks
Zicheng Zhang"
Yinglu Liu²
Congying Han't
Tiande Guo'
Ting Yao2
Tao Mei²
University Chinese Academy of Sciences
2JD Al Research
zhangzicheng @ mailisggiimmmiss.....mm...mm www.m.aaas.a...mmmm ucas.ac.cn
liuyinglul @jd.com
hancy @ ucas.ac.cn
tdguo@@ ucas.ac.cn
tingyao. gmail.com
tmei @ live.com
Source
Figure 1: Illustrations of the generalized one-shot GAN adaptation task. Target images with the corresponding binary entity masks are provided adapt the pre-trained GAN the domain of similar style and entity as target images. The first three masks are all zeros and the others have value in the areas bounded by green lines. Our adapted model is of strong cross-domain correspondence to realize both style and entity transfer.
Abstract
The adaptation of a Generative Adversarial Network (GAN) aims to transfer a pre-trained GAN to target domain with limited training data. In this paper, we focus on the one-shot case, which more challenging and rarely explored in previous works. We consider that the adaptation from source domain to a target domain can be decoupled into two parts: the transfer of global style like texture and color, and the emergence of new entities that do not belong the source domain. While previous works mainly focus on style transfer, we propose novel and concise framework to address the generalized one-shor adaptation task for both style and entity transfer, in which reference image and its binary entity mask are provided. Our core idea to constrain the gap between the ternal distributions of the reference and syntheses by sliced Wasserstein distance. To better achieve it, style fixation is used at first to roughly obtain the exem- plary style, and an auxiliary network is introduced to the generator to disentangle entity and style transfer. Besides, to realize cross-domain correspondence, we propose the variational Laplacian regularization to constrain the smoothness of the adapted generator. Both quantitative and qualitative experiments demonstrate the effectiveness of our method in various scenarios. Code is available at https; One shot GAN AAMM-Adapttiion
//github. com/ htta///haagnc2///eeeaaaaae.
Introduction
Benefiting from numerous excellent pre-trained GANs (e.g., StyleGAN [16 and BigGAN 6)) GAN adaptation has become an important research topic, which leverages the knowledge of GANs
"Work done during internship JD Al Research. CCorreponningg author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

pre-trained on scale datasets to mitigate the lack of data and speed up the training on new domain. The adaptation can be divided into few-shot, one-shot and zero-shot [T] cases. The few-shot
(usually > 10) case has been extensively studied in the past years 42
26 22 The latest works (29 43
46 seek the cross-domain correspondence that keeps the shapes or contents of syntheses before and after adaptation invariant. Recently, the more challenging one-shot case has been explored by several works [47 55 20), which mainly focus on adapting GANs to the target domain of similar style as the given image. By utilizing GAN inversion [45]. the adapted models can be applied to
some creative tasks like image style transfer and manipulation [55].
However, we perceive there exist limitations in previous task settings 47 55 20], due to the fact that an exemplar contains rich information more than artistic style including color and texture. As shown in Fig. [ the source domain refers to natural faces, and the exemplars of target domains are artistic portraits with some entities (e.g,, hat and accessories). Previous works only concern the artistic style transfer while the entities are neglected. This may lead to two problems: 1) In some cases, the entities considered as an important part of domain feature, should be transferred with the style simultaneously. 2) The entities of large size, like the mask in Fig. [ easily bias the style extraction of face region and cause undesired artifacts in adapted syntheses. Therefore, we perfect the task settings with the aid of an extra binary mask, which labels the entities we are interested in and helps to define the target domain more exactly and flexibly. We name the new task generalized one-shot GAN adaptation. The scenarios in previous works can be taken as the special case when the binary mask is full of zeros. We believe the exploration of the novel task is quite meaningful, which not only serves to artistic creation for more general and complex scenarios, but also facilitates further discovery and utilization of the knowledge in pre-trained models.
To tackle the new adaptation problem, we propose a concise and effective adaptation framework for StyleGAN (17), which is a frequent base model in previous works [47 555 20). Firstly, we modify the architecture of the original generator to decouple the adaptation of style and entity, where the new generator comprises an additional auxiliary network to facilitate the generation of entities, and the original generator is dedicated to generating stylized images. Secondly, unlike previous works using CLIP similarity (55) or GAN loss [47] to learn the domain knowledge, our framework directly minimizes the divergence of the internal distributions of the exemplar and syntheses by sliced Wasserstein distance [5] for both style and entity transfer. Combined with the style fixation that attaches all syntheses with exemplary style by the transformation in latent space, our framework is efficient and potent enough to obtain compelling results. Thirdly, inspired by the cross cross-domain consistency (29 and classic manifold learning [12 2). we propose the variant Laplacian regularization that smooths the network changes before and after adaptation to preserve the geometric structure of the source generative manifold, and prevent the content distortion during training. We conduct extensive experiments on various references with and without entities. The results show that our framework can fully exploit the cross-domain correspondence and achieve high transfer quality. Moreover, the adapted model can handily perform various image manipulation tasks.
2 Related works
GAN adaptation leverages the knowledge stored in pre-trained GANs to mitigate the lack of data and speed up the training on new domain. [42 is the first to study and evaluate different adapting strategies. It shows that transferring both pre-trained generator and discriminator can improve the convergence time and the quality of the generated images. Since then, more and more adapting strategies [28, 41 32 533 are proposed for various GAN architectures [14 6 25 As StyleGAN [16] and its variants [7 [5] continue to make breakthrough progress in the generation of various classes of data, designing the adapted strategies of StyleGAN has gained lot and sustained attention. Specifically, (30, [9] have explored the zero-shot GAN adaptation, which transfers StyleGAN into
the target domain defined by the text prompts with the help of CLIP B1) 22 29 21
43
46 18
study to transfer the StyleGAN to the domain defined by the given few-shot images.
Among the above works, our work has a close relation to FSGA 29,, which proposes the cross- domain consistency (CDC) loss to maintain the diversity of the source generator, and makes the transferred and source samples have corresponding relation. Although CDC loss is ineffective for FSGA when the training data is only one-shot, its mechanism inspires us to consider a more effective regularization to maintain the relation, which will be discussed in Sec. 4.4

---

have explored the one-shot adaptation task. [55 20 align the adapted samples and reference image in the CLIP feature space. [9] learns a style mapper by constructing a substantial paired dataset. 44 introduces the extra latent mapper and classifier to the original GAN. In contrast, we focus on both style and entity adaptation which has never been studied before. And we prove that under the decoupling of style and entity, aligning the internal distributions with the proper regularization is effective for GAN adaptation.
Recently, [55 20 9 47
3 Preliminaries
StyleGAN A canonical StyleGAN 17 can be summarized in two parts: Firstly, given a noise distribution P(z), mapping network transforms the noise from space Z = [z|P(2) & 0} C R Rxx112 into the replicated latent space W = ([F(=);...:: F<>)]|= E 2} C R where F is a fully- connected network. Secondly, a synthesis network G composed of convolutional blocks transfers W space into image space MM- = [G(w) w E W) S "xxxxx3,, Another important concept the extended latent space W [[F((]); = E Z}. For V w1 u2 E W, the latent code manipulation merges their information by the linear combination diag(()) )uy +diag (r2) W W+, 01 02 E R. Empirically, W space is inferior to W in generation fidelity and editability, but superior in GAN inversion that finds a code Wref to reconstruct a reference yres E Hx X3, In this paper we only focus tuning G. and fix F for keeping the distribution of source latent spaces.
Manifold learning Considering the sample matrix X ; =h' from source manifold M,, W = [wt,]Ky-1 is the weight matrix, where Wi,j is the weight of and z, usually defined as e --1aa-=>;11"" //0 Given task-fpecific function f and y = f(x), manifold learning requires the trans- formed samples Y [y: v on the new manifold Mr should preserve the geometric structure of Ma, which means the data nearby on M, are also nearby on M2, and vice versa. The requirement is mostly realized by minimizing the manifold regularization term [4] SM, |V M. d.(x),, V.M. is the gradient along manifold, and P(x) is the probability measure. If .M; is submanifold of R, the former is equalized with JM, yAM, ((r) by Stokes theorem. Á called the saplace-Beltaane op- erator, is a key geometric object the Riemannian manifold, thus the integral is also called Laplacian regularization. In practice, it estimated by the discrete form 2tr(Y"T LY) = Ex, Wa, || , L= D - W is known as the graph Laplacian operator, D is the degree matrix where the diagonal element d, = Ei= Way and other elements are zero. The convergence between L and A can be found in (3]. Intuitively, the regularization encourages the local smoothness of If data distribute densely in an area, the function should be smooth to avoid local oscillations. If æz and æ, are close, in the discrete form y will impose a strong penalty to the distance between y, and y
Task definition The generalized one-shot adaptation can be formally described as: Given a reference Fref, a mask mref (0. } is provided to label the location of entity contained in Pres- The target domain T is defined to contain images of similar style and entities as Yres- With the knowledge stored in a generative model G, pre-trained source domain S, generative model Ga is learned from Yref and m,ref to generate diverse images belonging to domain T. Note that we can set miref when there no interested entities existed in yref- Moreover, the adaptation should satisfy the cross-domain correspondence. i.e., V w W, G.(w) and G. (w) should have similar shape or content in visual. With the correspondence, adapted models can be applied for not only synthesis, but also the transfer and manipulation of source images.
4 Methodology
4.1
Overall
Our framework for tackling the generalized one-shot GAN adaptation task is illustrated in Fig. For the sake of description, in the following part we will take the StyleGAN pre-trained FFHQ an example to describe each component of the framework in detail. We will demonstrate our framework can be applied to various domains in the experiments.
Network architecture Different from previous works [55
20 that directly adapt the source generator Gs, our target generator G, (as Fig. shows) consists of a generator G, inherited from Gg, and an auxiliary network (aur) trained from scratch. We are inspired by the fact that G, can
3

---

Yver
Yref
Block
Legend
Eyer
Conv Black
Conv Conv
Cono Conv
Conv Block
Conv SanvBlock
Conv Black
Conv
Flow
Cone
Conv
Lone
Wrof
Flow of
Yrer
Apet
(a)
Reconstruction
Style injector
Corv Block
Corv Block
Corv Block
Corw Block
Corv Block
Block
Excyle
Style
Comw Block
Conu
(b) Random synthesis
(c) Loss
Gg(W)
(d) Amx
Black
Figure 2: The proposed framework for generalized one-shot GAN adaptation. The convolutional blocks are inherited from the original StyleGAN [17]. The black arrows represent the flow of G., and the red arrows represent the flow of G,. For the details please refer Sec. 4.1
approximate arbitrary real faces but cannot handle most entities, due to that G, captures the prevalent elements (clear natural faces) of FFHQ rather than the tails (faces with entities like hats ornaments) of the distribution 49 27). The design makes aua serve to entities, and G. just focus stylizing clear face to benefit from the prior knowledge stored in G.. As shown in Fig. dd), for each w, its feature map generated by the convolutional block of StyleGAN is fed into aux, where UNet (33) predicts the feature map fene entity and the mask m that exactly labels the position of the entity, then they are merged with f in by Eq. ① to get the feature map of face covered with entity.
aur(f (1- fi + m C fent
+ denotes the Hadamard product. We hope the shape of the entity is only influenced by the content of synthesis, hence we place auz after the fourth block of StyleGAN, where the 32 32 resolution contains the most content information and the least style information of synthesis (see Sec. Because the global style and local entity are usually independent, the disentangled design also prevents the style of faces from being contaminated by that of entities (see Sec. 4.3
Training process Our training process be divided into three main parts: I Style fixation and exemplar reconstruction. In Fig. 2(a), we get the latent code wref [w whee of yref by the GAN inversion technique [39], and train G, to reconstruct Yref with reconstruction loss Cree- In Fig. b),, each w will be transformed into the style-fixed code wn = [wo : to roughly obtain the exemplary style. 2 Internal distribution learning. We minimize the divergence of internal patch distributions between syntheses and exemplar for style and entity transfer. Instead of using GAN loss that is hard to optimize, we take sliced Wasserstein distance (SWD) [5] as style loss Lutyle and suppress
entity loss Cent to learn internal distribution efficiently. 3 Manifold Regularization. the content distortion during training, we propose the variational Laplacian regularization Lytapn to smooth the change from G, G, by preserving the geometric structure of the source manifold. As shown in Fig. c). the overall loss is
min Cree X2Lstyle + AgGent + XaLy Lapst G.
(2)
The details of training process will illustrated and discussed in the following sections. For convenience of description and self-consistency, some abbreviations have been taken: G.(wf),
G.(w"), Veee Ge(wref), Uree Gr(Wref), x G,(w"), Pree G.(Wref).
4.2 Style fixation and exemplar reconstruction
well known that StyleGAN is equipped with the disentangled latent spaces W and W+ where the latter vectors of code mainly determine the style of synthesis whereas the earlier vectors determine the coarse-structure or content of the synthesis. Taking advantage of this trait, we first roughly transfer the exemplary style to other syntheses by fixing the style code. The exemplar yref is

---

fed into the pre pre-trained GAN inversion encoder e4e [3 to get the latent code wre/ = y Weef where wief E R and wre/ (((8-1)x512 encode content and style information of exemplar respectively, For each w E W, it will be transformed to w before fed into networks,
w² d (x) + diag(1 - a.)rref, a; = 1i<=((),, i I...., 18.
The content part of w is reserved and style part is replaced by wief- and we denote the new space W Because style fixation should not change the content of the original synthesis, by using the pre-trained Areface model [10) to compare the identity similarity before and after style fixation, we find setting / 8 is acceptable which gets 65% average similarity and obtains the adequate exemplary style in visual. In MTG 55 style mixing assists as a post-process post-processing step to improve the style quality, but the experiments (Fig. 3 show that the style of syntheses is underfitting and not consistent. In our framework, is used as a pre-processing step to facilitate the following learning.
Because Frec is just the projection of yreff the source domain, there often exists obvious distinction between them in visual as shown in Fig. Hence we adopt the reconstruction loss Lrce to narrow the gap,
Lree dssimmy rear Dref) + lpips (Frect Dref) + >smssemm mref).
dssim denotes the negative structural similarity metric 44 and lpips is the perceptual loss [50], mfee the upsampled reconstructed mask from oux. worth noting free acts on G, rather than the vanilla architecture Gy. When mref = O, aux does not work and G2 is equivalent to Gi.
4.3 Internal distribution learning
prove that the internal patch distribution of single image contains rich meaning. Minimizing the divergence of internal distributions works out many tasks like image generation, style transfer, and super-resolution. It is usually achieved by the adversarial patch discriminator [13. However, is well known that the training of GAN is time-consuming. unstable, and requires large GPU memory. Previous few-shot GAN adaptation works 29) using GAN loss almost have the serious model collapse phenomenon. We find that slice Wasserstein distance (SW D) [5] can lead to the same destination but with greater efficiency. For two tensor B, E RR Nxx xd the SWD of their empirical internal distributions is defined as
Recent works about internal learning 35 37 51 52
SWD(A,
||sot((pro((A, 0)) sort(proj(B,, ())))) do,
(5)
where Sd = (0 E Rd |0| 1}, proj Hx R HW that projects each pixel from the d channels to a scalar by 0, and sort denotes the sort operator ordering all values. SWD D can be easily implemented by convolution of randomized 1 x 1 kernel along with quick-sort algorithm.
To realize style and entity transfer, distribution of the syntheses and reference. The style loss is defined
SW D minimize the divergence of the empirical internal
m
Latyle
Σ
Σ
sw Swp(((()) (((ree))
(6)
n: Polplee SE
2 Pttgle is set of convolutional layers from the pre-trained (pips network to extract spatial features. Note that the generator learns the style of ÿree rather than that of yref to prevent the style of the entity from leaking into the other area. The entity loss is defined
Lent
Σ GE
Σ
SW s(p(m o v.). simmi @ Ovee)))
73 Pence
Pent is also a set of convolutional layers from the pre-trained lpips net, m is upsampled by m to get the same resolution the final synthesis. Taking the reconstructed entity m 3 Vrec as the target can be viewed as an implicit augmentation technique to avoid overfitting.
4.4 Manifold regularization
The training losses, particularly the style loss, inevitably distort the content of source syntheses and change their relative relationship. To achieve cross-domain correspondence, we use the variant
5

---

FSGA
MTG
JoJoGAN
Targer
Figure 3:
Comparison of our framework with other state-of-the-ard The source domain contains natural face images of 1024 x 1024. Please zoom for better visual effects.
Laplacian regularization LapR to alleviate the distortion. Since we do not tune the StyleGAN mapping network, the W space coupled with the source generator G, does not change, define
LyLopR = J/x (wwo(Gr(wu)) - V (GG, dP(w)),
(8)
where é is pre-trained semantic extractor, here we the pre-trained CLIP image encoder [31 that is sensitive to both style and content variation. Lv can interpreted from two perspectives. The first is the smoothness of the function: Because the integrand also represents the smoothness of the residual (GG, (w²)) (GG, (w"),, minimizing Laiahh expects syntheses from G, and G, across W have a smooth semantic difference. In the ideal case, the integrand is nearly zero, for various w in a local area, G (w=) and G, (w*) have similar differences in terms of style and entity. The second perspective is the preservation of geometric structure: According to the Laplacian regularization theory (3] the integral form in Eq. (8) can be estimated in discrete form;
Lv =
Wis (1( - ó(v)) + )(x,) - (2:))
= 21r ( LR),
(p))
where R (0( T <(x) a )r. ply T of (n )'), wt.j =|
the Laplacian matrix (see Sec. 3 In Eq. (9h if and wr, are close in latent space, way will be a large scalar that encourages (v()) o0y,) and [}(x;) Φ(x;)) to be the same. This means that the relative distances of adjacent syntheses before and after adaptation are isometric in the feature space. In practice, the loss can be efficiently computed using Eq. 9p). the derivation is provided in the supplementary materials.
Relation between LVIApR and Lenc Recently the influential work FSGA 29 introduces the cross-domain distance consistency loss Lone to achieve cross-domain correspondence. For arbitrary and w, without loss of generality, FSGA defines the conditional probability Pylı and Qyj in Eq. (10) to form the similarity distributions, then minimizes their Biclbacck-Libble divergence Eq. [1] that encourages the syntheses of G2 to have the same similarity property as that of G.-
EE [1(mm;) MF 931e e (1,) -
-110(m;)) -|(0)))))(<=11
Pyj
(10)
LCDe EE KL 112 = ER Ey Py
log
Pil
(11)
Zi
In the above equation, Φ denotes the feature map the corresponding generator. From the perspective of manifold learning, P.4 and 4. ; depict the neighborhood structure of æ; and y, respectively, LeDe tries to place a; in new space so as to optimally preserve neighborhood identity. Actually, we find
that this idea is equivalent to the Stochastic Neighbor Embedding [12 40), which is also a popular

---

Table 1: Quantitative results. The target images are shown in Fig. B
US denotes the user preference.
NME and IS are not counted on the Mask since the masked face cannot be identified by 7 and [0]
Sketch
Disney
Arcane
Hat
Zelda
Mask
NME,
1sT USTIMEE
IST
UST
NME.
1St
US
NMET
1ST
UST
NMEL
1ST UST
UST
FSGA
0.14
0.11 0.09
0.26
0.00
0.09
0.17
0.04
0.09
0.19
0.03 0.10
0.28
0.00 0.09
0.10
MTG
0.08
0.31 0.09
0.19
0.14 0.11
0.08
0.24 0.10
0.09
0.29 0.10
0.15
0.05 0.09
0.10
JoJoGAN
0.09
0.24 0.13
0.11
0.17 0.11
0.10
0.17 0.11
0.08
0.21 0.10
0.09
0.09
0.11
0.10
OSCLIP
0.11
0.20 0.09
0.11
0.19 0.11
0.10
0.19
0.09
0.13
0.23 0.10
0.21
0.13 0.10
0.10
Ours
0.08
0.40 0.60
0.09
0.26 0.58
0.07
0.25
0.61
0.08
0.36 0.60
0.09
0.25
0.61
0.60
w/o fottle
0.07
0.45
0.07
0.40
0.07
0.32
0.08
0.40
0.08
0.35
w/o LyLaph
0.09
0.35
0.10
0.11
0.08
0.21
0.10
0.29
0.12
0.15
method for preserving the manifold structure. LCDC has the following disadvantages. Firstly, large batch is almost inacceptable for a high-resolution GAN to estimate p.j: and g-jo- LapR directly constrains the distance and performs well even when the batch is 2. Second, LCDD is non-convex and hard to optimize, in practice, the loss value is usually around Ic-5, implying that LCDC does not provide an effective penalty for the differences. Finally, because the so fimar form of Pyla and Zyje is scale- invariant about the inputs, LCDC cannot guarantee the isometric relationship of the source and adapted syntheses like LapR, thus is hard to avoid mode collapse. In the supplementary materials, we show that replacing LODG with LV LapR can greatly remedy the collapse of FSGA.
5 Experiments
Our implementation is based on the official code of StyleGAND If myef * O. the total epoch is 2000. We adopt the Adam optimizer with learning rate 1e 3, 31 0, B2 = 0.999. The cosine annealing strategy of the learning rate is adopted to reduce the learning rate to Le 4 gradually. In
Eq. 2 A = 10. A2- = 0.2, Aa- 2, X4 1. and in Eq. (4) As- 100. We use the pre-trained lpips
network with vgg architecture 38 containing five convolutional blocks. In Eq. (6) and Eq. 7
Lstyle {ugg1:1. t991:)), J. Pent {uggi.1. Vg91:2, V991.3. v991-1). and = 1. In Eq. 9 =
including the samples from reconstruction and internal distribution learning. a for calculating the graph weights is 128. We use the Monte Carlo simulation that randomly samples 256 vectors on the unit sphere to compute the integral in Eq. ( If Mref = O, we change X2 to 2, A to 0.5, and epoch to 1000 for speeding up training. The augmentation of the reference only includes the horizontal flip. More experimental details and results can be found in the supplementary materials.
5.1 Comparison with SOTA methods
We take the few-shot adaptation method FSGA 29). one-shot adaptation methods MTG 55 OSCLIP [20). and one-shot face style transfer method JoJoGAN 9 for comparison on the commonly studied face domain. As shown in Fig. since these works cannot deal with the entity yet, for a fair comparison we select three portraits without entity (Sketch, Disney, Arcane) from previous works and three portraits with entities (hat, Zelda decorations, and mask) from AAHQ 23 Their masks can be obtained via manual annotation, or alternatively by semantic segmentation
34 9
Qualitative results Fig. 3 shows the qualitative results. The top row images are the source natural faces, and they are inverted by e4e (39] to obtain the latent codes. From the results, we can draw the following conclusions: Firstly, our syntheses are of competitive style as the given references. Though shown in thumbnails, they have more obvious high-frequency details to look sharp and realistic, whereas the results of JoJoGAN are very fuzzy. Secondly, our method achieves the strongest cross-domain correspondence that preserves the content and shape of the source image, which is a great advance compared with other methods without a doubt. It also implies method can keep the diversity of the source model without content collapse. Thirdly, our method can generate high-quality entities, which are adaptive to the various face shapes and look harmonious with the rest area of the syntheses. Experiments prove that our method has sufficient visual advantages over competitors.
Quantitative results Previous works mainly conduct user studies to compare the visual quality. Following them, we survey user preferences for different methods. We first randomly sample 50 latent codes in W space and feed them into various models. We provide the references, source syntheses,
7

---

Target
Style livation - + Ltyle
Lent
+yapp
Estgle
Stylion
Ge
Source
GE
Figure 4: Ablation study. The arrows indicate how the current model has changed from the previous model. The upper and lower syntheses come from G. and G2 respectively.
and adapted syntheses to volunteers. They consider both style and content and subjectively vote for their favorite adapted syntheses. Moreover, to objectively evaluate the cross-domain correspondence, we use the face alignment network 7 to extract facial landmarks and calculate the Normalized Mean Error (NME) of landmarks between source syntheses and adapted syntheses.
T Σ
[mnk(x)) - -lkk(y))l*
NME G.)
(12)
VHW
where Imk denotes the 68 points two-dimension landmarks extractor, and n is set 1000. NME reflects the face shape difference before and after adaptation. We further report the Identity Similarity (IS) predicted by the Areface [10] to metric the preservation of identity information. The model with lower NME and higher IS shows better cross-domain corresponding property. The quantitative results have been shown in Table Our method undoubtedly obtains the best scores in terms of NME and IS. It performs stably and gets similar NME in different domains. FSGA and OSCLIP obtain relatively worse scores due to overfitting as shown in Fig. 3 their common characteristic is that the GAN discriminator is used for training. MTG performs very unsteadily especially when the target domains have strong semantic style like Disney and Zelda. Compared with our approach, JoJoGAN works well on NME, but poorly on IS. For the user study, we finally collect valid votes from 53 volunteers, and a user averagely spends 4 seconds on one question. Our method obtains about 60% of the votes on each adapted model, whereas other methods evenly acquire the rest of the votes. It proves that our method is much more popular with users.
Training and inference time Our method costs about 12 minutes for myref * 0 and 3 minutes for myrf 0 NVIDIA RTX 3090. FSGA, MTG, JoJoGAN, OSCLIP cost about 48, 9, 2, 24 minutes respectively. All of them take about 30ms to generate an image.
5.2 Ablation study
We start from the basic reconstruction loss Lree and continuously improve the framework observe the qualitative changes of the syntheses. As illustrated in Fig. 4 the target exemplar is a hard case depicting man with beard and cap, and style fixation can bring a rough but remarkable improvement to the style of synthesis. Although Letyle can enhance the style of the face, it deteriorates the hair and introduces noise like the beard to the female face. Lent plays a critical role in the generation of the entity. Without it, the cap cannot be synthesized. In particular, Ly Lapft has a significant suppression of the noise in previous syntheses. Finally, we remove Latyle and style fixation respectively, then the syntheses are no longer of the target style. It means that both of them play vital roles in style transfer. We also provide the quantitative ablation in Table T to study the impacts of Lstyle and LV shows that removing Latytee has positive effect on NME and IS while removing LVI.appt has an opposite effect. The quantitative results confirm that LV Lapn necessary for preventing content from being distorted by style loss.
5.3 More results
Adaptation of various domains In addition to the success of the face domain, our framework can be applied to other source domains. We take StyleGANs pre-trained on the 512 x 512 AFHQ dog, cat data set g and the 256 x 256 LSUN church dataset [48] as exemplars. As shown in Fig. [5] for each source domain, provide images with entities, and without entities, the source images
8

---

Souree
Source
Figure 5: Our framework can transfer various source domains to various target domains.
Target/Source
Rec/Trunsfer
Smile
Age
Rotation
Style
Removing entity
Smile
Figure 6: Applying the adapted model to manipulate target image and transferred image, including semantic editing, style transfer and entity removal.
randomly sampled from the source space. For images without entities, the adapted syntheses realistic and have a strong corresponding relationship in visual. In other cases, our approach also yields satisfactory results. For example, the cartoon eyes of the dogs are in the right place and very similar to that the reference. It proves that our framework is valid and has a wide universality.
Image manipulation Because our adapted model retains the geometric structure of the source generative space, can efficiently perform semantic editing on the image using the semantic direction found in the source space. As shown in Fig. 6 since our model is capable of accurately reconstructing the target exemplar, we can edit both the exemplar and the synthesis. We select three representative semantic directions (36) to change smile, age, and pose rotation, and the results show that the adapted generator achieves exact control not only on the face but also on the entity. addition, our model can also change the style by altering the style code, which is hard for previous works. Furthermore, framework be applied to remove the entity with G,, and the result can be manipulated similarly. We consider these functions will be helpful for artistic creation.
6 Conclusion and limitations
In this paper, we construct new GAN adaptation framework for the generalized one-shot GAN adaptation task. Because our framework relies heavily on learning of internal distributions, inevitably has some limitations. We think the most important one is that it cannot precisely control the position of the entity, which may lead to failures when the pose changes too much. Another limitation learn by patch distribution.
that the entity cannot be too complex, otherwise, it is difficult However, these extreme cases require additional consideration. In general, our framework is effective various scenarios. We believe that the exploration of the novel task and the introduction of manifold regularization are significant for future work.
Acknowledgements
This paper is supported by National key research and development program of China (2021YFA 1000403). the National Naturll Science Foundation of China (Nos. U19B2040), the Strategic Priority Research Program of Chinese Academy Sciences, Grant No. XDA27000000, and the Fundamental Research Funds for the Central Universities.

---

References
[1] Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? In Proceedings of the IEEE international conference computer vision, pages 4432 4441, 2019.
[2] Mikhail Belkin and Partha Niyogi. Laplacian eigenmaps for dimensionality reduction and data representation. Neural Computation, 15:1373-1399 2003.
[3] Mikhail Belkin and Partha Niyogi. Towards theoretical foundation for laplacian- manifold methods. J. Comput. Syst. Sci., 14:1289-1309 2008.
[4] Mikhail Belkin, Partha Niyogi, and Vikas Sindhwani. Manifold regularization: A geometric framework for learning from labeled and unlabeled examples. Mach. Learn, Res., 7:2399- 2434, 2006.
[5] Nicolas Bonneel, Julien Rabin, Gabriel Peyré, and Hanspeter Pfister. Sliced and radon wasser- stein barycenters of measures. Journal of Mathematical Imaging and Vision, 51:22 45, 2014.
[6] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019.
[7] Adrian Bulat and Georgios Tzimiropoulos. How far are we from solving the 2d & 3d face alignment problem? (and a dataset of 230,000 3d facial landmarks). In International Conference on Computer Vision, 2017.
[8] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung- Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. 2020 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8185-8194, 2020.
[9] Min Jin Chong and David Forsyth. Jojogan: One shot face stylization. ArXiv, abs/2112.11644 2021.
[10] Jiankang Deng. J. Guo, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4685-4694, 2019.
[11] Rinon Gal, Or Patashnik, Haggai Maron, Gal Chechik, and Daniel Cohen-Or. Stylegan- nada: Clip-guided domain adaptation of image generators, 2021.
[12] Geoffrey E. Hinton and Sam T. Roweis. Stochastic neighbor embedding. In NIPS, 2002.
[13] Phillip Isola, Jun- Yan Zhu, Tinghui Zhou, and Alexei A. Efros. Image-to-image translation with conditional adversarial networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5967-5976, 2017.
[14] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint 1rXiv:1710.10166, 2017. 2017.
[15] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. ArXiv, abs/2006.066766 2020.
[16] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4401-4410, 2019.
[17] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila, Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8110 81188119,,111 2020.
[18] Chaerin Kong, Jeesoo Kim, Dong Yan Han, and Nojun Kwak. Smoothing the generative latent space with mixup-based distance learning. ArXiv,
2021.
[19] Gihyun Woon and Jong-Chul Ye. Clipstyler: Image style transfer with single text condition. ArXiv, abs/2 12.00374, 2021.
10

---

[20] Gihyun Kwon and Jong- Chul Ye. One-shot adaptation of gan in just one clip. ArXiv, ubs/2203.09301, 2022.
[21] Hyuk-Gi Lee, Gi-Cheon Kang, Changhoon Jeong, Han-Wool Sul, and Byoung- Tak Zhang. C: Contrastive learning for cross cross-domain correspondence in few-shot image generation. 2021.
[22] Yijun Li, Richard Zhang, Jingwan Lu, and E. Shechtman. Few-shot image generation with elastic weight consolidation. ArXiv, abs/2012.0078// 2020.
[23] Mingcong Liu, Qiang Li. Zekui Qin, Guoxin Zhang, Pengfei Wan, and Wen Zheng. Blendgan: Implicitly gan blending for arbitrary stylized face generation. In Advances in Neural Information Processing Systems, 2021.
[24] Yinglu Liu, Hailin Shi, Hao Shen, Yue Si, Xiaobo Wang, and Tao Mei. A new dataset and noundary-attentinn semantic segmentation for face parsing. In AAAI, 2020.
[25] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. ArXiv, abs/1802.05957, 2018.
[26] Sangwoo Mo, Minsu Cho, and Jinwoo Shin. Freeze discriminator: simple baseline for fine-tuning gans. ArXiv, abs/2002. 10964, 2020.
[27] Ron Mokady, Michal Yarom, Omer Tov, Oran Lang, Daniel Cohen-Or, Tali Dekel, Michal Irani, and Inbar Mosseri. Self-distilles stylegan: Towards generation from internet photos. ArXiv, abs/2202. 12211, 2022.
[28] Atsuhiro Noguchi and Tatsuya Harada. Image generation from small datasets via batch statistics adaptation. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 2750-270 2750,2778,110 2019.
[29] Utkarsh Ojha, Yijun Li, Jingwan Lu, Alexei A. Efros, Yong Jae Lee, Eli Shechtman, and Richard Zhang. Few-shot image generation via cross-domain correspondence. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10738- 2021. 2021.
[30] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and D. Lischinski. Styleclip: Text-driven manipulation of stylegan imagery. 2021 IEEE/CVE International Conference on Computer Vision (ICCV), pages 2065-2074, 2021.
[31] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In ICMI., 2021.
[32] Esther Robb, Wen-Sheng Chu, Abhishek Kumar, and Jia-Bin Huang. Few-shot adaptation of generative adversarial networks. arXiv preprint arXiv:2010. 1943, 2020.
[33] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015.
[34] Amit R.Sharma and Prakash. R. Devale Prakash. Devale. Face photo-sketch synthesis and recognition. International Journal of Applied Information Systems, 1:46-52, 2012.
[35] Tamar Rott Shaham, Tali Dekel, and T. Michaeli. Singan: Learning generative model from single natural image. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 4569 2019.
[36] Yujun Shen, Ceyuan Yang, Xiaoou Tang, and Bolei Zhou. Interfacegan: Interpreting the disentangled face representation learned by gans. arXiv preprint arXiv:2005.0966 2020.
[37] Assaf Shocher, Bagon, Phillip Isola, and M. Irani. Ingan: Capturing and retargeting the "dna" of natural image. 2019 IEEE/CVF International Conference Computer Vision (ICCV), pages 4491-4500, 2019.
[38] K. Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, Abs//440911550 2015.

---

[39] Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or. Designing an encoder for stylegan image manipulation. ACM Transactions on Graphics (TOG), 40:1 14. 2021.
[40] Laurens van der Maaten and Geoffrey E. Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9:2579-2605, 2008.
[41] Yaxing Wang, Abel Gonzalez-Garien David Berga, Luis Herranz, Fahad Shahbaz Khan, and Joost van de Weijer. Minegan: Effective knowledge transfer from gans to target domains with few images. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9329 9338, 2020.
[42] Yaxing Wang, Chenshen Wu, Luis Herranz, Joost de Weijer, Abel Gonzalez-Garcin and Bogdan Raducanu. Transferring gans: generating images from limited data. In European Conference Computer Vision, 2018.
[43] Yue Wang, Ran Yi, Ying Tai, Chengjie Wang, and Lizhuang Ma. Ctlgan: Few-shot artistic portraits generation with contrastive transfer learning. 2022.
[44] Zhou Wang, Alan Conrad Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. Image quality assessment: from error visibility to structural similarity, IEEE Transactions on Image Processing. 13:600-612, 2004.
[45] Weihao Xia, Yulun Zhang, Yujiu Yang. Jing-Hao Xue, Bolei Zhou, and Ming-Hsuan Yang. Gan inversion: survey. arXiv preprint arXiv:2101.05527, 2021.
[46] Jiayu Xiao, Liang Li, Chaofei Wang, Zhengjun Zha, and Qingming Huang. Few shot generative model adaption via relaxed spatial structural alignment. ArXiv, abs/2203.04121, 2022.
[47] Ceyuan Yang, Yujun Shen, Zhiyi Zhang, Yinghao Xu, Jiapeng Zhu, Zhirong Wu, and Bolei Zhou. One-shot generative domain adaptation. ArXiv, 02222111.09877 2021.
[48] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 550600336, 2015.
[49] Ning Yu, Ke Li, Peng Zhou, Jitendra Malik, Larry Davis, and Mario Fritz. Inclusive gan: Improving data and minority coverage in generative models. ArXiv, abs/2004.03355, 2020.
[50] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
[51] Zicheng Zhang, Congying Han, and Tiande Guo. Exsingan: Learning an explainable generative model from a single image. ArXiv, abs/2105.07350, 2021.
[52] Zicheng Zhang, Yinglu Liu, Congying Han, Hailin Shi, Tiande Guo, and Bo Zhou. Petsgan: Rethinking priors for single image generation. ArXiv, abs/2203.0148.. 2022.
[53] Miaoyun Zhao, Yulai Cong, and Lawrence Carin. On leveraging pretrained gans for generation with limited data. In ICML, 2020.
[54] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
[55] Peihao Zhu, Rameen Abdal, John C. Femiani, and Peter Wonka. Mind the gap: Domain gap control for single shot domain adaptation for generative adversarial networks. ArXiv, abs/2110.08398, 2021.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No] The code and the data are proprietary.
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [No] The images we use are widely used previous works or from open dataset.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [No] The images we use are widely used previous works from open dataset.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [No]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "uzAn-WduQh6", "reviewer": "Reviewer_fun1", "summary": "This paper considers the one-shot domain adaption problem, it decoupled adaptation into two parts: style and entity transfer. Unlike most previous works that mainly focus on style transfer, the proposed method utilises binary entity mask with concise manifold regularized GAN design. The author modifies the architecture of the original generator to decouple the adaption of style and entity, and proposes the variant Laplacian regularization to smooth the network. Extensive experiments are conducted on various references with and without entities.\n", "questions": "How to get binary entity masks? The author just mentioned it in the abstract and fig.1, should add descriptions in the experiment part.\n\nWhy did this paper chose sliced Wasserstein distance?  Need more insight and theoretical depth.\n\nIn L139 'each w will be transformed into the style-fixed code...', how to do transform? Need more details. \n", "limitations": "The authors adequately addressed the limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper focuses on an interesting one-shot domain adaption problem, the idea of disentangling style and entity transfer is straightforward.\n\nThe motivation of this paper is to solve one-shot domain adaption problem. However, I have some concerns as follows:\n1. About the problem setting: the paper only explores the domain adaption with a clear binary entity\nmask on the target image, which does not always exist for the general one-shot domain adaption setting. \n\n2. The disentangled GAN structure has been well studied by many existing works, it is hard for the reviewer to identify the real contributions of this paper.\n\n3. The experiments are insufficient. The evaluation size is too small, the main result Tabel.1, only uses Fig. 3's images for quantitative evaluation.\n\n4. Since the proposed method focuses on manifold regularization, some related Manifold GAN methods [1][2][3] are missing in comparison.\n[1]MR-GAN: Manifold Regularized Generative Adversarial Networks\n[2]MMGAN: Manifold-Matching Generative Adversarial Network\n[3]Manifold-preserved GANs", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "gQ3-gX4tK3m", "reviewer": "Reviewer_5BYZ", "summary": "This paper proposes a manifold regularized GAN adaption framework to deal with the generalized one-shot GAN adaption problem (i.e. the target domain to transfer contains both artistic styles and entities). To tackle this novel task, the paper modifies the architecture of the original generator by adding an additional auxiliary network to facilitate entity generation. To reduce the domain gap, they employ the sliced Wasserstein distance to minimize the divergence of the internal distributions between the exemplar and synthesis. Besides, they propose to use the variational Laplacian regularization $L_VlapR$ to mitigate the content distortion during training by preserving the geometric structure of the source manifold.\n", "questions": "See Weakness", "limitations": "Adequate", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written and easy to follow. Both quantitative and qualitative experiments show that the proposed method has sufficient visual advantages over the competition. In addition, their image processing results look interesting, which I believe will be helpful for artistic creation.\n\nMy only concern is the definition of style. In my opinion, the geometric feature is also part of the style and not just the color. I feel the results for Disney and Zelda are not so successful. Domain adaptation should also include some geometric changes, but the Zelda results in the paper don't look like CG images, and the Disney results cannot keep the big eyes and exaggerated expressions.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "E03tlvtrlRW", "reviewer": "Reviewer_AEH2", "summary": "This paper performs one-shot domain adaption of StyleGAN model and the model additionally supports adding new entities compared to previous methods. The proposed method is able to do so thanks to a handful of points. Among them, two techniques are interesting: 1) using sliced Wasserstein distance to compute the internal distribution distance between two images; 2) adding variant Laplacian regularization to alleviate cross-domain correspondence distortion. ", "questions": "What is the reason to produce a mask using UNet since we can directly use the mask provided by users (if I understand correctly)? The underlying motivation is unclear. Plus, the authors should conduct an experiment to explain the benefit of doing it.\n\nThe paper missed comparison with other internal distribution distances like mse between gram matrices. To me, entity loss and style loss can be directly implemented with MSE between the gram matrices of deep features from two images. The authors are advised to try these loss functions to show the advantage of using sliced Wasserstein distance.\n\nWhy do you give the task of this paper a new name--generalized one-shot GAN adaption? From my understanding, the method additionally can add entities but that doesn't mean it is generalized.", "limitations": "The authors provide a section to talk about the limitations of this work. I agree with the authors about these limitations. I think one viable solution for the entity position problem is to inquire the users for an extra mask representing the desired location of the entity in the synthesized image.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe experimental results are promising. Using sliced Wasserstein distance and variant Laplacian regularization make sense. The authors provide code and additional materials so the method seems reproducible. The learning speed of this approach is faster than previous methods.\n\n\nWeaknesses:\n\nThe writing of this paper can be improved. From the abstract and introduction, it is hard to identify the contributions of the proposed approach. Fig. 2 is hard to follow. It seems weird to have Fig. 1 below the title. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ACwlD9gVT3C", "reviewer": "Reviewer_Wx24", "summary": "This paper presents a method for one-shot domain adaptation, where a pre-trained GAN is leveraged to fine-tune the generator on one target image. First, given a source image, a latent vector is obtained by GAN inversion. Then, the generator is trained using a reconstruction loss, style and entity losses and a Laplacian regularizer, in order to transfer the style content of the target image into the source image while retaining the entity of the latter. Moreover, the proposed method admits a binary mask to select parts of the target image and to blend them into the source image, which is a novel contribution. Quantitative and qualitative results are presented when transferring from a few source images to a few target images from AAHQ dataset, as well as qualitative results for a few source and target images on AFHQ dog dataset and some source and target images on LSUN church dataset, showing a successful one-shot style transfer with optionally transferring objects from the target image. \n", "questions": "As it was written above, I would like to see averaged quantitative metrics over many source/target samples.\n\n\nA sentence in l.176-177 claims that slice Wasserstein distance “leads to the same destination but with greater efficiency”, compared to the GAN loss. There is no reference cited nor I have seen an ablation experiment in the paper. Could the authors elaborate on this with empirical evidence or otherwise, remove the sentence altogether?\n\nIn l.35 it is mentioned that other methods produce artifacts when the entities are big. Could you point out where can this be observed in the figures?\n\nHow were the hyper parameters chosen? No discussion is provided with respect to that.\n\n\n\n*Details on clarity*\n* The task definition in l.106 is not clear and also fails to mention one of the most important details: the generator is fine-tuned on only one image from the target domain. As it stands, it appears as if a collection of target images were used instead. \n* The methodology section focuses on one particular case, pretrained on FFHQ. Instead, the methodology should be general and only introducing the datasets and particular examples in the experimental section and Figure 2 as an illustration. \n* Figure 2 is not clear, misses notation to quickly cross-reference it with the text. Moreover, the figure is not self-contained as there is some notation that is not explained in the caption. Additionally, this figure is referenced in the text before anything about the model is explained, which generates confusion. \n* The explanation of the auxiliary net is lacking, specially to understand how the feature map $f_{ent}$ and mask $m$ are obtained. Is the mask predicted on the feature maps from the style-gan architecture? And if so, did you use a pre-trained U-Net or you trained one from scratch? How did you adapt a pre-trained U-Net in this case?\n* Equation (1) misses $f_{ent}$ and $m$ as inputs to $aux$.\n* l.187: “layers from pre-trained LPIPS” this is a experimental detail that does not belong in the methodology section.\n* l.187: “upsampled by m” what does hat mean?\n* Paragraph in lines l. 158-164 would be better suited in the experimental section.\n\n* Where is the source of the images “Sketch, Disney and Arcane” that are mentioned in l.241? No reference is provided. \n\n", "limitations": "The limitations were discussed and the broader impact briefly touched upon.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nThe work is well motivated and the introduction of a binary mask to optionally import objects from the target image is a novel contribution. It targets a useful application and can be of use to the community and for content creators. \n\nThe usage of quantitative metrics to evaluate the results is a good addition to the user studies. Moreover, the two metrics used are reasonable in the context of face to face translation. \n\nThe methodology is sound and the claims are adjusted to the results presented in the experimental section. \n\n**Weaknesses**\n\nThe paper lacks clarity and misses important information (more details in \"Questions\"). The related work is not well redacted and fails to situate the proposed method within other methods in the literature. \n\nI would have expected the quantitative metrics to be computed and averaged across multiple samples for the target and source domains. Instead, those are only computed on hand-picked source-target images and risks providing only a biased positive view of the results. \nThe experimental setup is lacking  to properly quantitatively evaluate the method with respect to the baselines. Instead of computing them on a handful of samples, they should be computed over a dataset (or a randomly chosen subset of data) for the source and target domains in order to draw more robust conclusions.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept