Mind the Gap: Domain Gap Control for Single Shot Domain Adaptation for Generative Adversarial Networks
Authors: Peihao Zhu
Rameen Abdal
John Femiani
Peter Wonka
Venue: ICLR 2022 Poster (2022)
Keywords: GAN
StyleGAN
Clip
Domain Adaptation
Style Transfer
Single Shot

Abstract: We present a new method for one shot domain adaptation. The input to our method is trained GAN that can produce images in domain A and a single reference image I_B from domain B. The proposed algorithm can translate any output of the trained GAN from domain A to domain B. There are two main advantages of our method compared to the current state of the art: First, our solution achieves higher visual quality, e.g. by noticeably reducing overfitting. Second, our solution allows for more degrees of freedom to control the domain gap, i.e. what aspects of image I_B are used to define the domain B. Technically, we realize the new method by building on a pre-trained StyleGAN generator as GAN and a pre-trained CLIP model for representing the domain gap. We propose several new regularizers for controlling the domain gap to optimize the weights of the pre-trained StyleGAN generator to output images in domain B instead of domain A. The regularizers prevent the optimization from taking on too many attributes of the single reference image. Our results show significant visual improvements over the state of the art as well as multiple applications that highlight improved control.

Published as a conference paper at ICLR 2022
MIND THE GAP: DOMAIN GAP CONTROL FOR SINGLE SHOT DOMAIN ADAPTATION FOR GENERATIVE AD- VERSARIAL NETWORKS
Peihao Zhu KAUST
Rameen Abdal
John Femiani Miami University
Peter Wonka KAUST
KAUST
Learn domain B
Predict
Edit age+exp
domain A
domain B
pose
gender
glasses
G
a
Gg
Figure One-shot domain adaptation: (left) single reference image from domain is used to refine GAN GA to learn GB; (center) every image in domain A has analog in domain B that shares a latent code and many salient attributes; (right) because salient attributes are preserved the new domain, many latent-edits meaningful in the new domain.
ABSTRACT
We present new method for shot domain adaptation. The input method is a trained GAN that can produce images in domain A and single reference image Ig from domain B. The proposed algorithm can translate any output of the trained GAN from domain A to domain B. There are two main advantages of our method compared to the current state the art: First, our solution achieves higher visual quality, e.g. by noticeably reducing overfitting. Second, our solution allows for more degrees freedom to control the domain gap, i.c. what aspects the image Lg are used to define the domain B. Technically, we realize the new method by building on a pre- trained StyleGAN generator as GAN and pre-trained CLIP model for representing the domain gap. We propose several new regularizers for controlling the domain gap to optimize the weights of the pre-trained StyleGAN generator that it will output images in domain B instead of domain A. The regularizers prevent the optimization from taking on too many attributes of the single reference image. Our results show significant visual improvements over the state of the art well as multiple applications that highlight improved contro
INTRODUCTION
We propose a new method for domain adaptation based on single target image. As shown in Fig. given a trained GAN for domain A, and a single image Ig from domain B, our approach learns to find a corresponding image in domain B for any image in domain A. We can achieve this by fine- tuning the GAN for domain A to obtain second GAN that generates images in domain B. The two GANs share latent space that single latent code will generate in domain and one domain B. The main selling point our method is that achieves superior quality than the state of the art in single shot domain adaption. Our method is computationally lightweight and only takes a few minutes on a single GPU, so that it can be widely applied.
corresponding images, one
In order to do this, we leverage multiple existing components, including networks: First, we use StyleGAN2 (Karras et al.) (220bb) as a pre-trained GAN. A follow-up ver- sion has been published on arXiv (Karras et al. 2021), but the code only became available after
excellent pre-trained
Code available at httphhhss//ppeeuuggghppt

---

Published as a conference paper at ICLR 2022
finished all experiments. Second, ford et al. 2021). ). to encode images as vectors. Third, we use the pioneering idea of StyleGAN- NADA (Gal et al. 202ID. which builds upon StyleCLIP (Patashmik et al. 2021), to encode domain gap (or domain shift) as vector in CLIP embedding space. Fourth, we leverage 112S (Zhu et al.) 20206) GAN embedding method to transfer image Ta into domain A to obtain better estimation of the domain gap.
pre-trained network for image embedding, CLIP Radd
Even though the visual quality of StyleGAN-NADA is already impressive when used as a single image domain adaption method, we identified multiple technical issues that can be improved to achieve another large jump visual quality. First, and most importantly, StyleGAN-NADA was designed for zero-shot domain adaptation, and does not have a good solution to model the domain gap based on single example image. Their reference implementation models the domain gap as vector from the average image in domain A to the given image Т₁ in CLIP embedding space. However, this leads to overfitting in practice and the transfer results lose attributes of the input images, that input images from domain A get mapped to images that are all too similar to L in domain B. We identify a better solution to this problem. In fact, the domain gap should be modeled vector from the image Ig to its analog in domain A, so that the image in domain A shares salient doobin-domain attributes with the reference image. We therefore need to solve an inverse B-to-A comain-transfer problem, which we propose tackle using the state-of-the-art GAN embedding method 112S (Zhu et al. (020b). key insight is that we can use a heavily regularized version of the II2S GAN inversion method do the reverse problem of transferring any related image (from domain B) into the domain A, helping to characterize the semantic domain gap better than previous work. Further extensions enable us to fine tune the modeling of the domain gap to explicitly model which attributes of the input image should be kept. Second, we propose multiple new regularizers to improve the quality. Third, we propose a technical improvement the heuristic layer selection proposed in StyleGAN-NADA that is more straightforward and robust.
In summary, we make the following contributions:
1. We reduce the mode collapsccoverfitting problem which often occurs in one-shot and few- shot domain adaptation. Our results look similar to the target domain images with fewer artifacts. These results also faithful to the identities of the source domain images and able to capture fine details.
2. Our domain adaptation provides more freedom to control the "similarity" between im- ages across domains that share common latent-code, which makes large number of downstream applications possible, e.g., pose adaptation, lighting adaptation, expression adaptation, texture adaptation, interpolation, and layer mixing, using state-of-the-art image editing frameworks.
2 RELATED WORK
Domain adaptation. Domain adaptation is the task of adapting a model to different domains. Different works in this area (Bousmalis et al., [016]] 2017 Na et al. 2020 Wang & Breckon 2020) Kang et al. 2019) try to learn diverse domain independent representations using the source domain to make predictions, such image classification, in the target domains. More importantly, gen- erating diverse representations of images by combining natural language supervision has been of interest to the computer vision and NLP research communities (Frome et al.) 2013). Recently, Ope- nAI's Contrastive Language-Image Pretraining (CLIP) (Radford et al. 2022 work established that transformer, and large datasets, could generate transferable visual models. In CLIP, both images and text are represented by high dimensional semantic-enn -emeddddge vectors, which can then be used for zero-shot learning.
GAN-based domain adaptation. In the GAN domain, various models and training strategies have been proposed for few-shot domain adaptation tasks (Bousmalis et al. 2017 ZHANG et al. [018 Li Liu et al. 2019). Most relevant to our work, the domain adaptation methods (Patashnik
ct al. 2020
et al. 2021 Gal et al. 2021 Jang et al., 2021 Song et al. 202I) that build upon StyleGAN (Kar- ras et al. / 2020b a) demonstrate impressive visual quality and semantie interpretability in the target domain. These methods can be broadly classified into shot and single-shot domain adap- tation methods.
2

---

Published as a conference paper at ICLR 2022
A notable few-shot method, StyleGAN-ADA (Karras et al., 2020al proposes adaptive discrimi- nator augmentation method to train StyleGAN on limited data. Another work, DiffAug (Zhao et al, 2020), applies differentiable transformations to the real and generated images for robust training. A discriminator related approach, FreezeD (Mo et al. 2020), freezes lower layers of the discrimina- tor to achieve domain adaptation. Toonify
interpolates between the model- weights of different generators to generate samples from a novel domain. A more recent work (Ojha al. 2021), reduces overfitting on limited data by preserving the relative similarities and differences in the instances of samples in the source domain using cross domain correspondence.
Latent space interpretation and semantic editing. GAN interpretation and understanding of the latent space has been topic of interest since the advent of GANs. Some notable works in this
domain (Bau et al.] 20188] 2019, Härkönen et al. 2020 Shen et al. 2020: Tewari et al. 2020a) have led to many GAN-based image editing applications. More recent studies into the activation space of StyleGAN have demonstrated that the GAN can be exploited for downstream tasks like unsupervised and few-shot part segmentation (Zhang al. 2021 Tritrong et al. 2021 Abdal al,
2021a
Collins et al. 2020 Bielski & Favaro 2019), extracting 3D models of the objects (Pan et al,
2021
Chan al. (2020) and other semantic image editing applications (Zhu et al. 2021 Tan al.
2020
Wu et al. 2020 Patashnik et al. 2021).
Image embedding is one of the approaches used to study the interpretability of the GANs. To enable the semantic editing of a given image using GANs, one needs to embed/project the image into its latent space. Image2StyleGAN (Abdal et al. 2019) embeds images into the extended StyleGAN space called W+ space. Some followup works (Zhu et al. [2020a] Richardson et al.) 2020 Tewari ct al., (020b)) introduce regularizers and encoders keep the latent code faithful to the original space of the StyleGAN. regularize the embeddings for high-quality image reconstruction and image editing. We use this method to embed real images into the StyleGAN and show that our domain adaptation preserves the properties of the original StyleGAN in See 4
of
Image2tylleGAN (I12S) (Zhu et al.) 2020b) uses PN space to
Image editing is another tool to identify the concepts learned by a GAN. In the StyleGAN do- main, recent works (Härkönen et al.] 2020; [Shen et al. 2020 Tewari et al., 2020a Abdal et al.) 202 extract meaningful linear and non-linear paths in the latent space. InterfaceGAN (Shen) ct al.] 2020) finds linear directions to edit latent-codes in a supervised manner. On the other hand, GANSpace (Härkönen et al., 2020) extracts unsupervised linear directions for editing using PCA in the W space. Another framework, StyleRig (Tewari et al. (2020a), maps the latent space of the GAN 3D model. StyleFlow (Abdal et al.) (022b) extracts non-linear paths in the latent space to enable sequential image editing. In this work, we will use StyleFlow to test the semantic editing of our domain adapted images.
In the area of text-based image editing, StyleCLIP (Patashnik et al. 2021 extends CLIP to perform GAN-based image editing. StyleCLIP uses the CLIP embedding vector traverse the StyleGAN manifold, by adjusting the latent-codes of a GAN, in order to make a generated image's CLIP em- bedding similar to the target vector, while remaining close to the input in latent space. A downside to this approach is that these edits are unable to shift the domain of a GAN outside its original man- ifold. However, their use of CLIP embeddings inspired StyleGAN-NADA (Gal al. (02T), which creates a new GAN using refinement learning to do zero-shot domain adaptation. Although unpub- lished, they also demonstrate one-shot domain adaptation in their accompanying code. The original and target domain represents a direction used to shift the domains. Although in the accompanying source-code (ni nongal/StyleGAN NADA), they bootstrap-estimate of the mean CLIP image embedding of the original domain, and use a reference image or its CLIP image embedding to represent the new domain.
represented by CLIP text embeddings. The difference of the embeddings
3 METHOD
Our approach involves fine-tun tne-tuning&& a GAN trained for some original domain A, e.g. FFHQ faces, to adapt it to new related domain B. In our approach, the images in A and the images in B are related to each-other by a common latent code. Any image which can be generated or embedded in domain A can be transferred to corresponding and similar image in B. We use the CLIIP
3

---

Published as a conference paper at ICLR 2022
embeddings semantic-space in order to model the difference between domains A and B, and we use StyleGAN (Karras et al., 2018 2020b) as the image generator. A key to our approach is to preserve directions within and across domains as illustrated in Fig. Before fine-tuning the GAN for domain (to obtain the GAN for domain B), we determine a domain-gap direction. This direction, called yf is a vector in CLIP embedding space which points towards a reference image 1g which is in domain B from its corresponding image LA in which is in domain A. We use the CLIP image-embedding model Eū to find
of -
- EI(AA).
(1)
Finding La in domain for given image in domain is a significant limitation in the current state of the art, StyleGAN-NADA (Gal et al. 202T), as they use the mean of domain A. The mean of domain A a very crude approximation for LΛ. Instead, we propose an inverse domain adaption step, by projecting the image İg into the domain A to find sample that is more similar and specific to the reference image than the mean of domain A. In principle, this problem is also a domain trying to solve, just in the inverse direction. The
adaption problem similar the problem major difference is that we have a pre-trained GAN available in domain A.
We use the I12S GAN-inversion method (Zhu al. 2020b) in order to find latent code for an image similar to LB that is plausibly within domain A. The I2S and I12S methods use an extended version of W space from StyleGAN2. The W code is used 18 times, once for each style block in StyleGAN2. When allowing each element to vary independently, the resulting latent space is called W+ space Abdal et al. (2019) 2020); : Zhu et al. (2020b). I2S showed that the additional degrees of freedom allow GAN inversion for a wider set of images with very detailed reconstruction capabilities, and II2S showed that an additional regularization term to keep the latent codes close to their original distribution made latent-code manipulation more robust. I12S uses a hyperparameter, X, which can increased in order to generate latent codes using more regularization, and therefore in higher density regions of the W latent space. The effect of this parameter is shown in Fig. 2 The value suggested in the original work was À = 0.001, however, low values of lambda allow II2S to find latent codes that too far away from the latent-codes produced by the mapping network of the original GAN and thus produce images that are less plausible to have come from domain A, underestimating the gap between domains. In the context of domain shift we find it is useful to = 0.01 as illustrated in Fig. 2 The result is a latent code w in W+ space which is shifted towards a high-density portion of the domain A. Then the image generated from that code, IA. an image in domain A that corresponds to Ip.
Ref
A
1e-3
=
1e-2
Figure 2: An illustration showing how I12S embeds L in the original StyleGAN domain A, shown for two different values of X. Reference images from other domains are shown in the top row. The value recommended by Zhu et al.] (2020b) is shown in the second row, and the value used in this work is shown in the third row, Although there is some subjectivity involved, we believe that the large value A= = le-2 is needed for II2S to find images that plausibly could belong to the domain A, which in this case is FFHQ faces.
4

---

Published as a conference paper at ICLR 2022
Training As illustrated in Fig. to Ly but still plausibly within domain A. In principle, it possible that II2S finds L so that Ig is similar enough to be considered the same, in which case the two domains overlap. However, concerned with the cases where the domains are different, and the vector od indicates the direction of gap, shift, between domain A and domain B. We refinement learning train new generator, G so that images generated from G are shifted parallel to ye in CLIP space, relative images from GA. The desired shift is indicated by the red arrows Fig. 3 During training, latent codes w. are generated using the mapping network of StyleGAN2. Both GA and Gg are used to generate images from the same latent code, but the weights of G are frozen and only G updated during training. The goal of refinement learning is that G will preserve semantic information that within domain but also that it will generate image shifted across a gap between domains. When refining the generator for domain B, we freeze the weights of the StyleGAN2 "ToRGB' layers, and the mapping network also frozen. The overall process of training is illustrated in Fig.
II2S to find image LA which we consider to be similar
Domain B
IB
style mixing
Gn(w)
yref
Gn(
IA
samp
Domain
GA(w)
W
G
Figure 3: The vectors in the CLIP image embedding space, E₁, which control domain adaptation. Each domain depicted here as a dashed outline; the vectors ye and ysamp cross between the two domains and are used to refine generator for domain B. Corresponding images should be shifted in the same direction. The vectors AA and UB model important semantic differences within each domain that should also be preserved by domain transfer. G (w) and Ga(w) are corresponding images for an arbitrary latent-code w encountered during training. Style mixing (shown on the right) shifts part of the latent code towards the reference image effecting the result in both domains.
The goal of training is shift CLIP embeddings from domain A in direction parallel gh We use the vector yaamp to represent the current domain shift of the network GB during training, on a single sample. We have
samp
Er((((w) (T(G)(w)
as cross-domain vector for corresponding images generated from the same w latent code using the two generators. We use the loss
Lelip.across sim ( samp)
(3)
where sim(a, b) A |||||||||| the cosine similarity score. This loss term is minimized when the domain shift vectors are parallel.
It is important that the reference image Iø matches the generated image. Gz(w")), both in mantic sense, as measured by the similarity of the CLIP embeddings, and also in a visual sense. We accomplish this using two losses: Eett and Tepe The first loss measures the change in the CLIP-embeddings of the original and reconstructed reference image,
(4) ensuring that the GB reconstruct the embedding. Unlike Lelip accress: this loss term is not based on a change in embeddings between the two domains, instead it guides G by aligning it to a global embedding in CLIP space, ensuring that / remains fixed in the domain of Gn.
Let.etip 1 - sim (E, (IB). E, (GB(w
The second loss term a reconstruction loss based on perceptual and pixel-level accuracy,
Licfuee LeIPs (IB, Gp (w) + L2 (1B, (B(
(5)
5

---

Published as a conference paper at ICLR 2022
One-Shaf Shot LB
Uses GA
GA
A
CLIP Embed
EI
Embed
Subtract Vector
ytef
CLIP Embed
Ei(LB
REFINEMENT TRAINING
Ei(B
B
Reconstruction
Lree_rec
Cosine Similarity
ceep
Losses
G at (
CLIP Embed
E ((
to
Ge
Cosine Similarity
(Leamanle Weights)
2 zaaamp
Random
Mapping Network
10
G
CLIP Embed
BG
Lejig
Ga
Subtract
UB
Ga
Ca(w)
CLIP
((())(w)
Subtract
Embed
Casine
Es(la
Subtract
U
Similarity
Lliip_within
Figure 4: process diagram for domain transfer. White rectangles indicate calculations, computed values are shown on the connecting lines. The four loss-calculations are indicated by blue rectangles, and the learnable weights of StyleGAN2 (all weights except the mapping network and the ToRGB layers) are indicated in green.
where LePs is the perceptual loss from Zhang et al. (2018), and L2 is the squared euclidean differ- ence between pixels. The purpose this loss is to ensure that the visual, and not just the semantic, qualities of the image embeddings do capture many semantic and visual qualities the image, there are still many percep- tually distinct images that could produce the same CLIP embedding. This is visible in Fig, 6 without the reconstruction loss GB fails to preserve some important visual qualities (such as symmetry) of the input.
preserved. This is necessary in addition to Leefslip because, while the CLIP
There is a tendency for GANs to reduce the variation during training, especially in few-shot fine- tuning. We combat this by preserving the semantic information that is not related the domain gap. A semantic change that is not related to the change in domains should not be affected by GB- Therefore, the vector connecting the reference and sample images within the domain A should be parallel to the corresponding vector in domain B. Let = (G connecting a sample image with latent-code represents semantic changes that are within domain A, and we want the matching semantic changes to occur within the domain Let UB F, (T)) denote the corresponding vector in domain B. We introduce the loss
E (IA) be vector the reference image in the CLIP space. This vector
witii sim U UB),
which is minimized when the two within-domain changes are parallel.
The final loss is then weighted sum of losses
Lelipaacros + within Lelipwwithin Aref_elip Irre_clip + Arei Inef
with empirically determined weights of Aclip_within 0.5, Ayer_elip = 30, and Aref = 10. Together, these four loss terms guide the refinement process for Gg. Among these losses, Lelip_across was proposed by StyleGAN-NADA (Gal et al. (021I. The other losses are novel contributions this work.
Style Mixing After the training step. the generator GB generates images that are semantically similar to the reference image Tg. However, have observed that the visual style may not be

---

Published as a conference paper at ICLR 2022
sufficiently similar. We attribute this to the idea that the target domain may be a subset of the images produced by the new generator Gg. This issue was addressed in StyleGAN-NADA (Gal et al. 202I) using second latent-mining network in order to identify distribution of latent codes within the domain of GB that better match the reference image. Our approach exploits the structure of latent codes in W+ space. Latent vectors in W space can he divided into 18 blocks of 512 elements, each impacting different layer of StyleGAN2. Empirically, the latter blocks of the W+ code have been shown have more effect on the style (e.g. texture and color) of the image whereas the earlier layers impact the coarse-structure or content (Zhu et al. (02T) of the image. We partition the latent code in the image into w = (we, Ws) where we consists of the first m blocks of the W+ latent code that capture the content of the image, and ws consists of the remaining blocks and captures the style. this work, we will use m = 7 unless otherwise specified. Then we transfer the style from a reference image using linear interpolation, to form i = (we; ws) where
úg = (1 a) + a (u
(8)
and us is last (18 m)) blocks of =e Consider the distribution of images generated from random w drawn according to the distribution of latent codes from the mapping network of StyleGAN2. If
O, then the distribution of images Gn (1) includes the reference image, but encompasses wide variety of other fine visual styles. If a = 1, then the images GB( will still have a diverse content, but they will all very closely follow the visual style of Lg. An important application of this method is in conditional editing of real photographs. To achieve that, first we take a real input image Teaa and invert it in domain A using I12S on the generator GA in order to find a W+ latent code Wreal. Then G (Waaa) generates a corresponding image in domain B. We can then compute Wraal by interpolating the style codes (8) so that the final image G ((rrea) similar to Ieaal but has both content and the visual style shifted towards domain B.
4 RESULTS
In this section, we will show qualitative and quantitative results of our work. The only other pub- lished method that accomplishes similar one-shot GAN domain adaptation which we are aware of is Ojha et al. (202T). They focus on few-shot domain adaptation, but they also demonstrate a capa- bility solve the one-shot problem. The most closely related work to our approach is StyleGAN- NADA (Gal et al. (022). which is unpublished at the time of submission, however we compare to it as the main competitor. The paper mainly discusses zero-shot domain adaptation, but the approach can also accomplish one-shot domain adaptation, as demonstrated in their accompanying source- code. Moreover, it demonstrates impressive improvements over the state of the art and even beats many SOTA few-shot methods considering the visual quality. As our method can still significantly improve upon the results shown in EtyleGAN-NADA this underlines the importance of our idea in reducing overfitting. We compare against additional approaches in the appendix.
Given a reference image, the training time for our method is about 15 minutes for 600 iterations on a single Titan XP GPU using ADAM as optimizer with the same settings as Gal et al. (002). We use a batch size of 44 At inference time, there are different appli- cations. In a basic operation, GAN generated images can be transferred with single forward pass through a GAN generator network, which works in 0.34 seconds. Considering a more advanced operation, where existing photographs are embedded into GAN latent space, the additional em- bedding time has to be considered. This embedding time is only 0.22 seconds using e4c (Tov et al.) 202T) and about two minutes using I12S (Zhu et al. 2020b).
Training and Inference Time.
Visual Evaluation. In Fig. we show a comparison of our results on faces against the two most relevant competing methods - StyleGAN-NADA (Gal et al.] 202T) and few-shot-domain- adaptation (Ojha et al., 202T). The results show that our method remains faithful to the original identity of the embedded images in domain A, while the other two methods suffer from overfitting, i..., collapsing narrow distributions which do not preserve salient features (for example the iden- tity of a person). We show additional visual results in the supplemental materials, including results on cars and dogs and results for fine-tuning the domain adaptation.
User Study. We also perform a user study by collecting 187 responses from Amazon Mechanical Turk in order to compare the visual quality and the domain transfer capabilities of our framework
7

---

Published as a conference paper at ICLR 2022
Ours
StyleGAN-NADA
cew-shot-gan-adaptaie
Figure 5: Comparison of our framework with thie-off-hheaas art frameworks for StyleGAN domain adaptation. We compare with StyleGAN-NADA (Gal et al. 202T and the few-shot method of of al. (2021). Each row corresponds to different reference image J, and each column a different real image Treal from domain A. Notice that method is able to match the styles of the reference images, while StyleGAN-NADA fails to maintain the content of the images in domain A (for ex- ample the identity of a person is lost). On the other hand, the few-shot method suffers from severe mode collapse.
compared to the competing methods. When asked which method generates higher quality images from domain B, 73% users preferred our approach to StyleGAN-NADA, and 77% selected ours over Few-shot (Ojha et al.) 202T). When asked which method is better at maintaining the similar- ity to a corresponding source image in domain A, we found that 80% of the responses chose our approach over StyAAGAN-NADA, and 91% preferred our approach to Few-shot. Our method out- performs the competing works in terms of the quality of the generated image, and the similarity of the generated image the source image from domain A. According to the user study, the other methods produced images that are more similar to J, but that is also an indication of overfitting and mode collapse.
Ablation study. We perform an ablation study to evaluate each component of our framework. In Fig. d we show the effect of I12S embedding, different losses and style mixing/interpolationn on the output.
Ref
Dom. A
StyleGA N-nada +T2S Embedding
+ LoyeLeelip
+ Lreduve
+ +ellowwwttin
Style Mixing
Figure 6: Ablation study of the losses and style mixing used in our framework. From left to right: the reference image TA and several images from domain A, the baseline approach (StyleGAN-NADA), adding II2S instead of using the mean of domain A, adding Lnd Letip within and then using style mixing. The top row shows reconstructions the image IA using GB-
8

---

Published as a conference paper at ICLR 2022
Image editing capabilities. Another important aspect of our method is that able preserve the semantic properties of the original StyleGAN (domain A) in domain B. We can make edits to the images in domain B via the learned generator Gn without retraining the image editing frameworks on the new domain. Fig. shows image editing capabilities in the new domain B. We use StyleFlow edits such lighting, pose, gender etc. to show the fine-grained edits possible in the domain.
DOm. A
Ref
Dom. B
Eyeglasses
+Gender
+Pose
Style
Lighting
+ Pose
+Gender
+Style
Figure 7: Image editing capabilities of the new domain B using StyleFlow (Abdal al. 202.bb This figure shows the editing results of the embedded real image Teat transferred to domain B. Notice that method preserves the semantic properties of the original StyleGAN.
Limitations Our method has several limitations (See Fig. 8) Some of these limitations are inher- ent due to the challenging nature of the problem of single-shot domain adaptation. Other limitations can be addressed in future work. First, when we find the initial image domain that corresponds to the input in domain B, do not attempt to control for the semantic similarity. Future work should encourage the images to have similar semantics. Second, we can only transfer between re- lated domains. For example transferring FFHQ faces into the domain of cars paper. Third, also relevant to the original distribution the StyleGAN, embeddings into the Style- GAN work best when the objects transformed to the canonical positions (for example face poses that are the same as FFHQ). Extreme poses of the objects in the reference images sometimes fail.
explored in this
Dom. A
Ref
P
Figure 8: Some failure cases of method. In these examples, we observe that the identity of the face compromised bit more than in typical examples of our method.
CONCLUSIONS
We propose novel method for single shot domain adaption. The main achievement of this work to obtain results of unprecedented quality while reducing overfitting observed in previous work. The technical key components of our work are method model the domain gap vector in CLIP embedding space, a way to preserve within-domain variation, and several extensions for fine- ane-grained seribute-hased control. We also introduce several regularizers and style mixing approach.

---

Published as a conference paper at ICLR 2022
REFERENCES
Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4432 4441, 2019.
Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan+:: How to edit the embedded im- ages? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8296-8305, 2020.
Rameen Abdal, Peihao Zhu, Niloy Mitra, and Peter Wonka. Labels4free: Unsupervised segmenta- tion using stylegan, 2021a
Rameen Abdal, Peihao Zhu, Niloy J. Mitra, and Peter Wonka. Styleflow: Attribute-conditioned images using conditional continuous normalizing flows. ACM Trans. Graph., 40(3), May 2021b. ISSN 0730-0301. doi: 10.1145/3447648. URL https:
exploration of
//doi. 10. 11/ 1445344444444
David Bau, Jun- Yan Zhu, Hendrik Strobelt, Bolei Zhou, Joshua Tenenbaum, William T. Freeman, and Antonio Torralba. Gan dissection: Visualizing and understanding generative adversarial net- works, 2018.
David Bau, Jun- Yan Zhu, Jonas Wulff, William Peebles, Hendrik Strobelt, Bolei Zhou, and Antonio Torralba. Seeing what gan cannot generate. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 4502-4511, 2019.
Adam Bielski and Paolo Favaro. Emergence object segmentation in perturbed generative models. arXiv preprint arXiv:1905. 12663, 2019.
Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan. Domain separation networks, 2016.
Konstantinos Bousmalis, Nathan Silberman, David Dohan, Dumitru Erhan, and Dilip Krishnan. Unsupervised pixel-level domain adaptation with generative adversarial networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jul 2017. doi: 10.1109/vvpr.
2017.18. URL httt::/://d http:://xxxdd..oo///.o.oo www..////.....////P......... 2017.
Eric R. Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein, pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis, 2020.
Hila Chefer, Sagie Benaim, Roni Paiss, and Lior Wolf. Image-based clip-guided essence transfer. arXiv preprint arXiv: 20 12427, 2021.
Edo Collins, Raja Bala, Bob Price, and Sabine Süsstrunk. Editing in style: Uncovering the local semantics of gans, 2020.
Andrea Frome, Greg Corrado, Jonathon Shlens, Sany Bengio, Jeffrey Dean, Mare' Aurelio Ranzato, and Tomas Mikolov. Devise: A deep visual-semantic embedding model. 2013.
Rinon Gal, Or Patashnik, Haggai Maron, Gal Chechik, and Daniel Cohen-Or. Stylegan-nada: Clip- guided domain adaptation of image generators, 2021.
Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pp. 2414-2423, 2016.
Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. Ganspace: Discovering interpretable gan controls. arXiv preprint 200i::2004.025460 2020.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.
Gans trained by two time-scale update rule converge local nash equilibrium. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, pp- 6629-6664 Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781 110860964.
10

---

Published as a conference paper at ICLR 2022
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normal- ization. In ICCV, 2017.
Wonjong Jang, Gwangjin Ju, Yucheol Jung. Jiaolong Yang, Xin Tong, and Seungyong Lee. Style- carigan. ACM Transactions on Graphics, 40(4):1-16, Aug 2021. ISSN 1557-7368. doi: 10.1145/9450066634599666
justinpinkneyttoonffyy Toonify. https//s/ssttttiicccc///////aaaa
Guoliang Kang, Lu Jiang, Yi Yang, and Alexander G. Hauptmann. Contrastive adaptation net- work for unsupervised domain adaptation. 2019 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), Jun 2019. doi: 10.1 10.1109/cy:22219000505 URL http:
//dx. //dx.doi.oorg 10. 110..1111//PPP.
00500
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. arXiv preprint arXiv: 04944, 2018.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4401-4410, 2019.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. In Proc. NeurIPS, 2020a.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of StyleGAN. In Proc. CVPR, 2020b.
Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. CoRR, abs/2106. 12423,, 2021.
Tuomas Kynkäänníemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. CoRR, abs//194.06699 2019.
Kai Li, Yulun Zhang, Kunpeng Li, and Yun Fu. Adversarial feature hallucination networks for few- shot learning. 2020 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2020. doi: 10. 10.11090ew44200022200001133
URL http:://et // doi
cvpr42600. 2020 01348
Ming- Yu Liu, Xun Huang, Arun Mallya, Tero Karras, Timo Aila, Jaakko Lehtinen, and Jan Kautz. Few-shot unsupervised -aaae-tt-imaaeee translation. 2019 IEEE/CV International Conference on Computer Vision (ICCV), Oct 2019. doi: 10.1 10.110991c..2211.0006.. URL http:://ooo doi.
00g//00.109//00 2019 01065)
Sangwoo Mo, Minsu Cho, and Jinwoo Shin. Freeze the discriminator: a simple baseline for fine- tuning gans, 2020.
Jaemin Na, Heechul Jung, Hyung Jin Chang, and Wonjun Hwang. Fixbi: Bridging domain spaces for unsupervised domain adaptation, 2020.
Utkarsh Ojha, Yijun Li, Jingwan Lu, Alexei A. Efros, Yong Jae Lee, Eli Shechtman, and Richard Zhang. Few-shot image generation via cross-domain correspondence, 2021.
Xingang Pan, Bo Dai, Ziwei Liu, Chen Change Loy, and Ping Luo. Do 2d {gan}s know 3d shape? unsupervised 3d shape reconstruction from 2d image {gan}s. In International Confer- ence on Learning Representations, 2021. URL httpt:t/////t....t /tt.ttttss://pperre /wwp//eeeeeeeeeeeeeeee eww net/ forum?iceoo
FGqi DSBUKLO..
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text- driven manipulation of stylegan imagery, 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh. Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.

---

Published as a conference paper at ICLR 2022
Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or. Encoding in style: a stylegan encoder for image-to ponge-to-image translation. arXiv preprint
00f1::2008.009511, 2020.
tinongal/StylGGAN NADA. StyleGAN nada
Stylegan-nada.
http:://gtttb..omm/rimmi
Yujun Shen, Ceyuan Yang, Xiaoou Tang, and Bolei Zhou. Interfacegan: Interpreting the disentan- gled face representation learned by gans. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
Guoxian Song, Linjie Luo, Jing Liu, Wan-Chun Ma, Chunpong Lai, Chuanxia Zheng, and Tat-Jen Cham. Agilegan: Stylizing portraits by inversion-consistent transfer learning. ACM Transactions on Graphics (Proc. SIGGRAPH), jul 2021.
Zhentao Tan, Menglei Chai, Dongdong Chen, Jing Liao, Qi Chu, Lu Yuan, Sergey Tulyakov, and tetti-ipuut-condiioonnn hair image generation for portrait editing, ACM Transactions on Graphics (TOG), 39(4):1-13, 2020.
Nenghai Yu. Michigan:
Ayush Tewari, Mohamed Elgharib, Gaurav Bharaj, Florian Bernard, Hans-Peter Seidel, Patrick Pérez, Michael Zollhofer, and Christian Theobalt. Stylerig: Rigging stylegan for 3d control over portrait images. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6142-6111 6151, 2020a.
Ayush Tewari, Mohamed Elgharib, Mallikarjun BR, Florian Bernard, Hans-Peter Seidel, Patrick Pérez, Michael Zöllhofer, and Christian Theobalt. Pie: Portrait image embedding for semantic control. volume 39, December 2020b. doi:
Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or. Designing an encoder for stylegan image manipulation. arXiv preprint arXiv:2102.0277 2021.
Nontawat Tritrong, Pitchaporn Rewatbowornwong, and Supasorn Suwajanakom. Repurposing gans for one-shot semantic part segmentation, 2021.
Qian Wang and Toby Breckon. Unsupervised domain adaptation via structured prediction based selective pseudo-labeling. Proceedings of the AAAI Conference on Artificial Intelligence, 34
(04):6243-6250, Apr 2020. ISSN 2159-5399.
doi: 10. 10.1609/aaii.334144.0000 //dx. wwx.doi.oom org. 10. 1 609 /AAAI VV4404..609)
URL http:
Zongze Wu, Dani Lischinski, and Eli Shechtman. Stylespace analysis: Disentangled controls for stylegan image generation. arXiv preprint urXiv:2011. 2020.
Richard Zhang, Phillip Isola, Alexci A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric, 2018
Ruixiang ZHANG, Tong Che, Zoubin Ghahramani, Yoshua Bengio, and Yangqiu Song. Metagan: An adversarial approach to few-shot learning. In Bengio, H. Wallach, H. Larochelle, K. Grau- man, N. Cesa- Bianchi, and R. Gameet (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL http::t:tpppppaeeaap
neurips...rpppp.. cc/ 0804 7bc3 Tatccc11a0eeeeeeee7700000aaaaaaa
Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, Jean-Francoir Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler. Datasetgan: Efficient labeled data factory with minimal human effort, 2021.
Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for data-efficient gan training, 2020.
Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou In-domain gan inversion for real image editing. In European Conference on Computer Vision, pp. 992-608. Springer, 2020a.
Peihao Zhu, Rameen Abdal, Yipeng Qin, John Femiani, and Peter Wonka. Improved stylegan em- bedding: Where are the good latents?, 2020b.
Peihao Zhu, Rameen Abdal, John Femiani, and Peter Wonka. Barbershop: Gan-based image com- positing using segmentation masks, 2021.
12

---

Published as a conference paper at ICLR 2022
A APPENDIX: ADDITIONAL RESULTS
A.1 VISUAL EVALUATION OF STYLE TRANSFER.
We provide additional visual evaluation of the results. In Fig. 9 and 10 we show results of domain adaptation applied to faces. The input photographs are in the top row and the reference images are the first column. We can see that the results take on the style of the reference image, even though the reference image is far outside the original GAN's latent space. Also, we notice that overfitting is successfully limited, as each result maintains several important aspects the input image. Fig. 13 and [4 we show results for cars, cats, and dogs on the same task. This shows that our method is consistent across different StyleGAN objects/datasets.
A.2 QUANTITATIVE COMPARISON OF SKETCH IMAGES.
We calculate the FID (Heuell al. 2017) between 1,000 generated images and the entire sketch dataset. Additionally, we report the precision and recall metric (Kynkäänniemi et al. (019) to measure the quality and diversity respectively. As shown in Tab. [ our method outperforms the contemporary methods Few-Shot (Ojha et al. 2021), and StyleGAN-NADE GGa et al. 2021 on
both metrics.
Another contemporary method, TargetCLIP (Chefer et al. 2021 is capable of one-shot 'essence transfer' using latent-edit, however the weights the generator are not modified their approach is restricted to the manifold of the original generator. Because it cannot shift to completely new domain, TargetCLIP failed to produce any sketch images and has a precision=O. Because the images it did generate in the original space of StyleGAN it has high recall (0.29), but this number is not meaningful.
Unsurprisingly, all one-shot domain transfer methods have low recall (low diversity) but it is signif- icant that ours the only approach with positive recall to within 2 significant digits.
Table 1: Quantitative comparison on one-shot adaptation between dew-shot-dommai---dppaeee StyleGAN-NADA, and our method. Evaluation metrics include FID, precision, and recall (higher means higher diversity).
One Shot Method
FID
precision
recall
Few-shot (Ojha et al. SG-NADA (Gal al.
2021)
158.86
0.00
0.00
202T
124.55
0.12
0.00
Ours
78.35
0.33
0.02
A.3 MULTI-SHOT DOMAIN ADAPTATION.
Although it was designed for one-shot domain adaptation, our method can be extended to few-shot domain adaptation by using multiple input/reference image pairs (I LB). In Fig. We show the visual improvement obtained using 3-shot reference images.
A.4 CONTROLLING THE STYLE GAP
Our method provides a way to control the domain gap between the domain and domain B by explicitly controlling the style of the images sampled from or embedded in domain A. [2 shows that we can control the degree to which style from the reference image is preserved by increasing the style-mixing parameter a, which not possible with any of the competing methods. This gives users more control over content generation and editing.
A.5 ADDITIONAL COMPARISON
In addition to our comparison with StyleGAN-NADA (Gal et al. 2021 and few-shot domain adap- tation (Ojha al. 2021), compare against three additional methods in Fig. [5] These include one concurrently developed method called TargetCLIP (Chefer et al., 2021) as well as two other methods that work on lower resolution images for one-shot domain transfer. These are the method
13

---

Published as a conference paper at ICLR 2022
Dom. A
Ref
Figure 9: Style transfer results obtained by our method after style interpolation in domain B. The top row represents the real images embedded in the latent space GA (domain A) whose latent codes then used by Gg (domain B). The first column represents the reference images Ig which are input to our domain adaptation framework.
of Gatys al Gatys et al. (2016) and the the AdaIN approach (Huang & Belongie 2017). Our visual results compare favorably against the new methods in Fig. [3] with respect to preserving the identity of the original image while also generating images that belong the new domain.
14

---

Published as a conference paper at ICLR 2022
A.6 INFERENCE AND EDITING TIME
Our proposed approach uses I12S for training and inference and StyleFlow (Abdl et al., bb for editing in the new domain. GAN inversion using I12S HD (1024 1024) images takes 150 seconds on average, and each latent-code edit operation takes 0,47 seconds. Generating the images afterwards takes an addition 0.34 seconds. Note that the run-time is dominated by GAN -inversion using I12S, however as we show in Fig. [6 once training is completed, we can use other GAN inversion methods to accomplish the edits. With e4e (Tov al. 2021) inversion only 0.22 seconds and the entire process of inversion, editing, and generating the edited image can be accomplished in approximately one second.
15

---

Published as a conference paper at ICLR 2022
Dom. A
Ref
5
Figure 10: The structure of rows and columns is the same in Fig, 9 Note: method also works
well when reference images are real face images.
16

---

Published as a conference paper at ICLR 2022
single-shot
Dom. A
3-shot
Figure 11: Our method extends to deal with multiple reference images. The figure compares the re- sults using reference images and using single reference image. It can observed that our method can better catch the general style and achieve more stable results when given multiple reference images.
17

---

Published as a conference paper at ICLR 2022
Ref
Ref
Dom. A
Style Direction
Dom. Style Direction
Figure 12: Style interpolation results achieved by our framework. Unlike the competing methods, method has an explicit control over the styles in the domain B. Each sub figure shows reference image and images embedded domain A. Notice that we can control the amount of variation in style depending parameter that can be specified by user.
Dom.
Ref
Figure 13: Our domain transfer results on cars. The structure of rows and columns is the same as in
Fig. g
18

---

Published as a conference paper at ICLR 2022
Dom.
Ref
Dom.
Ref
Figure 14: Our domain transfer results on cats and dogs. The structure of rows and columns is the same as Fig. 9
19

---

Published as a conference paper at ICLR 2022
bom
Ref
TargetCLIP
Gatys et al.
AdaIN
Figure 15: Additional comparisons with other baseline methods including the concurrent method TargetCLIP CCeffer et al.] 2021) as well as two lower-resolution methods from Gatys et al. (2016) and AdaIN (Huang & Belongie 2017). One-shot reference images from domain
shown in the
left column. Each image is the result of transferring the image in the top row into the new domain. Compare these images our method in Fig. 7 our proposed approach has fewer artifacts while preserving the identity of the image in domain A.
Dom. B
Style
-Eyeglasses
+Gender
+ Pose
Dom.
Ref
1125
+0.475
e4e
0.22s
10.47 Edits
Inversion
Figure 16: Comparison Nomain-transfer and editing using I12S vs e4e. The new GAN is always trained using I22, but once training is complete, e4e can be used to transfer images into the new domain. II2S takes 2.5 minutes embed the image, while e4e needs about 0.22 seconds. StyleFlow editing takes 0.47 seconds, and StyleGAN image generation takes about 0.34 seconds.
20


Reviews:
[{"review_id": "xybNqxOad2K", "reviewer": "Reviewer_eZmY", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths:**\n\n1. The visualization results demonstrate the proposed method can generate much higher quality images than the baseline StyleGAN-NADA [1], which seems promising to me. The ablation study shows the effect of each regularizer in the loss function.  \n2. The framework itself is clear and novel.  \n3. The training and testing are GPU-friendly and the cost time is acceptable.  \n4. The method is versatile in that it is also equipped with functions like image editing and domain gap controlling.\n\n**Weaknesses:**\n\n1. The biggest problem of the current version of the paper is writing and presentation, especially in Section 3. The writing and presentation is kind of messy to me. Just to mention a few: $v^{\\text{ref}}$ first appears on l.9 in Section 3 and later it is defined in equation (1), which means I do not have any idea what $v^{\\text{ref}}$ exactly is before reading till equation (1). What is $W+$ space? More details of II2S should be provided for readability.  \n2. The compared baselines are simply StyleGAN-NADA [1] and few-shot-gan-adaptation [2]. I think there should be more baseline methods in the style transferring area.  \n3. No quantitative inference time comparison with the baselines are provided.\n\nSome minor points: \n\n(1) Page 5 simply consists of two big figures, leaving much blank space and making the reading experience unsatisfying.  \n(2) There should be a comma at the end of eq (5).  \n(3) In the paragraph \"Training and Inference Time\"  of Section 4, there is something missing in the last sentence \"This embedding time is a few seconds using xx\".  \n(4) There is something wrong with the font style of \"P-norm\" in the paragraph \"latent space interpretation and semantic editing\".\n\n**References**\n\n[1] Rinon Gal, Or Patashnik, Haggai Maron, Gal Chechik, and Daniel Cohen-Or. Stylegan-nada: Clip-guided domain adaptation of image generators, 2021.  \n[2] Utkarsh Ojha, Yijun Li, Jingwan Lu, Alexei A. Efros, Yong Jae Lee, Eli Shechtman, and Richard Zhang. Few-shot image generation via cross-domain correspondence, 2021.  ", "summary_of_the_paper": "This paper proposes a novel method for single shot domain adaptation for GANs. The approach achieves visually better performance than the current SOTA, allows more degrees of freedom, and is meanwhile efficient to deploy. ", "summary_of_the_review": "My current recommendation is borderline leaning to accept. I may change my score in the rebuttal if my concerns are not addressed well.\n\n--------\nThe authors addressed most of my concerns and hence I raise my score to 8. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "bGxxTlrDE6w", "reviewer": "Reviewer_d3yx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. This proposed method by combining StyleGAN and CLIP is elegant.\n2. The experiments demonstrate the effectiveness of their method, and the visual results in the paper are promising.\n\nWeakness:\n\n1. About Figure 6, why use `+` before every semantics?  It is obvious that the glasses were removed from the man's face instead of adding. And what is the difference between the `Ref` and `Dom.B` in the bottom group?\n\n\n==========Update==========\n\nAt first, the results in this paper left an impressive result in my mind, and then I gave a high score. However, after reading some papers that also have been submitted to ICLR2022, which do the same task, compared with them, I found some weaknesses in this paper and therefore changed my score.\n\n1. The whole paper lacks quantitative results, and all the presented results are some cherry-picked images.  And for lacking comparisons with SOTA proposed by other reviewers, although you have added some, there is still a lack of quantitative comparison.\n2. The main experiments are only style transfer on human faces and lack the attributes transfer. This task can not be called `generative domain adaption.`  Although some extra experiments on cars and cats are shown in the appendix, those results are also misleading. For instance, in figure 11, the caption says this is a style transfer task, but the dogs' structure changed vastly.\n3. The diversity of the transferred model is not validated as well, which is an important index to evaluate the generative models.\n4. When compared with other methods, using the CLIP model is kind of unfair.\n", "summary_of_the_paper": "This paper mainly deals with domain transfer tasks in pre-trained GANs, such as transferring the generated images from domain A to domain B by a single reference image from domain B. And experiments show that the proposed method surpasses the SoAT.", "summary_of_the_review": "The results in the paper are a little amazing and can be used in style transfer on human faces.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "JTPcdZTVt0J", "reviewer": "Reviewer_Zd5v", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The proposed method is novel based on best of my knowledge. The paper designed a framework to leverage the prior knowledge in pretrained CLIP, StyleGAN for adaptation. The idea of modelling domain shift as parallel vectors is interesting and seems to be effective.\n- The achieves results are very impressive, notably better than competing methods as demonstrated in fig 4. Also due to the latent space is coupled, it can be combined with works in latent code editing (e.g. Styleflow) for attribute manipulation.\n- The related work part is well-written and gives a comprehensive review of the related topics.\n\nWeakness:\n- Some of the claims are not fully supported by experiments. For e.g. authors claim that \"We greatly reduce the mode collapse/overfitting problem which often occurs in one-shot and few-shot domain adaptation.\", however I found it's not fully sufficient to support this with only fig 4. Besides the user study, perhaps more quantitative measurements can be reported for ablation study etc.\n \n- The writing can be further refined. Currently the methodology part is a bit difficult to follow and many details seem to be missing. In particular, what do the learnable weights in fig 3 represent? Would the authors elaborate further on which parameters are being learned?\n\nMinor questions:\n- For stylegan-nada and few shot gan in fig4, how many reference images are used?\n", "summary_of_the_paper": "The paper proposed a method for adapting a trained GAN to another domain using a single image from the target domain. The main technical contributions include a training framework which leverages pretrained CLIP, StyleGAN for adaptation, several new regularizers and other improvements.\n", "summary_of_the_review": "Overall I think the paper is interesting, as it proposes a novel way for one-shot GAN adaptation. However I do have some concerns about insufficient experimental validation and clarity in details.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a novel method for the single-shot domain adaptation with the help of Generative Adversarial Nets. The proposed method is interesting, novel, and versatile. Moreover, the performance is impressive and better than the existing methods. However, the writing needs some improvement for better readability. More quantitive results should be provided in the revision for completeness.
Decision: Accept (Poster)