Few-shot Hybrid Domain Adaptation of Image Generator
Authors: Hengjia Li
Yang Liu
Linxuan Xia
Yuqi Lin
Wenxiao Wang
Tu Zheng
Zheng Yang
Xiaohui Zhong
Xiaobo Ren
Xiaofei He
Venue: ICLR 2024 poster (2024)
Keywords: Few shot; Domain Adaptation; Image Generator
Primary area: generative models

Abstract: Can a pre-trained generator be adapted to the hybrid of multiple target domains and generate images with integrated attributes of them? In this work, we introduce a new task -- Few-shot $\textit{Hybrid Domain Adaptation}$ (HDA). Given a source generator and several target domains, HDA aims to acquire an adapted generator that preserves the integrated attributes of all target domains, without overriding the source domain's characteristics. Compared with $\textit{Domain Adaptation}$ (DA), HDA offers greater flexibility and versatility to adapt generators to more composite and expansive domains. Simultaneously, HDA also presents more challenges than DA as we have access only to images from individual target domains and lack authentic images from the hybrid domain. To address this issue, we introduce a discriminator-free framework that directly encodes different domains' images into well-separable subspaces. To achieve HDA, we propose a novel directional subspace loss comprised of a distance loss and a direction loss. Concretely, the distance loss blends the attributes of all target domains by reducing the distances from generated images to all target subspaces. The direction loss preserves the characteristics from the source domain by guiding the adaptation along the perpendicular to subspaces. Experiments show that our method can obtain numerous domain-specific attributes in a single adapted generator, which surpasses the baseline methods in semantic similarity, image fidelity, and cross-domain consistency.

Published as a conference paper at ICLR 2024
FEW-SHOT HYBRID DOMAIN ADAPTATION OF IMAGE GENERATOR
Hengjia Li
Yang Liu
Linxuan Xia'
Yuqi Lin'
Wenxiao Wang
Tu Zheng 1
Zheng Yang
Xiaohui Zhong
Xiaobo Ren
Xiaofei Hel
State Key Lab of CAD&CG, Zhejiang University 2 Alibaba Cloud
Zhejiang University
Fabu Inc.
"Ningbo Beilun Third Container Terminal Co., Ltd
Ningbo Zhoushan Port Co., Ltd
ABSTRACT
Can a pre-trained generator be adapted the hybrid of multiple target domains and generate images with integrated attributes of them? In this work, we intro- duce a new task - Few-shot Hybrid Domain Adaptation (HDA). Given a source generator and several target domains, HDA aims to acquire adapted generator that preserves the integrated attributes of all target domains, without overriding the source domain's 's characteristics. Compared with Domain Adaptation (DA), HDA offers greater flexibility and versatility to adapt generators to more composite and expansive domains. Simultaneously, HDA also presents more challenges than DA as we have access only to images from individual target domains and lack authentic images from the hybrid domain. To address this issue, we introduce a discriminator- free framework that directly encodes different domains' images into well-separable subspaces. To achieve HDA, we propose a novel directional subspace loss comprised of a distance loss and direction loss. Concretely, the distance loss blends the attributes of all target domains by reducing the distances from generated images to all target subspaces. The direction loss preserves the characteristics from the source domain by guiding the adaptation along the per- pendicular to subspaces. Experiments show that our method can obtain numerous domain- speucifc attributes in a single adapted generator, which surpasses the base- line methods in semantic similarity, image fidelity, and cross domain consistency. Project page is at /tetttss:/echhppllttt .iroo...iii github..... 1o FHDF prroeeccrooe
INTRODUCTION
Few-shot generative domain adaptation (DA) aims adapt a pre-trained image generator (Karras ct al.) 2019; Brock et al.) [0188] Vahddat et al. 2021 Rombach et al., (2022) from the source domain to new target domain using only few reference images. Existing methods (Mo et al.) 2020 Li et al. 2020; Xiao et al. 2022; Mondal et al.) generally seek to achieve realistic and diverse generation which acquires salient characteristies of the target domain and preserves the variations learned from the source domain. Building upon previous promising progress toward DA, it is straightforward to derive the adapted model given images from sketch domain as shown in Fig. (Left). However, a challenge remains when we require to generate images with integrated attributes of sketch, smile, and baby, given real images from individual domains. Besides, in real-world scenarios, images from the hybrid domain (e.g., smiling baby with the style of sketch) tend to be more difficult to collect compared with single domain. Under such circumstances, conventional DA becomes less feasible.
Instead of DA, we investigate this novel task Few-shot generative Hybrid Domain Adaptation (HDA). Given a source generator and few-shot images of several target domains, HDA aims to acquire adapted image generator that preserves the integrated attributes of all target domains, without overriding the original characteristics. Compared with DA, HDA offers greater flexibility and versatility adapt generators more composite and expansive domains. For example, given the
Equal contribution; Corresponding authors.

---

Published as a conference paper at ICLR 2024
skuteh
skerch, smatc baby
GT.
Gs
GTUTUTT
Domain Adaptation
Source Generator
Hybrid Domain Adaptation (Ours)
Figure I: (Middle) Given a source generator (Gs) pre-trained on a large-scale dataset, we propose to adapt it to the new target domain. (Left) Domain Adaptation adapts the generator from the source domain (human(sketch). The adapted model GT, captures the target distribution using extremely few-shot references. (Right) Our newly introduced Hybrid Domain Adaptation aims to adapt to GTAUTAT that generate images with integrated attributes from Ti (sketch), To (smile), and Ta (baby), given few-shot references from multiple individual domains.
few-shot references from the individual sketch, baby and smile domains shown in Fig. (Right), HDA aims to adapt the generator to keetch-smile-bbby domain. Concurrently, these images retain the primary characteristics of the source domain, which upholds cross-domain consistency.
Compared with conventional DA, HDA is more challenging two aspects: (1) Existing DA ap- proaches typically employ the discriminator to discern whether generated images belong the target domain. However, we only have images sampled from individual domains and lack real images of the hybrid domain, which presents challenges for designing the discriminattr-basee adaptation framework. (2) Since there are extremely few reference images, the discriminator could easily overfit the easy-to-learn characteristics of certain target domains (Ojha et al.) 2021), leading missing the characteristics from other target domains in the generator.
To solve these issues stemming from the reliance on discriminator, we propose diseriminator- framework for HDA. Instead of directly distinguishing whether generated images are real or fake, we endeavor project distinct domains into separate embedding subspaces. Inspired by image classification task (Dosovitskiy et al. 2020 Oquab et al. [2023] Liu et al. (021)) that encodes images into high- nigh-dimensional embedding space and extract wela-separable features, we posit that the pre- trained image encoder intrinsically functions as a hybrid domain classifier. Leveraging this property, we utilize these well-known pre-trained encoders to embed different domains' images into several distinct embedding subspaces.
Employing these well-separable subspaces, we introduce the directional subspace loss comprised of a distance loss and a direction loss to achieve HDA. Specifically, our objective is for generated images to progressively approach all the target subspaces. To this end, we propose the distance loss to minimize the distances from the generated images to all target domains" subspace. Solely employing the distance loss can result in model collapse, since distinct generated images may project onto the same point the subspace and compromise cross-domain consistency. To preserve more characteristics from the source domain, we further propose the direction loss to guide the adaptation easily use our directional subspace loss
along the perpendicular to subspaces. Additionally, for single domain adaptation since DA is a special form of HDA.
We validate the effectiveness of our method across multiple target domains. Compared with other potential approaches for achieving hybrid domain (Wu et al. 2023 Gal et al. 2021). our method accomplishes the fastest adaptation without training models on multiple individual domains. For both DA and HDA, our method achieves superior semantic similarity with the target domain, better image fidelity, and more characteristics preservation of the source domain compared with existing approaches. Overall, our contributions are summarized as follows:
We introduce a novel task Hybrid Domain Adaptation (HDA). Compared with DA, HDA offers greater flexibility and versatility to adapt generators more composite and expansive domains.
2

---

Published as a conference paper at ICLR 2024
We propose novel discriminator-fres framework with directional subspace loss for HDA. Com- pared with other potential approaches, our methods accomplish fast and versatile adaptation without training models on multiple individual domains.
For both DA and HDA, our method surpasses existing methods on the semantic similarity with the target domain, image fidelity, and cross-domain consistency. Qualitative and quantitative results demonstrate the effectiveness of our method.
2 RELATED WORKS
Few-shot Generative Domain Adaptation. Few-shot generative domain adaptation aims to adapt pre-trained image generator to a new target domain with a limited number of reference images. Due to the scarcity of training images, existing works (Mo et al. 2020 Li et al., 2020 Ojha et al. 202 Zhao et al.] 2022b; Xiao et al. 2022; Mondal et al.) often adopt extra regularization terms to avoid overfitting, For example, CDC OOjaa et al., 2021) proposes the instance distance consistency loss to preserve the distance between different instances in the source domain. RSSA (Xiao et al. 2022) proposes relaxed spatial structural alignment method preserve the spatial structural information of the source domain. AdAM (Zhao et al. 2022a) proposes Adaptation-Aware kernel Modulation to address general FSIG of different source- target domain proximity. While previous works achieve promising progress toward generative domain adaptation, they rely fundamentally on the discriminator which makes it difficult to handle hybrid domain adaptation.
Inference Time Interpolation for Hybrid Domain Adaptation. Taking advantage of the disen-
tanglement in the latent space StyleGAN (Härkönen et al. 2020; [hhen & Zhou 202T Xu et al.)
2022, Shen et al. 2020, Wu et al. 2020), some works (Wu et 20033 Nitzan et al. 2023 Gal et al. 2021) are capable of producing images from the hybrid domain through the interpolation technique.
For example, DoRM (Wu et al. 2023) adapts the source generator to each domain individually and then interpolate multiple domain's 's latent codes at inference time to generate images from the hybrid domain. However, the approach of inference time interpolation needs to train models on multiple individual domains, which necessitates multiple times the model size and training time. In contrast, method can acquire one model for the hybrid domain in just few minutes.
Pre-trained Image Encoder as Implicit Domain Classifier. Image encoder has been extensively explored to extract salient and representative features, which typically projects the images into well-separable embedding space for classification. Recently, beginning with VIT (Dosovitskiy et al.,
2020 Caron al. 2021 Oquab et al. 2023 Liu et al. 2021), Transformer-based network achieves revolutionary performance on the image classification challenge. To address large variations in the scale of visual entities and the high resolution of pixels images, Swin (Liu et al.) (202T) proposes a hierarchical Transformer whose representation is computed with shifted windows. On the other hand, self-supervised efforts represented by DINO (Caron et al. 202T) and DINOv2 (Oquab et al.) 2023) have also been devoted to developing effective image encoder. In this paper, we propose to utilize the pre-trained image encoder as an implicit domain classifier to encode the reference images from different domains into the distinct embedding subspace (see Appendix A.
Discriminator-F Domain Adaptation. Style-NADA (Gal et al.. 2021) proposes text-driven DA and presents a discriminator-free method which utilizes CLIP (Radford et al. (02T) model to produce the CLIP-space direction. Although the method can also be applied for image-driven domain adaptation, it suffers from the semantic bias stemming from the CLIP encoder and is not guaranteed to converge to the exact realization of the style such as the sketch domain. Besides, the directional loss with CLIP cannot handle small attribute edits such the sunglasses domain. Instead, we propose the directional subspace loss with the image encoder pre-trained on large classification dataset, which facilitates the adaptation of small attributes and significantly alleviates the bias.
METHOD
3.1 PROBLEM FORMULATION
Following previous works Ojha al. 2021 Xiao et al.) (022). start with pre-trained Style- GAN2 (Karras et al. 2019) generator G that maps from noise 2 to images in a source domain S, and set of N target domains Ti, E {1,-.. N). Domain adaptation (DA) finetunes Gs with few-shot
3

---

Published as a conference paper at ICLR 2024
G8
Y.
fo
A ft fs
2
fz
Ac-pp 8
Y
Encoder
Laist |p
D 21.27
Edirect
Y
Pi
5
| [4
B(Cii A
8GT
Figure 2: The diagram of our discriminator-fii framework with the directional subspace loss Caist and Ldirect Y; is the reference images from the i-th domain. Y, and Y, are images generated by frozen source generator and training target generator. The frozen image encoder extracts features to constitute the subspace
P, After that, we project the target embedding ft onto the subspace P; to obtain f.- Then we minimize the distance between ft and fr. Simultancously, we minimize the angle between f. J' and f fe to guarantee that Se moves along the perpendicular from fa to P..
images Y, from each target domain T; to yield a generator GT,- The adapted Gy; can generate images Y, similar to domain T. Differently. hybrid domain adaptation (HDA) aims to acquire generator Gr that models hybrid domain T = - Ti and generate images Y with integrated attributes.
3.2 DISCRIMINATOR-FREE DOMAIN ADAPTATION
Conventional DA methods commonly incorporate a discriminator differentiate whether the gener- ated images pertain to the target domain. However, we only have images sampled from individual domains and lack real images of the hybrid domain, which poses challenges in the development of discriminator-baat based methods. Moreover, owing the scarcity of reference images (Ojha et al., 2021), the discriminator inevitably exhibits a tendency to overfit to easily learned attributes certain target domains. Consequently, this leads the generator ignore other target domains' attributes within the context of HDA.
To address the issues, we propose discriminator-fier framework for HDA. Specifically, utilize the pre-trained image encoder E, such as Swin (Liu et al.) (002)) and Dinov2 (Oquab et al. (203) in image classification task, encode the few-shot reference images Y, from T; into features f::
f: E(Y), F.
Σ f.
/Ef:
where F; is the average mean of fi and [fal is the cardinality of f. that equals to the number of images in T. As shown in Fig. the features f in fr constitute a separable embedding subspace P. (see Appendix A.I).
3.3 DIRECTIONAL. SUBSPACE LOSS
Then we introduce a directional subspace loss for the adaptation. Technically, we encode the images generated by the frozen source generator Gs and the adapted generator G7 with the same noise z.
Y, Gs(2),
GT(22)
(2)
= E(Y)) = E(Y).
Our objective is for generated images progressively approach the target subspace. Inspired by (Simon et al. 2020), we propose the distance loss from the generated images to the target subspace. Concretely, we first project the embedding Je onto the subspace P. to get the projected point f::
J M,M (Jr Ja) + +J.
where M, the projection matrix computed from SVD factorization of all elements in f.. We then minimize the distance between Je and subspace P. spanned by f::
Laist = däsl( P.) = |
(4)
4

---

Published as a conference paper at ICLR 2024
However, solely employing Ldist still result in model collapse, i.e., distinct generated images may project onto the same point on P. which compromises cross-domain consistency. To preserve more characteristics from Y, we propose to enforce the collinearity between fa. fi, and f; such that f becomes the closest point to f. on the subspace P. To this end, minimize the angle between ft f--Ja fa and f - ft to guarantee that ft moves along the perpendicular from fa to the subspace P::
Do-st
T De-pp f Fe.
Ag-st D
Ldivect
||s--el A
Besides, to alleviate the biases introduced by individual image encoder, we employ the ensemble method that exploits several image encoders during the adaptation:
LDA Egpp(s) BEE where E is the set of pre-trained image encoders and À is the balancing factor.
Σ(Cdis XLdivect),
(6)
3.4 HYBRID DOMAIN ADAPTATION
Naturally, we can extend the directional subspace loss for HDA, since DA is special form of HDA. For the distance loss, we reduce the distances between ft and all target subspaces ensure that the adapted generator owns their attributes:
Clist Σ a -ail
where is pre-defined domain coefficient for the i-th domain.
On the other hand, we also extend the direction loss for HDA preserve the characteristics of the source domain fo. Specifically, we minimize the angle between ft - fa and the weighted sum of fi - Fe (the perpendicular from ft to the i-th subspace):
A
as Fi),
(8)
Ao-2 A/-op
Cdirect
|
Overall, our final objective for HDA is
LHDA Expp(E)
(C'iset + XL'direc):
(9)
BEB
4 EXPERIMENTS
4.1 DATASETS AND METRICS
Datasets: Following previous literature, consider Flick-Facees-HQ (FFHQ) Karras al. 2019 as one of the source domains and adapt to the combination of the following individual target domains (a) FFHQ-baby, (b) FFHQ-sunglasses, (c) sketch, and (d) FFHQ-smile. As in previous works, all our experiments use 10 randomly sampled targets for each domain. Unless stated otherwise, we operate on 256 256 resolution images for both the source and target domains.
Metrics: A key difference between the proposed HDA task against previous DA (Ojha et al. 2021
Mondal al., Nitzan al., 2023) is that there are no real images in the hybrid target domains, In terms of the quantitative eval luation, we focus on these evaluation metrics in our experiments: (1) CLIP-Score (Nitzan et al. 2023 Radford et al. 202I) measures the compatibility which can be thought of as the semantic similarity with the target domain. (2) The Inception Score (IS) is used to assess the fidelity of images, which is practical in the few shot setting (Xiao et al. )022). (3) Identify similarity (ID) measures cross-domain consistency where the adapted images retain consistency with their corresponding source images in domain invariant aspects like pose and identity, following (Zhang et al. 2022 Wu al. 2023
image-text pairs,

---

Published as a conference paper at ICLR 2024
Seq
DoRM
Real sunglasses
Ours
Real smille
Source
Seq
DoRM
Real babies
NADA
Ours
Real
Source
Figure 3: Qualitative results on 10-shot HDA of smile- sunglasss and baby-smile.
Figure 4: Qualitative results on 3 domain 10-shot HDA of sketch-smile-bbby.
4.2 METHODOLOGG AND BASELINES
Methodology:
Following previous domain adaptation (DA) works (Ojha et 2021 Xiao al.
2022, et al. Zhao et al.) 2022b), we apply our method on the StyleGAN2 Karras et al. 2020 architecture. To show the flexibility of our method, we conduct the experiments on both DA and HDA. As described in Section 3.2 we utilize Swin (Liu et al. 2021) and Dinov2 (Oquab et al. 2023 as the image encoder, which are pre-trained on ImageNet 22k dataset (Deng et al. 2009))
Baselines: To demonstrate the effectiveness of our method, we compare our method against the
following baselines that could potentially achieve hybrid domain: (1) Sequential: sequentially adapted to individual target domains to achieve hybrid domain; (2) DoRM: Domain Remodulation (Wu et al.
2023) for hybrid domain as discussed in Section 2 (3) NADA: Following NADA (Gal et 2021), we interpolate the model parameters in different domains to achieve hybrid domain. Additionally for
DA, we compare ours with CDC (Ojha et al. 2021). DoRM (Wu et al. 2023). RSSA (Xiao et al.
2022
AdAM (Zhao et al.] 2022a). and NADA (Gal al. 2021).
4.3
RESULTS ON HYBRID DOMAIN ADAPTATION
Qualitative Results: Fig. Fig.3show, shows the qualitative result of the baseline and our method on HDA, all of
which start from the source domain FFHQ (Karras al. 2019) to the combinations of individual

---

Published as a conference paper at ICLR 2024
smile-sunglass
baby-smile
baby-skerch
Method
CS (1)
(t)
ID
CS (t)
IS
ID (t)
CS (t)
(t)
(t)
Seq
17.92
1.82
0.261
17.36
1.7
0.33
22.69
2.28
0.198
DoRM
18.99
1.94
0.312
16.38
2.12
0.28
17.46
2.69
0.21
NADA
20.23
1.34
0.27
20.45
1.68
0.34
22.06
2.61
0.18
Ours
25.65
2.4
0.373
23.08
2.94
0.34
24.91
2.97
0.223
Table 1: Quantitative evaluations on 10-shot HDA including CLIP-Score (CS), IS, and ID discussed
Section 4.1 Note that indicates higher better.
2-domain
5-domain
10-domain
Method Inter
Dis. free
size (1)
time (U)
size (U)
time
size (L)
Seq
24M
240min
24M
600min
24M
1200min
DoRM
30M
240min
54M
600min
84M
1200min
NADA
48M
6min
120M
15min
240M
30min
Ours
24M
3min
24M
3min
24M
3min
Table 2: Comparison of model size and training time. Note that Inter. and Dis. free indicate the inference time interpolation and diseriminator-free methods.
baby, sketch, smile and sunglasses domains, respectively. For naive Sequential learning, we find that suffers from catastrophic forgetting (McCloskey Cohen 1989) and tends to generate images of the last adapted domain, which fails to generate desired images (e.g.. baby with the style of sketch). DoRM (Wu et al., 2023) are unstable to generate images of the hybrid domain (e.g.. the images of smile-sunglasses lack the attribute of smile and lose the characteristics of the source domain), since the simple interpolation of latent codes at inference time is not enough to semantically align multiple domains. NADA could somehow generate images with integrated attributes, but exhibits significant bias stemming from the CLIP model. For instance, the sketch domain deviates sty stylistically from the reference images, and it fails learn the sunglasses domain. In contrast, our method exhibits desirable performance and shows its stable capability to generate hybrid target images and preserve the characteristics of the source domain. Furthermore, we also conduct experiments on in Fig. 4 where our generated images blend
the hybrid of three target domains seeethh-mmil--aee the domain-specific attributes and preserve the primary characteristics of the source domain, which surpass all the baseline methods. More results for HDA are included in Appendix.
To quantify the quality and diversity of the generated images, we evaluate all methods with CLIP-Score, IS, and ID respectively. As shown in Table our method achieves best scores for all metrics. Especially for CLIP-Score our method significantly outperforms other methods, indicating that generated images with method effectively integrate the multiple attributes from distinct domains. Furthermore, our method achieves better IS and ID, indicating that generated images has higher fidelity and preserve more characteristics of the source domain.
Quantitative Results:
Additionally, we calculate the model size and training time to verify the efficiency of our method as shown in Table Compared with the discriminator baased baselines, obviates ad adversarial training, thereby saving substantial training time. Furthermore, compared with interpolation-basse approaches inference time, our method saves multiple times the model size and training time, completing the adaptation in just three minutes.
discriminator-fir free method
4.4 RESULTS ON SINGLE DOMAIN ADAPTATION
Qualitative Results: To show the flexibility of our proposed method, we also conduct the experiments on DA, as shown in Fig. 5 Similar to the performance on HDA, our method surpasses all baseline approaches. Specifically, previous methods exhibit severe overfitting the few-shot reference images, inadequately preserving source domain attributes, while also demonstrating relatively inferior image fidelity. While NADA demonstrates cross-domain consistency some extent, it exhibits significant
7

---

Published as a conference paper at ICLR 2024
NAN
CDC
DoRM
RSSA
AdAM
NADA
Ours
Real smile
Source
CDC
DoRM
RSSA
AdAM
NADA
Ours
Real sketches
Source
Figure 5: Qualitative results on 10-shot DA smile and sketch.
baby
smile
sunglasses
sketch
Method
(t)
(t)
(S)
(t)
18
ID (t)
CS (t)
IS
ID (1)
CDC
17.84
2.22
0.326
18.93
1.22
0.479
22.19
2.33
0.318
20.83
2.12
0.214
DoRM
17.10
2.30
0.335
17.10
2.18
0.490
22.26
2.59
0.389
19.42
2.06
0.365
RSSA
18.42
2.57
0.238
18.18
2.20
0.410
21.63
2.52
0.256
21.40
1.66
0.231
AdAM
21.18
0.210
19.79
1.61
0.280
22.50
2.50
0.197
21.18
2.22
0.109
NADA
21.52
3.36
0.182
19.95
1.18
0.305
20.95
1.51
0.233
20.9
1.86
0.335
Ours
21.92
3.71
0.353
20.38
2.23
0.587
24.13
2.67
0.414
22.83
2.39
0.417
Table 3: Quantitative evaluations on 10-shot DA.
bias stemming from the CLIP encoder. In contrast, method is capable of accurately acquiring the visual attributes of target domain while retaining strong fidelity and cross-domain consistency.
To demonstrate the effectiveness of our method DA, we evaluate all the methods with CLIP-Score, IS, and ID respectively. As reported in Table 3 our method surpasses all the baselines. Concretely, CLIP-Score Furthermore, outperforms priors on and ID, indicating that our generated images have better fidelity and cross-domain consistency.
Quantitative Results:
method achieves higher semantic similarity with the target domain in
8

---

Published as a conference paper at ICLR 2024
10
v/o Cars!
64
viti
0/2
Source
w/o Catrect
with Cdirect
Figure 6: Qualitative ablation for Ldirect sunglasses.
Target
Source
NADA
Ours-
Ours-Ensembbr
Figure 7: Qualitative ablation for the ensemble sketch.
4.5
ABLATION STUDY
First, we conduct the ablation study to ver- ify the effectiveness of our proposed direc- tional subspace loss. As shown in Fig. 6 Ldirect greatly improves the cross-domain consistency (e.g., clothes and hair). Consis- tently, we observe that the adaptation with directional loss maintains higher ID scores throughout training. Quantitatively, we report the ablation for Ldist and Lairect in T It can be clearly observed that the Lasee and Ldsrect significantly improve image fidelity and cross-domain consistency. Besides, we report the qualitative ablation for the encoders ensemble in Fig. 7 Our method with Swin (Liu et al.) 202T) significantly alleviates the bias compared with NADA (Gal al. (02T) and the ensemble technique further help to converge to the exact realization of the style of the sketch domain.
sunglass
baby
Laiat
Ldivect
18 (f)
ID
(5)
ID (f)
1.91
0.249
2.56
0.178
2.61
0.372
3.5
0.304
2.67
0.414
3.71
0.353
Table 4: Ablation for Laist and Litivest on 10-shot DA.
CONCLUSION & LIMITATION
Conclusion. In this paper, we propose a new task - Hybrid Domain Adaptation (HDA), which aims to adapt pre-trained generator the hybrid target domain generate images with integrated attributes. To achieve this, we introduce a novel discriminator-frer framework, which utilizes the pre- pre-trained image encoder to encode the reference images from different domains into separate embedding subspaces. Furthermore, we introduce a directional subspace loss to guide the adaptation and retain cross-domain consistency. We believe that our work is an important step towards few-shot HDA since have demonstrated that the source generator can be effectively adapted to hybrid domain without accessing the real images from the hybrid domain.
Limitation. While our method realizes HDA and achieves better superior metrics compared with state-of-the the-thh-rt DA methods, also has the limitation. We observe that for hybrid domain adaptation, the weights of different domain in our loss not entirely equivalent to the proportionality of visual characteristics, which needs empirical adjustment. Nevertheless, we believe that some solutions could be integrated into our HDA pipeline to facilitate the alignment in the future.

---

Published as a conference paper at ICLR 2024
6 ACKNOWLEDGEMENT
This work was supported part by The National Nature Science Foundation of China (Grant No: 62303406), in part by Yongjiang Talent Introduction Programme (Grant No: 2023A-194-G).
REFERENCES
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. ICLR, 2018.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings the IEEE/CVF international conference on computer vision, pp. 9650-9660, 2021.
Eric R Chan, Connor Z Lin, Matthew Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry-awen 3d generative adversarial networks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp- 16123-16133, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-248222 255. leee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 11929, 2020.
Rinon Gal, Or Patashnik, Haggai Maron, Gal Chechik, and Daniel Cohen-Or. Stylegan-nada: Clip-guided domain adaptation of image generators. arXiv preprint arXiv:2108.0000 2021.
Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. Ganspace: Discovering interpretable gan controls. Advances in Neural Information Processing Systems, 33:9841-9850, 2020.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401-4410. 2019.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 10-8119, 2020.
Yijun Li, Richard Zhang, Jingwan (Cynthia) Lu, and Eli Shechtman. Few-shot image generation with elastic weight consolidation. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 15885-15896. Cur- ran Associates, Inc., 2020. URL
www.rnngw....ruueeeeee neur iss.iips... ps. cc cctpppppeee2e0//77//////
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021.
Michael McCloskey and Neal Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 165. Elsevier, 1989.
Sangwoo Mo, Minsu Cho, and Jinwoo Shin. Freeze the discriminator: a simple baseline for fine- tuning gans. In CVPR Al for Content Creation Workshop, 2020.
Amab Kumar Mondal, Piyush Tiwary, Parag Singla, and AP Prathosh. Few-shot noss-domain image generation via inference-time latent-code learning. In The Eleventh International Conference on Learning Representations.
10

---

Published as a conference paper at ICLR 2024
Yotam Nitzan, Michaël Gharbi, Richard Zhang, Taesung Park, Jun- Yan Zhu, Daniel Cohen-Or, and Eli Shechtman, Domain expansion of image generators. arXiv preprint arXiv:2301 230202200 2023.
Utkarsh Ojha, Yijun Li, Jingwan Lu, Alexii A Efros, Yong Jae Lee, Eli Shechtman, and Richard Zhang. Few-shot image generation via cross-domain correspondence. In Proceedings of the IEEF/CVF Conference Computer Vision and Pallern Recognition, pp. 10743-10752, 2021.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprini arXiv:2304.07193, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. PMLR, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22500-22510, 2023.
Yujun Shen and Bolei Zhou, Closed-form factorization of latent semantics in gans. Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 1532-153000 2021.
Yujun Shen, Ceyuan Yang, Xiaoou Tang, and Bolei Zhou. Interfacegan: Interpreting the disentan- gled face representation learned by gans. IEEE transactions on pattern analysis and machine intelligence, (4(4):2004-2011 2020.
Christian Simon, Piotr Koniusz, Richard Nock, and Mehrtash Harandi. Adaptive subspaces for few-shot learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4136-4145, 2020.
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. Advances in Neural Information Processing Systems, 14:11287-1130, 2021.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
Yi Wu, Ziqiang Li, Chaoyue Wang, Heliang Zheng, Shanshan Zhao, Bin Li, and Dacheng Ta. Domain re-modulation for few-shot generative domain adaptation. arXiv preprint arXiv:2302.02550, 2023.
Zongze Wu, Dani Lischinski, and Eli Shechtman. StyleSpace analysis: Disentangled controls for
StyleGAN image generation. arXiv:201 12799, 2020.
Jayy Xiao, Liang Li, Chaofei Wang, Zheng-Jun Jun Zha, and Qingming Huang. Few shot genera- tive model adaption via relaxed spatial structural alignment. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 11204-11213, 2022.
Guodong Xu, Yuenan Hou, Ziwei Liu, and Chen Change Loy. Mind the gap in distilling stylegans. In Computer Vision-ECCCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XXXIII, pp. 423-439. Springer, 2022.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large- scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506 500.. 550.0333,6 2015.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 586-595, 2018.
11

---

Published as a conference paper at ICLR 2024
Zicheng Zhang, Yinglu Liu, Congying Han, Tiande Guo, Ting Yao, and Tao Mei. Generalized one-shot domain adaptation of generative adversarial networks. Advances in Neural Information Processing Systems, 35:13718-13730, 2022.
Yunqing Zhao, Keshigeyan Chandrasegaran, Milad Abdollahzadeh, and Ngai-Man Man Cheung. Few- shot image generation via adaptation-aware kernel modulation. Advances in Neural Information Processing Systems, 35:19427-19440, 2022a.
Yunqing Zhao, Henghui Ding, Houjing Huang. and Ngai-Man Cheung. closer look at few-shot image generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9140-9150, 2022b.
12

---

Published as a conference paper at ICLR 2024
30
baby
sketch
20
sunglasses
10
0
-10
-10
-20
baby
sketch
30
sunglasses
20
-10
10
10
20
(a) Swin
(b) Dinov2
Figure 8: T-SNE visualization of the embedding space encoded by Swin (Liu et al. 2021 and
Dinov2 (Oquab al. 2023 for images from baby, sketch, and sunglasses domain.
Intra-LPIPS (t)
smile
sketch
sunglasses
NADA
0.529
0.491
0.535
CDC
0.616
0.453
0.562
RSSA
0.625
0.480
0.563
AdAM
0.615
0.495
0.598
Ours
0.642
0.506
0.615
Table 5: Intra-cluster pairwise LPIPS distance 10-shot DA.
A APPENDIX
A.1 EMBEDDING SPACE ANALYSIS
Since the image encoder in image classification typically projects the images into well-separable embedding space, such as Swin (Liu et al., (021) and Dinov2 (Oquab et al., (023). we can employ the separable subspaces achieve HDA. To validate that, visualize the embeddings of real images from multiple domains by t-SNE (Van der Maaten & Hinton 2008). As shown in Fig. X distinct subspaces from different domains manifest separation, which facilitates us adapt the generator to the target subspace corresponding to the target domain.
A.2 INTRA-LPIPS
Following CDC (Ojha et al. 2021). we utilize Intra-cluster pairwise LPIPS distance (Zhang et al. 2018) evaluate the diversity of generated images for single domain adaptation. Specifically, 1000 generated images and 10 real images from the target domain to compute Intra-L.PIP.. As shown Table our method achieves higher Intra-LPIPS distance than baseline methods, indicating more distinct images being generated.
A.3 MORE QUALITATIVE RESULTS
As the complement of qualitative results in the main paper, we conduct the experiments on more target domains. We present the results in Fig. and Fig. for DA and HDA respectively. It can be observed that the generated images generated by our methods integrate the attributes, which significantly surpasses the baseline methods.
13

---

Published as a conference paper at ICLR 2024
Seq
DoRM
Real skatches
NADA
Ours
Real babies
Source
Seq
DoRM
Real sunglasses
NADA
Ours
Real babies
Source
Seq
DoRM
Real sketches
NADA
Curs
Real smile
Source
Figure Qualitative results on 10-shot HDA.
A.4 IMPLEMENT DETAILS
As depicted in Eq. and Eq. S). pre-defined domain coefficient to modulate the attributes from multiple domains. For most experiments on two domains, we use a; 0.5 except for baby- sunglasses with abaaby 0.. and Osunglasses 0.7. For the experiment of three domains, set Obbbby 0.7, Osketth = 0.4, and Osymll 0.3.
Following the setting of previous DA methods, utilize the batch size of 4 and training session typically requires 300 iterations in roughly 3 minutes on a single NVIDIA TITAN GPU. Besides, we set the balancing factor À as 1 in Eq. (6) and Eq. (9.
14

---

Published as a conference paper at ICLR 2024
CDC
DoRM
RSSA
AdAM
NADA
Ours
Real aanglssess
Source
DoRM
RSSA
AdAM
NADA
Ours
Real babies
Source
Figure 10: Qualitative results on 10-shot DA of sunglasses and baby.
Domain
Prompt
baby sketch sunglasses
"face of baby"
"face of person with the style of sketch"
"face of person with sunglasses"
smile
"face of person with smile"
emile-sunglasser baby-smile
"face of person with smile and sunglasses"
"face of a baby with smile"
baby- sketch
"face of a baby with the style of sketch"
Table 6: Prompts of different domains for CLIP-Score.
A.5 PROMPTS FOR CLIP-SCORE
To measure the semantic similarity with the target domain, adopt CLIP-Score as the evaluation metric in Section 4.1 As shown in Table 6 we present the prompts corresponding to different target domains.
15

---

Published as a conference paper at ICLR 2024
Babg
Smile
Sunglassea
+
Smill
ReL.
Source
10-shot
5-shol
1-shot
Figure 11: Impact of different shots for our method in HDA. Results 5-shot are close to those of 10-shot. Compared with them, the results of 1-shot exhibit relatively lower cross-domain consistency.
10-shot
Source
Red hair
Red hair Smile
Mask
Source
Sunglasses
Sunglasses-
Conditional generation of our method in DA and HDA. We adapt the generator with masked images for red hair and sunglasses respectively. The red masks represent red hair, while the blue masks represent faces wearing sunglasses.
Figure 12:
A.6 RESULTS USING DIFFERENT NUMBERS OF SHOTS
We perform experiments lower shots (5-shot and 1-shot) as suggested using various datasets. As shown in Fig. the results of 5-shot are close to those 10-shot, which integrates the attributes and maintains the consistency with source domain. Although multiple attributes have been learned, the results of I-shot exhibit relatively lower cross-domain consistency compared with 10-shot and 5-shot. This is because when there is only reference image per domain, the subspace in directional subspace loss degenerates into a single point (see Fig. 2). Then distinct generated images corresponding different noise tend to converge towards the same image in CLIP's embedding space, which comprises cross-domain consistency as depicted in Section 3.3
16

---

Published as a conference paper at ICLR 2024
Baby
Raby
Smile
10-abot
Source
Swin
Dinov2
Ensemble
Figure 13: Effect of different pre-trained image encoders in Fig. e.g,, Swin, Dinov2, and their ensemble.
A.7 RESULTS OF CONDITIONAL. GENERATION
As shown in Fig. 12 we conduct the experiments on conditional generation. Specifically, we collect masks to separate these attributes and
10-shot images with red hair and sunglasses. Then adapt the generator with masked images for red hair and sunglasses respectively. We can observe that the generated images possess the corresponding attribute for both single DA and hybrid DA. Simultaneously, these images also maintain consistency with source domain.
A.8 EFFECT OF DIFFERENT PRE-TRAINED IMAGE ENCODERS
As shown in Fig. 13 conduct experiments pre-trained Swin and Dinov2 explore the impact of different image encoders on generated images. Our method agnostic to different pre-trained image encoders. Although they exhibit slight stylistic differences, these due to their different approaches to extract features into separated subspaces, as depicted in Fig. g To converge to the exact realization the target domain, our method employs the ensemble technique that exploits both Swin and Dinov2. As shown in the figure, the results closely resembles the attributes of the target domain while maintaining the best consistency with source domain.
A.9 TRAVERSAL FOR DOMAIN COEFFICIENT
In our method for hybrid domain adaptation, this parameter controls the composition ratio of the attribute from each domain. As depicted in Appendix AA.4 we use a; 0.5 for most experiments without the need for complex and intricate adjustments. To further explore the sensitivity. we conduct the study for simple traversal of a; As shown Fig. [4] the attributes of generated images transit smoothly between domains. Our method produces the similar attribute blending effect when ag {0.4, 0.5, 0.6).
A. THE RESULTS OF MORE DOMAINS
Consistent with prior work for few-shot generative domain adaptation (like CDC (Ojha et al. 202 RSSA (Xiao et al. 2022) and DoRM (Wu et al. 2023) ), our experimental data encompasses both global style (sketch and baby) and local attributes (smile and sunglasses). The combination of these domains demonstrates the effectiveness our method since they encompass all types of generative domain adaptation.
17

---

Published as a conference paper at ICLR 2024
Baby
Smile
Sunglasses
Smile
0
0,2 0.4
0.5
0.6
= 0.8
Figure 14: The study of simple traversal a; for hybrid domain adaptation. The attributes of generated images transit smoothly between two domains. Our method produces the similar attribute blending effect when og {0.4, 0.5, 0.6}. Note that Olbaby and Olanglasses are Qsmile.
10-shot (Naphae!)
Souree
Raphael
Raphael-Hoote
Heepael-Smiic
10-shot (Caricature)
Suurce
Caricature
Coricature-Baby
Carioatare-Sletehh
Figure 15: Single and hybrid domain adaptation Raphael and Caricature.
To provide more comprehensive evidence validity, we conduct additional experiments on Raphael and Caricature for both single and hybrid domain adaptation. As shown in Fig. [5] the results integrate the characteristics from multiple target domains and maintains robust consistency with source domain, which further demonstrates the effectiveness of method.
A. THE GENERALIZABILITY TO OTHER DOMAINS
To verify the generalizability of our method to other domains, we conduct experiments on church domain following prior work (Ojha et al. 2021 Xiao et al. 2022 Wu et al.) 2023 RSSA (iia et al. (022) and DoRM (Wu et al. (023)). We adapt the pre- trained generator from LSUN Church (Yu et al. 2015) Van Cogh's 's house paintings, haunted houses, and the combination of them. As shown in Fig. [6 the results acquire the corresponding style and showcase the preservation of good
cross-domain consistency. This aligns with the results observed in the face domain.
18

---

Published as a conference paper at ICLR 2024
10-shot ( Van Gogh)
10~shot (Haunted)
Source
Van Gogh
Van Gogh- Haah--Hnnted
Hounted
Figure 16: Results of domain adaptation from LSUN Church to Van Gogh's 's house paintings, haunted houses, and the combination of them.
Input
Blue eyes
Bowlcut
Pose
Pose
Bowleut
Source
Baby
Baby
Smile
Figure 17: Editing results on the images generated by original and adapted model for both single and hybrid domain adaptation. The first line is the source images. The second and third lines are images of the adaptation.
A.12 THE EDITABILITY BEFORE AND AFTER THE DOMAIN ADAPTATION
We conduct the editing the images generated by original and adapted model for both single and hybrid domain adaptation. As shown in the Fig. 17 the results indicate the adapted generator maintains similar editability like pose the original generator. This verifies that the method effectively preserves original generator's attributes.
19

---

Published as a conference paper at ICLR 2024
Source
Sunglasses
Sunglasses-Sne Smill
Figure 18: Single and hybrid domain adaptation in 3D GAN. We adapt pre-trane pre-trained EG3D (Chan
al. 2022 with our method using 10-shot training images per domain.
A.13 SINGLE AND HYBRID DOMAIN ADAPTATION IN 3D GAN.
We conducted experiments using the popular 3D-av aware image generation method, EG3D (Chan et 2022). Specifically, we replace the discriminator we did in Fig. for both single and hybrid domain adaptation. As shown in Fig. T88, we adapt the pre-trained generator from FFHQ to sunglasses and the hybrid of sunglasses and smile with 10-shot training images per domain. We can observe that the results effectively integrate the attributes and preserve the characters and poses of source domain.
A.14 COMPARISON TO METHODS FOR DIFFUSION MODEL PERSONALIZATION
Current trend in customized text-to-image models like DreamBooth Ruiz et al. 2023) aim to mimic the appearance of subjects in a given reference set. Similar to ours, DreamBooth fine-tunes a pre-trained generator for the personalization. However, there are two key differences between personalization and HDA.
(I) DreamBooth aims to retain the individuals from the training images, requiring similar individuals across the training set. Differently, our goal is to acquire the attributes of target domain like and preserve consistency with the source generator, ensuring that images generated from the same noise exhibit similar individuals.
(2) Dreambooth utilizes text-to-image generator which necessitates intricate and laborious adjust- ments of prompts synthesize images with specific attributes. Additionally, certain attributes challenging to accurately describe using text, such as artistic paintings. Conversely, our adapted model adeptly preserves the domain-specift attributes of reference images without the need for intricate prompt engineering.
20


Reviews:
[{"review_id": "u2SOmGevjE", "reviewer": "Reviewer_fvXy", "summary": "The paper introduces the few-shot generative Hybrid Domain Adaptation (HDA) task for image-based data. It is assumed that an image generator is accessed from a source domain and then from several target domains. Adaptation is performed on all target domains. To perform HDA, the paper presents a discriminator-free approach with a style GAN pre-trained on a source domain. It is adapted to individual target domains using a few shots from each domain and generates an image style showing composite attributes of each target domain separately.  The proposed approach relies on features extracted from standard transformer-based architectures. In addition, distance and direction losses are used to guide the generator to produce images with integrated features of the target domains while preserving the features of the source domain. Experiments are conducted using the FFHQ dataset and three evaluation metrics are used to measure performance and compare with different baselines.", "questions": "- How would the proposed generator compare with well-known generator models such as DALLE, Imagen and etc.?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written and easy to understand. The method is well described and the experiments are clearly presented. \n\n- The discriminator-free approach presented is interesting as it reduces the computational cost of fitting compared to existing approaches. \n\n- The approach shows promising results for all metrics compared to the baselines on the FFHQ dataset.", "weaknesses": "- (Major) The paper does not provide strong arguments as to why hybrid domain adaptation is a meaningful task. It is not clear why the current approaches and benchmarks, e.g. domain adaptation and domain generalisation, are not sufficient to address the problem presented. These points need further elaboration to motivate the paper. \n\n- (Major) results are reported using 10 shots, while it would be interesting to see results using different numbers of shots, e.g. 1-shot and 5-shot.\n\n- It is unclear whether the methodology is applicable to the generation of conditioned image styles. For example, generating an image that is conditioned on one or more styles of the target domain, rather than an image that spans all attributes of the target domains. It could discussed how the proposed methodology generalises to conditional generation. \n\n- A study of the effect of different pre-trained image encoders would add value to the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vREsJRVeuJ", "reviewer": "Reviewer_2zhA", "summary": "Studies the problem of adapting an image generator (GAN) to additionally incorporate the attributes of multiple (hybrid) target domains with access only to individual target domain data. This is accomplished by optimizing a directional subspace loss that blends target attributes without mode collapse. Presents results on several generative DA benchmarks.", "questions": "Please address the weaknesses listed above.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "– The proposed problem – hybrid generative DA –  is interesting, and well-motivated. It is plausible that it might be easier to collect data for several independent target domains rather than for each possible combination thereof. \n\n– The paper is well-written and very easy to follow. The presented qualitative results are compelling and experimental results are convincing, with strong+ and recent baselines such as DoRM\n\n– The proposed method is simple, intuitive, and easy to follow", "weaknesses": "– I’m unsure about why this paper claims to introduce Hybrid generative DA, after acknowledging that prior work studying this problem exists (eg. DoRM). I understand that DoRM necessitates training target-specific models, but do they not also present some results on hybrid generative DA (eg. Figure 1 of their paper)?\n\n– The approach requires manually tuning $\\alpha_i$, and does not provide a principled method for hyperparameter selection. How sensitive is the approach to this parameter?\n\n– The paper would be strengthened by a discussion and additionally, a comparison to recent methods for diffusion model personalization (eg. Dreambooth). While the methods would not necessarily be directly comparable, it would be a valuable point of reference\n\n– The paper would benefit from a more detailed conceptual comparison to Nitzan et al., CVPR 2023: particularly, their approach of repurposing dormant directions in latent space for a new domain, seems quite related to the proposed approach\n\n– The training time comparisons in Table 2 lack sufficient details: were the baseline numbers obtained by running publicly available code? Is it a completely apples-to-apples comparison, i.e. have all necessary parameters (eg. number and type of GPUs) been controlled for?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5jaWJ9R8ii", "reviewer": "Reviewer_bxKn", "summary": "The paper proposes a method called  Hybrid Domain Adaptation (HDA), which aims to align a GAN generator to several target domains without affecting the attributes of the source domain. As opposed to domain adaptation, this task makes sure that the attributes from the target domains do not override the attributes of the source domain. So, in essence, it is a more difficult task than conventional domain adaptation where only two domains are involved. To approach this problem, the paper proposes to use two ideas 1) use a discriminator-free framework that directly encodes different domains’ images into well-separable subspaces.  2) The paper proposes to use two types of CLIP-based losses i) distance loss to make sure the features of the source and the target domains are aligned and ii) direction loss to push the features of the target domain perpendicular to the source domain. Various qualitative results are shown throughout the paper. The paper also compares quantitatively with other domain adaptation methods.", "questions": "1) Can you explain the rationale behind focusing on a limited set of attributes (age, smile, and sketch) and how this choice demonstrates the validity of the proposed method in diverse domains?\n\n2) Given that the method mainly revolves around face datasets and lacks face-specific components, how does this impact the generalizability of the method to other domains?\n\n3) Could you discuss and provide insights into the advantages and disadvantages of your method compared to other approaches like DualStyleGAN and 3DAvatarGAN, which utilize few-shot domain adaptation with discrimination?\n\n4) In cases where attributes like smile and age can be manipulated using existing StyleGAN editing methods, what benefits does your hybrid domain adaptation method bring to these scenarios?\n\n5) To determine whether the original generator's attributes are retained, could you elaborate on the results of comparing editability before and after the domain adaptation task? \n\n6) How does the method compare with \"Mind the GAP\" and \"StyleCLIP,\" which employ similar loss mechanisms and concepts in domain adaptation?", "limitations": "", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The paper proposed a task more challenging than domain adaption. This task aims to adapt multiple domains in a single generator without affecting the characteristics of the original generator. To achieve this, the paper employs a discriminator-free approach and CLIP-based losses to ensure the fidelity of the outputs.\n\n2) The paper shows results in a few shot settings and compares them with the competing domain adaptation methods. The quantitative results show that the proposed method is also faster than these methods.\n\n3) In the context of the face domain, the paper uses to adapt sketch images which have a significant domain gap with the faces. The results show better domain adaptation compared to the competing methods. The authors also show ablation of the components used in their method.", "weaknesses": "1) One notable concern is the lack of diverse domain results in the paper. The primary focus of the paper revolves around attributes like age, smile, and sketch, which are maintained throughout the paper. This limited scope may not provide sufficient evidence to establish the method's overall validity.\n\n2) Building on the previous point, the paper predominantly concentrates on face datasets, which restricts the method's evaluation. This is particularly problematic since the method itself lacks face-specific components. \n\n3) The paper overlooks discussions and validation concerning the advantages and disadvantages of employing few-shot domain adaptation techniques with discrimination other than StyleGAN-NADA, as seen in methods such as DualStyleGAN and 3DAvatarGAN.\n\n4) Attributes like smile and age can be manipulated using existing StyleGAN editing methods. It's challenging to discern why hybrid domain adaptation is necessary in these cases.\n\n5) To ensure that the original generator's attributes are preserved, a useful approach could involve comparing the editability before and after the domain adaptation task. For instance, evaluating the ability to edit attributes like pose using a 2D generator. Such evaluation is lacking in the paper. This also raises the question of how the method would perform in a 3D GAN setting.\n\n6) It is worth noting that there are other relevant papers in the field, such as \"Mind the GAP,\" which employ similar loss mechanisms to constrain domains. Additionally, \"StyleCLIP\" introduces directional loss terms based on CLIP.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)