Few-shot Image Generation via Adaptation-Aware Kernel Modulation
Authors: Yunqing ZHAO
Keshigeyan Chandrasegaran
Milad Abdollahzadeh
Ngai-man Cheung
Venue: NeurIPS 2022 Accept (2022)
Keywords: Few-shot Image Generation
Modulation of Deep Neural Networks
Transfer Learning
Knowledge Preserve

Abstract: Few-shot image generation (FSIG) aims to learn to generate new and diverse samples given an extremely limited number of samples from a domain, e.g., 10 training samples. Recent work has addressed the problem using transfer learning approach, leveraging a GAN pretrained on a large-scale source domain dataset and adapting that model to the target domain based on very limited target domain samples. Central to recent FSIG methods are knowledge preserving criteria, which aim to select a subset of source model's knowledge to be preserved into the adapted model. However, a major limitation of existing methods is that their knowledge preserving criteria consider only source domain/source task, and they fail to consider target domain/adaptation task in selecting source model's knowledge, casting doubt on their suitability for setups of different proximity between source and target domain. Our work makes two contributions. As our first contribution, we re-visit recent FSIG works and their experiments. Our important finding is that, under setups which assumption of close proximity between source and target domains is relaxed, existing state-of-the-art (SOTA) methods which consider only source domain/source task in knowledge preserving perform no better than a baseline fine-tuning method. To address the limitation of existing methods, as our second contribution, we propose Adaptation-Aware kernel Modulation (AdAM) to address general FSIG of different source-target domain proximity. Extensive experimental results show that the proposed method consistently achieves SOTA performance across source/target domains of different proximity, including challenging setups when source and target domains are more apart. Project Page: https://yunqing-me.github.io/AdAM/

Few-shot Image Generation via Adaptation-Aware Kernel Modulation
Yunqing Zhao
Keshigeyan Chandrasegaran' keshigeyandsutd. edu.
sutd. edu.
Milad Abdollahzadeh*
Ngai-Man Cheung!
milad_abdollll wtd..aaabbalaaadeeeuduu......... edw.. edu.sg
ngaiman_ chaaaanncchunnuuuuucccaeo edu
Singapore University of Technology and Design (SUTD)
Abstract
Few-shot image generation (FSIG) aims learn to generate and diverse sam- ples given an extremely limited number of samples from a domain, egg., 10 training samples. Recent work has addressed the problem using transfer learning approach, leveraging a GAN pretrained on a large-scale source domain dataset and adapting that model to the target domain based very limited target domain samples. Central to recent FSIG methods are knowledge preserving criteria, which aim to select subset of source model's knowledge be preserved into the adapted model. However, major limitation of existing methods is that their knowledge preserving criteria consider only source domain/source task, and they fail consider target domain/udqptation task in selecting source model's knowledge, casting doubt on their suitability for setups of different proximity between source and target domain. Our work makes two contributions. As our first contribution, we revisit recent FSIG works and their experiments. Our important finding is that, under setups which assumption of close proximity between source and target domains is relaxed, existing state-of-the-ar (SOTA) methods which consider only source domain knowledge preserving perform no better than baseline fine-tuning method. To address the limitation cxisting methods, as our second contribution, we propose Adaptation-Ail kernel Modulation (AdAM) to address general FSIG of different source-target domain proximity. Extensive experimental results show that the proposed method consistently achieves SOTA performance across source/target domains of different proximity, including challenging setups when source and target domains more apart. Project Page: https://yunqing me. iothub.io/AdAM/
1 Introduction
2 3 have been applied to a range of important 5 image-to-image translation [6. 7. image editing 8.9,. anomaly detection [10], and data augmentation [2]. However, a critical issue that these GANs often require large-scale datasets and computationally expensive resources to achieve good performance. For example, StyleGAN 4 is trained on Flickr-Faces-HC (FFHQ) 4 that contains 70,000 images. However, in many practical applications only few samples available (e.g., photos rare animal species skin diseases). Training a generative model problematic in this low-data regime, where the generator often suffers from mode collapse blurred generated images To address this,few-shi. image generation (FSIG) studies the possibility of generating
Generative Adversarial Networks (GANs) T
applications including image generation 4
13 14 [5
"Equal Contribution
'Corresponding Author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Table 1: Transfer learning for few-shot image generation: Various criteria proposed to augment baseline transfer learning to preserve subset of source model's knowledge into the adapted model.
Method
Knowledge preserving criteria
Source domain/task aware
Target domain/sdaptation aware
IGAN [T6
Not available
FreezeD
Preservation of lower layers of the discriminator pre pre-trained on the source domain.
17
x
EWC 18
Preservation of weights important the source generative model pre-trained on the souree domuin.
/
CDC 14
Preservation of pairwise distances of generated images by the source the source domain.
generative model pre- pre-trained Preservation of muliieeve semantic diversity of the generated images by the source generative model pre-trained on the source domain. Preservation of kemels important in adaptation of source model to target.
DCL 19
/
AdAM
(Our work)
sufficiently diverse and high quality images, given very limited training data (e.g.. 10 samples), FSIG also attracts an increasing interest for some downstream tasks, e.g., few-shot classification [12
FSIG with Transfer Learning, Recent works in FSIG are based on transfer learning approach 20 i..., leveraging the prior knowledge of a GAN pretrained on a large-scale, diverse source dataset (e.g., FFHQ [4] or ImageNet (2l) and adapting it to a target domain with very limited samples (e.g..
face paintings (22)). As only very limited samples are provided define the underlying distribution. standard fine-tuning of a pre-trained GAN suffers from mode collapse: the adapted model can only generate samples closely resembling the given few shot target samples [16] 14. Therefore. recent works [8] [4] [9] have proposed to augment standard fine-tuning with different criteria to carefully preserve subset of source model's knowledge into the adapted model. Various criteria has been proposed (Table I and these knowledge preserving criteria have been central in recent FSIG research. In general, these criteria aim to preserve subset of source model's knowledge which is deemed to be useful for target-domain sample generation, e.g., improving the diversity of target sample generation.
Research Gaps. One major limitation of existing methods is that they consider only source domain in preserving subset of source model's knowledge into the adapted model. In particular, these methods fail to consider target aamaannadaatation task in selection of source model's knowledge (Table D For example, EWC [88 applies Fisher Information 23 to select important weights entirely based on the pretrained source model, and it aims to preserve these selected weights regardless of the target domain in adaptation. Similar to EWC [8] CDC (14 proposes an additional constraint to preserve pairwise distances of generated images by the source model, and there is no consideration of target knowledge preserving criteria in recent works raise question regarding their suitability in different source/target domain setups. It should be noted that existing FSIG works (under very limited target samples) focus largely on setups where source and target domains in close proximity (semantically) e.g., Human faces (FFHQ)-Bal faces 14 or Carss-Aaandom Cars [9] It is unclear about their performance when source/target domains are more apart (e.g., Human faces (FFHQ) Animal faces [5)..
domain/adaptation. These
Contributions. In this paper we take an important step to address these research gaps for FSIG. Specifically, our work makes two contributions. As our first contribution, we revisit existing ofte--o--the-arr (SOTA) algorithms and their experiments. Importantly, we observe that when the close proximity assumption is relaxed in experiment setups and source/target domains are more apart, existing SOTA methods perform no better than a baseline fine-tuning method. Our observation suggests that recent methods considering only source domain/source task in knowledge preserving may not be suitable for general FSIG when source and target domains are more apart. To validate our claims, we introduce additional experiments with different source/target domains, analyze their proximity qualitatively and quantitatively, and examine existing methods under a unified framework.
Informed by our analysis, as our second contribution, we propose an adaptation-ave -awrr kernel modulation approach to address general FSIG of different source/target domain proximity. In marked contrast to existing works which preserve knowledge important to source task, our method aims to preserve subset of source model's knowledge that are important to the targel domain and the adaptation task. More specifically, we propose an importance probing algorithm to identify kernels which encode important knowledge for adaptation to the target domain. Then, we preserve the knowledge of these kernels using parameter-efficieet constransrr kernel modulation.

---

FSIG Problem Setup
2
Our Main Findings
180
Source Generator G
TGAN
135
FreezeD
EWC
90
adapt G,
CDC
target:
DCL
Ours
Target Generator G
Bables
Sunglasses
MetFaces
Cat
Dog
Wild
3 Adaptation-Aware Kernel Importance Probing
Importance measured generator for few-shot adaptation
Low FI Kernels
G
G,
High FI Kernels
2333
aleces L.000 osesse 1.0000 cabaaa scole Estimated Fisher Inforamtion
A Convolutional Kenel
Measure importance of kernel's knowledge for adaptation using Fisher Information (FI)
Kernel to be fine-tuned during adaptation (Low FI)
Kernel to be modulated during adaptation (High FI)
Figure 1: Overview and our contributions. Q: We consider the problem of FSIG with Transfer Learning using very limited target samples (i.e. 10-shot). 2: Our work makes two contributions, We discover that when the close proximity assumption between source-target domain is relaxed, SOTA FSIG methods (EWC 18), CDC [4,, DCL. (19) which consider only source domain/source task in knowledge preserving perform better than baseline fine-tuning method (TGAN (I6)) (Sec We propose novel adaptation-awar kernel modulation for FSIG that achieves SOTA performance across source / target domains with different proximity (Sec 3 Schematic diagram of our proposed Importance Probing Mechanism: We measure the importance of each kernel for the target domain after probing and preserve source domain knowledge that is important for target domain adaptation (Sec44) The same operations are applied to discriminator.
We conduct extensive experiments to show that our proposed method consistently achieves SOTA performance across source/target domains of different proximity, including challenging setups when source/target domains are more apart. Our main contributions are summarized as follows:
We revisit existing FSIG methods and experiment setups. Our study uncovers issues with existing methods when applied to source/target domains of different proximity.
We propose Adaptation Aware kemeel Modulation (AdAM) for FSIG. Our method consis- tently achieves SOTA performance both visually and quantitatively across source/target domains with different proximity.
Related Work
Few-shot image generation. Conventional few shot learning 24 25 26 aims at learning a discrim-
inative classifier for classification (27
28 29
segmentation 31 32 or detection [3 34 35
tasks. Differently, few-shot image generation (FSIG) 14
19 aims at learning generator for new and diverse samples given extremely limited samples (e.g., 10 shots). Transfer learning has
18
been applied to FSIG. For example, Transferring GAN [16] (TGAN) applies simple GAN loss [| to

---

FEHQ
Babies
Sunglasses
MetPaces
Cat
Dog
Wild
Inception-v3 Features
LPIPS (el)xNet) Features
Targel Domain
Size
FID 1 LPIPS
100
FFHQ A
70.0K
Babies [1]
2.49K
147
0.274
Sunglasses
14
2.68K
108
0.347
MetFaces
16
1.33K
107
0.358
20
Cal
[
5.15K
227
0.479
Dog
4.74K
210
0.442
Wild
4.74K
272
0.484
120
100
100 131
103 100
Figure 2: Qualitative Quantitative analysis of source-target domain proximity: We use FFHQ
3 as the source domain. We show source-target domain proximity qualitatively by visualizing Inception-v3 (Left) (37] and LPIPS (Middle) (3 using AlexNet [39] backbone features. and
quantitatively using FID LPIPS metrics (Right). For feature visualization, we use t-SNE [40] and show centroids (A) for all domains. FID LPIPS is measured with respect to FFHQ There are two important observations: I Common target domains used in existing FSIG works (Babies, Sunglasses, MetFaces) are notably proximal to the source domain (FFHQ). This can be observed from the feature visualization and verified by FID / LPIPS measurements. @ We clearly show using feature visualizations and FID LPIPS measurements that additional setups - Cat [5), Dog [5] and Wild [3] represent target domains that are distant from the source domain (FFHQ). We remark that large FID values in this analysis are reasonable due to the distance between the source (FFHQ) and different target domains observed from centroid distance / feature variance. The effect of limited sample size (target domains) for FID LPIPS measurements are minimal and we include rich supportive studies in Supplementary. Additional experiments and source/target setups in Supplementary to further support our analysis.
fine-tune all parameters of both the generator and the discriminator. FreezeD 17 fixes a few high- resolution discriminator layers during fine-tuning. To augment and improve simple fine-tuning, more recent works have focused on preserving specific knowledge from the source models. Elastic weight consolidation (EWC) [18] identifies important weights for the source model and tries to preserve these weights. Cross-domain Correspondence (CDC) [4] preserves pair-wise distance of generated images from the source model to alleviate mode collapse. Dual Contrastive Learning (DCL) [19] applies mutual information maximization to preserve multi-level diversity of the generated images by the source model. In this work, we observe that these SOTA methods perform poorly when source and target domains are more apart. Therefore, their proposed source knowledge preservation criteria may not be generalizable. Based on analysis, we propose adaptation-awar knowledge selection which is more generalizable for source/target domains with different proximity.
3 Revisiting FSIG through the Lens of Source-Target Domain Proximity
through the lens of source-target domain proximity. Specifically, we scrutinize the experimental setups of existing FSIG methods and observe that SOTA 18 (semantically) proximal to the source domain: Human Faces (FFHQ) Baby Faces; Human Faces (FFHQ) - Sunglasses; Cars- - Abandoned Cars; Church - Haunted Houses [18 14 19]. This raises the question as to whether existing source-target domain setups sufficiently represent general FSIG scenarios. Particularly, real-world FSIG applications may not contain target domains that are always proximal to the source domain (e.g..: Human Faces (FFHQ) Animal Faces). Motivated by this, we conduct an in- in-depth qualitative and quantitative analysis on source-targe target domain proximity where we introduce target domains that are distant from the source domain (Sec Our analysis uncovers an important finding: Under our additional setups where the assumption of close proximity between source and target domain is relaxed, existing SOTA FSIG methods [18 [4.19] which consider only source domain/source task in knowledge preserving perform no better than baseline fine-tuning method. We show this due to the strong focus of existing SOTA methods in preserving source domain knowledge, thereby not being able to adapt well to distant target domains (Sec 3.2
In this section, we revisit existing FSIG methods (10-shot) [6
18
14 19
14. 19 largely focus on adapting to target domains that are

---

3.1 Source-Target Domain Proximity Analysis
Introducing target domains with varying degrees of proximity to the source domain. this section, we formally introduce source-target domain proximity with in-depth analysis to scrutinize existing FSIG methods under different degrees of source-target domain proximity. Following prior FSIG works [16 [77 [8] 14 19), we use FFHQ 3 the source domain in this analysis. We remark that existing works largely consider different types of human faces as target domain (i.e.: Babies [14], Sunglasses 14), MetFaces [6)). To relax the close proximity assumption and study general FSIG problems, we introduce more distant target domains namely Cat, Dog and Wild (from AFHQ [}
consisting of 15,000 high-quality animal face images at 512 x 512 resolution) for our analysis.
Characterizing source-target domain proximity. Given the wide success of deep neural network features in representing meaningful semantic concepts 41
42 43
we visualize Inception-v3 (37
and LPIPS 38 features for source and target domains to qualitatively characterize domain proximity. Further, we use FID and LPIPS distance to quantitatively characterize source-target domain proximity, We remark that FID involves distribution estimation (first, second order moments) [44] and LPIPS computes pairwise distances (learned embeddings) (38) between source / target domains.
Analysis. Feature visualization and FID/ LPIPS measurement results are shown in Figure Our results both qualitatively (columns I, 2) and quantitatively (column 3) show that target domains used in existing works (Babies (3. Sunglasses (], MetFaces (36]) are notably proximal to the source domain (FFHQ), and our additionally introduced target domains (Dog, Cat and Wild (5) are distant from the source domain thereby relaxing the close proximity assumption in existing FSIG works.
3.2 FSIG methods under Relaxation of Close Domain Proximity Assumption
Motivated by our analysis in Section 3.1 we investigate the performance of existing FSIG meth- ods [16 17 18 [4 [9] by relaxing the close proximity assumption between source and target domains. We investigate the performance of these FSIG methods across target domains of different proximity to the source domain, which includes our additionally introduced target domains: Dog, Cat and Wild. The FID results for FFHQ - Cat are: TGAN (simple fine-tuning) [6]: 64.68, EWC [8] 74.61, CDC [14]: 176.21, DCL [19]: 156.82. Full results can be found in Table 2 We emphasize that our investigation uncovers an im-
portant finding: Under setups which the assumption of close proximity between source and target domain is relaxed (Dog, Cat, Wild), existing SOTA FSIG methods 19) perform no better than a baseline method [16. This can be consistently observed in Table 2
2
22
23
24
G2
[8 14
This finding critical as exposes a serious drawback of SOTA FSIG methods [18 14. 19 when close do- main proximity (between source and target) assumption is relaxed. We further analyse generated images from SOTA FSIG methods and observe that these methods are unable to adapt well to distant target domains due to only considering source domain/ / task knowledge preservation. This can be clearly observed from Fig- ure 3 We remark that TGAN (simple baseline) [16] also suffers from severe mode collapse. Given that our investigation uncovers an important problem in SOTA FSIG methods, we tackle this problem in Sec Figure 3 (last row) shows a glimpse of our proposed method.
EWC
CDC
DCL
AdAM (Ours)
Figure 3: G. is the source generator (FFHQ). Adapting from the source domain (FFHQ) to distant target domain (Cat) using SOTA FSIG
4 Adaptation-Awarg Kernel Modulation
methods EWC [18), CDC 14. DCL 19 (rows 2. 3. 4) results in observable knowledge trans- fer that is not useful to the target domain. i.e. Source task knowledge such as Caps (21, 24). Hair styles/color brown (22), red-hair (2a). Eye glasses (zs) from FFHQ are transferred to Cats during adaptation which is not appropriate.
We focus on this question: "Given a pretrained GAN on a source domain Ds, and a few- samples from a target domain Dr, which part of the source model's 's knowledge should be preserved, and which part should be updated, during the adaptation from D, to D.?" In contrast to
Our method (last row) can alleviate these issues.
5

---

Algorithm 1: Few-Shot Image Generation via Adaptation-Aware Kernel Modulation (AdAM)
Require:
Pre-trained GAN: G, and Ds, prabes iler adapts threshold quantile I, learning rate
Importance Probing:
Freeze all kernels { (W iii=1 in pre-trained networks Gs. and D. 2 Randomly initialize modulation matrix M, for each kernel W;
for k k++, while k probe do
Perform kernel modulation for all kernels using Eqn to obtain modulated weights W
Update M - M - a V W) /* lightweight, i.e., ileTprobe << ileTodapt */
6 end
Measure importance of each kernel W, by computing Fl for the corresponding M, using Eqn
8 Compute the index set A of important kernels using quantile /. of FI values as threshold Main Adaptation:
it' j A then
10
Initialize the kernel by W and freeze the kernel, randomly initialize Mj
11 else
12
Initialize the kernel by W
13 end
14 for k k++, while < ateradapt do
15
if j A then
16
Modulate kernel using Eqn | to obtain modulated weights W
17
Update M,- - M2 a VM, W)
18
else
Update W, - W; - a Vw, W)
19
20
end
end
SOTA FSIG methods [88 14 19I, we propose an adaptation-awart FSIG that also considers the target domain / adaptation task in deciding which part of the source model's knowledge to be preserved. In a CNN, each kernel is responsible for a specific part of knowledge (e. g., pattern or texture). Similar behaviour is also observed for both generator [45] and discriminator [46 in GANs. Therefore, in this work, we make this knowledge preservation decision at the kernel level, i.e., casting the knowledge preservation decision problem of whether kernel is important when adapting from D. to Dr.
Our FSIG algorithm has two main steps: (i) a lightweight importance probing step, and (ii) main adaptation step. In the first step, i.e., importance probing, we adapt the model using parameter- efficient design to the target domain for a limited number of iterations, and during this adaptation, we measure the importance of each individual kernel for the target domain. The output of importance probing are decisions of importance unimportance of individual kernels. Then, in the second step, i.e., main adaptation, unimportant kernels. The overview the proposed system shown in Figure and the pseudocode is shown in Algorithm
preserve the knowledge of important kernels and update the knowledge of
Proposed Importance Probing for FSIG, Our intuition for the proposed importance probing is: "The source GAN kernels huve different levels of importance for euch target domain." " For example, different subsets of kernels could be important when adapting a pretrained GAN on FFHQ to Babies [14] compared to adapting the same pretrained GAN to Cat [5]. Therefore, we aim for knowledge preservation critcrion that is target domain/adappationnaaa -wwae (Table [.. In order to achieve adaptation-awwrreess.
propose light-weight importance probing algorithm which considers adaptation from source to target domain. There are two important design considerations: probing under (i) extremely limited number of target data and (ii) low computation overhead.
As discussed, in this importance probing step, we adapt the source model to the target domain for limited number of iterations and with a few available target samples. During this short adaptation step, we measure the importance of kernel for the adaptation task. To measure the importance, we use Fisher information (FI) which gives the informative knowledge of that kernel in handling adaptation task 47. Then, based on Fl measurement, we classify kernels into important / unimportant. These kernel-level importance decisions then used in the next step, i.e., main adaptation.

---

In the main adaptation step, we propose to apply kernel modulation to achieve restrained update for the important kernels, and simple fine-tuning for the unimportant kernels. As will be discussed. the modulation is rank-constrained and has restricted degree-of- -frreddo--r therefore, it is capable to preserve knowledge of the important kernels. On the other hand, simple fine-tuning has large degree- of-freedom for updating knowledge of the unimportant kernels. Furthermore, the rank-constrained kernel modulation is parameter-effiiien.. Therefore, we also apply this rank-constrained kernel modulation in the probing step to determine the importance of kernels.
Kernel Modulation. The kernel modulation is used in the main adaptation step to preserve knowledge of important kernels into the adapted model. Furthermore, it is also used in the probing step as a parameter-effictt technique to determine importance of kernels. Specifically, we apply Kernel ModuLation (KML) which is proposed very recently [29]. In [29], KML is proposed for multimodal few-shot classification (FSC). In particular, in [29], KMI. has been found be effective for knowledge transfer between different classification tasks of different modes under few-shot constraint. Therefore, in our work, we apply KMI. for knowledge transfer between different generation tasks of different domains under limited target domain samples.
Specifically, in each convolutional layer of CNN, the 7t kernel of that layer W. Rei.. xkxk is convolved with the input feature X E IR xhxw to the layer to produce the 1th output channel (feature map) Y; E R xw", i.e., Y; = W; X + bi, where b, E IR denotes the bias term. Then, KML [29] modulates W, by multiplying with the modulation matrix M, E Re x plus an all-ones matrix J E IR Sin x
W; = W + M,)
(1)
where O denotes Hadamard multiplication. In Eqn. [ using J allows to learn the modulation matrix in a residual format. Therefore, the modulation weights are learned perturbations around the pretrained kernels which helps to preserve source knowledge. The exact pretrained kernel can also be transferred to the target model if optimal. There are some important differences between discriminative version of KML in (2 and our version, please see Supplementary for details.
This baseline KML learns individual modulation parameter for each coefficient of the kernel. Therefore, it could suffer from parameter explosion when using in recent GAN architectures (e.g., more than 58M parameters in StyleGAN-V2 [3] To address this issue, instead of learning the modulation matrix, we learn a low-rank version of it (29 48). More specifically, for Conv layer within CNN, with a total number of dont kernels to be modulated, instead of learning M = (M.) dane
we learn two proxy vectors m, E Rdout and m E (kxxk) and construct the modulation matrix using the outer product of these vectors, i.e., M m. X m2- Furthermore, as we are using KML for adaptable knowledge preservation, we freeze the base kernel W, during adaptation. Therefore, trainable parameters are m. m2. This reduces the number of trainable parameters significantly, and has better performance on restraining the update of important kernels (see Supplementary). As it will be discussed later, the value of dout equals to the total number of kernels a layer (Cont) during probing, and for main adaptation, it is determined by the output of our probing method (dout & Cout).
Importance Measurement. Recall our FSIG has two main steps: (i) importance probing step (Lines 1-8 in Algorithm D and (ii) main adaptation step (Lines 9-21 in Algorithm IT. In probing, we also apply KML as parameter-efficient technique to determine importance of individual kernels. In particular, for probing, we propose to apply KML to all kernels (in both generator and discriminator) to identify which of the modulated kernels are important for the adaptation task. To measure the importance of the modulated kernels, we apply Fisher information (FT) to the modulation parameters. In our FSIG setup, for a modulated GAN with parameters O, Fisher information F can computed as:
02
F(O) E| S02 [x)
where L(x(O) is the binary cross-entropy loss computed using the output of the discriminator, and includes few-shot target samples, and fake samples generated by GAN. Then, Fl for a modulation matrix F(M,) can be computed by averaging over FI values of parameters within that matrix. As we are using the low-rank estimation to construct the modulation matrix, we can estimate F(M,) by FI values of the proxy vectors. In particular, considering the outer product in low-rank approximation,
we have M, = r m m
(Ca, xk*k)]), where |mgl = Cin x k x k. Then we
7

---

Table FSIG (10-shot) results: We report FID scores (L) our proposed adaptation -aaare FSIG and compare with existing FSIG methods. We emphasize that Cat, Dog and Wild target domains are additional experiments included this work. (Sec Our experiment results show two important findings: 1) Under setups which assumption of close proximity between source and target domains is relaxed (Cat, Dog. Wild), SOTA FSIG methods EWC, CDC, DCL. which consider only source domain in knowledge preserving perform better than a bascline fine-tuning method (TGAN). 2) Our proposed adaptation-awes -wware FSIG achieves SOTA performance in all target domains due to
preserving source domain knowledge that is important for few-shot target domain adaptation. We
generate 5,000 images using the adapted generator to evaluate FID on the whole target domain. We
also report the corresponding KID, Intra-LPIP' and standard deviations in Supplementary.
Target Domain
Babies [14]
Sunglasses [14]
MetFaces
36
AFHQ-Cat [S]
AFHQ-Dog [5]
AFHQ-Wild [S]
IGAN [16]
101.88
33.97
76.81
64.68
131.46
81,30
TGAN+ADA
36
97.91
53.64
75.82
80.16
162.63
81.55
FreezeD 1t
96.25
46.95
73.33
63.60
157.98
77.18
EWC 18
79.93
49.41
62.67
74.61
158.78
92.83
CDC 14
69.13
41.45
65.44
176.21
170.95
135.13
DCL 19
56.48
37.66
62.35
156.82
171.42
115.93
AdAM (Ours)
48.83
28.03
51.34
58.07
100.91
36.87
use the unweighted average of FI for parameters of my and m2, proportional to their occurrence frequency in calculation of M,, as an estimate of F(M,) (details in Supplementary):
mal
1
F(M)) F(m))
F(me)
m2
After calculating F(.)) for all modulation matrices in both generator and discriminator, we use the 1% quantile of these values threshold (separately for generator and discriminator) to decide whether modulation of a kernel is important or unimportant for adaptation to the target domain. If the modulation of a kernel is determined be important (during probing), the kernel modulated using KML during main adaptation step; otherwise, the kernel is updated using simple fine-tuning during main adaptation. In all setups, we perform probing for 500 iterations. We remark that in probing only modulation parameters m1, My are trainable, and FI is only computed on them, therefore the probing a very lightweight step and can be performed with minimal overhead (details in Supplementary). The output of probing step are the decisions to apply kernel modulation or simple fine- thee-tunig on individual kernels. Then, based on these decisions, the main adaptation is performed. The proposed FSIG scheme is summarized in Algorithm
5 Empirical Studies
5.1 Experiments Results
18 14 19 in Experiment Details. For fair comparison, we strictly follow prior works [16 17 the choice of GAN architecture, target adaptation setups and saper-parameters. We use StyleGAN- V2 K as the GAN architecture and FFHQ as the source domain. Our experiments include setups with different source-target proximity: Babies/Sunglasses [14, MetFaces (36) and Cat/Dog/Will (AFHQ) (5) (See Sec. (3.. Adaptation is performed with 256 x 256 resolution and batch size 4 on a single Tesla V100 GPU, We apply importance probing and modulation on base kernels of both generator and discriminator. We focus 10-shot target adaptation setup in the main paper.
Qualitative Results. We show generated images with our proposed AdAM along Baseline [6 17 and SOTA FSIG methods [18 14. 19 for two target domains, Babies and Cat with different degrees of proximity FFHQ before and after adaptation. The results are shown in Figure 4 top and bottom, respectively. By preserving source domain knowledge that is important for target domain, our proposed adaptation-aware FSIG method can generate substantially high quality images with high diversity for both Babies and Cat domains. We also include FID 44 and Intra-I [14 (for measuring diversity) to quantitatively show that our proposed method outperforms SOTA FSIG We show more generated samples in Supplementary.
methods [18 19..
8

---

(1)
Intra-IPPIPS (T)
T3AN
101.58
0.517
13AAN
ADA
FreezeD
96.25
0.518
2MC
79.93
0.521
89.13
DCC
56.48
0.580
(Ouzs)
10-Shot Beal Babies
FID (1)
Intra-LPIPS (1)
84.68
0.490
22AN
80.16
0.513
1 ADA
Freezen
63.60
2BC
74.61
0.587
DCC
156 82
0.816
(Dues)
58.07
0.557
10-Shot
Rea" Data
Figure 4: Qualitative and quantitative comparison of 10-shot image generation with different FSIG methods. Images of each column are from the same noise input. Left: 10 real target images for few-shot adaptation. Middle, Right: For target domain with close proximity (e.g.Babies, top). our method can generate high quality images with more refined details and diversity knowledge, achieving best FID and Intra-LPIPS socre. For target domain which is distant (e.g., Cat, bottom), TGAN/FreezeD overfit to the 10-shot samples and others fail. In contrast, our method preserves meaningful semantic features at different levels (e.g., posture and color) from source, achieving good trade off between quality and diversity. In particular, our Intra-LPIPS approaches that of EWC, while our generated images have much better quality qualitatively and quantitatively.
Quantitative Results. We show complete FID (J) scores in Table Our proposed AdAM for FSIG achieves SOTA results across all target domains of varying proximity to the source (FFHQ). We emphasize that it is achieved by preserving source domain knowledge that is important for target domain adaptation (Sec 4). We also report Intra-LPIPS (f) as an indicator of diversity, as Figure
5.2 Analysis
Ablation study of Importance Probing. The goal of importance probing (denoted as "IP"") is to identify kernels that are important for few-shot target adaptation as shown in Figure 5 (Top). To justify the effectiveness of our design choice, we perform an ablation study that discards the IP stage and regard kernels as equally important for target adaptation. Therefore, we simply modulate all kernels without any knowledge selection. As one can observe from ((outo))BBottm)) knowledge selection plays vital role in adaptation performance. Specifically, the significance of knowledge preservation is more evident when the target domains are distant from the source domain.
(Bottom),

---

Low Fl Kernels
High Fl Kernels
TCAM
FreezeD
PMC
CTC
DCI
Dure
180
FFIQ + Babies
FFBQ + AFHQ Cat
FFHQQ + Babies
FI1Q + AFRQ Cat
152
220
171
1,40
60
154
43
67
27
88
5009 5041 caca 10030 Jose cake Estimated Riseer Inforamiton
<kow xiaea caow kiles ciow saicie
Estimated Essher Inforamtion
10
200
100
50
25
10
shod
10
200
100
50
25
10
shot
Target Domain
Babies
Sunglasses
MetFacce
AFHQ-Cat
AFHQ-Dog
AFHO-Wild
AdAM (Wlo probing)
54,46
33,66
60.43
82.41
160.87
81.24
AdAM (Ours)
48.83
28.03
51.34
58.07
100.91
36.87
Figure 5: (Top Left) Our proposed IP identifies and preserves source kernels important (high FI) for target adaptation. (Bottom) FID score on different datasets. We validate the effectiveness of IP by modulating all kernels without IP. On the other hand, if we fine-tune all parameters without IP and modulation (TGAN), it suffers mode collapse (Table 2 and Figure 4). (Top Right) We evaluate the performance of different number of shots (10, 25, 50, TO0, 200) Babies and AFHQ-Cat. We show that our method consistently outperforms other FSIG methods in all setups. In Supplementary. we also show the generated images given different number of shots on more target domains.
Number of target samples (shots). The number of target domain training samples is important factor that can impact the FSIG performance. In general, more target domain samples can allow better estimation of target distribution. We study the efficacy of our proposed method under different number of target domain samples. The results are shown in Figure 5 and we show that our proposed adaptation-ant aware FSIG method consistently outperforms existing methods in all setups.
Discussion
Conclusion. Focusing on FSIG, we make two contributions. First, we revisit current SOTA methods and their experiments. We discover that SOTA methods perform poorly in setups when source and target domains are more distant, as existing methods only consider source domain/task for knowledge preservation. Second, we propose new FSIG method which is farget/adaptation-awaae (AdAM). Our proposed method outperforms previous work across all setups of different source-target domain proximity. We include extended experiments and analysis in Supplementary.
Broader Impact. Our work makes contribution to generation of synthetic data applications where sample collection is challenging, e.g., photos of rare animal species. This is an important contribution to many data-centric applications. Furthermore, transfer learning of generative models using a few data sample enables data and computation-efficien impact on environmental sustainability and reduction of greenhouse gas emission. While our work targets generative applications with limited it parallely raises concerns regarding such methods being used for malicious purposes. Given the recent success of forensic detectors [49
model development. Our work has positive
50 51 52), we conduct a simple study using Color-Robust forensic detector proposed in 49 on our Babies and Cat datasets. We observe that the model achieves 99.9 99.8% and 99.9% average precision (AP) respectively showing that AdAM samples can be successfully detected. We also remark that our work presents in a broader context.
opportunities for improving knowledge transfer methods [53 54 55
56)
Limitations. While our experiments are extensive compared to previous works, in practical applica- tions, there are many possible target domains which cannot be included in our experiments. However, as our method is target/adaptation aware, we believe our method can generalize better than existing SOTA which are agnget-.g
Acknowledgment
This research supported by the National Research Foundation, Singapore under its Al Singapore Programmes (AISG Award No.: AISG2-RP-200 2021-22----0 AISG Award No.: AISG-100F201 8-005). This project is also supported by SUTD PIE-SGP-AI-2018-000 We thank anonymous reviewers for their insightful comments.
10

---

References
[1] lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[2] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. International Conference Learning Representations, 2018.
[3] Tero Karras, Samuli Laine, Mika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 81 10-8119, 19, 2020.
[4] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401 4410, 2019.
[5] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Lung-Woo Ha. Stargan v2: multiple domains. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020.
Diverse image synthesis for
[6] Jun- Zhu, Taesung Park, Phillip Isola. and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference computer vision, pages 1223-2232, 2017.
[7] Ming- Yu Liu, Xun Huang, Arun Mallya, Tero Karras. Timo Aila, Jaakko Lehtinen, and Jan Kautz. Few- shot unsupervised taage-to-image translation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10551-10560, 2019.
[8] Ji Lin, Richard Zhang, Frieder Ganz, Song Han, and Jun- Yan Zhu. Anycost gans for interactive image synthesis and editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19986-14966, 2021.
[9] Hongyu Liu, Ziyu Wan, Wei Huang, Yibing Song. Xintong Han, Jing Liao, Bin Jiang, and Wei Liu. Deflocnet: Deep image editing via flexible low-level controls. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10765 10774, 2021.
[10] Swee Kiat Lim, Yi Loo, Ngoc Trung Tran, Ngai Man Cheung, Gemma Roig, and Yuval Elovici. Doping: Generative data augmentation for unsupervised anomaly detection with gan. In 18th IEEE International Conference on Data Mining, ICDM 2018, pages 1122 1127. Institute of Electrical and Electronics Engineers Inc., 2018.
[11] Ngoe-Trung Tran, Viet-llung Tran, Ngoc-Bao Nguyen, Trung- Kien Nguyen, and Ngai-Man Cheung. On data augmentation for gan training. IEEE Transactions on Image Processing, 30:1882-1897, 2021.
[12] Lucy Chai, Jun-Yan 7hu, Eli Shechtman, Phillip Isola, and Richard Zhang. Ensembling with deep generative views. In CVPR, 2021.
[13] Qianli Feng, Chenqi Guo, Fabian Benitez-Quiron and Aleix M Martinez. When do gans replicate? on the choice of dataset size. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 6701-6710, 2021.
[14] Utkarsh Ojha. Yijun Li, Jingwan Lu, Alexci A Efros, Yong Jac Lee, Eli Shechtman, and Richard Zhang Few-shot image generation via cross-domain correspondence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10743-10752, 2021.
[15] Atsuhiro Noguchi and Tatsuya Harada. Image generation from small datasets via batch statistics adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2019.
[16] Yaxing Wang, Chenshen Wu, Luis Herranz, Joost van de Weijer, Abel Gonzalez- and Bogdan Raducanu. Transferring gans: generating images from limited data. In Proceedings of the European Conference Computer Vision (ECCV), pages 218-234, 2018.
[17] Sangwoo Mo, Minsu Cho, and Jinwoo Shin. Freeze the discriminator: a simple baseline for fine-tuning gans. In CVPR Al for Content Creation Workshop. 2020.
[18] Yijun Li, Richard Zhang, Jingwan (Cynthia) Lu, Eli Shechtman. Few-shot image generation with elastic weight consolidation. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 15885 15896. Curran Associates, Inc., 2020.
11

---

[19] Yunqing Zhao, Henghui Ding. Houjing Huang, and Ngai-Man Cheung. A closer look at few-shot image generation. In CVPR, 2022.
[20] Sinno Jialin Pan and Qiang Yang. A survey transfer learning. IEEE Transactions on knowledge and data engineering. 22(10):1345 1359, 2009.
[21] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248 255, 2009.
[22] Jordan Yaniv, Yael Newman, and Ariel Shamir. The face of art: landmark detection and geometric style in portraits. ACM Transactions on graphics (TOG), 38(4): 2019.
[23] Alexander Ly. Maarten Josine Verhagen, Raoul PPP Grasman, and Eric-Jan Wagenmakers. A tutorial on fisher information. Journal of Mathematical Psychology. 80:40-55 55, 2017.
[24] Chelsca Finn, Pieter Abbeel, and Sergey Levine. Modcl-agnostic meta-lcaming for fast adaptation of deep networks. International conference on machine learning, pages 1126 1135. PMLR, 2017.
[25] Li Fei-Fei, Rob Fergus, and Pictro Perona. One-shot learning of object categories. IEEE taansctions on pattern analysis and machine intelligence, 28(4):594-611, 2006.
[26] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in neural information processing systems, pages 4077-4087, 2017.
[27] Yiluan Guo and Ngai-Man Cheung. Attentive weights generation for few shot learning via information maximization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13499-13508, 2020.
[28] Jiamei Sun, Schastian I. Lapuschkin, Wojeiech Samek, Yunqing Zhao, Ngai-Man Cheung, and Alexander Binder. poolanatioonguideen training for cross-domain few-shet classification. In 2020 25th International Conference on Puttern Recognition (ICPR). pages 7609-7616. IEEE, 2021.
[29] Milad Abdollahzadeh, Touba Malekzadeh, and Ngai-Man Man Cheung. Revisit multimodal meta-learning through the lens of multi task learning. Advances in Neural Information Processing Systems, 35, 2021.
[30] Yunqing Zhao and Ngai-Man Cheung. Fs-ban: Born-again networks for domain generalization few-shot classification. arXiv preprint arXiv:2208. 10930, 2022.
[31] Weide Liu, Chi Zhang, Guosheng Lin, and Fayao Liu. Crnet: Cross-reference networks for few-shot segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 4165 4173,, 2020.
[32] Malik Boudiaf, Hoel Kervadee, Ziko Imtiaz Masud, Pablo Piantanida, Ismail Ben Ayed, and Jose Dolz. Few- shot segmentation without meta learning A good transductive inference is all you need? In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 19979-1398 2021.
[33] Gongjie Zhang, Kaiwen Cui, Rongliang Wu, Shijian Lu, and Yonghong Tian. Pnpdet: Efficient fow-shot detection without forgetting via loug-and-play sub- networks. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3823-3832, 2021.
[34] Zhibo Fan, Yuchen Ma, Zeming Li, and Jian Sun. Generalized few-shot object detection without forgetting. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 4527- 4536, 2021.
[35] Jia Gong, Zhipeng Fan, Qiuhong Ke, Hossein Rahmani, and Jun Liu. Meta agent teaming active learning for pose estimation. Proceedings the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 11079 11099, 2022.
[36] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. Advances in Neural Information Processing Systems, 33:12104- 12114, 2020.
[37] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818 2886222,22222 2016.
[38] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metrie. In CVPR, 2018.
12

---

[39] Alex Krizhevsky, 1lya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25:1097-1105, 2012.
[40] Laurens van der Maaten and Geoffry Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(8))2557-2205, 2008.
[41] Hossein Talebi and Peyman Milanfar: Learned perceptual image enhancement. In 2018 JEEE international conference computational photography (ICCP), pages 1-13. IEEE, 2018.
[42] Hossein Talebi and Peyman Milanfar. Nima: Neural image assessment. IEEE transactions image processing, 27(8):3998 4011, 2018.
[43] Stanislav Morozov, Andrey Voynov, and Artem Babenko. On self-supervised image representations for [gan] evaluation. In International Conference on Learning Representations, 2021. 2021.
[44] Martin Hcusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[45] David Bau, Jun Yan Zhu, Hendrik Strobelt, Bolei Zhou, Joshua B Tenenbaum, William T Freeman, and Antonio Torralba. Gan dissection: Visualizing and understanding generative adversarial networks. In Proceedings of the International Conference on Learning Representations (ICLR). 2019.
[46] David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantify- ing interpretability of deep visual representations. In Proceedings the IEEE conference on computer vision and puttern recognition, pages 1541-6549, 2017.
[47] Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless Fowlkes, Stefano Soatto, and Pictro Perona. Task2vec: Task embedding for meta-learning In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 6430-6439, 2019.
[48] Christian Simon, Piotr Koniusz, Richard Nock, and Mehrtash Harandi. On modulating the gradient for meta-learning, In European Conference on Computer Vision, pages 556-572. Springer, 2020.
[49] Keshigeyan Chandrasegaran, Ngoe-Trume Tran, Alexander Binder, and Ngai- Man Cheung. Discovering Transferable Forensic Features for CNN-ggnneratd Images Detection. In Proceedings of the European Conference on Computer Vision (ECCV), Oct 2022.
[50] Sheng- Yu Wang, Oliver Wang, Richard Zhang, Andrew Owens, and Alexei A. Efros. CNN-Generated Images Are Surprisingly Easy to Spot... for Now. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[51] Keshigeyan Chandrasegaran, Ngoe-Trune Tran, and Ngai-Man Cheung. A Closer Look at Fourier Spectrum Discrepancies for ONN-Generated Images Detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 7200-7209, June 2021.
[52] Joel Frank, Thorsten Fisenhofer, Lea Schönherr, Asja Fischer, Dorothea Kolossa, and Thorsten Holz. Leveraging frequency analysis for deep fake image recognition. International conference on machine learning, pages 3247-3258. PMIR, 2020.
[53] Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. In NIPS Deep Learning and Representation Learning Workshop, 2015.
[54] Keshigeyan Chandrasegaran, Ngae-Trung Tran, Yunqing Zhao, and Ngai-Man Cheung. Revisiting Label Smoothing and Knowledge Distillation Compatibility: What was Missing? In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning. volume 162 of Proceedings of Machine Learning Research, pages 1590029166 PMLR. 17-23 Jul 2022.
[55] Bycongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi. A compre- hensive overhaul of feature distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1921-1930, 2019.
[56] Utku Evci, Vincent Dumoulin, Hugo Larochelle, and Michael C Mozer. Head2toe: Utilizing intermediate representations for better transfer learning. In International Conference Machine Learning, pages 6009-6033. PMLR, 2022.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See Sections 3 and more details in Supplementary.
(b) Did you describe the limitations of your work? [Yes] See Section 6
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
See
Supplementary
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material as URLY? [Yes] We include code, data URL, pre-trained GAN models and reproducibility details in Supplementary. (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We specify all training details Supplementary.
(c) Did you report error bars (e-g.. with respect the random seed after running experi- ments multiple times)? [Yes] We report standard deviations in Supplementary.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We include all compute details in Supplementary.
4. If you using existing assets (e.g., code, data, models) or curating/releasing
assets... (a) If your work uses existing assets, did you cite the creators? [Yes]] We have cited all original papers and included URLs for the code and datasets used.
(b) Did you mention the license of the assets? [Yes] We mention the license of the assets in Supplementary.
(c) Did you include any assets either in the supplemental material We are releasing code and pre-trained GAN models. The URL details are included in Supplementary.
URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A] All assets are publicly available and contain no personally identifiable information offensive content.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "en2tQYyCMZq", "reviewer": "Reviewer_4Rin", "summary": "This paper tackle the problem of few-shot image generation, which aims to learn to generate image given a limited number of samples from a domain. The author first conducts a probing analysis of existing methods and finds out the reasons for degenerated performance of these methods under the assumption of relaxed proximity between source and target domains. Secondly, the author propose adaptation-aware kernel modulation to address general FSIG of different source-target domain proximity. This results in good performance, in terms of generated image quality in multiple few-shot settings.", "questions": "1. More analysis of the proposed importance probing is recommended. The author proposes important probing to identify the importance of each kernel in the GAN model. To further verify the generalizability of this probing method, I would suggest the authors try the importance probing on different types of pre-trained GAN models. \n\n2. Is it possible to visualize the importance probing results in Figure 5? In this way, we are able to reveal what kind of information is preserved (w/ high IF) in the modulation, and this can serve as support for the effectiveness of this probing analysis.\n\n3. More discussion on (potential) failure cases is recommended in the main paper or appendix.\n\nMinor comment:\n\n1. The layout of the main paper should be reorganized in the future version. The margins between some figures and tables (e.g., Figure 5 and 6) are too small, making the captions a bit hard to read.  ", "limitations": "Yes, the author discusses the potential societal impact in Section F1 of the appendix.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The paper is well-written and easy to follow.\n\n2. There are good performance gains. The proposed approach is (to the best of my knowledge) novel, and also theoretically interesting.\n\n3. The model is evaluated on multiple few-shot settings and compared to multiple baselines, which shows its outperformance. The proposed method obtains SOTA on few-shot image generation performance, with a good ablation study.\n\n4. Code is provided, and reproducibility is considered.\n\nWeaknesses:\n\n(Please refer to \"Questions\".)  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "e6HZc3bkjy3", "reviewer": "Reviewer_ekht", "summary": "This paper focuses on the few-shot image generation, especially when there might be a huge gap between the source and target domains. By setting up a valiation pipelline, authors find that existing approaches perform worse than the directly fine-tuning when the source and target domains are apart. In order to tackling such challenging setting, the proposed method leverages Fisher informance to measure the importance on the target domain and performs the corresponding modulation. Experimental results demonstrate the effectiveness of the proposed approache, especially across the relatively challenging domains: \\eg from human faces to animal faces. ", "questions": "Major concerns:\n- As CDC evaluates its approach on skteches as well, I am wondering how well the proposed method performs under such drawing style changes. Ideally, there might be only several kernels affecting the drawing style when adapting from face to sketch faces. Thus, evaluating the proposed method under such style changes is very necessary. \n-  How much the proximity between the source and the target could be relaxed. As the main focus is to challenge the domains that are apart to some extent, what if we fine-tune a face model on other objects like cars? Are all kernels useless? That is, the limit across challenging domains of the proposed method is worth to explore. \n- How many images of target domains are required to determine the importance of kernels. As all experiments are conducted with 10-shot samples, what if we continue to decrease the number of training samples, like 5, even 1 image? And what the FID becomes if we continue to involve more target images, (maybe involving all AFHQ) as Fig.6 does (but more than 200 images). \n- The technical contributions could be better clarified, compared to EWC. It seems that this paper measures the importance on the target domain while EWC does it on the source domain. \n\nOverall, the exploration on such challenging domains is very necessary for the future research of few-shot generation. Thus, I will change my score depending on the response. \n\n\n\n", "limitations": "The limitations of the proposed method could be discussed more comprehensively. ", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- This paper is well-organized and easy to follow.\n- The focus on relatively unrelated domains is very interesting and challenging as existing methods just fail under this. \n- Anlysis of evaluating previous approaches is also well-built to uncover the issues.\n- Experimental results demonstrate the effectiveness of the proposed adaptation method that explicitly considers the target domain. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "TMaxD1ioSwE", "reviewer": "Reviewer_gbAJ", "summary": "This paper proposes a method to do few-shot (e.g. 10 images) image generation, by transferring a pretrained GAN from a source domain (e.g. FFHQ) to a target domain (e.g. dogs/cats). The method is simple and the empirical performance is good.", "questions": "1.  I see connections between Fisher information (FI) and class saliency [a] or other similar attribution methods. But these attribution methods use first-order derivatives only, so they are much cheaper. Have you verified that FI is indeed advantageous over first-order attribution methods? In other words, Is using second-order derivatives indeed necessary?\n2. In Fig. 2, the Sunglasses dataset is from [32]. But in remaining text, it's from [8]. However, I couldn't find the Sunglasses dataset from either of these two papers.\n3. The references [32] and [33] are duplicate.\n4. Why the first face (the young lady) of G_s in row 7 is the same as the 9th face of G_s in the last row? \n5. Suggest the authors to use superscripts $\\boldsymbol{m}_1^i$ instead of $\\boldsymbol{m}_1(i)$ in Eq.(3) for better clarity.\n\n[a] Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps, 2014.", "limitations": "1. It is natural that a user of FSIG can take few-shot images of a particular person and generate diverse images of them. Have the authors tried this setting? How well does it work? If the performance is good, it could be a security threat to face recognition systems.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Few-shot (e.g. 10 images) image generation is practically useful.\n2. The empirical performance is good.\n\nWeaknesses:\n1. In eq.(3), the size of the fisher information matrix in (3) is square of the number of parameters, i.e., $(c_{in}\\cdot k\\cdot k)^2$. This could be huge. However, the authors claim this is very lightweight with minimal overhead. Please justify. \nEDIT: I've read the appendix and found the derivations. However, $|\\boldsymbol{m}_2|$ seems to mean the number of parameters in $\\boldsymbol{m}_2$ (Eq.3 appendix)? But in the text before Eq.3 (main text), $|\\boldsymbol{m}_2|=k$, the number of conv channels.\n2. The chosen target-domain datasets, babies, metfaces, cat, dog, wild are actually quite similar to the source domain (human or animal faces). According to [33], the domain gaps between human and animal faces is not so big as they look.\n3. In Fig. 4 and 6, the authors should include TGAN+ADA as an important baseline, instead of focusing on the much weaker TGAN. According to [33], when there are 1k target-domain images, the FID is quite low (around 25, Fig. 9d [33]), when adapting from FFHQ to Cat. So I expect that at the left side (200 shots) of Fig 6, TGAN+ADA should have competitive performance.\n4. Table 2, Fig 5 and Fig 6 are crammed in Page 9.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept