Masked Generative Adversarial Networks are Data-Efficient Generation Learners
Authors: Jiaxing Huang
Kaiwen Cui
Dayan Guan
Aoran Xiao
Fangneng Zhan
Shijian Lu
Shengcai Liao
Eric Xing
Venue: NeurIPS 2022 Accept (2022)
Keywords: Generative Adversarial Network
Data Limited Image Generation
Data Efficient GAN

Abstract: This paper shows that masked generative adversarial network (MaskedGAN) is robust image generation learners with limited training data. The idea of MaskedGAN is simple: it randomly masks out certain image information for effective GAN training with limited data. We develop two masking strategies that work along orthogonal dimensions of training images, including a shifted spatial masking that masks the images in spatial dimensions with random shifts, and a balanced spectral masking that masks certain image spectral bands with self-adaptive probabilities. The two masking strategies complement each other which together encourage more challenging holistic learning from limited training data, ultimately suppressing trivial solutions and failures in GAN training. Albeit simple, extensive experiments show that MaskedGAN achieves superior performance consistently across different network architectures (e.g., CNNs including BigGAN and StyleGAN-v2 and Transformers including TransGAN and GANformer) and datasets (e.g., CIFAR-10, CIFAR-100, ImageNet, 100-shot, AFHQ, FFHQ and Cityscapes).

Masked Generative Adversarial Networks are Data-Efficient Generation Learners
Jiaxing Huang': Kaiwen Cui': Dayan Guan', Aoran Xiao', Fangneng Zhan',
Shijian Lu" Shengeai Liao", Eric Xing
1 School of Computer Science and Engineering, Nanyang Technological University, Singapore
Max Planck Institute for Informatics, Germany
Inception Institute of Artificial Intelligence (IIAI). Abu Dhabi, UAE
School of Computer Science, Carnegie Mellon University, USA
Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE
(Jiaxing Huang, Kaiwen Cui, Dayan,Guan, Aoran Aoram.Xiao, fzhandmpi-inf wpg. de, shengcai www.eiiiccptiiniii..rrrr. epxingoes.. comiingossccu.... edu, Eric. Xing@mbzuai ac..zauuii.c............ ac.
Luniiu. edu.
Abstract
This paper shows that masked generative adversarial networks (MaskedGAN) are robust image generation learners with limited training data. The idea of MaskedGAN is simple: randomly masks out certain image information for effective GAN training with limited data. We develop two masking strategies that work along orthogonal dimensions of training images, including a shifted spatial masking that masks the images in spatial dimensions with random shifts, and a balanced spectral masking that masks certain image spectral bands with self- adaptive probabilities. The two masking strategies complement each other which together encourage more challenging holistic learning from limited training data, ultimately suppressing trivial solutions and failures in GAN training. Albeit sim- ple, extensive experiments show that MaskedGAN achieves superior performance consistently across different network architectures (e.g., CNNs including BigGAN and StyleGAN-v2 and Transformers including TransGAN and GANformer) and datasets (e.g., CIFAR-10, CIFAR- 100, ImageNet, 100-shot, AFHQ, FFHQ and Cityscapes).
Introduction
"What cannot create, do not understand." -Richard Feynman. Generative Adversarial Network (GAN) [9 such an unsupervised framework learning to understand and create: the discriminator aims to understand "what are real and fake images" by binary discrimination while the generator strives to produce realistic images according to discriminator's feedback. In recent years, GANs have achieved great success under the presence of large amounts of training data. When only limited training data is available, they instead experience obviously increased generation failures [58] 31 By reversing the dictum, get "What do not understand, can create" perfect metaphor of such generation failures. Specifically, with limited training data, the discriminator tends to achieve discrimination via certain less meaningful shortcuts by memorizing training data merely focusing on casy-toddiscrmminate image locations and spectra without holistic understanding of images, ultimately leading to trivial solutions that fail to "create" realistic images [58] 31 as shown in Fig.
43 which might be the
"indicates equal contribution. corresponning author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

(4) with 100% CIFAR-10 Data
(4) with 20% CIFAR-10 Data
FID (4) with 10% CIFAR-10 Data
26.0
100
100
BigGiW
BigGAW
BigGAN NaskerGaN
17.5
NaskecGAM
30
NaskesGAW
20
15.0
60
00
12.5
40
40
20
20
7.5
1.00
140
260
250
300
350
400
20
60
06
100 170
1:00
1850
20
80
100
Training Iteration
203
Training Iteration
-0:
Training Iteration
203
Figure I: -fate-of-the-art GANs such as BigGAN often experience clear training collapses with limited training data such as 20% and 10% of CIFAR-10. The proposed MaskedGAN (i.e., Masked- BigGAN) effectively stabilizes adversarial training and suppresses trivial solutions and training failures by encouraging more challenging holistic understanding of images.
Trivial solutions are ubiquitous in unsupervised learning 3 (MAE) 48
6 7
10 and so in Masked Autoencoders
21 [5]] as one prevalent unsupervised representation method. MAE works by masking and reconstructing the image, where image masking is the core design that suppresses trivial solutions and enables learning high-level semantic features 5] For instance, [21] presents a high- ratio masking strategy for MAE: 1) it can impede shortcuts in image reconstruction, e-g., local spatial interpolation that recovers a masked patch from adjacent patches without high-level understand- ing 51: 2) it encourages global spatial interpolation that requires modelling inter- paterr context information [54] 25 globally, ultimately learning nontrivial and meaningful representations [21] (}.
42 18 2
In this work, we explore the idea of image masking training for GAN, aiming to develop robust image generation learners with limited training data. By viewing image masking as one general means of suppressing training shorteuts, we hypothesize that image masking in GAN training should be conducted over both spatial and spectral dimensions for preventing discriminators from merely focusing on easy-to-discriminate image locations and spectra. In another word, applying random spatial and spectral masking creates more challenging tasks for discriminator training which forces discriminators to learn more holistic instead of easy-to toy-ttoordiriinatee information only. At the other end, as applying masking to discriminators slows down their learning, it is crucial to decelerate the generator learning synchronously for overall balanced and stable adversarial learning.
Driven by this motivation, we design MaskedGAN, a robust image generation network that can learn effectively with limited training data. The idea of MaskedGAN is simple: it randomly masks certain image information to suppress trivial solutions in GAN training. We develop two masking strategies that work along orthogonal dimensions of images. The first is shifted spatial masking that masks the image in spatial dimensions with a random shift. Intuitively, random masking along spatial dimensions enforces discriminator to learn over all image locations instead of merely focusing on -osy-toddiscriminate locations. The second is balanced spectral masking that decomposes an image into multiple bands in spectral space and masks portion of spectral bands with self self-tdaptive probabilities. Random masking along spectral dimension encourages discriminator to learn hard-to- discriminate bands (e.g., high- frequrrry bands capturing shapes and structures) instead of largely on easy bands (e.g., low-frequency bands capturing color and brightness). The two masking strategies complement each other which together suppress trivial solutions and training failures by encouraging more challenging holistic discriminator learning from limited training data.
MaskedGAN also applies the two masking strategies to the generator training. Differently, on generator side, the two image masking operations work on outputs by sparsifying the training signals backpropagated from the discriminator. This design slows down the generator learning and stabilizes the overall adversarial learning by keeping similar learning paces for discriminator and generator. It ensures that MaskedGAN can converge to a Local Nash Equilibrium under certain conditions [22].
We summarize the contributions of this work in three aspects. First, this paper investigates image masking training for GAN, aiming for building robust image generation learners with limited training data. Second, we propose MaskedGAN and design spatial and spectral masking for training robust and effective GAN with limited data. The two masking strategies complement and encourage more challenging holistic understanding of images which help suppress trivial solutions and stabilize the overall adversarial training. Third, extensive experiments show that simple implementation of MaskedGAN achieves superior generation performance consistently across different network architectures and datasets.
2

---

2
Related Work
This work is related to two main fields of research, namely, generative adversarial networks for image generation and masked autoencoders for unsupervised representation learning.
Generative Adversarial Network (GAN) 19 is an unsupervised data synthesis framework that aims generate model distribution that mimics a given target distribution. consists of a generator that generates model distribution and a discriminator that distinguishes the model distribution from the target. In recent years, series of GANs have been proposed to improve the performance and training stability from different perspectives, such as designing more stable training objectives [[]
36
20, 37 451, building better network architectures 44
38 39
26 14 involving
various training strategies [66
30 35 etc. Despite the great success obtained, they require large amounts of training data for training effective generation models. When only limited training data is available, they often suffer from clearly increased generation failures with severely degraded In this work, we focus on training effective GANs with limited training
performance [58 31 27
data.
Training GAN with limited data: Recently, the task of training effective GANs with limited data has attracted increasing attention for alleviating the requirement of large amounts of training data. Most existing efforts address this challenging task from two perspectives. The first data augmentation [58] 31 [27 that works via massive hand-crafted data augmentation policies, e.g., [58]
employs different types of differentiable augmentation to stabilize GAN training. The second is
model regularization 46 13
which stabilizes GAN training by regularizing the discriminator
training via label flipping 46 network co-training
13
lottery ticket hypothesis [9) etc.
We tackle this challenging task from perspective of image masking training. Specifically, we design MaskedGAN that exploits two orthogonal image masking strategies for encouraging more challenging holistic understanding of images. Extensive evaluations show that MaskedGAN stabilizes the adversarial training process and suppresses trivial solutions and training failures effectively.
Masked Autoencoders (MAE) has greatly advanced the research of unsupervised representation learning recently. Despite different designs, MAE is essentially denoising autoencoder (DAE) [47] that learns semantic representations by corrupting and reconstructing the input signals. Most existing work can be viewed as generalized DAE under different types of corruptions, such as masking 18 21). or masking image color channels (57). For example, Context Encoder 44 masks random image regions of different shapes and learns to reconstruct them. Color Encoder [42) masks image color channels and learns to reconstruct them
image pixels, patches, and regions 48 8
from grayscale images. Recently, [8] 2 2T mask random image patches for MAE training. In particular, it is demonstrated in (21 that various image masking strategies can help suppress trivial solutions in MAE training while large mask ratios help learn better representations.
Different from most existing work that explores image masking in MAE training for learning better unsupervised representations, we explore the idea of image masking for GAN training and aim to develop robust image generation learners with limited training data. We design two orthogonal image masking strategies, namely, shifted spatial masking and balanced spectral masking that help learn robust image generation models effectively, more details to be described in the ensuing subsections.
3 Method
3.1
Task Definition
This work focuses the task of training GAN with limited training data. GAN is an unsupervised framework that aims to generate a model distribution that mimics a given target distribution: the generator G' learns to generate images G(z) given the latent code while the discriminator D learns to distinguish the generated images G(z) from the real images x. Conventionally, GAN is optimized alternatively with discriminator loss La and generator loss Lg that can be defined as:
La = E [fa(-D(z)) z~plu
+ E [f((D(G(2))] zmpp
(1)
Cg = E (g(-D(G()))) a<p:
(2)
3

---

Raw Image
Masked Image
Fourter Decompuse
Shiftad Spatial Masking
Image Masking Spatial Dimensions
Image Conposilion
Masked Image
Masked Image Localion
B Balanced
Unmasked Image Spectrum
Spectial Masking
Misked Tmage Spectrum
Spatial Dimensions
Spectral Dimension
Image Masking Spectral Dimension
Figure 2: Overview of the proposed MaskedGAN: MaskedGAN randomly masks certain image information for effective GAN training with limited training data. We develop two masking strategies that work along orthogonal image dimensions, including Shifted Spatial Masking that masks images along spatial dimensions with random shifts and Balanced Spectra Masking that masks certain image spectral bands based on self-adaptive probabilities.
where Pdata denotes the distribution of the provided real images and pz the prior distribution (e.g.- Gaussian distribution). The notations fa and fg stand for the loss mapping functions [19 38).
GAN training failures with limited data. Prior studies [58 31 show that GAN often experiences generation failures with severely degraded generation performance when only limited training data available. Specifically, with limited training data, the discriminator tends discriminate via meaningless shortcuts by merely focusing on casy-to-discrimintee image locations and spectra instead of holistic understanding of images. This can be viewed clearly Fig. of the appendix where the Gini Coefficient [17 of discriminator's spatial attentions increases quickly along the training iteration (when only limited training data is available). Note that the Gini coefficient [17] is negatively correlated with equality, i.e., the discriminator will pay more unevenly distributed attention to each spatial location while the Gini coefficient increases from '0' to 'I' For image generation with GAN, the large Gini coefficient thus means that the discriminator starts to focus on certain spatial locations (easy to discriminate) while ignoring other spatial locations (hard to discriminate), ultimately leading to over-confident discriminator and training collapse.
3.2 Masked Generative Adversarial Network
We tackle the challenging task of training GAN with limited data from perspective of image masking training [21] Specifically, we design two orthogonal image masking strategies, namely, shifted spatial masking and balanced spectral masking as illustrated in Fig. 2 The two masking strategies complement each other and encourage more challenging holistic understanding of images which help stabilize the adversarial training process and suppress trivial solutions and training failures.
Shifted spatial masking masks random image patches in spatial dimensions, aiming to force discriminator to learn over all image locations instead of merely focusing on easy-to-discriminate locations. first generates a random patch- mask and then introduces a random shift to the generated mask and finally applies the shifted mask to the training image. The shift operation allows to conduct spatial masking continuously along spatial dimensions: a masked patch could appear on any possible locations instead of fixed grid locations [2] 2. This feature is critical to image generation that requires high-fidelity along spatial dimensions.
Given an image E H first generate random patch-based spatial mask Mspatial (0, NW and apply random spatial shift d [0, D) on Mspatial acquire the shifted spatial mask Mapaiaa- The shifted spatial masking function can be defined by: Mispattal (r) = x X mpuuiaa In the patch-based mask generation,
set the grid size N x N, i.e., the patch size is # W N. We set the random shift range D as half of patch size, which allows the masked patch to appear on any possible locations. For images with multiple color channels, each channel are masked in the same manner.
Relations to the spatial masking in MAE. MAE aims to encode an image patch into a feature vector, which is downsampling process and does not require high precision along spatial dimensions (i.e., pixels within a patch are considered as the same). GAN, on the contrary, aims for decoding a latent code into a high-resolution image, which is an upsampling process and requires high fidelity along

---

spatial dimensions (i.e,, pixels within patch should not be completely the same). We therefore introduce an additional "mask shift" to allow the masked patches appear at any image locations (instead of fixed grid locations only as in the spatial masking in MAE 21 2 which helps generate images with high-fidelity along spatial dimensions.
Balanced spectral masking masks random spectral bands of an image in the decomposed spectral space, aiming to encourage the discriminator learn from all spectral bands instead of focusing on easy bands (e.g., low-frequency bands capturing color and brightness) only. Specifically, balanced spectral masking decomposes an image into multiple bands in spectral space and masks portion of spectral bands with adlfiade probabilities, i.e., spectral band is masked with higher (or lower) probability contains more (or less) contents. The balanced masking crucial to the spectral masking as image contents are usually unevenly distributed along the spectral dimension. Specifically, the contents of natural images are distributed very densely around low-free www-frequency bands but very sparsely in high-frequency bands. Therefore, uniform masking along the spectral dimension imbalanced and does not work well for image generation.
Given an image x E # we first employ Fourier transformation 53 23 24 56 to decompose it
into multi-band representation mal k#xxxx x'C', where C denotes the number of spectral bands. The image contents in each band I E Rº can then be computed by summing all contents in each band, L..,, 1 Σn Ew w (H,, and followed by normalization operation: I T( / 1
With the calculated In The balanced spectral mask Mppectral {0, 1 can be generated by masking e-th band with probability 1((() Finally, the balanced spectral masking function can be defined by: Mappectaal (x) ( (H,W.C) x m spppetaal
Loss functions. We apply the two image masking strategies to both discriminator and generator training, where the overall training losses can be formulated as follows:
La E (M(-D(M(=)) E [f((D(((G()))). z~PPmm
(3)
zmp.
fgg E z-pp
(4)
where M(.) Mappecrall (Mispanial) is the combination of shifted spatial masking and balanced
spectral masking.
Note the masking strategies work in different manners in the discriminator and generator training. Specifically, by applying image masking on input images in discriminator training, we could create challenging discriminator task that requires holistic learning beyond certain easy-to-discrimiatte information. Differently, in generator training, the image masking operation works on output side by sparsifying the training signals sack-propagated from the discriminator. One major consideration is to slow down the generator training so that it has similar learning pace as the discriminator which further stabilizes the overall adversarial learning between the discriminator and the generator.
3.3 Theoretical Insights
The proposed Masked Generative Adversarial Network (MaskedGAN) is inherently connected with the theory of stochastic approximation.
Proposition 1
The MaskedGAN can be modeled as an instance of the Two Time-Scale Update Rule.
Proposition 2
The MaskedGAN converges to a Local Nash Equilibrium under certain conditions.
The proofs of Propositions and 2 are provided in the Appendix.
Experiments
We evaluate MaskedGAN over different network architectures and public datasets. Sections
presents experiments with BigGAN over datasets CIFAR-10 34 CIFAR-100 34 and ImageNet [5]
Section 4.2 reports experimental results with StyleGAN-v2 over datasets 100-shot (58) AFHQ
and FFHQ 32). Section 4.3 presents experiments with two transformer-bssed GANs (TransGAN and
5

---

Table I: Conditional image generation with BigGAN on CIFAR-10 and CIFAR-100. MaskedGAN (i.e., Masked -iggGAN) achieves superior performance especially with limited training data. We calculate FID (L) scores with 10K generated samples and the validation set, as in [58]
CIFAR-10
CIFAR-IO1
Method
100% Data
20% Data
10% Data
100% Data
20% Data
10% Data
Non-saturated GAN
9.83 0.06
41.99
0.18
13.87 0.08
32.64 0.19
70.5 0.38
LS-GAN 36
9.07 = 0.01
0.16
41.68
0.18
12.43 + 0.11
27.09 0.09
54.69 0.12
RAHinge GAN
29
11.31 0.04
23290
48.13 0.33
14.61 0.21
18.79
52.72 0.18
StyleGAN-v2 33
11.07 + 0.03
23.08 0.11
36.02 0.15
16.54 # 0.04
32.30 0.11
45.87 # 0.15
BigGAN 4
(baseline)
9.74
48.08 0.28
13.60
32.99
66.71 = 0.01
LeCam-GAN 46
8.00
15.27
3.12
11.88
25.51
49.63 0.16
GenCo 13
8.83
1.057
2.108
1.09
2.015
40.98 0.09
ADA 31
0.09
1.09
0.11
0.02
0.16
27.08 0.15
DA [58]
0.03
0.110
2.09
11.99
2.06
35.39 0.08
MaskedGAN
8.41
0.009
15.8
11.65 0.03
18.33
24.02 0.12
GANformer) over datasets CIFAR-10, CIFAR-100 and Cityscapes [12. In addition, we present abla- tion studies in Section 4.1 and discuss different features of the proposed MaskedGAN in Section 4.4 The implementation and dataset details are provided in appendix.
4.1
Conditional image generation with BigGAN CIFAR-10, CIFAR-100 and ImageNet
Table 2: Ablation studies of MaskedGAN with BigGAN (baseline) on 10% CIFAR-10 data.
Shifted Spatial Masking
Balanced Spectral Masking
Method
FID (1)
Spatial Masking
Random Shift
Spectral Masking
Self-adaptive Probubility
BigGAN
48.08
28.35
22.56
35.20
28.78
MaskedGAN
15.89
Table 3: Conditional image generation with BigGAN on ImageNet. MaskedGAN (Le., Masked- BigGAN) improves the image generation consistently across different setups of the large-scale ImageNet. We evaluate the models with 10K generated samples and the whole training set.
10% training data
5% training data
2.5% training data
Method
ISt
FID
ISt
FIDI
ISa
FID
BigGAN 4 (baseline)
10.94 0.35
3
6.13 + 0.09
91.16
3,92 + 0.07
133.80 + 0.76
ADA 31
12.67 =0.31
31.89 20.17
90.25
8.54
56.83 0.48
DA 58
0..3
32.82
9.63 0.21
56.75 0.335
8.17 0.28
63.49 + 0.51 0.51
MaskedGAN
13.34
26.51 # 0.12
12.85 0.40
3..70
12.68 = 0.27
38.62 = 0.37
class-conditional image generation results with baseline BigGAN 4 on CIFAR-10 and CIFAR-100. All models are trained with 100%, 20% or 10% training data (i.e., 50K, 1OK or 5K images), and evaluated over the validation set (10K images). It shows that MaskedGAN (i.e., Masked Masked-BigGAN BigGAN) achieves superior performance over CIFAR- and CIFAR-100 as compared with state-of taa---ffthhhteee art methods. Results are averaged with three runs.
CIFAR-10 and CIFAR-100.
Ablation studies over CIFAR-10. We perform ablation studies with the widely adopted BigGAN over 10% CIFAR- 10 data as shown Table the core of the proposed MaskedGAN, we examine how our designed shifted spatial masking and balanced spectral masking contribute to the overall performance of data-limited image generation. As shown in Table 2, the baseline (BigGAN) in the first row does not perform well with limited training data. Including the basic Spatial Masking in

---

Table Unconditional image generation with StyleGAN-v2 on 100-shot. With only 100 training images, MaskedGAN (ie., SaskeedSStleGAAN... outperforms the baseline StyleGAN-v2 signifi- cantly in FID (L). It also ouiperforms the state-of-the-an consistently, with and without pre-training.
Pre-training
100-shot
Methods
w/ 70K images
Obama
Grumpy Cat
Panda
Scale/shift 41
Yes
50.72
34.20
21.38
MineGAN [49
Yes
50.63
34.54
14.84
TransferGAN [50]
Yes
48.73
34.06
23,20
TransferGAN DA
58
Yes
39.85
29.77
17.12
FreezeD 40
Yes
41.87
31,22
17.95
StyleGAN-v2
33
(baseline)
No
80.20
48,90
34.27
ADA [31
No
45.69
26.62
12.90
LeCam-GAl GAN 46
No
38.58
41.38
19.88
GenCo 13
No
36.35
33.57
15.50
AdvAug 9
No
52.86
31.02
14.75
DA [58
No
46.87
27.08
12.06
APA 27
No
43.75
28.49
12.34
InsGen 52
No
45.85
27.48
12.13
MaskedGAN
No
33.78 + 0.27
20.06 0.13
8.93 # 0.06
Table 5: Unconditional image generation with StyleGAN-v2 on AFHQ and FFHQ. MaskedGAN (i.e., Masked-StyleGANee works effectively with limited training data face generation tasks. We calculate FID (4) scores with 10K generated samples and the whole training set.
APHQ-Cal
AFHQ-Dog
Method
20% Data
10% Data
5% Data
20% Data
10% Data
5% Data
StyleGAN-22 33 (baseline)
12.51 0.09
17.67 + 0.11 15.27 = 0.09
26.27 0.21
$..24
$6.84 = 0.36 46.28 0.25
95.32 0.51 46.68 0.27
MaskedGAN
12.03 0.10
0.13
42.25 0.22
AFHQ- FilQ-wild
FFHQ
Method
20% Data
10%
5%
5% Data
2.5%
1.25% Data
StyleGAN-22
33
(baseline)
+
33.85 +
30.06 0.25
127
MaskedGAN
7.76 = 0.02
10.39 = 0.09
=
28.53 0.14
3..04:0.21
0.234
GAN training improves the baseline clearly shown in the 2nd row, while further involving Random Shift on top of Spatial Masking brings clear improvement in FID as shown the 3rd row. On the other hand, the basic Spectral Masking (masks spectral bands with the same probability) does work well as shown in the 4th row as image contents are usually unevenly distributed along the spectral dimension. However, Balanced spectral masking which includes the proposed Self-adaptive Probabiliries shown in the 5th row, improves image generation clearly in FID (19.30 above the baseline), showing that the balanced masking strategy is crucial for effective spectral masking. In addition, the two image masking strategies work in orthogonal dimensions (i.e., spatial and spectral dimensions), which complement each other data-limited image generation. We can observe clearly that combining these two masking strategies (MaskedGAN in the last row) performs the best.
ImageNet. Table 3 reports caas-conniiional generation results with BigGAN 4 over ImageNet 64 x 64 resolution. The models are trained with different amounts of training data including - 10% (100K images), ~5% (50K images) and -2.5% (25K images), and evaluated set (~1.2 million images). Experiments show that MaskedGAN works effectively with different data setups over the very diverse and large-scale ImageNet. Results are averaged with three runs.
the whole training
4.2 Unconditional image generation with StyleGAN-v2 on 100-shot, AFHQ and FFHQ
100-shot. Table 4 reports unconditional image generation results with StyleGAN-v2 on 100-shot datasct. All models trained and evaluated over 100 images of Obama, Grumpy Cat and Panda,
7

---

Table 6: Unconditional image generation with TransGAN on CIFAR-10 and CIFAR-100.
MaskedGAN works effectively with transformer based GAN (TransGAN) as well. We calculate FID ()) scores with 10K generated samples and the validation set.
CIFAR-10
CIFAR- 100
Method
100% Data
20% Duta
10% Data
100% Data
20% Data
10% Data
TransGAN 28 (buseline)
22.53 0.18
53.58 +.38 0.31 18.29 + 0.13
68.59 0.40
25.73 0.22
58.61 = 0.37
73.95 0.42 32.28 # 0.25
MaskedGAN
8.71 # 0.09
29.23 = 0.21
16.88 + 0.15
26.17 = 0.23
Table 7: Unconditional image generation with GANformer on Cityscapes. MaskedGAN brings significant gains over the baseline GANformer consistently. We calculate the FID (L) scores with 10K generated samples and the whole training set.
Training Data
2.5% training data
1.25% training data
Method
1St
FID
1St
FID
IS1
FIDU
GANformer 26 (baseline)
1.61 +: 0.02
34.87 + 0.25
1.56 0.02
62.23 + 0.42
1.44 0.01
108.73 = 0.58
MaskedGAN
+ 29.84 + 0.21
1.60 0.02
55.74 0.32
1.56 0.02
75.75 + 0.45
respectively. Experimental results show that the proposed MaskedGAN works effectively with only 100 images, i.e., brings significant gains over StyleGAN-v2 (baseline) and outperforms state-of-the art methods by clear margins. Besides, MaskedGAN surpasses all transfer learning methods but does not require pre-training over large-scale datasets. Results are averaged with three runs.
AFHQ and FFHQ. Table presents unconditional image generation results with StyleGAN-v2 4 on AFHQ and FFHQ For AFHQ, the models are trained with different amounts of training data including -20% (1K images), ~10% (500 images) and -5% (250 images), and evaluated over the whole training set 5K images). For FFHQ, the models are trained with ~10% (7K images), -5% (3.5K images) and -2.5% (1.75K images) data, and evaluated over the whole training set (70K images). It can observed that MaskedGAN works effectively with limited training data over animal and human face generation tasks. Results are averaged with three runs.
4.3 Unconditional image generation with TransGAN and GANformer
We evaluate MaskedGAN with recent transformer-based architectures including TransGAN 28 and GANformer 266 Specifically, TransGAN is the first work that uses purely transformers for image generation. GANformer employs bipartite structure for computing soft attention, which circumvents the computation constraints in standard transformers and leads to improved image generation.
CIFAR-10 and CIFAR-100 with TransGAN. 6 reports unconditional image generation with TransGAN CIFAR-10 and CIFAR-100. The models trained with 100%, 20% 10% training data (i.e., 50K, 10K or 5K images), and evaluated over the whole training set (50K images). We can observe that image masking strategies work effectively with wansformer-based GAN well. Results are acquired with three runs.
Cityscapes with GANformer. Cityscapes. The models are trained with ~5% ~2.5% or -1 1.25% training data (i.e., 1.25K, 0.63K or 0.31K images) and evaluated over the whole training set (i.e., ~25K images ). Experiments show that our image masking strategies bring significant gains over the bascline GANformer consistently over scene generation task. Results acquired with three runs.
Tables 7 reports unconditional image generation with GANformer on
4.4 Discussion
Generalization across different GAN architectures: We examine the generalization of the proposed MaskedGAN by evaluating it with four representative GAN architectures, including two CNN-based (i.e., BigGAN and StyleGAN-v2) and two Transformer-based (i.e., TransGAN and GANfomer).. Experimental results Tables [T- show that our MaskedGAN brings clear generation improvements consistently over different GAN architectures.
8

---

FID (4) 10% CIFAR-10 Data
(4) with 10% ImageNet Data
FID (4) 2.5% ImageNet Data
100
100
200
BiGAAN
BigDAM
40
Lecams
50
MasceduAM
150
MascedGAM
40
MeskedGAN
60
100
40-
40
20
20
25
50
75
160
13%
150
175
700
70
1000
50
Training Iteration
Le3
Training Iteraaioon
143
Training Iteraaioo
143
FID (4) with 100-shot Obama
FID (4) with 2.5% Cityscapes Data
FID with 10% CIFAR-10 Data
200
175-
150-
100
50
50
50
100
150
200
250
300
350
50
5
10 15
20
25
30
35
40
10
20
30
40
50
60
70
80
Training Iteration
Training Iteration
Training ltteration
Figure 3: Convergence of GAN training with limited training data: MaskedGAN converges well consistently across different datasets (e.g., CIFAR-10, ImageNet, 100-shot and Cityscapes), different amounts of training data (e.g., 10% and 2.5% of ImageNet), and different generation architectures (e.g., BigGAN, StyleGAN-v2. GANformer and TransGAN). Most existing GANs instead experience clear training collapses with limited training data. The recent TransGAN can converge but it trains severely degraded generation model (-68 in FID).
StyleGAN-22 (beselime)
MaskedGAN (ours)
Figure 4: Qualitative illustration and comparison over 100-shot Obama (top) and AFHQ- Wiil with 5% data (bottom): With limited training data, introducing our image masking into GAN training helps generate more realistic and high-fidelity images, especially in terms of image shapes and structures.
Generalization across different generation tasks: We study the generalization of the proposed MaskedGAN from the perspective of image generation tasks. Specifically, we perform extensive evaluations over object generation tasks with CIFAR-10, CIFAR-100 and ImageNet, face generation tasks with 100-shot AFHQ, and FFHQ, and scene generation tasks with Cityscapes. Experimental results in Tables I 7 show that the proposed MaskedGAN achieves superior generation consistently over different generation tasks.
Generalization over different numbers of training samples: We study the generalization of the proposed MaskedGAN from the perspective of the number of training samples. Specifically, we benchmark MaskedGAN on CIFAR-10 and CIFAR- with 100%, 20% and 10% data, ImageNet with 10%, 5% and 2.5% data, AFHQ with 20%, 10% and 5% data FFHQ and Cityscapes with 5%, 2.5% and 1.25% data and 100-shot dataset with only 100 images. Experiments Tables T show that MaskedGAN achieves superior performance consistently with different amounts of training data.
Convergence comparison across differrntt network architectures and datasets: We examine the convergence of the proposed MaskedGAN by benchmarking it over various network architectures (e.g., based including BigGAN and StyleGAN-v2 and Transformers-basee including TransGAN and GANformer) and datasets (e.g.. CIFAR- 10, ImageNet, 100-shot and Cityscapes). Fig. provides the line charts of FID versus training iteration. It shows clearly that most state-of -fate-of-the-arr art GANs B33 experience generation failures (or called training collapses) with severely degraded generation performance when only limited training data available. In addition, prior studies on data-limited generation (e.g., data augmentation method (58) and model regularization method (46)) could improve the performance in some degree but still suffer from generation failures and training collapses. As comparison, the proposed MaskedGAN converges well consistently across different amounts of

---

AFHO-DOO Dog
AFHQ- Wild
FFHQ
StyleGAN-V2 (Baseline)
MaskedGAN
Figure 5: Qualitative illustration and comparison over AFHQ-CA AFHQ-Dog and AFHQ Wild with 5% training data and FFHQ with 1.25% data: With limited training data, introducing the proposed image masking strategies into GAN training helps generate more realistic and high-fidelity images, especially in terms of image shapes and structures.
training data, network architectures and datasets. The great convergence of MaskedGAN is largely attributed to two factors: its image masking designs suppress trivial solutions and training failures directly; it keeps similar learning paces for discriminator and generator which ensures that network converge to a Local Nash Equilibrium under certain conditions [22]
We provide qualitative illustrations in Fig. and which show that MaskedGAN outperforms the baseline and state-of-the-ar models clearly. Specifically, with limited training data, introducing the proposed image masking strategies into GAN training helps generate more realistic and high-fidelity images, especially in terms of image shapes and structures. Due the space limit, we provide additional discussions and qualitative illustrations (including comparisons with the state-of-the the-arts)s) in appendix.
Conclusion
This paper presents MaskedGAN, a robust image generation network that can learn effectively with limited training data. MaskedGAN introduces two image masking strategies that work along orthogonal dimensions of training images, namely, shified spatial masking and balanced spectral masking. The two masking strategies complement each other which together encourage more challenging holistic learning from limited training data, ultimately suppressing trivial solutions and failures GAN training. Extensive experiments show that MaskedGAN achieves superior performance consistently across different network architectures (e.g., CNNs including BigGAN and StyleGAN-v2 and Transformers including TransGAN and GANformer) and datasets (e.g., CIFAR- 10, CIFAR- 100, ImageNet, 100-shot, AFHQ, FFHQ and Cityscapes). Moving forward, will explore data masking training in other generation tasks such multi-modality generation.
Acknowledgement
This study is supported under the RIE2020 Industry Alignment Fund Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from Singapore Telecommunications Limited (Singtel), through Singtel Cognitive and Artificial Intelligence Lab for Enterprises ESCALE@NTI This research was also carried out on the High Performance Computing resources Inception Institute of Artificial Intelligence, Abu Dhabi.
10

---

References
[1] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pages 214-223. PMLR, 2017.
[2] Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106. 08254, 2021.
[3] Piotr Bojanowski and Armand Joulin. Unsupervised learning by predicting noise. In International Conference on Machine Learning, pages 517- 526. PMLR, 2017.
[4] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. International Conference on Learning Reprexentations, 2018.
[5] Shuhao Cao, Peng Xu, and David A Clifton. How understand masked autoencoders. arXiv preprint arXiv:2202.0367 2022.
[6] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Mathhijs Douze. Deep clustering for unsupervised learning of visual features. Proceedings of the European conference on computer vision (ECCV), pages 132-149, 2018.
[7] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsu- pervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33:9912 19-9942,,,4. 2020.
[8] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International Conference on Machine Learning, pages (691-1703. PMLR, 2020.
[9] Tianlong Chen, Yu Cheng, Zhe Gan, Jingjing Liu, and Zhangyang Wang. Data-efficient gan training beyond (just) augmentations: lottery ticket perspective. Advances in Neural Information Processing Systems, 34, 2021.
[10] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 15750-15758, 2021.
[11] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. Proceedings of IEEE/CVF conference on computer vision and pattern recognition, pages 8188 8197, 2020.
[12] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler. Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9213-3223, 2016.
[13] Kaiwen Cui, Jiaxing Huang, Zhipeng Luo, Gongjie Zhang, Fangneng Zhan, and Shijian Lu. Genco: generative co co-training for generative adversarial networks with limited data. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 499-507, 2022.
[14] Kaiwen Cui, Gongjic Zhang, Fangneng Zhan, Jiaxing Huang, and Shijian Lu. Fbe-gan: diverse and flexible image synthesis via
composition. arXiv preprint arXiv:2107.0311 2021.
[15] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. lcce, 2009.
[16] Emily Denton, Soumith Chintala, Rob Fergus, et al. Deep generative image models using a laplacian pyramid of adversarial networks. Advances in neural information processing systems. 28, 2015.
[17] Robert Dorfman. A formula for the gini coefficient. The review of economics and statistics, pages 146 149, 1979.
[18] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv: 1929,, 2020.
[19] Ian Goodfellow, Jean Pouget-Abadee Mehdi Mirza, Bing Xu, David Warde Fardee, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672-2680, 2014.

---

[20] Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. Advances in neural information processing systems, 30, 2017.
[21] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint arXiv: 1.0:337, 2021.
[22] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[23] Jiaxing Huang, Dayan Guan, Aoran Xiao, and Shijian Lu. Fsdr: Frequency space domain randomization for domain generalization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 6891-6902, 2021.
[24] Jiaxing Huang, Dayan Guan, Aoran Xiao, and Shijian Lu. Rda: Robust domain adaptation via fourier adversarial attacking. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1988,999,, 2021.
[25] Jiaxing Huang, Shijian Lu, Dayan Guan, and Xiaobing Zhang. Contextual-relation consistent domain adaptation for semantic segmentation. In European Conference on Computer Vision, pages 705-722. Springer, 2020.
[26] Drew A Hudson and Larry Zitnick. Generative adversarial transformers. In International Conference on Machine Learning, pages 4487-4499. PMLR, 2021.
[27] Liming Jiang, Bo Dai, Wayne Wu, and Chen Change Loy. Deceive d: Adaptive pscudo augmentation for gan training with limited data. Advances in Neural Information Processing Systems, 34, 2021.
[28] Yifan Jiang. Shiyu Chang, and Zhangyang Wang. Transgan: Two pure transformers can make one strong gan, and that can scale up. Advances in Neural Information Processing Systems, 34, 2021.
[29] Alexia Jolicoeur- The relativistic discriminator: a key element missing from standard gan. arXiv preprint 807.00034, 2018.
[30] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv: 10196, 2017.
[31] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lchtinen, and Timo Aila. Training generative adversarial networks with limited data, Advances in Neural Information Processing Systems. 33:12104-12114 14, 2020.
[32] Tero Karras, Samuli Laine, and Timo Aila. A style- generator architecture for generative adversarial networks. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 4401-4410, 2019.
[33] Tero Karras. Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8110-88 8119, 2020.
[34] Alex Krizhevsky, Geoffrey Hinton, et Learning multiple layers features from tiny images. 2009.
[35] Steven Liu, Tongzhou Wang, David Bau, Jun- Yan Zhu, and Antonio Torralba. Diverse image generation via self-conditioned gans. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4428-14425 2020.
[36] Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang. and Stephen Paul Smolley. Least squares generative adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 2794-2802, 2017.
[37] Lars Mescheder, Andreas Geiger, and Sehastian Nowozin. Which training methods for gans do actually converge? In International conference on machine learning, pages 3481-3490. PMIR, 2018.
[38] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957 2018
[39] Takeru Miyato and Masanori Koyama. egans with projection discriminator. In International Conference Learning Representations, 2018
[40] Sangwoo Mo, Minsu Cho, and Jinwoo Shin. Freeze discriminator: A simple baseline for fine-tuning gans. arXiv preprint arXiv:2002.10964, 2020.
12

---

[41] Atsuhiro Noguchi and Tatsuya Harada. Image generation from small datasets via batch statistics adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2750-2758, 2019.
[42] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2536 2544, 2016.
[43] Manuel Porear, Antoine Danchin, Victor de Lorenzo, Vitor A Dos Santos, Natalio Krasnogor, Steen Rasmussen, and Andrés Moya. The ten grand challenges of synthetic life. Systems and synthetic biology, 5(1):1-9, 1.
[44] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511006434, 2015.
[45] Tim Salimans, lan Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
[46] Hung- Yu Tseng, Lu Jiang, Ce Liu, Ming-Hsuan Yang, and Weilong Yang. Regularizing generative adversarial networks under limited data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7921-7931, 2021.
[47] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and fierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. Proceedings of the 25th international conference on Machine learning, pages (096-1103, 2008.
[48] Pascal Vincent, Hugo I. Larochelle, Isabelle Lajoie, Yoshua Bengio, Picrre-Antoine Manzagol, and Léon Bottou. Stacked denoising autoencoders: Learning useful representations a deep network with local denoising criterion. Journal of machine learning research, (12)), 2010.
[49] Yaxing Wang, Abel Gonzalez-Gareen Garcia, David Berga. Luis Herranz. Fahad Shahbaz Khan, and Joost van de Weijer. Minegan: effective knowledge transfer from gans to target domains with few images. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9332-9341, 2020. 2020,
[50] Yaxing Wang, Chenshen Wu, Luis Herranz, Joost van de Weijer, Abel Conzalez-Garche and Bogdan Raducanu. Transferring gans: generating images from limited data. In Proceedings of the European Conference on Computer Vision (ECCV), pages 234,, 2018.
[51] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. arXiv preprint arXiv:2111 1.09886, 2021.
[52] Ceyuan Yang. Yujun Shen, Yinghao Xu, and Bolei Zhou. Data-efficient instance generation from instance discrimination. Advances in Neural Information Processing Systems, 34:9378-9390, 2021.
[53] Yanchao Yang and Stefano Soatto. Fda: Fourier domain adaptation for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4085 4095, 2020.
[54] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Generative image inpainting with contextual attention. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 5505-5514, 2018
[55] Han Zhang, lan Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial PMLR, 2019.
networks. In International conference on machine learning, pages
[56] Jingyi Zhang, Jiaxing Huang, Zichen Tian, and Shijian Lu. Spectral unsupervised domain adaptation for visual recognition. In Proceedings of the IEEE/VVF Conference on Computer Vision and Pattern Recognition, pages 9829-9840, 2022.
[57] Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European conference on computer vision, pages 649 666. Springer, 2016.
[58] Shengyu Zhao Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for data- efficient gan training. Advances in Neural Information Processing Systems, 33:7559-7570, 2020.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contribu- tions and scope? [Yes]
(b) Did you describe the limitations your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Following please find them. This work could mitigate ethical concerns privacy violation for using massive images in training GAN, Current GANs require using large amounts of training data for training effective generation models which could raise concerns in data privacy when collecting large amounts of training data. This work proposes MaskedGAN for data-limited image generation. which aims learn effective GANs with limited data (e.g.. 100 images). Thus, it could relieve the reliance on large amounts training data and so risk privacy violation.
(d) Have you read ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state full set of assumptions of theoretical results? [Yes]
(h) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as URL)? [N/A]
(b) Did you specify the training details (e.g., data splits, hyperparamcters, how they were chosen)? [Yes]
(c) Did you report error (e.g., with respect to the random seed after running experiments multiple times)? [Yes]
(d) Did you include total amount of compute and the type resources used (e.g., type GPUs, internal cluster, or cloud provider)? [Yes]
If you using existing assets (e.g., code, data, models) curating/relcasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're us-
ing/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable informa- tion or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you descrihe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "yE9hR5bWMB", "reviewer": "Reviewer_LGsj", "summary": "This paper proposes two novel masked-based data augmentation methods for image generation task with GAN, namely shifted spatial masking and balanced spectral masking. Extensive experiments are conducted to demonstrate the effectiveness and generalizability of the proposed methods.", "questions": "1. Is the proposed method orthogonal to existing augmentation methods like [1][2]? The author might consider to add an experiment that using both the proposed mask-based augmentation with either [1] or [2] to see whether this would give an extra performance boost.\n\n[1] Zhao, S., Liu, Z., Lin, J., Zhu, J. Y., & Han, S. (2020). Differentiable augmentation for data-efficient gan training. Advances in Neural Information Processing Systems, 33, 7559-7570.\n\n[2] Karras, T., Aittala, M., Hellsten, J., Laine, S., Lehtinen, J., & Aila, T. (2020). Training generative adversarial networks with limited data. Advances in Neural Information Processing Systems, 33, 12104-12114.", "limitations": "The author may consider to put more visualization results in the paper.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pro:\n1. The paper is well-written and easy to follow.\n2. The proposed method is technically sound and intuitive. The paper is well-motivated. \n3. Extensive experiments are conducted to support the author's points.\n4. Competitive performances are achieved by using the proposed method. The proposed method is generalizable enough to a lot of GAN architectures.\n\nCons:\n1. Missing augmentation comparison: in table 3, I think a proper comparison would be 1. biggan, 2. biggan + diffaug[1], 3. biggan + [2], 4. biggan + masked-aug (MaskedGAN). Same issue also exists in table. 5~7.\n\n[1] Zhao, S., Liu, Z., Lin, J., Zhu, J. Y., & Han, S. (2020). Differentiable augmentation for data-efficient gan training. Advances in Neural Information Processing Systems, 33, 7559-7570.\n\n[2] Karras, T., Aittala, M., Hellsten, J., Laine, S., Lehtinen, J., & Aila, T. (2020). Training generative adversarial networks with limited data. Advances in Neural Information Processing Systems, 33, 12104-12114.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "xeV6DWa8hwx", "reviewer": "Reviewer_aPDi", "summary": "This paper propses MaskedGAN to help GANs learn from the limited data by introducing two image masking strategies. One is the shifted spatial masking and the other is the balanced spectral masking. Those two masking strategies complement each other which together encourage GANs learning effectively and robustely on the limited data.", "questions": "1. Differences from data augmentation? As it lacks the reconstruction part of MAE, the masking strategy can be regarded as another type of data augmentation. From this perspective, what if we have sufficient data, is improvement consistent? Besides, the relation to data augmentation is required to be discussed in related work.\n\n2. As claimed in the contribution summary, the holistic understanding of images is encouraged. Is there any experimental support? ", "limitations": "The authors do not adequately addressed the limitations and potential negative societal impact of their work.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe motivation of this paper is clear, and the authors did lots of experiments to demonstrate the effectiveness of their method. But the experiment setting part needs to be improved as listed in the weaknesses or question part.\n\n\nWeaknesses:\n\n1. More data augmentation methods are supposed to be involved in the comparison of CIFAR and Imagenet. For example, StyleGAN2-ADA is required rather than the original StyleGAN2, especially with limited training data. Overall, some tables seem unfair since many baselines are without data augmentations, while the proposed method could be regarded as one novel data augmentation. \n\n2. Some important baselines are missing. For instance, InsGen and APA (both are NeurIPS 2021 work) achieved strong performances with limited training data on FFHQ, and AFHQ.\n\nInsGen: Data-Efficient Instance Generation from Instance Discrimination, Yang. et al, NeurIPS 2021.\nAPA: Deceive D: Adaptive Pseudo Augmentation for GAN Training with Limited Data, Jiang, et al. NeurIPS 2021.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "enUYom-mzlM", "reviewer": "Reviewer_S4RA", "summary": "This paper proposes MaskedGAN to deal with the problem of training GAN with limited data. MaskedGAN uses two masking strategies, shifted spatial masking (masking spatial patches) and balanced spectral masking (masking spectral bands), on both real images and generated images during the training of GAN. Experimental results shows the effectiveness of the proposed method over the original GAN, as well as the superiority over other techniques for training GAN with limited data, across different architectures and datasets.", "questions": "1. My major concern is why the comparison results with other techniques for training GAN with limited data on AFHQ, FFHQ, and ImageNet are missing. The experimental results are not convincing enough without them.\n\n2. Masking, including spatial masking and spectral masking, can also be viewed as a kind of data augmentation strategy. What makes masking, or the specially designed masking in this paper, superior to other data augmentation strategies?\n\n3. How much does the computation overhead increase after adding the masking strategy during GAN training?", "limitations": "The authors did not state the limitation of the proposed method clearly in the paper. They only claimed possible future applications, e.g., applying MaskedGAN to multi-modality generation, but those are not the limitation of the method or result itself.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n1. The idea of developing masking strategies to enhance the training of GAN with limited data is great. It shows the power of masking in the generative field. \n\n2. The results are strong. This paper tests the proposed method with different architectures and datasets, and shows consistent and remarkable improvements over baselines, especially when the ratio of available data is small. \n\nCons:\n\nIn the experiments, the proposed method compares with other techniques for training GAN with limited data (e.g., DA, GenCo, etc) only on very small datasets (e.g., CIFAR, 100-shot) but not on large datasets (e.g., ImageNet, FFHQ). On large datasets, it only compares with the baseline.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "EeUwAi4XSQ", "reviewer": "Reviewer_mM3P", "summary": "This work suggests two strategies for masking in GANS and it claims that their Mask Generative Adversarial Networks (MaskedGAN) are robust with limited training data.\nThe two strategies are:\n\n1) Shifted spatial masking (random shifts in the spatial domain)\n2) Balanced spectral masking (multiple bands with self-adaptive probabilities)\n\nHypothesis: masking helps the model to learn hard-to-discriminate band and create a challenging scenario\nSupport for the hypothesis: training on multiple architectures (not covering all datasets)\nLimited dataset setup:\n10%(5K), 20%(10K), and 100%(50K) of data on CIFAR10, 100 \n2.5%(25K), 5%(50K), and 10%(100K) of data on ImageNet\n\n", "questions": "- What do you mean by Masking? Are you zeroing some pixels? In that case, are you using Gated or Partial Convolutions? If yes, then the models are different, and if no, how do you handle the normalization of the output to adjust for the fraction of missing data?\n- Is the sampled data heterogeneous or homogeneous? How have you handled it?\n- What are the learning rate, batch-size, and epochs of training for each of the experiments?\n- In Fig2, I don't understand how the masked image is used in the model? Specifically, can you make it clear if the model receives the right images directly? If it receives the lower image (Masked Image in the spectral domain), it seems a completely different image! Do some of the pixels retain the original pixel values?\n", "limitations": "No. They mentioned the limited data training, which is the main setting of the paper, but as far as I understand, listing the limitation means to discuss where the methodology does not work.", "rating": 3, "confidence": 5, "soundness": 1, "presentation": 1, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper introduces an interesting subject for the community.\n- There are multiple experiments.\n- Multiple models are used for the experiments.\n\nWeaknesses:\n- The writing is somewhat incoherent and unclear: in the abstract and introduction, problem statements lack motivation and clarity.\n- The literature review section does not provide coherent statements for each cited work, and it bulk references some papers that might not be directly related. e.g. for Masked Autoencoders, it references basic papers of autoencoders(45, 39, 8, 17, 2, 20, 48), which needs to be discussed in the Autoencoder section.\n- The mentioned shifting and Fourier domain manipulation (masking) are not clearly discussed.\n- It is not clear how the sampling happens in the limited data scenario and if it is heterogeneous or homogenous.\n- The number of training rounds is not specified.\n- It seems it works as an augmentation, and with multiple random masking, we actually increase the number of data samples. I think after mentioning the number of iterations, it would be appropriate to compare with SOTA augmentation techniques in GANs.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept