Mitigating Mode Collapse by Sidestepping Catastrophic Forgetting
Authors: Karttikeya Mangalam
Rohin Garg
Jathushan Rajasegaran
Taesung Park
Venue:  (2021)
Keywords: mode collapse
catastrophic forgetting
multi-adversarial training

Abstract: Generative Adversarial Networks (GANs) are a class of generative models used for various applications, but they have been known to suffer from the mode collapse problem, in which some modes of the target distribution are ignored by the generator.  Investigative study using a new data generation procedure indicates that the mode collapse of the generator is driven by the discriminator’s inability to maintain classification accuracy on previously seen samples, a phenomenon called Catastrophic Forgetting in continual learning. Motivated by this observation, we introduce a novel training procedure that dynamically spawns additional dis-criminators to remember previous modes of generation. On several datasets, we show that our training scheme can be plugged-in to existing GAN frameworks to mitigate mode collapse and improve standard metrics for GAN evaluation.

Under review as a conference paper at ICLR 2021
MITIGATING MODE COLLAPSE By SIDESTEPPING CATASTROPHIC FORGETTING
Anonymous authors Paper under double-blind review
ABSTRACT
Generative Adversarial Networks (GANs) are a class of generative models used for various applications, but they have been known suffer from the mode col- lapse problem, in which some modes of the target distribution are ignored by the generator. Investigative study using new data generation procedure indicates that the mode collapse of the generator is driven by the discriminator's inability to maintain classification accuracy on previously seen samples, a phenomenon called Catastrophic Forgetting in continual learning. Motivated by this observation, we introduce a novel training procedure that dynamically spawns additional dis- criminators to remember previous modes of generation. On several datasets, we show that our training scheme can be plugged-in to existing GAN frameworks to mitigate mode collapse and improve standard metrics for GAN evaluation.
INTRODUCTION
Generative Adversarial Networks (GANs) (Goodfellow et al. 2014) are extremely popular class of generative models that is not only used for text and image generation, but also in various fields of science and engineering, including biomedical imaging (Yi al. 2019 Nie et al. 2018: Wolterink itt al. 2017), autonomous driving (Hoffman et al.) 2018 Zhang et al. 2018), and robotics (Rao et al. 2020 Bousmalis al., 2018). However, GANs are widely known be prone to mode collapse, which refers to situation where the generator only samples a few modes of the real data, failing to faithfully capture other more complex or less frequent categories. While the mode collapse problem is often overlooked in text and image generation tasks, and even traded off for higher realism of individual samples (Karras et al., 2019: Brock et al., 2019), dropping infrequent classes may cause serious problems in real-world problems, in which the infrequent classes represent important anomalies. For example, a collapsed GAN can produce racial/gender biased images (Menon et al. 2020).
Moreover, mode collapse causes instability in optimization, which can damage not only the diversity but also the realism of individual samples the final results. As an example, we visualized the training progression of the vanilla GAN (Goodfellow et al. 2014) for simple bimodal distribution in the top row of Figure I At collapse, the discriminator conveniently assigns high realism to the region unoccupied by the generator, regardless of the true density of the real data. This produces a strong gradient for the generator to move its samples toward the dropped mode, swaying mode collapse to the opposite side. In particular, the discriminator loses its ability to detect fake samples was previously able to, such as point X.. The oscillation continues without convergence.
From this observation, we hypothesize that the mode collapse problem in GAN training is closely related to Catastrophic Forgetting (McCloskey & Cohen 1989. McClelland et al. 1995] Ratcliff 1990) in continual learning. That is, since the distribution of the generated samples is not stationary. the discriminator forgets to classify the previously generated samples as fake, hindering convergence of the GAN minimax game. promising line of works (Zhang et al., 2019b; Rajasegaran al.. 2019: Rusu et al.) [0166] Fernando et al. 2017) tackle the problem in the supervised learning setting by instantiating multiple predictors, each of which takes charge in a particular subset of the whole distribution. Likewise, we also tackle the problem of mode collapse in GAN by tracking the severity of Catastrophic Forgetting by storing a few exemplar data during training, and dynamically spawning an additional discriminator if forgetting is detected, as shown in Figure [] The key idea is that the added discriminator is left intact unless the generator recovers from mode dropping of that sample, essentially sidestepping catastrophic forgetting.

---

Under review as a conference paper at ICLR 2021
Vanilla GAN
DMAT (ours)
Discriminator
Discriminator 2 (new)
400
800
1200
1600
#iter
Spawn new discriminator
Figure I: Visualizing training trajectories: We visualize the distribution of the real (green dots) and fake (blue dots) over the course of the vanilla GAN (top row) and our method (the second row and below). The background color indicates the prediction heatmap of the discriminator with blue being fake and warm yellow being real. Once the vanilla GAN falls into mode collapse (top row), it ends up oscillating between the two modes without convergence. Moreover, the discriminator's prediction at point X oscillates, indicating catastrophic forgetting in the discriminator. With DMAT procedure, a new discriminator is dynamically spawned during training. The additional discriminator effectively learns the forgotten mode, guiding the GAN optimization toward convergence.
While the mode collapse problem has been tackled by many previous works, as discussed in Section 2 we show that our approach based on Catastrophic Forgetting can be added to any existing GAN frameworks, and is the most effective in preventing mode collapse. Furthermore, the improved stability of training boosts the standard metrics on popular GAN frameworks, To summarize, our contributions are:
We propose novel GAN framework, named Dynamic Multi Adversarial Training (DMAT), that prevents Catastrophic Forgetting in GANs by dynamically spawning additional discrim- inators during training.
We propose a computationally efficient synthetic data generation procedure for studying mode collapse in GANs that allows visualizing high dimensional data using normalizing flows. We show that mode collapse occurs even in the recent robust GAN formulations.
Our method can be plugged into any state-of titte-ofthie the-rtt GAN frameworks and still improve the quality and coverage of the generated samples.
2 RELATED WORKS
Previous works have focused on independently solving either catastrophic forgetting in supervised learning or mode collapse during GAN training. Among the efforts addressing mode collapse, a few prior works have proposed anleisaaversaiaall solutions for mitigating mode collapse, similar to our work. In this section we review these works in detail and discuss our commonalities and differences.
2.1 MITIGATING MODE COLLAPSE IN GANS
Along with advancement in the perceptual quality of images generated GAN (Miyato et al. 2018
Karras et al. 2019 Brock et al., 2018 Karras et al. 22020) a large number of papers DDrugkar et al.
2016 Metz et al., 2016 Arjovsky al. | [017 Srrivastava et al., 2017 Nguyen et al.) 2017 Lin et al. 2018 Mescheder et al. 2018; Karras et al. (019) identify the problem of mode collapse in GANs and aim mitigate it. However, many of them do attempt to directly address mode collapse,
was seen as secondary symptom that would be naturally solved as the stability of GAN optimization progresses (Arjovsky al., 2017, Mescheder et al.) 2018) Bau al. 2019). While the magnitude of mode collapse is certainly mitigated with more stable optimization, we show that it is still not solved problem. To explicitly address mode collapse, Unrolled GAN (Metz et al. 2016) proposes an unrolled optimization of the discriminator optimally match the generator objective, thus preventing mode collapse. VEEGAN (Srivastava et al., 2017) utilizes the reconstruction loss on the latent space. PacGAN (Lin et al. 2018)) feeds multiple samples of the same class to the discriminator when making the decisions about real/fake. contrast, our approach differs in that our method can be plugged into existing state-of-the-arr ahr- GAN frameworks yield additional performance boost.
2

---

Under review as a conference paper at ICLR 2021
2.2 MULTI-ADVERSARIAL APPROACHES
The idea of employing more than one adversarial network in GANS to mitigate mode collapse or to improve the quality of generated images in general has been explored by several previous works. MGAN (Hoang et al. 2018) uses multiple generators, while D2GAN (Nguyen et al., (017) uses two discriminators, and GMAN (Durugkar al. 2016) and MicrobatchGAN (Mordido al., 2020) possibly more than two discriminators that can be specified as hyperparameter. On the other hand, based on our hypothesis on catastrophic forgetting, our method can dynamically add discriminators at training time, achieving superior performance than existing works.
2.3 OVERCOMING CATASTROPHIC FORGETTING IN GAN
Catastrophic forgetting was first observed connectionist networks by McCloskey Cohen (188).. From then there has been plethora of works to mitigate catastrophic forgetting neural networks. These methods can be categorized into three groups: a) regularization based methods (Kirkpatrick
et al. 2017) b) memory replay based methods (Rebuffi et al., 2017) c) network expansion based methods (Zhang et al. 2019a Rajasegaran et al. 2019). Our work is closely related to the third category of methods, which dynamically adds more capacity the network, when faced with novel tasks. This type of methods, adds plasticity to the network from new weights (fast- weights) while keeping the stability of the network by freezing the past-weights (slow-weights). Although our work incrementally adds more capacity to the network, we enforce stability by letting a discriminator to focus on a few set of classes, not by freezing its weights. The possibility of catastrophic forgetting of GANs has been discussed by Thhanh-Tung & Tran (2020). However, the impact of catastrophic forgetting was mostly limited to theoretical analysis, and we found that the proposed solution deteriorates the performance on several GAN architectures such as BigGAN (Brock et al. 2019).
3 PROPOSED METHOD
In this section, we first describe our proposed data generation procedure that we use as a petri dish for studying mode collapse in GANs. The procedure uses random normalizing flows for simultaneously allowing training on complex high dimensional distributions yet being perfectly amenable 2D visualizations. Next, we describe our proposed Dynamic Multi Adversarial Training (DMAT) algorithm that effectively detects catastrophic forgetting and spawns a new discriminator to prevent mode collapse.
3.1 SYNTHETIC DATA GENERATION WITH NORMALIZING FLOWS
Mode dropping in GANs in the context of catastrophic forgetting of the discriminator is difficult problem to investigate using real datasets. This because the number of classes in the dataset cannot be easily increased, the classes of fake samples are often ambiguous, and the predictions of the discriminator cannot be easily visualized across the whole input space. In this regard, we present simple yet powerful data synthesis procedure that can generate complex high dimensional multi-modal distributions, yet maintaining perfect visualization capabilities.
The proposed procedure begins by sampling from simple two dimensional Gaussian distribution. The samples are then augmented with biases and subjected to an invertible normalizing flow (Karami et al. 2019) parameterized by well conditioned functions 9i _ + _ Optionally, this function can be followed by a linear upsampling transformation parameterized by a de di+p dimensional matrix A (Algorithm D. The transformations are constructed be analytically invertible thus allowing mapping the high dimensional output space to input (see Appendix for more information). Note that the entire transform is deliberately constructed to be bijective function so that every generated sample in IR' can be analytically mapped to R.a allowing perfect visualization on 2D space. Furthermore, by evaluating a dense grid of points in IR.2, we can also have a useful insight into discriminator's learned probability distribution on manifold heatmap
2D plane.
This synthetic data generation procedure enables studying mode collapse in a controlled setting. This also gives practitioners the capability to train models chosen data complexity with clean wwo-dimensional visualizations of both the generated data and the discriminator's 's learnt distribution. This tool can be used for debugging new algorithms using insights from the visualizations. For
3

---

Under review as a conference paper at ICLR 2021
example, in the case of mode collapse, quick visual inspection would give the details of which modes face mode collapse or get dropped from discriminator's 's learnt distribution.
3.2 DYNAMIC MULTI ADVERSARIAL TRAINING
Algorithm 1 Synthetic Data Generation
Algorithm 3 D-MAT:
Dynamic
Multi-
Adversarial Training
Input: Mean { and standard deviation {ax) for initialization, {9:]=1 well condi-
wg. On initial discriminator & gen- erator parameters, greediness parameter 6,
Require:
tioned IR? + I22 functions
Sample weights w ~ Dirichlet(KK)
(T&) spawn warmup iteration schedule
18 Sample from 2D mixture of gaussians */
D < [/}
X2D ΣN w.
while has not converged do
Sample {2 (2(0) Ji=1 p(2)
X2n [ 1), [
Sample [x P,
1s Randomly Initialized Normalizing Flow */
Sample (i)
Uniform K)
for to = K do
Sample {a{1)) Bernoulli(c)
i even then
/* Loss weights over discriminators */
x² = o o (xh)))
Sample weights
Dirichlet(K)
else
5((0 (=(=
x² = [+t (xf)), at]
2 (i) f arg (+(())
end if
end for
* Discriminator responsible for 2(0 #/
2, (i) <- a(t) (1) + - a(i))
y Discriminator responsible for w(1) #/
0a (1) < (i)
Algorithm 2 DSPAWN:
Diseriminator
/* Training Discriminators #/
Spawning Subroutine
Lu f Ex= 1/ ( (xx) 1]-
Require: Exemplar Data {e]
[ (a::) + 1]+
Input: Discriminator set D = {f fi=l
1
for k 1 k = DD do
1a Check forgetting over exemplar images */
w + ADAM L.n)
for i 1
do 8 sk] - f (e,) V k E (1... K}
end for
/ Training Generator */
if K max(s)))))))))))))())))))))) 0g Σx s[k] then Initialize fk+1 +1 with random weights w 1e Spawn new discriminator */ Initialize random weight w K+1
s[k]] < >E fh (2(0) V E (1... (D) Weighed mean over discriminators */
La sort(m) (sr()s)sss))))
+ ADAM(LL)))
B + ( U pK+1
if more than T. warm-up iterations since the last spawn then
break end if end for
TD DSPAWN((F..)
end if end while
return Discriminator Set 13
Building upon the insight relating catastrophic forgetting in discriminator to mode collapse in generator, we propose a multi adversarial generative adversarial network training procedure. The key intuition is that the interplay of catastrophic forgetting in the discriminator with the GAN minimax game, leads to an oscillation generator. Consequently, as the generator shifts to a new set of modes the discriminator forgets the learnt features on the previous modes. However if there are multiple discriminators available, each discriminator can implicitly specialize on a subset of modes. Thus even if the generator oscillates, each discriminator can remember their own set of modes, and they will not need to move to different set of modes. This way we can effectively sidestep catastrophic forgetting and ensure the networks do not face significant distribution shift. A detailed version of our proposed method is presented in Algorithm 3 Spawning new discriminators: We initialize the DMAT training Algorithm 3 with a regular GAN using just one discriminator. We also sample a few randomly chosen exemplar data points with maximum of one real sample per mode, depending dataset complexity. The exemplar data points are used to detect the presence of catastrophic forgetting in the currently active set of discriminators DD and spawn
discriminator if needed. Specifically
4

---

Under review as a conference paper at ICLR 2021
V indicates that the generator could effectively learn all the data modes, while means despite best efforts with tuming the training suffers from mode collapse (more than a quarter of the data modes are dropped). For each level. we show results with the SGD (left) & ADAM (right) optimizers. MNIST results with ADAM optimizer are provided for reference. We observe that MNIST is a relatively easy dataset, falling between Level I and II in terms of complexity.
Table 1:
(z)
Ag92x2
2
MLP
MLP, Ag92x2
MNIST
Label
Level I
Level II
Level
III
Level IV
Level V
GAN-NS Goodfellow al. 2014
X
X
X
X
WGAN Arjovsky et al. 201 7
/
/
x
/
X
X
X
X
/
Unrolled GAN (Miz et al. 2016
/
/
/
/
V
J
X
X
D2GAN (Nguyen et al. 2017
V
J
V
V
X
X
GAN-NS DMAT
V
X
we propose that if any discriminator among D has an unusually high score over an exemplar data point e;, this is because the mode corresponding to e, has either very poor generated samples or has been entirely dropped. In such a situation, if training were to continue we risk catastrophic forgetting in the active set TD, if the generator oscillate to near eq,- This is implemented by comparing the max score over at e; to the average score over and spawning a new discriminator when the ratio exceeds ar 1). Further, we propose to have or (r 1) a monotonically increasing function of [ID), thus successively making it harder to spawn each new discriminator. Additionally, we use a warmup period T; after spawning each new discriminator from scratch to let the spawned discriminator train before starting the check over exemplar data-points.
(Algorithm 2).
Multi- multi-Diseriminator Training: We evaluate all discriminators in D on the fake samples but do not update all of them for all the samples. Instead, we use the discriminator scores to assign responsibility of each data point to only one discriminator. We use an <-greedy approach for fake samples where the discriminator with the lowest output score is assigned responsibility with a probability 1 and a random discriminator is chosen with probability e, In contrast, for real samples the responsible discriminator is always chosen uniformly randomly, In effect, we slightly prefer to assign the same discriminator to the fake datapoints from around the same mode to ensure that they do not forget the already learnt modes and switch to another mode. The random assignment of real points ensure that the same preferentially treated discriminator also gets updated on real samples. Further for optimization stability, we ensure that the real and fake sample loss incurred by each discriminator is roughly equal in each pack-propagation step by dynamically reweighing them by the number of data points the discriminator responsible for. We only update the discriminator on the losses of the samples they are responsible for. Generator Training: We take a weighted mean over the discriminators scores on the fake datapoints for calculating the generator loss. At each step, the weights each discriminator in ID gets is in decreasing order of its score on the fake sample. Hence, the discriminator with the lowest score is given the most weight since it is the one that is currently specializing on the mode the fake sample is related to. In practice, we sample weights randomly from a Dirichlet distribution (and hence implicitly they sum to 1) and sort according to discriminator scores to achieve this. We choose soft weighing over hard binary weights because since the discriminators are updated in an e greedy fashion, the discriminators other than the one with the best judgment on the fake sample might also hold useful information. Further, choose the weights randomly rather than fixing a chosen set to ensure DMAT is more deadset agnostic since the number of discriminator used changes with the dataset complexity so does the number of weights needed. While a suitably chosen function for generating weights can work well on a particular dataset, we found random weights to work as well across different settings.
Table 2: Quantitative Results on the Stacked MNIST dataset: adversarial training (DMAT) procedure to simple DCGAN achieves perfect mode coverage, better than many existing methods for mode collapse.
Applying our proposed dynamic multi
GAN
UnrolledGAN
D2GAN
RegGAN
DCGAN
with DMAT
# Modes covered
628.0 0 1 140.9
817.4 4 37.9
1000 + 0.0
955.5 + 18.7
849.6 + 62.7
1000 +: 0.0
KL (samples || data)
2.58 0.75
1.43 0.12
0.080 0.01
0.64 = 0.05
0.73 + 0.09
0.078 + 0.01
5

---

Under review as a conference paper at ICLR 2021
Table Quantitative Results on CIFARIO: We benchmark DMAT against other multi adversarial baselines as well as on several GAN architectures, observing consistent performance increase.
GAN-NS w/ ResNet
DMAT + GAN-NS
DMAT + DCGAN
Model
D2GAN
MicrobatchGAN
DCGAN
IS
7.15 = 0.07
6.77
6.7 + 0.06
8.1 + 0.04 16.35
6.03 = 0.05
6.32 = 0.06 30.14
FID
28.91
33.42
WGAN-GP
DMAT + WGAN-GP
DMAT +
DMAT + BigGAN
Model
w/ ResNet
SN-GAN
SN-GAN
BigGAN
7.59 0.10
7.80 0.07 17.2
8.22 + 0.05
8.34 + 0.04 13.8
9.22
9.51 = 0.06
FID
19.2
14.21
8.94
6.11
4 RESULTS
We test our proposed method on several popular datasets, both synthetic and real & report consistent increase in performance on popular GAN evaluation metrics such as Inception Score (Salimans) et al. 2016) and Frechét Inception Distance (Heusel et al. 2017) with proposed dynamic multi- adversarial training. Finally, we also showcase our performance in the GAN fine-tuning regime with samples the CUB200 dataset (Welinder al. 2010) which qualitatively are more colorful and diverse than an identical BigGAN finetuned without DMAT procedure (Figure 3).
4.1 SYNTHETIC DATA
We utilize the proposed synthetic data generation procedure with randomly initialized normalizing flows visualize the training process of simple DCGAN (Radford al. 2015) in terms of the generated samples as well as discriminator's probability distribution over the imput space. Figure visualizes such a training process for simple bimodal distribution. Observing the pattern of generated samples over the training iteration and the shifting discriminator landscape, we note clear mode oscillation issue present the generated samples driven by the shifting discriminator clear
output distribution. Focusing a single fixed real point in space any of the modes, oscillation in the discriminator output probabilities strongly indicating the presence catastrophic forgetting in the discriminator network. Further such visualizations more complex distributions (toy 8-D Gaussian rings) are added the Appendix(D [
Effect of Data Complexity Mode Collapse: We use the flexibility in choosing transformations to generate datasets of various data distribution complexities as presented in Table [ Choosing g(z) with successively more complicated transformations
produce synthetic datasets of increasing complexity, the first five of which we roughly classify as Levels. The Levels are generated by using simple transforms such as identitym constant mapping, small Multi layer perceptrons and well conditioned linear transforms (A). On this benchmark, we investigate mode collapse across different optimizers such as SGD & ADAM (Kingma & Ba, 2014) on several popular GAN variants such as the natusaturaiing GAN Loss (GAN-NS) (Goodfellow et al. 2014). WGAN (Arjovsky et al.) 2017)) and also methods targeting mitigating mode collapse specifically such as Unrolled GAN (Metz et al.) 2016) and D2GAN (Nguyen et al.] 2017). We also show results of our proposed DMAT training procedure with simple GAN-NS, which matches performance with other more complicated mode collapse specific GAN architectures, all of which robust mode collapse up to Level IV. The procedure can be extended to even more complicated distributions than Level V, but in practice find all benchmarked methods collapse Level V. This indicates that in contrast other simple datasets like MNIST (LeCun) 1998), Gaussian ring, or Stacked MNIST (Metz et al. (016)). the complexity of synthetic dataset can be arbitrarily tuned up or down gain insight into the training and debugging of GAN via visualizations.
4.2 STACKED MNIST
We also benchmark several models on the Stacked MNIST dataset following (Metz et al. 2016 Srivastava et al., 2017). Stacked MNIST is an extension of the popular MNIST dataset (LeCun et al. 1998) where each image is expanded in the channel dimension to 28 x 28 x 3 by concatenating 3 single channel images from original MNIST dataset. Thus the resulting dataset has 1000 overall modes. We measure the number of modes covered by the generator the number of classes that

---

Under review as a conference paper at ICLR 2021
Table 4: BigGAN + DMAT Ablations on CIFAR1O (A) A relaxed spawning condition with small a and short warmup schedule that leads to large number of discriminators (>7) (B) Long warm-up schedules that spawn new discriminators late into training (C) greedy strategy for assigning responsibility of fake samples (e 0) (D) Flipping the data splitting logic with responsibilities fake samples being random and of real being e-gredy and (E) Choosing the discriminator with lowest score for updating Generator instead of soft random weighting.
Rffect Ablation
Large [ID]
Spawn too late Long T2 schedule
Greedy VD
Random for fake
1-hot weight vector 772
Proposed Method
Small a. Short 7₃
0
e-groedy for real
IS
8.83 0.04
9.28 + 0.08
9.31 + 0.06
8.95 = 0.04
9.25 + 0.05
9.51 0.06
FID
14.23
9.37
8,6
12.5
9.25
6.11
Table 5: Per-class FID on CIFARIO: FID improves consistently across all classes.
Classes
Plane
Car
Bird
Cat
Deer
Dog
Frog
Horse
Ship
Truck
Avg
BigGAN
24.23
12.32
24.85
21.21
12.81
22.74
17.95
13.16
12.11
18.39
8.94
+ DMAT
20.50
10.30
23.48
18.48
11.51
19.41
11.50
12.24
10.69
12.94
6.11
A%
18.2
19.6
5.8
14.8
11.3
17.2
56.1
7.5
11.7
42.1
46.3
generated least once within pool of 25, 600 sampled images. The class of the generated sample is identified with pretrained MNIST classifier operating channel wise on the original stacked MNIST image. We also measure the KL divergence between the label distribution predicted by the MNIST classifier in the previous experiment and the expected data distribution.
Understanding the collet interplay: In Section I we discuss our motivation for studying catastrophic forgetting for mitigating mode collapse. We also design an investigative experiment to explicitly observe this interplay by comparing the number of modes the generator learns against the quality of features the discriminator learns throughout GAN training on the stacked MNIST dataset. We measure the number of modes captured by the generator through a pre-trained classification network trained in supervised learning fashion and frozen throughout GAN training. To measure the amount of 'forgetting" in discriminator, we extract features real samples from the penultimate layer of the discriminator and train small classifier the real features for detecting real data mode. This implicitly indicates the quality and information contained in the the discriminator extracted features. However, the performance of classification network on top of discriminator features is confounded by the capacity of the classification network itself. Hence we do a control experiment, where we train the same classifier on features extracted from a randomly initialized discriminator, hence fixing lower-bound to the classifier accuracy.
Referring Figure 2 we observe a clear relation between the number of modes the generator covers at an iteration and the accuracy of the classification network trained on the discriminator features the same iteration. In the vanilla single discriminator scenario, the classification accuracy drops significantly, indicating a direct degradation of the discriminative features which is followed by complete collapse G. In the collapse phase, the discriminator's learnt features are close to random with the classification accuracy being close to that of the control experiment. This indicates the presence of significant catastrophic forgetting in the the discriminator network.
Number Modes
Discriminator Test Accurasy
0.50
600
0.40
400
200
0.30
CruD
Centrdl
025
20
40
100
200
Number Epochs
Number of spochs
Figure 2: Investigating the forgetting-oollapse interplay: We investigate our hypothesis that catastrophic forgetting is associated with mode collapse. To this end, on the left pane, we plot the magnitude of mode collapse by counting the number of modes produced by the generator. On the right pane, we assess the quality of the discriminator features by plotting the accuracy of lincar classifier top of the discriminator features at each epoch. In the original DCGAN model (OneD), the coverage of modes and the quality of discriminator features are both low and decreasing. In particular, the test accuracy from the discriminator's features drops almost to randomly initialized weights (shown as control). On the other hand, adding DMAT (MultiD) dramatically improves both mode coverage and the discriminator test accuracy.
7

---

Under review as a conference paper at ICLR 2021
Figure 3: Sample Diversity CUB200: We showcase samples from BigGAN (pretrained on imagenet and finetuned on CUB200 with DMAT procedure (left four columns) and from an identical BigGAN finetuned without DMAT (right four columns). We observe that while the sample quality is good for both setups, the samples generated with DMAT are more colorful & diverse, exhibiting bright reds and yellow against a variety of backgrounds. While the samples from vanilla tine-tuning restricted to whites/grays & only hint of color.
In contrast, training the same generator with the proposed DMAT procedure leads to stable training with almost all the modes being covered and the classification accuracy increasing before saturation. Catastrophic forgetting thus effectively sidestepped by dynamic multi adversarial training which produces stable discriminative features throughout training that provide a consistent training signal to the generator thereby covering all the modes with little degradation.
4.3 CIFAR10
We extensively benchmark DMAT on several GAN variants including unconditional methods such as DCGAN (Radford et al. 2015), ResNet-WGG WGAN-GP (Gulrajani et al. 2017 He et al. (016)) & SNGAN (Miyato et al. 2018) and also conditional models such as BigGAN (Brock et al. 2018). Table 3 shows the performance gain on standard GAN evaluation metrics such as Inception Score and Fréchet distance of several architectures when trained with DMAT procedure. The performance gains indicate effective curbing of catastrophic forgetting in the discriminator with multi adversarial training. We use the public evaluation code from SNGAN (Miyato et al.] 2018) for evaluation. Despite having other components such as spectral normalization, diversity promoting loss functions, additional RI losses & other stable training tricks that might affect catastrophic forgetting to different extents, observe consistent increase in performance across all models. Notably the ResNet GAN benefits greatly with DMAT despite a powerful backbone - with IS improving from 6.7 to 8.1, indicating that the mode oscillation problem is not mitigated by simply using a better model.
DMAT can also improve performance by over 35% even on well performing bascline such as BigGAN (Table We also investigate the classwise FID scores of a vanilla BigGAN and identical BigGAN trained with DMAT on CIFAR10 10 and report the results in Table Performance improves across all classes with previously poor performing classes such as 'Frog' "Truck' experiencing the most gains. Further, we also ablate several key components of the DMAT procedure the BigGAN architecture with results reported in Table We observe all elements to be critical to overall performance. Specifically, having moderate schedule to avoid adding too many discriminators critical. Also, another viable design choice is to effectively flip the algorithm's 's logic and instead choose the fake points randomly while being greedy the real points. We observe this strategy to perform well on simple datasets but lose performance with BigGAN on CIFAR10 (Table 4).
CONCLUSION
In summary, motivated from the observation of catastrophic forgetting in the discriminator, we propose a new GAN training framework that dynamically adds additional discriminators to prevent mode collapse. We show that method can be added to existing GAN frameworks to prevent mode collapse, generate more diverse samples and improve FID IS. As future work, we plan to apply our method to large scale experiments prevent mode collapse in generating higher resolution images.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. arXiv: arXiv:1701.07775 2017.
Wasserstein gan. arXiv preprint
David Bau, Jun- Yan Zhu, Jonas Wulff, William Peebles, Hendrik Strobelt, Bolei Zhou, and Antonio Torralba. Sceing what a gan cannot generate. In Proceedings of the International Conference Computer Vision (ICCV), 2019.
Konstantinos Bousmalis, Alex Irpan, Paul Wohlhart, Yunfei Bai, Matthew Kelcey, Mrinal Kalakrish- nan, Laura Downs, Julian Ibarz, Peter Pastor, Kurt Konolige, et al. Using simulation and domain adaptation to improve efficiency of deep robotic grasping. In 2018 IEEE international conference on robotics and automation (ICRA), pp. 4243-4250. IEEE, 2018.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1096, 2018.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. 2019.
Ishan Durugkar, Ian Gemp, and Sridhar Mahadevan. Generative multi-adversarial networks. arXiv preprint 161 1.01673, 2016.
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint
2017.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- fion processing systems, pp. 2672-2680, 2014.
Ishaan Gulrajami, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. In Advances in neural information processing systems, pp. 5767-5777, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. In Advances in neural information processing systems, pp. 6626-6633 2017.
Quan Hoang, Tu Dinh Nguyen, Trung Le, and Dinh Phung. Mgan: Training generative adversarial nets with multiple generators. In International Conference on Learning Representations, 2018.
Judy Hoffman, Eric Tzeng, Taesung Park, Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In International conference on machine learning, pp. 1989-1998. PMLR, 2018
Mahdi Karami, Dale Schuurmans, Jascha Sohl- Sohl-Dickstein, Laurent Dinh, and Daniel Duckworth. Invertible convolutional flow. In Advances in Neural Information Processing Systems, pp. 5635- 5645, 2019.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. 2019.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. 2020.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv: arXiv:1412.6 44/266980, 2014.

---

Under review as a conference paper at ICLR 2021
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521-3522) 2017.
Grabska-Barwinskee Barwinska, et al. Overcoming
Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mmist// 1998.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2222 1998.
Zinan Lin, Ashish Khetan, Giulia Fanti, and Sewoong Oh. Pacgan: The power of two samples in generative adversarial networks. In Advances in neural information processing systems, pp. 1498-1507, 2018.
James L McClelland, Bruce L McNaughton, and Randall C O'RRelly. Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychhlogical review, 102(3):419, 1995.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi, and Cynthia Rudin. Pulse: Self-supervised photo upsampling via latent space exploration of generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? In International Conference on Machine learning (ICML), 2018.
Luke Metz, Ben Poole, David Pfau, and Jascha Sohl-Dickstein. Unrolled generative adversarial networks. arXiv preprint 1.100166,,,,,,,, 2016.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv:/882.05997,, 2018.
Gonçalo Mordido, Haojin Yang, and Christoph Meinel. microbatchgan: Stimulating diversity with multi-adversarial discrimination. arXiv preprint arXiv:2001.03377 2020.
Tu Nguyen, Trung Le, Hung Vu, and Dinh Phung. Dual discriminator generative adversarial nets, In Advances in Neural Information Processing Systems, pp. 2670-2680, 2017.
Dong Nie, Roger Trullo, Jun Lian, Li Wang, Caroline Petitjean, Su Ruan, Qian Wang, and Dinggang Shen. Medical image synthesis with deep convolutional adversarial networks. IEEE Transactions on Biomedical Engineering, 65(12):2720 2730, 2018
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks, arXiv preprint
2015.
Jathushan Rajasegaran, Munawar Hayat, Salman H Khan, Fahad Shahbaz Khan, and Ling Shao. Random path selection for continual learning. In Advances in Neural Information Processing Systems 32, pp. 12669-12679. Curran Associates, Inc., 2019.
Kanishka Rao, Chris Harris, Alex Irpan, Sergey Levine, Julian Ibarz, and Mohi Khansari. Rl-cyclegan: Reinforcement learning aware simulation-to-real. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11157-11166, 2020.
Roger Ratcliff. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review, (7(2)228,, 1990.
Sylvestre- Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
10

---

Under review as a conference paper at ICLR 2021
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint 1rXYi::160600661,, 2016.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans, 2016.
Akash Srivastava, Lazar Valkov, Chris Russell, Michael U Gutmann, and Charles Sutton. Veegan: Reducing mode collapse in gans using implicit variational learning. In Advances in Neural Information Processing Systems, pp. 3308-3318, 2017.
Hoang Thanh-Tung and Truyen Tran. On catastrophic forgetting and mode collapse in gans, arXiv preprint arXiv: 000015, 2020.
P. Welinder. S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongic, and P. Perona. Caltech-UCSD Birds 200. Technical Report CNS-TR-2010-001, California Institute of Technology, 2010.
Jelmer M Wolterink, Tim Leiner, Max A Viergever, and Ivana l8gum. Generative adversarial networks for noise reduction in low-dose ct. IEEE transactions on medical imaging, 36(12):553--255, 2017.
Xin Yi, Ekta Walia, and Paul Babyn. Generative adversarial network in medical imaging: A review. Medical image analysis, 58:101552, 2019.
Jeffrey O Zhang, Alexander Sax, Amir Zamir, Leonidas Guibas, and Jitendra Malik. Side-tuning:
Network adaptation via additive side networks. arXiv preprint arXiv: 1912.3393 2019a.
Jeffrey O. Zhang, Alexander Sax, Amir Zamir, Leonidas J. Guibas, and Jitendra Malik. Side-tuning:
Network adaptation via additive side networks. 2019b.
Mengshi Zhang, Yuqun Zhang, Lingming Zhang, Cong Liu, and Sarfraz Khurshid. Deeproad: Gan- based metamorphic testing and input validation framework for autonomous driving systems. In 2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 142. IEEE, 2018.


Reviews:
[{"review_id": "qYgWer5-JcD", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The contributions of this paper revolve around the simple but interesting idea that mode collapse and missing modes in GANs are due (at least in part) by catastrophic forgetting in the discriminator, as the discriminator (and the generator) see a non-stationary training distributions due to their interaction. The authors provide toy experiments clearly illustrating this phenomenon and then propose an algorithm (DMAT) based on multiple discriminators taking charge for different parts of the input space (different modes). They show on several synthetic datasets with varying complexity that DMAT helps where other approaches fail, although DMAT (and all the other methods) still fail on the harder cases. They also show numerical improvements in standard GAN metrics against other methods (I don't know enough to be sure if these are SOTA, though). They also have nice experiments suggesting a strong link between mode collapse and catastrophic forgetting, thus also reinforcing the main claim.\n\nHence I see several good contributions around what seems to be a good idea. I only had a few minor comments, otherwise.\n\n* there is generally a trade-off in generative models between missing modes and spurious modes; it would be good to make sure that we don't gain on one side by losing on the other.\n\n* the algorithm is actually complicated and contains a lot of bells and whistles whose justification is not always clear (hence the statement in 4.1 that other methods are 'more complicated' seems unfair)\n\n* the D2GAN performs as well as DMAT in table 2 but does not get the 'bold' treatment, this should be fixed.\n\n* I do not find the difference between with and without DMAT in figure 3 compelling, so conclusion about this should also be toned down.\n\n* several clarity issues:\n  - alpha_t in Alg. 2 is not specified in the algorithm (although it is in the text)\n  - the justification for alpha_t is not clear (and in particular the specifics of how it changes with t)\n  - the justification for the experimental setup associated with figure 2 is not clear enough; why not keep the capacity of the classifier fixed (and not necessarily linear)ˆ? \n  - 'with a maximum of one real sample per mode': how would you know, on real data, since the modes are not known a priori? hence this thinking only works for toy data where the modes are known.\n  - English needs proofreading in many places\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "dVebc34o6ZC", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes and approach to mitigate mode collapse in arbitrary GANs. This is relies on the assumption that mode collapse relates to catastrophic forgetting in the GAN discriminator. The authors design a synthetic experiments based on a normalizing flow generator, allowing to quantify mode collapse. They propose a heuristic to avoid mode collapse of the discriminators by adding additional ones during training. The results show improvements with respect to baseline regarding mode covering and FID.\n\nStrengths:\n- the synthetic experiment to quantify mode collapse looks elegant,\n- the authors report systemic improvement in FID over baselines.\n\nWeaknesses:\n- the authors fail to cite and compare their approach to another approach based on accumulating networks to improve GAN performance (\"AdaGAN: Boosting Generative Models\", Tolstikhin et al. 2017). There are also extra references therein.\n- The relation between mode collapse and catastrophic forgetting in not clearly supported. This can be seen in Fig. 2, were despite the authors statements, there is no clear covariation between forgetting and collapse.\n- While the systematic improvement of mode covering provided by the method is not surprising, the systematic FID improvement are weakened by the absence of detailed (e.g. supplemental) material to reproduce the results and make sure this result is solid.\n- The authors make a link between mode collapse and oscillations, although the later is now a well studied and general phenomenon with solid theoretical results, that can be dealt with by modifying the optimization algorithm (Daskalakis et al, Training GANs with optimism). \n\nOverall, this seems and interesting research direction, but evidence for the main claim of the paper is lacking. In the current state I tend to recommend rejection.\n\n\nPerhaps the authors could find investigate more in depth going through all their experiments, whether their is a clear causal link between mode collapse and catastrophic forgetting. This is likely tricky, in the sense that mode collapse will likely lead to a loss of discriminator performance. Also the authors could compare their approach with alternative solutions, that do not rely on this collapse-forgetting link, such as the two papers cited above.\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "_DUpWBth_Gc", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses the well-known phenomenon of model collapse in generative adversarial networks (GANs). In particular, this paper identifies catastrophic forgetting of the discriminator as a potential source of mode collapse and proposes a multi-discriminator framework called DMAT as a solution. DMAT takes inspiration from expansion-based continual learning approaches, which add network capacity (in this case entire discriminators) to preserve past knowledge.\n\nPros:\n1.\tThere are several excellent visualizations of generator oscillation behavior (Figure 1, Figure 4 <- I like the animations) that underpins DMAT’s design.\n2.\tBecause DMAT is primarily based on adding discriminators to the model, it can be fairly simply combined with many GAN approaches. This is demonstrated for example in Table 3.\n3.\tDiscriminators are dynamically added as needed, allowing the network to scale indefinitely to new modes. This also saves some computation at the beginning, when a large number of discriminators may not be necessary.\n4.\tA fairly wide range of experiments were run, and adding DMAT appears to lead to stronger results (higher Inception Score, lower Frechet Inception Distance) across a both an assortment of models and datasets.\n5.\tThe quality of the writing is generally good.\n\nCons:\n1.\tI have concerns over the novelty of this work. The primary motivation of the paper (GAN mode collapse being due to catastrophic forgetting, leading to generator oscillations) is a known phenomenon and has been thoroughly explored before in [1,2]. In particular, [2] also takes a continual learning-inspired approach (regularization, as opposed to expansion) to propose an easy-to-add method, and visuals very similar to this submission’s Figure 4 can be found in both [1] and [2]. Given the strong degree of conceptual overlap, these works probably deserve more mention than a small reference to [1] in the Related Works. This paper presents the “GAN catastrophic forgetting” hypothesis in the Introduction as if it is the authors’ novel contribution. Additionally, multi-discriminator networks have also been well-explored in the past, with several examples mentioned in the Related Works. There are only limited comparisons with these methods in the Experiments, but Table 2 for example does demonstrate that D2GAN (which has 2 discriminators) also manages to cover all modes.\n2.\tThe core methodology of DMAT is adding discriminators to capture new modes. While conceptually simple, this involves adding entire new networks, which can be expensive computationally. How many discriminators does DMAT typically add? In the continual learning literature, expansion of an entire network per task (equivalent to “per mode,” here) is considered very bad, and is the scenario that expansion strategies (some of which have sublinear parameter growth) specifically try to avoid.\n\n3.\tDiscriminator assignment to samples and spawning/initialization is rather ad hoc, relying primarily on heuristics and intuition. A Dirichlet Process Mixture Model/Mixture-of-Experts approach such as in [3] (also a continual learning set-up, albeit for more traditional supervised classification) would be much more principled. Additionally, my understanding is that new discriminators are randomly initialized, which may be slow and inefficient.\n4.\tI don’t find the “fair comparison” in Appendix C to be particularly fair, as it’s unclear if the discriminator can effectively use the capacity of a wider network, and if it could, it’d likely disrupt the delicate minimax game between the discriminator and the generator; too strong of either a generator or discriminator severely disrupts GAN learning.\n5.\tI don’t fully understand the benefits of the proposed data generation process in Section 3.1. Settings like mixture of 8/25 2D Gaussians are fairly commonplace; being 2D means the discriminator decisions are easy to visualize, and the number of classes can be trivially increased/decreased by adding/removing classes. Using normalizing flows don’t make these synthetic distributions any more realistic either.\n\nMiscellaneous:\n1.\tThe title is a little too vague, and in my opinion “sidestepping” is somewhat inaccurate: despite some successful results on simple datasets, DMAT’s design doesn’t necessarily avoid catastrophic forgetting entirely. Given the plethora of continual learning strategies, the title should be more specific to DMAT’s strategy (namely model expansion by growing the number of discriminators).\n2.\tAlg 3 is a little confusing. The way it’s currently presented, it appears at first glance that a new discriminator is spawned on every iteration after $T_t$. It would be better if either the spawning logic from Alg 2 appeared in Alg 3, or the call to DSPAWN (Alg2) in Alg 3 made it clearer that a new discriminator wasn’t created on every iteration. Another thing worth noting: DSPAWN sounds very similar to “despawn,” which implies removing discriminators, not adding them. \n3.\tBoth BigGAN and Side-tuning appear twice in the references.\n4.\tThe gap between the Figure 3 caption and the main body text is a little too narrow.\n\nDecision: \nThe experimental results presented in this paper are promising, but adding entire discriminator networks is computationally expensive, and I find DMAT’s formulation to be a little too unprincipled. Furthermore, while this specific combination may not have appeared in previous literature, discriminator catastrophic forgetting as a cause for mode collapse and multi-discriminator GANs have both been thoroughly explored before, leaving this submission with little novelty of its own. As a result, I recommend rejection.\n\n\n[1] Hoang Thanh-Tung and Truyen Tran. On catastrophic forgetting and mode collapse in gans. arXiv preprint arXiv:1807.04015, 2020.\n\n[2] Kevin J Liang, Chunyuan Li, Guoyin Wang, Lawrence Carin. Generative Adversarial Network Training is a Continual Learning Problem. arXiv preprint arXiv:1811.11083, 2018\n\n[3] Soochan Lee, Junsoo Ha, Dongsu Zhang, Gunhee Kim. A Neural Dirichlet Process Mixture Model for Task-Free Continual Learning. ICLR 2020.\n\n\n=========\nPost-rebuttal\n=========\n\nI thank the authors for their thorough response, which is well argued. Overall, yes, I agree with the authors that there are some differences between their work and prior work. I do not claim that there is zero novelty here. My concern is whether there is enough for this venue, given the high degree of similarity. As the authors do acknowledge that the hypothesis of \"catastrophic forgetting leads to GAN oscillations\" was introduced in other work, then the primary contribution here is replacing one continual learning method for preventing catastrophic forgetting in GAN discriminators with another, and despite the rebuttal, I still find the proposed solution to be rather ad hoc. I keep my score.", "extra_scores": {}, "extra_text": {}}, {"review_id": "XXrmp5xdFhn", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "-- Summary --\n\nThis paper studies specifically the mode collapse problem of GAN. It hypothesizes that mode collapse in training GAN is closely related to catastrophic forgetting, and proposes a method to improve the catastrophic forgetting issue to mitigate the mode collapse. The proposed method uses multiple dynamically-spawned discriminators, in which additional discriminators are spawned to remember a few exemplars of data modes. It enables the model to avoid the oscillation in the discriminators' predictions, and handle well different parts of the data. The experiments demonstrate this multiple-spawned-discriminators scheme can be plugged to improves into various baseline GAN models on benchmark datasets (Stacked MNIST, CIFAR-10, CUB500 (qualitative only)). The paper also proposed a synthetic data generation procedure that can generate high-dimensional synthetic data that can be visualized at low-dimension for studying the mode collapse. \n\nOverall, this paper is well-written and clear. Although the idea of multiple adversarial has been discussed in many existing works (some related works are missing in the paper, see below), this paper is different at posing a new scheme and specifically use it to address catastrophic forgetting. The experiments and study are sufficient to demonstrate the contributions.\n\nHowever, the hypothesis of a relation between mode collapse and catastrophic forgetting, which was discussed in [c] (missing also), is not new. The algorithms are not too clear (notations, explanations) that need to be improved in the updated version. Additionally, some claims in the paper need to be justified.\n\n[a] Multi-objective training of Generative Adversarial Networks with multiple discriminators.\n\n[b] Generative Multi-Adversarial Networks.\n\n[c] Self-Supervised GANs via Auxiliary Rotation Loss.\n\n-- Strength -- \n\nS1 – The proposed algorithms of dynamic-spawned discriminators to address the catastrophic forgetting is new to me.\n\nS2 – The experiments show significant improvements over baseline models.\n\nS3 – The synthetic data generation procedure looks useful.\n\n-- Weakness --\n\nW1 – Some parts of the algorithms are not clear.\n\nW2 – Quite missing to discuss some important works in the paper.\n\n-- Details --\n\n*Algorithms* - It would be a lot helpful and clearer if the authors explain the notations of Algorithms and relating them in the text discussions.\n\nD1 – Algorithm 1 is a bit confusing with superscripts and subscripts of notations. Not sure superscripts or subscripts are the indices of samples or dimensions? According to the Algorithm, $x_{2D}^0$ is not used after being declared before the loop? And what is $N$ in Algorithm 1?\n\nD2 – What are $[.]^-$ and $[.]^+$ of the loss function in Algorithm 3? Are the authors training GAN models with hinge losses but not discussed in the paper? \n\nD3 – The random weights m used once to compute weighted means over the discriminators do not looks make sense to me. This random may have much affect to update the generator but their random values are chosen not from any prior pieces of information. I found some discussion about this in the section of “Generator Training”, yet can the author provide the ablation study to show the importance of using these random weights?\n\n*Toy dataset* \n\nD4 – In the study of Effect of Data Complexity on Mode Collapse (Table 1): The authors test the mode collapse by \"more than a quarter of the data modes are dropped\". How is \"a quarter of the data modes\" measured? Is there any threshold to decide a mode is dropped or not? And What is z for Level 3 of Table 1?\n\n*Stacked MNIST experiment*\n\nD5 – In the Stacked MNIST experiment, it's surprised that the original DCGAN can achieve quite a good mode collapse average if following the standard setup of Unrolled GAN (Metz et al., 2016). What is the architecture used for Stacked MNIST experiments?\n\n*Forgetting-collapse interplay experiments*\n\nD6 – In the proposed method, there are multiple discriminators. Are all features extracted from all discriminators chosen for the quality evaluation in forgetting-collapse interplay experiments?\n\nD7 – The paper claimed: \"we show ..., and is the *most effective* in preventing mode collapse\". Although I agree that improving the catastrophic forgetting would improve the mode collapse, however, I would argue that there are likely other reasons causing mode collapse. For example, [d] shows the mode collapse still can happen in [c] due to the loss function of the classification. (Thanh-Tung et al., 2020) suggests the gradient penalty can address quite well the mode collapse. Can the authors compare the proposed method without gradient penalty compared to the same GAN models with only gradient penalty? Moreover, recent works [e,f,g] simply apply data augmentation for GAN training that significantly improves its performance. Therefore, I believe that data augmentation can also be helpful to address the mode collapse specifically in the case of limited data. \n\n[d] Self-supervised GAN: Analysis and Improvement with Multi-class Minimax Game.\n\n[e] Differentiable Augmentation for Data-Efficient GAN Training.\n\n[f] On Data Augmentation for GAN Training.\n\n[g] Training Generative Adversarial Networks with Limited Data.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose an approach to mitigate mode collapse phenomena in GANs. Motivated by the intuition that mode collapse stems from catastrophic forgetting of the discriminator, the authors propose a solution inspired by recent research in continual learning and dynamically add new discriminators during training. The authors empirically demonstrate that combining the proposed scheme with existing GANs leads to improvements in terms of Inception Score and FID. 

This paper is trying to address a significant problem for the generative modeling community. The reviewers appreciated the clarity of writing, the empirical results, and the idea of using normalising flows for an elegant visualisation. However, the reviewers have pointed out several major issues which were not adequately addressed by the authors. The first one is the clear failure to position the work with respect to related work. In fact, the main idea related to catastrophic forgetting was already established in [1,2] and subsequent works. Secondly, the improvements over the baseline come at a significant computational overhead which is extremely challenging and impractical. Finally, given the trend that large-scale models achieve significantly better results in practice, the proposed approach is not only impractical, but potentially extremely wasteful. Given very limited novelty, failure to position the work, and impracticality of the proposed solution, I will recommend rejection.

[1] https://arxiv.org/abs/1810.11598

[2] https://arxiv.org/abs/1911.06997
Decision: Reject