MCL-GAN: Generative Adversarial Networks with Multiple Specialized Discriminators
Authors: Jinyoung Choi
Bohyung Han
Venue: NeurIPS 2022 Accept (2022)

Abstract: We propose a framework of generative adversarial networks with multiple discriminators, which collaborate to represent a real dataset more effectively. Our approach facilitates learning a generator consistent with the underlying data distribution based on real images and thus mitigates the chronic mode collapse problem. From the inspiration of multiple choice learning, we guide each discriminator to have expertise in a subset of the entire data and allow the generator to find reasonable correspondences between the latent and real data spaces automatically without extra supervision for training examples. Despite the use of multiple discriminators, the backbone networks are shared across the discriminators and the increase in training cost is marginal. We demonstrate the effectiveness of our algorithm using multiple evaluation metrics in the standard datasets for diverse tasks.

MCL-GAN: Generative Adversarial Networks with Multiple Specialized Discriminators
Jinyoung Choi
Bohyung Han 1,2,3
ECE, IIPAI, AASRI Scoul National University, Korea [jino. choi, bhhan) (athann@snu.. kr
Abstract
We propose framework of generative adversarial networks with multiple discrimi- nators, which collaborate to represent a real dataset more effectively. Our approach facilitates learning generator consistent with the underlying data distribution based on real images and thus mitigates the chronic mode collapse problem. From the inspiration of multiple choice learning, we guide each discriminator have expertise in subset of the entire data and allow the generator to find reasonable correspondences between the latent and real data spaces automatically without extra supervision for training examples. Despite the use of multiple discriminators, the backbone networks shared across the discriminators and the increase in training cost is marginal. We demonstrate the effectiveness our algorithm using multiple evaluation metrics in the standard datasets for diverse tasks.
Introduction
With recent advances in deep generative models, Generative Adversarial Networks (GANs)
and Variational Autoencoders (VAEs) [2] have shown impressive achievements in generation of high-dimensional realistic images as well various computer vision applications including image-to- image translation 33.) image inpainting (6). image super-resolution (7). etc. GANs have particularly received lot of attention due their interesting framework of minimax games, where two agents, generator and a discriminator, compete against each other. Furthermore, compared to VAEs, GANs tend to produce acute, high-quality images. In theory, the generator learns the real data distribution by reaching an equilibrium point of the minimax game. However, in practice, the alternating training procedure does not guarantee the convergence to the optimal solution and often experiences mode collapsing, failing to cover the multiple modes of real data or, even worse, reaching trivial solutions.
This paper focuses on the mode collapse problem in training GANs by adopting multiple collaborating discriminators. Each discriminator learns to specialized in subset of reference data space, which is identified automatically during training, the ensemble of discriminators provides not only the differentiation of fake data but also more accurate predictions over the clusters of real data. this respect, a generator is encouraged to produce diverse modes that deceive set of discriminators. To achieve these goals, we employ a Multiple Choice Learning (MCL) (8) framework to learn multiple discriminators and update the generator via set of expert discriminators, where each discriminator associated with a subset of the true and generated examples. Our approach, based on a single generator and multiple discriminators, is called MCL-GAN, which is optimized by the standard objective of GANs combined with the objective of MCL on the discriminator side.
There are several GAN literature that adopts multiple discriminators (9-14). Among them, GMAN [10 is closely related to our approach in the sense that it utilizes an ensemble predic- tion of discriminators. It explores multi-diseriminator extensions of GANs with diverse versions of the aggregated prediction discriminators from a harsh trainer to a lenient teacher with softened
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

criteria. Meanwhile, there exist significant differences in the method of ensembling from our ap- proach. While GMAN focuses on the loss to the generator with parallel learning of discriminators, our strategy specializes in each discriminator for more informative feedback to the generator.
The training algorithm of the proposed approach is inspired by MCL [15], which is known to be effective in learning specialized models with high oracle accuracy in recognition tasks. Encouraged by this benefit, [16 apply MCL its variations [1 to the production of diverse and accurate outputs in several applications. For instance, Mun et al. 17 propose an MCL-KD framework to come up with a visual question answering (VQA) system based on multiple models that are specialized in different types of visual reasoning. DiverseNet (18) introduces control parameter as an input that diversifies the outputs of the network with an MCL loss. While these works generate multiple outputs explicitly and select one of them or take their ensemble at inference time, our approach adopts unique strategy for diversifying the mode with no additional cost at inference time.
The proposed method takes an advantage of MCL for GANs, which has hardly been explored before. Our approach does not require any supervision for model specialization, such class labels and other conditions, unlike the aforementioned works. Our main contributions are summarized as follows:
We propose a single-genen generaao multi-discriminntor GAN training algorithm to alleviate the mode collapse problem. Our approach provides simple yet effective learning objectives based on MCL to achieve the goal.
We present a balanced discriminator assignment strategy to facilitate the robust conver- genee of models and preserve the multi- modtitmodality of training data, where the number of discriminators is determined adaptively.
The proposed method is applicable to many GAN variants since there is no constraint on network architectures loss functions. Our method requires small extra overhead and trains models efficiently via feature sharing in the discriminators.
2
Related work
This section discusses the mode collapse issue in GANs and the GAN models with multiple discrimi- nators generators.
2.1 Handling mode collapse for diversity
Many variations of GANs propose either novel metrics for the discriminator loss or better alternatives to the discriminator design. For example, LSGAN [2]] substitutes the least square function for the binary cross-entrops loss the discriminator objective. WGAN (23) introduces critic function based 's distance rather than a binary classifter, and WGAN-GP (24) improves WGAN
on the by adding gradient penalty term. PacGAN (25) augments the discriminator's input by packing samples for a single label. EBGAN (26) models the discriminator as an energy function, which is optimized by the reconstruction loss in autoencoders. BiGAN (27) ALI (28). VEEGAN (29] and
Inclusive GAN (30) also learn reconstruction networks. In particular, VEEGAN (2 autoencodes the
latent vectors to learn the inverse function of the generator, and maps both the true and generated data to the latent distribution, e.g., a Gaussian distribution. Inclusive GAN (30) learns generator by matching between real and fake examples in the feature space. The mode collapse and diversity issues of generated outputs have been addressed explicitly in [12 31 32). They formulate the diversity metries that encourage the mode exploration of the generators and derive the loss function using the metrics.
2.2 GAN with multiple generators
This approach represents a data distribution with mixture model enforcing each generator to cover portion of the whole data space. It is naturally expected that mixture models approximate true distributions better than a single model, especially in high-dimensional spaces with multiple modes.
Another line of research is the integration of multiple generators B3 36
MAD-GAN 34 introduces an augmented classifier as a discriminator, which predicts whether a sample is real and which generator the sample is drawn from, to encourage individual generators to

---

learn distinctive modes. MGAN (35] has similar strategies to MAD-GAN but constructs separate branch its discriminator to perform the two tasks. MEGAN (36] adopts a gating network that produces one-hot vector to select the generator creating the best example. P2GAN (37 sequentially adds a new generator to cover the missing modes of real data.
2.3 GAN with multiple discriminators
Multiple discriminators are often employed to improve the performance of a single generator [ 13). D2GAN [9] conducts a three-player minimax game, where two discriminators are trained for the completely opposite objectives, minimizing the Kullback-Leibhe
(KL) divergence and the inverse KL divergence between the true and generated data distributions. The balancing of the two losses plays a role in seeking desirable and diverse modes. Albuquerque ef al. [33 propose general multi-objective optimization framework in the scenario with multiple discriminators. They present hypervolume maximization algorithm to obtain weighted gradients. Neyshabur et al. (14) train a GAN based on multiple projections. Each discriminator determines a random low-dimensional projection of a sample to address the instability of GAN training in high-dimensions. GMAN (10) presents diverse aggregation methods of multiple discriminators, where both hard and soft discriminator selection strategies are studied. Note that all the existing approaches learn the multiple discriminators independently. Thus, these discriminators may have strong correlations, which may not be appropriate for diversifying generated samples. Our approach, however, assigns each sample to the best-suited discriminator through interactions among the discriminators, and, consequently, each discriminator becomes an expert model for the assigned examples.
3 Multiple Choice Learning
We present the main idea of MCL (8] and its extensions briefly. Given a training dataset with N samples, D {(x,, J M models, 1M and task-specific loss function, ((.-), MCL
minimizes the following oracle loss:
N
LMCL (D)
Σ min E(V fm (x,)).
(1)
In other words, only the model with the smallest error out of M candidates is selected for each example. This optimization process makes each model fm(() become expert for subset of D, thus leads to forming a natural cluster in D.
A weakness of MCL is the possible mistakes caused by overconfidence issues. If non-specialized models make wrong predictions with high confidence in the score aggregation process, the average scores misleading and the ensemble model may result in puor-quality outputs. To alleviate this limitation, Confident Multiple Choice Learning (CMCL) (20) adopts a confident oracle loss that enforces the predictions of non-specialized model to be uniformly distributed using KL divergence, denoted by DxI. Assuming that fm (.) predicts the output distribution given data point x, i.e., P.n (x)),, the loss is modified
M
LCMCL(D) Σ
Vi,m (m()) + 3(1 vi.m) (am
where (yyy) the uniform distribution and the flag variable ft E {0, 1} allows the choices of the specialized models. Note that, if >M Ui,m k (k < .), each example is assigned to k models.
4 MCL-GAN
We describe our GAN extension with a generator G(:; and M discriminators (Dm Φm) ) M
Let p. and Pa be the distributions in latent space and real data space. respectively. Given z pz. the generator produces sample x = G(z; 0) and each the M discriminators makes a prediction on real example represents the probability that x belongs to the true data distribution. We now describe the loss functions L and L for training discriminators and the generator, respectively.
Pa the fake sample x. Each prediction, Dm (x; Φm), ranges in [0. 1 and
3

---

4.1 Expert training
Assuming that we draw Na real data and generate Ng examples in each training batch, denoted by and x, respectively, each network is trained as follows.
Expert discriminators are the ones that predict the highest scores for each example. With the indicator variable 2i,m for real example x,, the discriminators are trained minimize the following loss function:
Discriminators
Na
M
C.
(x)
Vi,m log( Dm (X:; (m)))
(3)
mmml
where we choose experts out of M discriminators for each example, i.e., Em=1 Vi.m = k. For fake sample, since all discriminators have to identify it correctly, the following loss is added to
N
C.0 (
Σ
log(1
Dm (G((s))):
Generator We train the generator using the gradients received from the expert models to encourage the generator to find the closest mode given With another indicator variable uj. for Zj. the expert loss for the generator is given by
c
Jjm log(1
Dm (G(zz;:)); om) and Σ
ujm k.
(5)
J=1
4.2 Non-expert training
The non-expert discriminators should not be overconfident real examples while it is desirable produce higher scores for real samples than fake ones. For this requirement, we give a uniform soft label, e-g., y [0.5, 0.5] for non-expert discriminators and regularize them with certain weight. To be precise, we obtain the following non-expert loss term corresponding to (3):
Na
M
Uim) (x;),
-
(6)
2=
1==1
where Vi.m is the same indicator variable defined in and (: y) is the cross-entropy loss function
given a target label y. The other counterpart for (5) is derived similarly as
N₂
M
Σ
Σ 1==1
The non-expert model training is effective to handle the overconfidence issue, but the model may still suffer from the data deficiency problem frequently happening in the standard MCL algorithms because each discriminator can observe only a subset of the whole dataset. To alleviate this limitation, our discriminators share the parameters of all layers in the feature extractor while branching the last layer only. This implementation is also sensible since the discriminators partially have the same objective to distinguish fake examples. The common representations of real samples are prone to be learned in the earlier layers despite being clustered in different subsets, whereas the critical information for high-level understanding is often encoded in deeper layers. Moreover, the number of model parameters and training time are saved significantly while still taking advantage of ensemble learning.
4.3 Balanced assignment of discriminators
On top of the adversarial losses, we introduce another loss for balanced updates of discriminators. As there is no supervision about the specialization target of a discriminator, e-g., class labels or feature embeddings, may be difficult to reasonably distribute real samples to expert models from

---

the beginning. Since the abilities of individual discriminators are severely off-balanced, they are highly prone to assign all samples a few specific models. Especially at an early phase of training. the model's capability is more sensitive to the number of updates in the discriminators.
To tackle this challenge, we propose another loss, called the balance loss, which gives discrimina- tors equal chances for updates. Let the selection of expert discriminators approximately follow categorical distribution with a parameter 14 = [ei..... J.M. Then the loss computed by the KL divergence of the probability distribution of discriminators for being selected as experts from To obtain the probability for discriminator selection, we apply the sof tmax function to the vector of M predictions from discriminators for each example since the discriminator with the highest score is guaranteed to be chosen as an expert. We average these probability vectors over the training batch, DM (x, T)) where B(;T) denotes a vector-valued softmar function with a temperature r given a logit. To sum up, the balance loss given by
ie., q = N, Ei =i=l s ((D (IDi(x))..) (x.)....
(x) (pllg)
In practice, we set Pm 1 Vm to update the discriminators evenly, which is because the true distribution unavailable. M assumption may not be congruent the real distribution of the dataset and excessively forced assignment would not result in an optimal clustering for specialization. We, therefore, decrease the weight for the balance loss gradually during training. Eventually, each example will be naturally assigned to its best model with very small weight of the balance loss. This adjustment helps stabilize training and naturally cluster the reference data. Note that the models are balanced within a few epochs and the weight reduction helps generate higher quality samples.
Likewise, a small constraint the distribution of the generator's output facilitates balanced generation while preventing the statistics of generated samples from being skewed. For this case, we use the distribution of the discriminators' assignments q instead of arbitrarily chosen μe, i.e.
L (x) (gllo)
where O 1, N,
((D((())))) DM (G(
4.4 Total loss
Altogether, the total losses for discriminators and the generator are summarized respectively as
_ = af + Ba and L C + af Pg
(10)
where and (Ba, 3g) are hyperparameters. Note that the proposed approach is applicable any GAN formulations with the corresponding adversarial losses.
4.5 Choice number of discriminators
A remaining concern is how to find the optimal number of discriminators while such information is not available in general as many clustering tasks. the number discriminators is much larger than the optimal one, it more desirable to focus training subset of discriminators than dividing the dataset into many minor clusters forcefully.
To ease this issue, we augment the & regularization loss to the discriminator side with weight y and encourage the sparsity in the discriminator selection for more desirable clustering results. Hence, even in the case that we are given an excessively large number of discriminators, our algorithm converges good points by using small subset of discriminators in practice. strategy may not always lead to the optimal number of discriminators and have conflicts with the balance loss (8). However, the balance loss fades away as training goes on, and our model identifies proper number of clusters by deactivating a fraction of discriminators adaptively. This sparsity loss would be useful when we learn on real world datasets, where the examples are drawn from unknown distributions.
true that this
5 Experiments
We evaluate the performance of MCL-GAN on unconditional and conditional image generation. In the results reported throughout this section, the asterisk (*) denotes the copy from other works.
5

---

1)
1)
MCL-GAN GAN
MCL-GAN 2)
IK
5K
10K
20K
30K
5K
10K
20K
50K
Figure 1: Snapshots of 256 random samples drawn from the generators of the baseline and MCL-GAN with (left) the standard GAN loss and (right) the Hinge loss after IK. 5K, 10K, 20K and 50K steps. Data sampled from the true distribution are in orange while the generated ones are in green.
7
7 = 0.00001
0.68
0.64
0.06
0.42
Discriminator
Discriminator ID
Figure 2: Effect of the & loss weight in MCL-GAN (r). The graphs show the ratio of training examples associated with each discriminator.
5.1 Results on synthetic datasets
We first perform toy experiments to verify the main idea of MCL-GAN intuitively. We consider mixture of 8 isotropic Gaussians in 2D whose centers are located on the circumference of circle with radius /2 while their standard deviation in each dimension is 0.05. We employ discriminators for training with the standard GAN loss while utilizing 2 discriminators with the Hinge loss [38]. We choose one expert discriminator for each sample (k 1) all experiments.
Figure Iillustrates the snapshots of generated examples, given by the baseline methods and MCL- GANs, through iterations. While the base models (M fail to cover all 8 modes, MCL-GANs manage to identify diverse modes quickly and produce samples belonging to all modes eventually, The models trained with the Hinge loss tend to generate discriminator; MCL-GAN also requires less discriminators the Hinge loss case to reconstruct the original data distribution. Figure illlstraaes how adaptively MCL-GANs select discriminators, given an excessive number of discriminators, e.g., M 20. They utilize 8 and 16 expert discriminators when 7 0 and 10-5 respectively; each mode tends be associated with or 2 discriminators depending on the value of y. We visualize the detailed mapping between examples and discriminators in Appendix A.I This observation implies that MCL-GAN covers all modes effectively while the E loss helps identify the proper number of discriminators to generate high-fidelity data.
diverse samples even only with single
5.2
Unconditional GANs on image dataset
5.2.1
Experiment setup and evaluation protocol
We run the unconditional GAN experiment on four distinct datasets including MNIST [39] Fashion- MNIST [40]. CIFAR-10 [41] and CelebA (42), where two types of network architectures, DC-
GAN 43 and StyleGAN2 444, are employed. Images are resized to 32x32 except for the CelebA dataset, for which DCGAN and StyleGAN2 adopt 64 x 64 and 128 128 images, respectively. For the
StyleGAN2 experiments on CelebA, we use the first and last 30K images from the "align&cropped" version of the train and validation splits following [30]. We apply architecture with three different GAN loss functions: the vanilla GAN loss []]. the LSGAN loss (2
method to the DCGAN
and the Hinge loss (38). Appendix Appendixx[describee more details of our setting.
We adopt Precision Recall Distribution (PRD) [46] and Frèchet Inception Distance (FID) 47 as our evaluation metrics. The PRD curve provides more credible assessment of generative models than

---

Table I: Precision and recall scores from PRD MNIST, Fashion-MNIST and CelebA
datasets with the DCGAN architecture. Note that GMAN fails to converge with the Hinge loss.
MNIST
Fashion-MNIST
CelchA Rec.t
Method
Loss
Rec.t
Prec.
Rec.t
Prec.t
FID
Prec.7
Base (DCGAN)
0.896
0.778
0.936
0.900
30,93
0.834
0.8.9
GMAN 10
0.968
0.976
0.909
0.955
31.66
0.888
0.873
GMAN T0
GAN
10
0.964
0.977
0.928
0.946
22.45
0.921
0.923
MCI-GAN
0.985
0.977
0.977
0.929
16.88
0.955
0.957
MCL-GAN
10
0.976
0.975
0.964
0.914
21.18
0.940
0,938
Basc (DCGAN)
0.977
0.957
0.928
0.866
19.87
0.923
0,943
GMAN [10]
LSGAN 22
10
0.966
0.973
0.953
0,952
22.72
0.934
0,906
MCL-GAN
10
0.983
0,980
0.963
0.911
17.81
0,950
0,952
Base (DCGAN)
0.790
0.785
0.936
0.853
23.56
0.905
0,883
MCL-GAN
Hinge
38
0.957
0.96.
0,959
0.916
20.49
0.914
0.925
MCL-GAN
10
0.978
0.968
0.949
0.885
21.23
0.928
0.931
Table 2: FID scores CIFAR-I0 with the DCGAN architecture, where denotes the usage of the different DCGAN architecture adopted in [45].
Method
# Disc.
# Gen.
FID
Requirements
Base (DCGAN)* 43
37.7
GMAN 10
10
37.11
Albaquerque al.
13
10
30.26
MCL-GAN
10
26.87
MGAN" 35
10
26.7
Multiple generators for inference
MSGAN
28.73
Class labels training
Base (GAN)"
31.90
Self-conditioned GAN"1
45
18.70
Clustering (with 100 clusters)
MCL-GAN!
17.15
Table 3: KL divergence and LPIPS using pretrained classifiers.
MNIST
Fashion- MNIST
CIFAR-10
Method
M
KL.
LPIPS1
KL
LPIPS f
KL
LPIPS
Base (DCGAN) 43
0.0268
0.0257
0.0437
0.0233
0.0532
0.0562
GMAN 10
5
0.0072
0.0238
0.0587
0.0289
0.1084
0.0594
MCL-GAN
5
0.0127
0.0259
0.0194
0,0294
0.0474
0.0562
single-valued metrics since it quantifies the model in two folds- coverage (recall) and quality of generated samples (precision). We measure the recall and precision of each model by computing the F& and F1/s scores from the PRD curve, respectively. We prefer higher scores and they are equal to 1 when the generated data distribution is identical to the reference. FID is popular evaluation metric of generative models, based on the distance between two datasets with multivariate Gaussian assumption. Lower FID scores mean that generated samples closer to the reference dataset.
5.2.2 DCGAN backbone
Table summarizes the precision and recall scores of our methods compared to other models with three different GAN objectives, when the number of discriminators is set to or 10 (M = 5 or 10) and the number of experts is 1 (k = 1). MCL-GAN achieves outstanding performance in terms of both metrics compared to the baseline and GMAN, which an existing approach adopting multiple discriminators, MNIST and CelebA. For Fashion-MNIST, observe that MCL-GAN focuses on the mode coverage (diversity) while GMAN cares about image quality rather than diversity.
As presented in Table MCL-GAN outperforms the compared GAN models (based on DCGAN) by large margins in terms of FID scores on CIFAR-IO, while being competitive with MGAN, which relies on multiple generators. MCL-GAN also achieves better results than clustering- approach, e.g., self-conditioned GAN (45 which requires additional computation due to clustering during training and the extra model parameters for conditioning the generator's inputs with cluster membership. The results imply that MCL-GAN is effective to maintain multi- modality in the underlying distribution with relatively small memory footprint and without extra supervision.
To analyze the semantic quality of generated images, we present their classification results given by the pretrained classifiers in Table We measure how much the predicted label distribution in each
7

---

Table 4: FID, precision and recall scores CIFAR- 10 and CelebA datasets with the StyleGAN2 architecture, where 10 and 5 discriminators are adopted, respectively, while k =.
CIFAR-10
CelchA30K*
Method
FID,-
Rec.7
Prec.
FIDL
Rec.f
Prec.
Train
Val
Train
Val
Train
Val
Basc (StyleGAN2)
9.06
0.979
0.984
9.37
9.49
0.730
0.741
0.855
0.844
Inclusive GAN
30
11,56
11.28
0.849
0.848
0.927
0.941
MCI-GAN
7.13
0.985
0.989
8.41
8.61
0.988
0.990
0.985
0.983
6
6
6
6
6
5
6
6
6
6
9
9
9
7
7
9
7
9
9
5
8
8
8
8
7
7
7
1
7
9
7
4
7
8
2
7
3
3
3
3
5
5
5
5
5
5
1
1
1
1
1
7
1
3
3
8
3
3
3
7
3
3
3
7
7
9
7
7
9
8
7
7
7
0
0
O
6
O
0
0
(a) MNIST (DCGAN)
(b) MaSHion-MNIST (DCGAN)
(c) CIFAR-10 (StyleGAN2)
Figure 3: Generated image clusters by MCL-GAN. Each row represents the cluster associated with each discriminator (M 10). Note that the images in the same row often have similar shapes and semantics but are not necessarily in the same class.
tested dataset deviates from the true (uniform) distribution using the KL divergence. We also calculate the average of class-wise LPIPS to measure the intra-class diversity of generated images. The overall results are favorable to MCL-GAN but there exist misleading points. For example, GMAN tends to achieve overly high class-wise LPIPS scores due to many diitr--disttibttion examples that confuse the classifier, illustrated in Appendix B.T
5.2.3 StyleGAN2 backbone
Table 4 presents that MCL-GAN is also effective in the state-of-the-art backbone model and out- performs StyleGAN2 (44) and its variation, Inclusive GAN (30), in terms of all metrics. Inclusive GAN employs the sample-wise reconstruction loss by regarding each image as a mode. This strategy appears improve recall but the model may suffer from sampling bias and scalability issues by estimating overly complex distributions. The results imply that MCL-GAN enhances the convergence and the reproducibility of large GAN models, practically leading to improved performance.
5.2.4 Discriminators specialization
Qualitative results Figure 3 qualitatively presents how successfully the discriminators in MCL- GAN are specialized to subsets of datasets. We learn the model with 10 discriminators, and illustrate the generated images with their memberships to discriminators; the images in the same row belong to the same diseriminator. We observe the semantic consistency of images within the same row in both MNIST and Fashion-MNIST clearly.
Analysis using attribute annotations To examine how effectively the discriminators represent the real data, we analyze the discriminator specialization statistics of MCL-GAN using the 40 binary attributes of each image in the CelebA dataset. We first compute the normalized histogram of discriminator assignments for the images with a certain attribute, and then compute the cosine similarity between the histograms obtain a 40 x 40 dimensional similarity matrix, where 40 the number of attributes. Figure 4 illustrates the similarity matrix, where we sample a subset of attributes for better visualization. We discover that the discriminator specialization statistics highly coincide with the latent semantic similarities of examples. Specifically, the four attributes more relevant to female-Arched Aeeeee-Arre___eeeebboee Fyebrows, Wearing Wearing_Farings, Farings, Wearing Lipstick and Heavy_ Makeup-aaaaa are clustered together, while another set of four attributes such as Goatee, Male, Mustache, and Sideburns construct
8

---

Arched Syetrows
Wearing, Eartinss
Wearing Lipstick
Heavy, Maituup
Blund Hair
Black, Hair
Goates
Mole
Mustacha
Sidebums
Archedd_Eyebross Eyebrows
Arched,
Wearlnng_arriinn Earrings
1.00
1.00
0.99
0:99
0.75
0.92
0.83
0.67
0.82
0.06
Wearing I Lipstick
1.00
0.99
1.00
1.00
0.75
0.89
0.78
0.80
0.76
0.79
Hesruy_Makeup Makeup
Heavy
1.00
0.99
1.00
1.00
0.74
0.88
0.76
0.779
0.75
0.78
Blond Hair
0.74
0.75
0.75
0.74
1.00
0.52
0.56
0.61
0.55
0.58
-3.4
Smmianty
Black_Hair Hair
0.90
0.92
0.89
0.88
0.52
1.00
0.94
0.94
0.94
0.95
Goetee
0.80
0.83
0.78
0.76
0.56
0.94
1.00
0.99
1.00
0.99
Male
0.83
0.88
0.80
0.79
0.61
0.94
0.99
1.00
0.98
1.00
Mustache 0.79
0.82
0.76
0.75
0.55
0.94
1.00
0.98
1.00
0.96
0.82
0.85
0.79
0.78
0.58
0.95
0.99
Figure The cosine similarity matrix among the attribute representations obtained by the discrimina- for assignments of MCL-GAN on the CelebA dataset.
Table Quantitative results of conditional GANs the image-to-image translation tasks.
Dataset
Metric
DRIT"
+ MS (DRIT +)" [12
+MCL
MMCL MS
1
57.24 = 2.03
51.85
53.77 + 1.36
49,74
NDB
25.60 1.14
22.80
25.40 1.14
30.00
2.55
Summer + Winter
JSD
0.066 0.005
0.046 0.006
0.036 1 0.004
0.044
0.005
LPIPS
0.115 0.000
0.147 1 0,001
0,199 1 0.002
0,263
0.003
FID 4
47.37 + 3.25
46.23 + 2.45
49.41 + 1.29
41.94 + 1.43
NDB
30.60 2.97
27.80 3.03
23.40 +: 1.52
24.20 3.27
Winter Summer
JSD
0.049 0.009
0.038 0.004
0.033 = 0.002
0.030
LPTPS
0.118 0.001
0.001
0.248 0.001
FID |
22.74 = 0.28
16.02 + 0.30
20.64 = 0.13
15.36 = 0.16
NDB
42.00 = 2.12
27.20 0.84
29,80 = 1.10
22.20 + 2.77
Cat - Dog
JSD
0.127 = 0.003
0.084 0.002
0,048 0.002
0.031 0.002
LPTPS
0.245
0.280 0.002
0.511
0.553 0.000
FID
62.85 0.21
29.57 0.23
20.61 0.05
27.16 0.20
NDB
41.00 0.71
31.00 1::00::0 0.71
16.40 = 0.89
20.20 = 1.48
Dog + Cat
JSD
0.272 0.002
0.068 1 0.001
0.024 0.001
0.031 1 0.001
LPIPS
0.102 = 0.001
0,214 1 0.001
0.429 1 0.001
0.482 - 0.000
Table 6: Quantitative results of conditional GANs in the text toxt-to-image synthesis task.
Dataset
Metric
StackGAN++
+MS [12]
+MCT.
+MCL+
FID 1
26
25.53:1188
22.91
25 25.4410.41
NDB
38.20:12.39
30.60::2.51
80--333336
23.20:13.03
CUB-200-2011
JSD
0.09-10.01
0.07+00.00
0.08++0.00
0.05=00000
LPTPS +
0.36++0.00
0.37+0.01
0.63++0.00
0.62+0.00
strong group. These results support that the individual discriminators learned by MCL-GAN are specialized to reasonable subsets of the dataset, which is helpful to generate more realistic images.
5.3 Conditional image synthesis
We apply MCL-GAN to image-to-inee to-image translation and text-to- text-to-image synthesis tasks, which require complex architectures to generate high-resolution images. In this experiment, we adopt MSGAN [12]. a technique with mode-seeking regularizer, as an additional component for alleviating the mode collapse in conditional GANs. Then, we observe whether the mode seeking and our multiple choice learning create synergy, based on FID, NDB/ISD (48), and L.IPS (49) following [12). Note that NDB counts the number of statistically different bins based on the clusters made by k-means clustering.

---

Table 7: Comparison with other discriminator assignment strategies such as Minimum, Random, and
GT- GT-Assiin. GT--Assign links an expert discriminator with real samples using the ground-trun trrnh class labels under our multi-discriminator framework but is unrealistic due to the requirement of the labels.
MNIST
Beshion-MNIST
CelebA
Strategy
M
k
Rec.1
Prec.
Rec.1
Prec.
Rec.1
Prec.
Buse (DCGAN) 43
0.896
0.778
0.936
0,900
0.834
0.839
Minimum
0.913
0.904
0.943
0.906
0.945
0.893
Random
0.971
0,954
0,930
0.917
0.930
0,946
MCL-GAN
0.985
0.977
0.977
0,929
0.955
0.957
GT-AAssin
10
0.978
0.966
0.969
0.935
Image-to-image translation We choose DRIT 4 5 an unpaired image-to-image translation method based on the cycle consistency, as our baseline. We employ MCL-GAN with M = 3 and k = 1 for distinguishing real and translated images. As shown in Table MCL-GAN significantly improves the diversity measure, LPIPS, while achieving high-fidelity data generation performance in terms of other metrics. In particular, our approach works better on more challenging task, cat = dog, since it effectively handles the changes both object shape and texture across domains by specializing discriminators to subset of modes. Note that the integration of MCL is mostly beneficial and the addition of the mode-secking module often leads to extra performance gains.
Text-fo-iman synthesis This experiment is based StackGAN+ [50] trained CUB-200- 2011 (5T] with a mode-seeking regularizer. StackGAN++ has hierarchical structure with a specialized pair of discriminator and generator to certain image resolution. We adopt its 3-stage version and trains an MCL-GAN with M = 3 and &o = only at the last stage, which handles images with size 256 256. Tablelojillustrates that the integration of MCL improves performance consistently, especially in terms of the diversity measure, LPIPS, where we observe the merit of the mode-seeking regularizer via the combination with MCL..
5.4 Discriminator assignment strategies
MCL-GAN is a model-agnostic ensemble algorithm with multiple discriminators, which is con- ceptually advantageous other discriminator assignment strategies. In practice, our specialized discriminators to the subsets of training data outperforms independent training of discriminators on the whole dataset (as in GMAN) or different discriminator assignment strategies such minimum-sen score discriminator selections, opposite to MCL-GAN, and random selections. Also, the performance of MCL-GAN is competitive the method assigning discriminators based on the ground-truth class labels, which exhibits the reliability of the discriminator specialization by MCL. Table 7 presents that MCL-GAN achieves outstanding performance compared to other strategies especially for the recall metric, even compared with GT-Assign without relying on class labels for discriminator specialization. The proposed approach is also efficient since it free from any time-consuming clustering procedure for sample assignments to discriminators during training, and incurs marginal extra cost despite the use of multiple discriminators because the discriminators share the feature extractor.
6 Conclusion
We presented a generative adversarial network framework with multiple discriminators, where each discriminator behaves as an expert classifier and covers separate mode in the underlying distribution. This idea is implemented by incorporating the concept of multiple choice learning. The combination of generative adversarial network and multiple choice learning turns out to be effective alleviate the mode collapse problem. Also, the integration of the sparsity loss encourages our model to identify the proper number of discriminators and estimate a desirable distribution with low complexity. We demonstrated the effectiveness of the proposed algorithm on various GAN models and datasets.
Acknowledgements This work was partly supported by Samsung Electronics Co., Ltd., and the In- stitute of Information & communications Technology Planning & Evaluation (ITTP) grants 2No. 0-00959, (Part 2) Few-Shot Learning of Causal Inference in Vision and Language for Decision Artificial Intelligence Graduate School Program (Seoul National Uni-
Making: No. 00.200110-0134 versity)) and the National Research Foundation of Korea (NRF) grant NNo.2022R1 A5A708390811 Trustworthy Artificial Intelligence] funded by the Korean government (MSIT).
10

---

References
[1] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B.. Warde-Farley, D., Ozair, S., Courville, A.,
Bengio, Y: Generative Adversarial Nets. In NeurIPS, (2014)
[2] Kingma, D.P., Welling, M.: Auto-encoding Variational Bayes. In ICLR. (2014)
[3] Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired Image-to-lmage Translation using Cycle- Consistent Adversarial Networks. In ICCV. (2017)
[4] Lee, H.Y., Tseng. H.Y., Huang, J.B., Singh, M.K., Yang. M.H.: Diverse Image-to-Image
Translation via Disentangled Representations. In ECCV. (2018)
[5] Lee, H.Y., Tseng, H.Y., Mao, Q. Huang, J.B., Lu, Y.D., Singh, M.K., Yang, M.H.: DRIT++ Diverse Image-to-Image Translation via Disentangled Representations. IJCV (2020)
[6] Yeh, R.A., Chen, C., Yian Lim, T., Schwing, A.G., Hossga M., Do, M.N.: Semantic Image Inpainting with Deep Generative Models. In CVPR. (2017)
[7] Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani,
A., Totz, J., Wang, Z., et al.: Photo-Realistic Single Image Super-Resolution using Generative
Adversarial Network. In CVPR. (2017)
[8] Guzman-Rivenn A., Batra, D., Kohli, P.: Multiple Choice Learning: Learning to Produce
Multiple Structured Outputs. In NeurIPS, (2012)
[9] Nguyen, T., Le, T., Vu, H., Phung, D.: Dual Discriminator Generative Adversarial Nets. In NeurIPS, (2017)
[10] Durugkar, I., Gemp, I., Mahadevan, S.: Generative
Networks. (2017)
[11] Doan, T., Monteiro, J., Albuquerque, I., Mazoure, B., Durand, A., Pineau, J., Hjelm, R.D.: On-Line Adaptative Curriculum Learning for GANs. In AAAL (2019)
[12] Mao, Q., Lee, H.Y., Tseng, H.Y., Ma, S., Yang, M.H.: Mode Seeking Generative Adversarial Networks for Diverse Image Synthesis. CVPR. (2019)
[13] Albuquerque, I., Monteiro, J., Doan, T., Considine, B., Falk, T., Mitliagkas, L.: Multi-Ohjectivee Training of Generative Adversarial Networks with Multiple Discriminators. In ICML. (2019)
[14] Neyshabur, B., Bhojanapalli, S., Chakrabarti, A.: Stabilizing GAN Training with Multiple
Random Projections. arXiv preprint arXiv: 705.07831 (2017)
[15] Lee. S., Prakash, S.P.S., Cogswell, M., Ranjan, V., Crandall, D., Batra, D.: Stochastic Multiple Choice Learning for Training Diverse Deep Ensembles. In NeurIPS. (2016)
[16] Chen, Q. Koltun, V.: Photographic Image Synthesis with Cascaded Refinement Networks. In ICCV, (2017)
[17] Mun, J., Lee, K., Shin, J., Han, B.: Learning Specialize with Knowledge Distillation for Visual Question Answering. In NeurIPS. (2018)
[18] Firman, M., Campbell, N.D., Agapito, L., Brostow, G.J.: DiverseNet: When One Right Answer Enough. In CVPR. (2018)
[19] Li, K., Zhang, T., Malik, Diverse Image Synthesis from Semantic Layouts via Conditional IMLE. In ICCV. (2019)
[20] Lee, K., Hwang, C., Park, K.S., Shin, J.: Confident Multiple Choice Learning. In ICML. (2017)
[21] Tian, K., Xu, Y., Zhou, S., Guan, J.: Versatile Multiple Choice Learning and its Application to Vision Computing. In CVPR. (2019)
[22] Mao, X., Li, Q., Xie, H., Lau, R.Y., Wang, Z., Paul Smolley, S.: Least Squares Generative Adversarial Networks. In ICCV. (2017)
[23] Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein Generative Adversarial Networks. In ICML. (2017)
[24] Guirajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A.C.: Improved Training of Wasserstein GANs. In NeurlPS. (2017)
[25] Lin, Z., Khetan, A., Fanti, G., Oh, S.: PacGAN: The Power of Two Samples in Generative Adversarial Networks. In NeurlPS, (2018)

---

[26] Zhao, J., Mathieu, M., LeCun, Y:: Energy- based Generative Adversarial Networks. In ICLR. (2017)
[27] Donahue, J., Krähenbühl, P., Darrell, T.: Adversarial Feature Learning. In ICLR. (2017)
[28] Dumoulin, V., Belghazi, I., Poole, B., Mastropictro, O., Lamb, A., Arjovsky, M., Courville, A.: Adversarially Learned Inference. In ICLR. (2017)
[29] Srivastava, A., Valkov, L.,, Russell, C., Gutmann, M.U., Sutton, C.: VEEGAN: Reducing Mode
Collapse in GANs using Implicit Variational Learning. NeurIPS, (2017)
[30] Yu, N., Li, K., Zhou, P., Malik, J., Davis, L., Fritz, M.: Inclusive GAN: Improving Data and
Minority Coverage in Generative Models. In ECCV. (2020)
[31] Liu, S., Zhang, X., Wangni, J., Shi, J.: Normalized Diversification. In CVPR. (2019)
[32] Yang. D., Hong, S., Jang, Y., Zhao, T., Lee, H.: Diversity-Sensitive Conditional Generative
Adversarial Networks. In ICLR. (2018)
[33] Tolstikhin, 1.O., Gelly, S., Bousquet, O., Simon-Gabriel, C.J., Schölkopf, B.: AdaGAN:
Boosting Generative Models. In NeurlPS. (2017)
[34] Ghosh, A., Kulharia, V., Namboodiri, V.P., Torr, P.H., Dokania, P.K.: Multi-Agent Diverse Generative Adversarial Networks. In CVPR. (2018)
[35] Hoang, Q.. Nguyen, T.D., Le, T., Phung, D.: MGAN: Training Generative Adversarial Nets
with Multiple Generators. In ICLR. (2018)
[36] Park, D.K., Yoo, S., Bahng, H., Choo, J.. Park, N.: MEGAN: Mixture of Experts of Generative Adversarial Networks for Multimodal Image Generation. In IJCAL (2018)
[37] Trung Le, Q.H., Vu, H., Nguyen, T.D., Bui, H., Phung, D.: Learning Generative Adversarial Networks from Multiple Data Sources. In IJCAI (2019)
[38] Lim, J.H., Ye, J.C.: Geometric GAN. arXiv preprint urXiv:1705.000 02889 (2017)
[39] LeCun, Y., Bottou, L., Bengio, Y., Haffner, Gradient-Band Baadee Learning Applied to Document Recognition. IEEE (1998)
[40] Xiao, H., Rasul, K., Vollgraf, R.: Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms. arXiv preprint 170..077770 (2017)
[41] Krizhevsky, A.: Learning Multiple Layers of Features from Tiny Images. Technical report (2009)
[42] Liu, Z., Luo, P., Wang, X., Tang, X.: Deep Learning Face Attributes in the Wild. In ICCV. (2015)
[43] Radford, A., Metz, L., Chintala, S.:
Unsupervised Representation Learning with Deep Convolu- tional Generative Adversarial Networks. In ICLR. (2016)
[44] Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T.: Analyzing and Improving the Image Quality of StyleGAN. In CVPR. (2020)
[45] Liu, S., Wang, T., Bau, D., Zhu, J.Y., Torralba, A.: Diverse Image Generation via Self-
Conditioned GANs. In CVPR. (2020)
[46] Sajjadi, M.S.M., Bachem, O.,, Lučić, M., Bousquet, O., Gelly, S.: Assessing Generative Models via Precision and Recall. In NeurlPS. (2018)
[47] Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Klambauer, G., Hochreiter, S.: GANs Trained by Two Scae- Update Rule Converge to a Nash Equilibrium. In NeurlPS. (2017)
[48]
Richardson, E., Weiss, Y.: On GANs and GMMs. (2018)
[49] Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang. O.: The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In CVPR. (2018)
[50] Zhang, H., Xu, T., Li, H., Zhang, S., Wang, X., Huang, X., Metaxas, D.N.: StackGAN++:
Realistic Image Synthesis with Stacked Generative Adversarial Networks. TPAMI (2018)
[51] Wah, C., Branson, S., Welinder, P., Perona, P., Belongic, S.: The Caltech-UCSD Birds-200-2011
Dataset. Technical Report CNS-TR-201 2N--TR--011-001 1-001, California Institute of Technology (2011)
[52] Kingma, D.P., Ba, J.: Adam: A Method for Stochastic Optimization. In ICLR. (2015)
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Appendix
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Appendix[F
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material as URL)? [Yes] See Appendix C
and and the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 2
and Appendix C
(c) Did you report error bars (e-g., with respect the random seed after running experi-
ments multiple times)? [Yes] Partly. See Section 3.3 5.3 and Appendix AA.2.5
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix AA.3
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent obtained from people whose data you're you're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "mcSfJKMI4Q", "reviewer": "Reviewer_Hs6x", "summary": "The authors propose a multi-discriminator generative adversarial network to address the issue of mode collapse. The discriminators are tied together through parameter sharing which helps reduce the training time.\n", "questions": "My major concern is whether the proposed method is overfitting on the training set due to several specialized discriminators. Please see and address point 1 in weakness above.", "limitations": "The authors have discussed the limitations of their work and potential negative societal impact. According to the authors:\n\"we believe that the proposed approach can alleviate the bias and fairness issues by identifying the minority groups in a dataset effectively.\"\n\nHow does the proposed method identify the minority groups in a dataset is not very clear to me.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n1.\tAim to solve the problem of mode collapse which is of interest to the community.\n\n2.\tThe number of discriminators is determined adaptively.\n\n3.\tThe proposed model is generic and can be applied to various variants of GANS as there is no architecture or loss specific constraints.\n\nWeakness:\n\n1.\tSince the proposed model leverages multiple specialized discriminators, I suspect that the generator might overfit the training data. I request the authors to provide visualization of nearest neighbours (from the training split) of some randomly generated samples. They should also compute and report (1) Pixel Memorization Score (PMS) and (2) Inception Memorization Score (IMS) as proposed in the following paper.\nhttps://proceedings.mlr.press/v161/mondal21a.html\n\n2.\tAlthough the authors report various metrics, there has been several drawbacks identified with the precision/recall metric. I suggest the authors to compute and compare density/coverage score instead as proposed in:\nhttp://proceedings.mlr.press/v119/naeem20a/naeem20a.pdf\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "hBfpZNsUWOS", "reviewer": "Reviewer_oqkx", "summary": "This paper proposes a generative adversarial network with multiple discriminators for GAN training. The proposed method guides each discriminator to have expertise in the subset of the entire data. For each sample, only k experts out of M discriminators are used. To address the over-confident problem, this paper proposes another loss, named non-expert training loss. To balance the assignment of discriminators, this paper proposes another loss, named balance loss. The proposed method is evaluated on several tasks, including unconditional image generation, image-to-image translation, and text-to-image synthesis. The results show that the proposed method outperforms the baselines.", "questions": "Please refer to Weaknesses.", "limitations": "In practice, it may be difficult to tune the non-expert training loss and balance loss. ", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The idea of using multiple discriminators is sensible. \n2. The proposed method is evaluated on various tasks, including unconditional image generation, image-to-image translation, and text-to-image synthesis. \n3. The experiments show that the proposed method outperforms GMAN. \n\nWeaknesses:\n1. The proposed method is an extension of GMAN. The difference is that the proposed method selects part of the multiple discriminators. In my opinion, this extension is incremental. \n2. The extension (i.e., using part of the multiple discriminators) also introduces two problems, i.e. over-confident problem and balance problem. Although the authors propose two specific losses to tackle the above problems, both the over-confident problem and balance problem seem difficult to tune in practice. \n3. The advantages of the proposed method over GMAN are not clearly explained. This paper stated that “While GMAN focuses on the loss to the generator with parallel learning of discriminators, our strategy specializes each discriminator for more informative feedbacks to the generator”. This is not convincing to me. \n4. The resolutions of images used in the experiments are 32x32, 64x64, or 128x128. I think the effectiveness of the proposed method should be verified on larger datasets, such as FFHQ with 1024x1024. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "O59sugVypd_", "reviewer": "Reviewer_bp3P", "summary": "This paper proposes a new GAN framework named MCL-GAN. MCL-GAN comprises a single generator and multiple discriminators and follows a multiple-choice learning (MCL) strategy to forward signals of samples to the generator. By conveying the plentiful signals to the generator, the authors insist the generator can generate diverse samples, which means MCL is effective in overcoming the chronic mode-collapse problem. By performing image generation experiments using standard benchmark datasets, the authors demonstrate that the proposed MCL-GAN helps in increasing the diversity of generated images.", "questions": "No", "limitations": "I understand that the authors are cautious about discussing the shortcomings of their work. So, I think it does not matter whether the authors provide the limitations of MCL-GAN or not. Instead, I highly recommend the authors write more about the negative societal impact of their work in the supplementary material.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n(+) The motivation of MCL-GAN is reasonable. Also, MCL-GAN is simple, so I think researchers can implement MCL-GAN easily. \n\n(+) MCL-GAN does not require too many parameters to be burdensome. Multiple discriminators share their parameters, and all we need is stacking multiple fully connected layers parallelly upon the shared network.\n\n(+) Toy experiments are appropriately designed and successfully support the main pillar of the paper. \n\n(+) Experiments using benchmark datasets demonstrate that the proposed MCL training is effective, and MCL-GAN can generate more diverse images than a baseline. \n\nWeaknesses\n\n(-) Limited novelty. Although experimental results show the efficacy of MCL-GAN, I think the technical novelty of MCL-GAN is limited. I think introducing MCL into some frameworks is not new to the community. \n\n(-) I am not sure whether the proposed MCL-GAN can be scaled up for high-resolution image generation. This paper does not contain any experiment using high-resolution image datasets, such as ImageNet, AFHQ, and FFHQ.\n\n(-) Using the old version of precision and recall [1] seems improper for measuring the diversity of generated images. I recommend the authors use the improved version of precision and recall [2] to quantify the diversity. The paper [2] has shown that the old precision and recall sometimes cannot capture the diversity well. The authors can compare their model with previous methods using StudioGAN library [3].\n\n[1] Sajjadi, M.S., Bachem, O., Lucic, M., Bousquet, O., & Gelly, S. (2018). Assessing Generative Models via Precision and Recall. NeurIPS.\n\n[2] Kynkäänniemi, T., Karras, T., Laine, S., Lehtinen, J., & Aila, T. (2019). Improved Precision and Recall Metric for Assessing Generative Models. ArXiv, abs/1904.06991.\n\n[3] https://github.com/POSTECH-CVLab/PyTorch-StudioGAN.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "N-PpOwKt2E", "reviewer": "Reviewer_FAZc", "summary": "The paper proposes a simple yet effective way to mitigate the mode collapse issue of GANs. To this end, the authors propose the utilization of multiple discriminators that specialize on different subsets of the distribution. This process is implemented using multiple choice learning.", "questions": "I suggest that the authors address the issues raised in \"weaknesses\" section.", "limitations": "The authors have included an ethical considerations part, however I would also like to see a discussion regarding the limitations of the proposed method.", "rating": 4, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n A. The main idea is simple and clearly presented.\n\n B. The experimental results highlight the efficacy of the method in the terms of a number of metrics.\n\n\nWeaknesses: \n\nA. In order to understand the significance of the proposed method, I’m missing a comparison with other clustering-based approaches, e.g.,\n\n[1]. ClusterGAN : Latent Space Clustering in Generative Adversarial Networks\n\n[2]. Diverse Image Generation via Self-Conditioned GANs\n\n\nB.  I would like to see some more metrics that are specific to the diversity of the generated samples. e.g., IvOM [3],  Pairwise Distance [3].\n\n[3]. Unrolled generative adversarial networks \n\n\nC. I would like to see how the approach performs on some mode-coverage benchmarks, e.g.,\n\n - Fashion-MNIST and partial MNIST of [4]\n\n - Stacked MNIST of  [5]\n\n[4]. Rethinking Generative Mode Coverage: A Pointwise Guaranteed Approach\n\n[5]. VEEGAN: Reducing Mode Collapse in GANs using Implicit Variational Learning\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept