CoGANs: Collaborative Generative Adversarial Networks
Authors: Omkar Joglekar
Goren Gordon
Venue: Submitted to ICLR 2023 (2023)
Keywords: GANs
Multiple generators

Abstract: In complex creative scenarios, co-creativity by multiple agents offer great advantages. Each agent has a specific skill set and a set of abilities, which is sometimes not enough to perform a general, large and complex task single-handed.
These kinds of tasks benefit substantially from collaboration. In deep learning applications, data generation is an example of such a complex, potentially multi-modal task. Previous Generative Adversarial Networks (GANs) focused on using a single generator to generate multi-modal datasets, which is sometimes known to face issues such as mode-collapse and failure to converge. The multi-generator based works such as MGAN, MMGAN, MADGAN and AdaGAN either require training a classifier online, the use of complex mixture models or sequentially adding generators, which is computationally complex.
  In this work, we present a simple, novel approach of training collaborative GANs (CoGAN), with multiple generators and a single critic/discriminator, without introducing external complexities such as a classifier model. We show that this method of workload division meets the state-of-the-art quality metrics, and makes GAN training robust. We present a proof-of-concept on the MNIST dataset, which has 10 modes of data. The individual generators learn to generate different digits from the distribution, and together learn to generate the whole distribution. We introduce a new component to the generator loss during GAN training, based on the Total Variation Distance (TVD) and show that it significantly improves stability during training and performance over state-of-the-art single generator GANs.

Under review as a conference paper at ICLR 2023
COMMUTE-GAN: COMPTETITIVE MULTIPLE EFFI- CIENT GENERATIVE ADVERSARIAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
In complex creative scenarios, co-creativity by multiple agents offers great advan- tages. Each agent has a specific skill set and set of abilities, which is generally not enough to perform large and complex task single-handed. These kinds of tasks benefit substantially from collaboration. In deep learning applications, data generation is example of such complex, potentially multi-modal task. Previous Generative Adversarial Networks (GANs) focused on using a single generator to generate multi-modaa datasets, which is sometimes known face issues such mode-collapse and failure to converge. Single generators also have to be very large that they can generalize complex datasets, this method can easily run into memory constraints. The current multi-generator based works such as MGAN, MMGAN. MADGAN and AdaGAN either require training classifier online, the use of complex mixture models or sequentially adding generators, which is computationally complex. In this work, we present a simple, novel approach of training competitive multiple efficient GANs (ComMutE- GANs,, with multiple generators and a single critic/diseriminator, without introducing external complexi- ties such as classifier model. We introduce new component to the generator loss during GAN training, based on the Total Variation Distance (TVD). Our method offers a robust, stable, memory efficient and easily parallelizable architecture. We present a proof-of concccep on the MNIST dataset, which has modes of data. The individual generators learn generate different digits from the distribution, and together learn to generate the whole distribution. We compare ComMutF-GGANG with larger single-generator GANs and show its memory efficiency and increased accuracy.
INTRODUCTION
With respect to human beings, "Creators" refer to any and all who engage in creative thinking. When people learn about new topics, they create cognitive structures that allow them to understand the topics; they generate concepts that are new to them, although possibly already very well known to others. This is creativity at a strictly intra-personal level. When working in a social setting, such as company classroom, one has to broaden this horizon include "Co-creativity" """rativtty""Crraiviiit through collaboration" summarizes the definition of co-creativity as defined by Lubart (2017). Often, collaborators have different complementary skills that enable them frequently produce shared creations that they could not or would not produce on their own (Lubart & Thornhill-Milne
2020).
Al-aided co-creation has also been proven improve general well-being (Yu 2021)).
Generative Adversarial Nets (GANs) are implicit generative models where one or more generators play a zero-sum game with a discriminator to recreate and potentially expand a chosen dataset. According to the definition established above, the generator models in modern day GANs such as those described in the works of Karras et al. (2018); [auer al. (2022); Kaaraa al. (020)):
Goodfellow et al. 42014); [Radford et al. (2015): Arjovsky et al. (211)) Guirajani et al. (2017) exhibit creativity on an intra-personal level. Accordingly, generative networks have been applied in many creative applications such as painting (Ganin al. 2018 Mellor al. 2019 Parikh & Zitnick 2020)
doodling (Ha & Eck 2017] Cao et al. 2019) and extending fictional languages (Zacharias et al. 2022). Most noticeable, in all the applications listed above, single, large, generative agent was applied to perform a complex task rather than breaking it down into smaller, more easily manageable sub-tasks. This approach, although effective, upper-bounded by memory constraints. Inspiration

---

Under review as a conference paper at ICLR 2023
from co-creativity research aims to resolve these constraints.
Other implementations of GANs that use collaboration, such as MGAN (Hoang et al. 2017). MM- GAN Pandeva & Schuberi 2019), MADGAN (Ghosh et al.] 2018) and AdaGAN (Tolstikhin et al.) 2017 try to rectify the missing co-creativity functionality of GANs by using a mixture of multiple generators, modeling the input latent space mixture model, requiring the discriminator to classify as well and sequentially train and add generators to the mixture, respectively. MGAN and MADGAN require the discriminator to have classification capabilities and both implementations force the gener- ators to generate different modes. The AdaGAN implementation poses problem of computational complexity because of the sequential training nature. MMGAN on the other hand focuses on using mixtures at the input latent space level rather than on separating generators.
Our work stands to provide an easier approach to co-creativity than the ones presented above. It does not require the online training of a classifier, we do use a pre-trained MNIST classifier to augment the generator loss with the Total Variation Distance (TVD) to enforce mode separation during training, but mode separation maybe enforced using other methods, not requiring this pre-trained classifier at all. We also show that this is easily and efficiently scalable to more than two generators, which can all be trained in parallel without making complex transformations to the latent space.
Analogous to human behaviour in a social setting, collaboration among two or more such generators allows each individual generator focus and specialize in specific sub-task, making the training process more stable, more efficient (memory-wise), the generated images clearer and the distribution of the generated images closer the actual distribution of the chosen MNIST dataset.
BACKGROUND
2.1 GANS, WGANS AND WGAN-GP
Generative Adversarial Nets (GANs) (Goodfellow et al. 2014) consist of two neural networks: generator (G(z)) that takes in randomly sampled noise, and generates fake images; a discriminator (D(x)), that takes in batches of real and fake data points and outputs a 0 for fake images or for real ones. They minimize the well-known GAN objective function
min max E (Dog( + Ex- [log(1 D(x) G D
Where x= G Z~ Pz, P, is the data distribution, Pg is the generator distribution and G and D the generator function and discriminator function, respectively.
A variation of this method is the Deep Convolution GAN (DCGAÑ) (Radford et al. 2015). that uses Convolutional Neural Networks (CNNs), improve image data generation significantly. Wasserstein GAN (WGAN) (Arjovsky et al., 2017) improves vanilla GANs by presenting and rectifying problems with the original GAN objective function. In WGANs, the Wasserstein distance between real data and generated data is minimized by optimizing over the objective
min max Ex- [xx)),((x)) - Ex-,, [())]. G DED
(2)
The WGAN-GP method (Gulaaaan et al. 2017) is an improvement over this method which uses their novel gradient penalty loss, instead of the former weight clipping to implicitly enforce the Lipschitz constraints. The new objective function solved by WGAN-GP is given by
min max Ex- (D(x) Ex- [D(x)] + À E B.R.R.((IVV (() 1)3), DED
where the first two terms are the same equation (2) and the third one is the gradient penalty (GP) term. Here, & = ex + (1 - x)x and e ~ 0 1]. P& the probability distribution associated with Modern GANs such as Karras al. (2018) 8 Sauer et al. (2022); Karras et al. (2020) that provide state-of-the-ant performance on large-scale scale image synthesis improve these basic methods by making their models larger and training procedures more complex. Our method explores another direction, namely, more instances of compact generators competing with each other. The idea takes inspiration from social dynamics and co-creativity research.
2

---

Under review as a conference paper at ICLR 2023
2.2 GANs FOR MULTI-MODAL DATASETS
In past works, there have been attempts with varied scope, to learn multi-modal datasets such as multi-class generation. The most notable are the MMGAN by Pandeva & Schubert (2019), MGAN by Hoang et al. (2017), MAD-GAN by Ghosh et al. (2018) and AdaGAN by [olstikhin et al. (217)) MMGAN approaches mudti-modal generation task by considering the latent space z as a Gaussian
mixture model consisting multiple modes. Initially, Gaussian distribution is sampled from cluster of Gaussian distributions (Col K, 1/K)), where the dataset has K "clusters". Then, z is sampled from this distribution and transformed by the generator create images. This method uses extra encoder (E) model to predict the cluster of data objects.
The MGAN method (Hoang et al.) 2017) implements multiple generators that generate K different batches. One of these batches is sampled randomly using a predefined distribution т. The generated batch fed into a "discriminator + classifier" network with shared weights in all layers except the output layer. The goal of the classifier is to predict which generator generated the images, and the discriminator has same functionality as the vanilla GAN method.
MAD-GAN (Ghosh et al., 2018) is method which combines the classifier and discriminator functionalities into a single model.
All of the previously mentioned methods require that an extra model be trained with the original GAN or multi-generator setup to force separation of modes among the generators. Unlike these methods, AdaGAN introduces sequential "training+adding generators" approach. This approach is computationally inefficient as shown by HHoag et al. (2017).
Although the results suggest that these methods work well, they require adding extra step in the training process. The method presented in this contribution manages to achieve promising results without the need of such additions.
2.3 GAN EVALUATION METRICS
The most commonly used metrics in the GAN community to evaluate the generated images is the Inception Score (IS) introduced in Salimans al. (2016). It uses the pre-trained Inception V3 model (Szegedy et al. (0144) to predict the class probabilities, the conditional probability of class prediction given the image p(yl.r). The marginal distribution p(y) calculated using the batch-wise average of p(y(z)). Finally the KL-Divergence is calculated using
KLD = p(y|=) * (log(p(ul=)) - log(ply)))
(4)
This is performed over multiple batches and the final score is reported as the mean and standard deviation of all the batches.
Over the past few years, IS has started be considered an unreliable metric for evaluation, especially for datasets that are not ImageNet (Barratt & Sharma (018). A better evaluation metric that has become more popular is the Frechet Inception Distance (FID) (Heusel al. 2017). This metric uses the InceptionVs. V3 model without the final classification layer. The latent distribution generated by the Inception V3 network for real and fake images compared. Specifically, the FID metric given by
FID = (Hy 22) + TT(CrrC + -C
(5)
where bi and μ12 are means of the latent representation of fake and real data, respectively and C1 and C2 are their covariance matrices. This metric was proposed as an improvement over the IS. However, suffers from some of the same issues as the IS because of the fact that Inception is pre-trained on a 1000 class dataset of RGB images. For our particular use case with MNIST (a grayscale dataset with only 10 classes), we used the tensor fl ow-gan library's built in method mni st www.ffeeeee..ee diaatanceee function. This func- tion uses pre-trained MNIST classifier, rather than the less relevant Inception to calculate the Frechet distance.
3

---

Under review as a conference paper at ICLR 2023
3 METHODS
3.1 DATASET
MNIST is very well-known dataset. It contains 60,000 images (train set) and 10,000 images (test set) of handwritten digits belonging to 10 classes (numbers 0-9) of 28x28x1 pixels per image. is widely used for mrof-of-concepee of various model architectures and training protocols related to image classification or generation. This is because it is easily available through TensorFlow and PyTorch, smaller images make it easier to implement on simplistic hardware and has well-defined modes in the dataset. This is the dataset we used for our experiments. The train set was used during the training procedures and the test set was used evaluate our models. We resized the images to 32x32 to maintain the dimensions as power of 2, to maintain even dimensions while using strided convolutions. We also normalized pixel values to lie between -1 and 1, as this has been shown to perform better in WGAN-GP (Gulrajani et al. 2017).
3.2 THE TVD LOSS COMPONENT
The Total Variation Distance (TVD) is a quantity that measures the distance between two categorical distributions and is defined as:
TVD(P.Q) 2
Σ P((x) - Q(x) xEX
(6)
Where P and Q are two categorical distribution vectors, X are the number of categories in the distribution and P(x) and (a) denote the probability of class in the distribution and Q respectively. We assumed each class in the MNIST dataset is mode. To enforce mode separation among individual generators, we augmented the generator loss in the original WGAN-GP paper (Gulrajani et al. 2017) with TVD based component. We chose this as the preferred metric because it has well defined bounds (TVD(P, Q) E [0. 1)) and is easy to compute for categorical distributions. It is also commutative, that is TV j) TVVD (j.
Subsequently, we defined the distance metric:
E TVD(C(()), N i= j=it1
A
which is just the average pairwise TVD. Here, C(x,) and C(K;)) are categorical distributions generated by a pre-trained MNIST classifier (C) on &; and ky. the images generated by generator i and j respectively. The classifier is a very basic convolution neural network. Refer to Appendix [ for more information N = n(n 1)/2 is the number of unique pairs of categorical distributions using 'n" generators. Finally, we introduced this term the loss of each generator get:
classifier training. We obtained peak accuracy of 99% after training.
LG
Pa ((x,) +7 (1 A)
i=
where the first term is the original generator loss from the WGAN-GP paper. The hyperparameter
y is the distance weight. We carried out experiments to find the optimal y which produces the best possible FID (refer to Section 4).
3.3 THE TRAINING ALGORITHM
The training algorithm remained the same for all the experiments. However, in each experiment, the effect of each newly introduced parameter (namely, 7 and n)) was tested. The specific values used for each experiment, are specified in Section All other hyperparameters are constant throughout all experiments. The batch size m has to be varied to maintain divisibility by n in step 4 of the algorithm. However, we restricted this value to one of two possible values [500, 504), to maintain approximate similarity. Specifically, for n. = 1, 1, 4, 5, 10, = 500 and for n = 3, 6. 7, 8, 9. m 504. We used the generator learning rate and discriminator learning rate ag = aa = 0.0002. We chose the number of discriminator training iterations per generator iteration, n. 3, following the recommendations of Gulrajani et al. (0017). Also following their recommendations, we used the Adam optimizer, setting 31 = 0.5 and Be 0.9 for all our experiments. The symbol denotes concatenation along batch dimension. X = 10 is the gradient penalty weight as described in Eq. equation 3, The algorithm pseudocode can be found in Algorithm Taatthhmmebblwe below.
4

---

Under review as a conference paper at ICLR 2023
Algorithm 1 Training algorithm
Require: B1, B2, at, the Adam parameters; the distance weight; the number of generators; na the number of discriminator iterations per generator iteration; GP coefficient; m the batch size; ag. Cdd, the learning rates; pre- pre-trainee trained classifier (C).
Require: initial discriminator parameters Wig, initial generator parameters d 1, 2, ..,,
1: while 0g have not converged do
2: for = 1, 1,.... Odd do
3:
Sample real data batch P,, latent variable batch p(z) and U00,11
4:
Split the latent vector batch into sub-batches (Zi...., Zn,)
5:
K; f (z)
Vi 1..,,,
6:
x - X1 02
Oxn
7:
- xx + c)x
8:
La t Σ (x) (x) + D(IV) - 1)
9:
w Adam! aa I.a, w. at (2)
10:
end for
11:
sample n latent variable batches z, ~ p(()
Vi 1,2,
12:
N; f- G.(z,)
Vi I, T
13:
7 + then
14:
A +
Di=1 1j=+1 TVD(C( ((R,))
Lg
E Lk=1 L-D
+ +7(1 A]]
Vi 1,2,.... ...
16:
else
Lg end if
E -D(E!
Vi= 1,2, .... n2
m
18:
0, < Adami Vo, 01,
Vi 2,2,...,
20:
end while
EXPERIMENTS
For all the experiments below, we used the same model architectures for all the generators and the discriminators, the details of which can be found in Appendix B
We performed 3 broad categories of experiments
Bascline experiments as proof of concept our new method.
2. Optimizing for the hyperparameter y.
3. Experiments comparing number of generators versus performance (in terms of FID).
In the baseline experiments, we ran two types of tests:
Model size experiments
TVD loss experiments
In the first type, ran the original WGAN-GP procedure, with bigger generators and compared performance with small generators using our method. used two different ways to make the generators bigger- using dense layers increase depth and using residual blocks. When using 4 small generators, we used y = 12.5. We kept the discriminator the same throughout these experiments. All three models were trained for 70 epochs. In the second type, 5.0, an arbitrary value to see the impact of the
two experiments,
with y 0
(termed as batch concalenation) and one with new loss component on convergence. Here trained all models until convergence was reached.
In the second category, we fixed the number of generators n = 5 and the number of epochs = 70. We carried out tests:
Constant " throughout training at discrete values 0, 5, 10, 11, 12, 13, 14, 15, 20).
Linear decay from 7 = 20 to 7 5, so that the effective weight is equal to the average of the two (Jeff = 12.5), which we found to be close to the optimal for the setup.
Sudden death, where 7 = 20 for the first 20 epochs of training and then is equal to 0 for the remainder.
5

---

Under review as a conference paper at ICLR 2023
In the final category, swept the number of generators n 21, ...., 10] to see how it affects the
performance. In this category of experiments, the constant parameter ry = 12.2 was chosen after the optimizations performed in the second category of experiments. Here, trained all models until convergence was reached.
RESULTS AND DISCUSSION
If run our experiments, with n 1, we can also evaluate the state-of-the-ar FID for a vanilla WGAN-GP. We performed a session of training under these conditions and as can be observed in the the current stale-of-the-art WGAN-GP gives an FID 4.25. In all the future discussions, we will compare the obtained values to this one.
Fig. 4
5.1
BASELINE EXPERIMENTS
5.1.1 MODEL SIZE EXPERIMENTS
In all experiments the discriminator was identical and had 4, 4,306, 177 parameters. In the first experiment, trained 4 generators using the TVD loss method (7 12.5). Each generator model was the same as presented in B and had 910, 660 parameters for a total of 3, 642, 640 parameters. Running these small generators resulted we added extra dense layer to the generator model, resulting in 69, 657, 657,924 parameters 766.5 times the original generator). After training using the WGAN-GP protocol, converged to an average replaced the generator with much deeper network that
final average FID 2.6. In the second experiment,
FID 5.56. In the third experiment, uses residual blocks. The total number of parameters in the generator is 3, 882, 049 (4.3 times the original generator). This model did not converge because the discriminator was casily overpowered. The average FID = 23.42. This model converges if we add more parameters to the discriminator. The models used in the latter experiments are also available in From the above results, it is clear that two or more competing generators exceed the performance of bigger and more complex ones. Our method can be efficiently parallelized due the architecture.
5.1.2 TVD LOSS EXPERIMENTS
We ran both of the two generators experiments until convergence 3 times, to test robustness. The convergence graphs for one training session are presented in Fig. [ Fig. Th]shows the convergence graph for batch concatenation (b.c.) TVD and Fig. Talsswws just batch concatenation (b.c.) based generator loss, We see from the graphs that both of the generators are converging at a similar pace (the lines overlap with each other), under both loss functions. The discriminator loss going indicates that it predicts negative and positive values with almost equal probability (meaning the system has converged). The network managed converge all 3 times for both loss functions (y and 7 = 5).
batch concatenation TVD convergence
batch concetenation convergence
4 Joss
00 loss
30
loss
1oss
10
loss
-10
-20
-20
-40
100
40
80
100
spochs
(b) Convergence graphs batch concatenation aug- mented with TVD
(a) Convergence graphs for batch concatenation only
Figure 1: 2 Generator convergence graphs

---

Under review as a conference paper at ICLR 2023
After convergence, measured 3 metries: FID, calculated according to Eq. equation 5
DVD(C(() (()((z))) and; average class probabilities (softmax) for each generators' gener- ations, as predicted by C. These metrics presented in Tables and
Table FID and TVD metric comparison
Loss
FID
TVD
b.c. only
4.738
0.9832 0.9965
TVD b.c.
3.602
Table Average class probabilities
b.c. only
3
4
5
6
7
8
generator 0
00.00
27.23
13.23
00.63
18.91
00.00
00.00
22.46
00.32
17.23
generator 1
19.15
00.01
08.94
19.75
00.06
17.19
18.35
00.29
14.93
01.31
TVD+b.c.
5
6
7
generator 0
18.98
21.77
00.47
00.03
18.96
00.00
00.13
19.47
00.49
19.69
generator 00.51
00.04
21.25
19.79
00.16
17.90
23.48
00.01
16.82
00.05
We see in Table that the two generators specialize in generating disjoint subsets of classes. The actual samples generated by each generator can seen in Fig We see that they are indeed from separate classes for different generators and of very high quality. We can also observe (Table how introducing a small non-zero constant fy = 5 drastically improves the FID as well as the TVD.
00
1
4
9
5 2
8
8
5
1
4
9
2
8
3
5
6
D
77
4
9
6
22665
7
2
7
7
9
6
65660
5
9
60
0
7 9
0 4 7
<
2 3
3 8
6
9
2 4
/
5 3
8
6
1
4
9
3
2.8 6
4249
9
3 3 3 O
(a) Samples generated by gen0 (tyd+b.c.)
(b) Samples generated by genl (tvd+b.c.)
(c) Samples generated by genO (b.c. only)
(d) Samples generated by genl (b.c. only)
Figure 2: Samples generated by both the methods
5.2
TVD LOSS WEIGHT EXPERIMENTS
5.2.1 DISCRETE CONSTANT
In this experiment we swept gamma across the following values [0, 10, 11, 12, 13, 15,20) 20) and kept it constant throughout the 70 epochs of training. We fixed the number of generators to 5, We evaluated each time using FID metrie. The plot the parametric sweep is seen Figure 3 We can see that there is an optimum value for the TVD loss weight (approximately 7 = 12.2).
5.2.2 LINEAR DECAY
In this experiment, we again fixed the number of generators to 5 and the number of epochs to 70. We linearly decayed the parameter y from 20 to 5, giving us an effective Yeff = 12.5. We evaluated the FID and TVD after training is complete and received FID 3.5379 and 7V D = 0.9999. method clearly gives an improvement over the ofate-of-the-ar FID as well as optimal mode separation.
5.2.3 SUDDEN DEATH
Keeping the other elements the setup exactly the same, we now keep = 20 for the first 20 epochs of training and then 7 for the remaining 50. We end up with FTD 2.6685 and
7

---

Under review as a conference paper at ICLR 2023
FID vs TVD loss weight
5.5
gamma vs FID
Sudden death
5.0
Linear decay
4.5
FID
4.0
3.5
3.0
0.0
2.5
5.0
7.5
10.0
12.5
15.0
17.5
20.0
TVD loss weight
Figure 3: FID vs y sweep
TVD 0.9998, which again is an improvement over the state-of-the-art FID along with great mode separation.
This result interesting as it suggests that one does not need to enforce mode separation (or compute the pairwise TVD) for all of the training epochs. Once the modes have sufficiently separated, remove the TVD loss component and the generators will keep generating separate modes and in much better quality than current state-of-the-art.
5.3 NUMBER OF GENERATORS SWEEP
From the above experiments found optimal value for 12.2 which fixed and kept constant for this experiment. We changed n from to 10 (1 representing current state-of-the-aryy We trained each setup until convergence 250 epochs, whichever came first. We set the early stopping criterion as: less than 0.01 change in the objective for 5 consecutive epochs. We decided to do this because when we tried running for a fixed number of epochs, we realized that for 3, 6, 7, the generators did not converge in the same number of epochs. Here notice that the mode separation problem much more difficult because of the non-divisibility of (the number modes) by n, which results in different training epochs. As we can also see Fig, the FID values for these specific values of n are much higher, denoting much worse performance. The FID for 4 and n 5 is the lowest, showing us that having more generators does not imply better performance and this is another parameter that can be optimized.
more modes of the dataset being assigned to different generator during
CONCLUSIONS
We introduced a novel approach in training WGANs in a collaborative manner which we call ComMutE-GANs This approach is inspired by the principles of co-creativity. Our approach gives improved generation performance under space constraints as we saw in (ectionS.1.II This method also manages improve overall stability, robustness and performance of GANs. We also introduced
8

---

Under review as a conference paper at ICLR 2023
FID
vs no. of generators
6.5
M
6.0
5.5
5.0
FID
4.5
4.0
3.5
3.0
2.5
2 4
10
no. of generators
Figure 4: FID vs no. of generators
a new loss component that can be added to the generator loss during training. We observed that even adding this loss for short amount of time in the initial training can drastically improve performance. We can also tune the weight of this loss to fit the number of generators we plan to use.
major disadvantage of the current method is the fact that need pre-trained classifier calculate a loss during training, which is not efficient. Majority of the future work will involve finding better losses enforce mode separation without the use of a classifier. The eventual goal of this research is to use multiple simple generator models to generate multi-modal data rather than single large, complex network. This research also contribution towards modelling co-creativity in humans. Study of this model can help better understand and implement creativity promoting interactions.
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou.
Wasserstein gan, 2017. URL https
7/arxiv. wrtp//ss///b/11111 07875
Shane Barratt and Rishi Sharma. A note on the inception score, 2018. URL httph:t//ps/ttttt brg/abs/1800 01973
Nan Cao, Xin Yan, Yang Shi, and Chaoran Chen. Ai-sketcher A deep generative model for producing high-qualin sketches. Proceedings of the AAAI Conference on Artificial Intelligence, 33:2564-222 2571, 07 2019. doi: 10.
Yaroslav Ganin, Tejas Kulkarni, Igor Babuschkin, S. M. Ali Eslami, and Oriol Vinyals. Synthesizing programs for images using reinforced adversarial learning, 2018. URL
abs/1804. 01118
Arnab Ghosh, Viveka Kulharia, Vinay Namboodiri, Philip H.S. Torr, and Puneet K. Dokania. Multi- agent diverse generative adversarial networks. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8513-8521, 2018.
doi: 10. N..1199/VVP..001 2018.00888.8

---

Under review as a conference paper at ICLR 2023
Ian J. Goodfellow, Jean Pouget-Abadde Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks, 2014. URL https
7/arxiv wwtpp//pgg........//.////
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. / neurpps
David Ha and Douglas Eck. A neural representation of sketch drawings, 2017.
URL https:
7/arxiv. org/ www./////4... 03477
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.
Gans trained by two time-scale update rule converge to a local nash equilibrium. 2017. doi:
|tttts:/:// org abaaaass/1000 wtg..bs////.... 08500
Quan Hoang, Tu Dinh Nguyen, Trung Le, and Dinh Phung. Multi- generative adversarial nets, 2017. URL httts:://///aiii httpa/aaww..raaaaaaaaaaaaaaaae/e
02556
Tero Karras, Samuli Laine, and Timo Aila. A style- generator architecture for generative adversarial networks, 2018. BtRLhttps: httpaariivvv/.////vva//aa./aaaiv wwg. org/ htth//b///888 04948
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8107-8116, 2020. doi: 10. 1.1109/CVP42260022000000000
Todd Lubart. The 7 c's of creativity. The Journal of Creative Behavior, 51(4):293-296, 2017. doi: wttp//oooo/r11.00/////oooooo/ abs/10.1002/ .990
URL pentps://one
Todd Lubart and Branden Thornhill-Mille. Creativity: An Overview of the 7C's of Creative Thought, pp. 277-305. 04 2020. ISBN ISBN 118-3-947732-35-77 (Hardcover) ISBN 978-3-947732-34--0 (Softcover) ISBN 978- 978-3-947732-33-3 (PDF).
doi: 10.17855/eeuu-4700.66777
John J. Mellor, Funbyung Park, Yaroslav Ganin, Igor Babuschkin, Tejas Kulkarni, Dan Rosenbaum, Andy Ballard, Theophane Weber, Oriol Vinyals, and S. M. Ali Eslami. Unsupervised doodling and painting with improved spiral, 2019. URL /tLt:ttss:///rr/
wwtp:/s////aa.....///////// 1910. 01007
Teodora Pandeva and Matthias Schubert. Mmgan: Generative adversarial networks for mudii- distributions, 2019. BtRl:s/s:: http::://axvvvvvooa org.
Devi Parikh and C. Lawrence Zitnick. Exploring crowd co-creation scenarios for sketches, 2020. http:httts..//xii..rrrr/oo/
07328
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised repprssetioo learning with deep convolutional generative adversarial networks, 2015. URL/ttpes//arxiv..crr//bbb/
06434
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Im- proved techniques for training gans, 2016. URL
03498
Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets, 2022. URL htt.thtt////a.ttst. org/ abs httpg//bb/2222..00000 2tt./2222.00222
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions, 2014. URL
https://arxiyy wttp:oo//////00/ 4842
Ilya O Tolstikhin, Sylvain Gelly, Olivier Bousquet, Carl-Johann SIMON-GABRIEL, and Bernhard Schölkopf. Adagan: Boosting generative models. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wal- lach, R. Fergus. S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL http / neurss
10

---

Under review as a conference paper at ICLR 2023
Haizi Yu, James A. Evans, Donna Gallo, Adam Kruse, William M. Patterson, and Lav R. Varshney. Ai- aided co-creation for wellbeing. In Andrés Gómez de Silva Garza, Tony Veale, Wendy Aguilar, and Rafael Pérez y Pérez (eds.). Proceedings of the Twelfth International Conference Computational Creativity, México City, México ( (Virtual), September 14-18, 2021, pp. 453-456 Association for Computational Creativity (ACC), 2021. URL http:::ttp..............p. httpttpppp/////otttt.o....mo wwwputat ona www.ooooaaaaeaaaeoooeo creat iveetivvity /oottteet//pplaacs//222///0////0 mee//cccc1//e /ttp://///////////o///////////// 2ttps////222////////hpp.p4..
Thomas Zacharias, Ashutosh Taklikar, and Raja Giryes. Extending the vocabulary of fictional languages using neural networks. CoRR, abs/2201.07288, 2022. URLhttps: /ttp/xxxx.vooo////oo
abs /2011 07288
A APPENDIX CLASSIFIER TRAINING
We present specific information about the classifier in this section. The network architecture Fig. and convergence graphs Fig. c are presented below. Each corv.block contains a convolutional layer with a 5x5 filter and number of filters going from 64 to 512 in powers of 2. Each convolutional layer is followed by BatchNormalizatin, ReLU and Dropout with drop-prob = 0.3. The MLP block
contains 2 layers of 128 neurons each followed by 10 neuron output layer. As observed in 6b the peak accuracy of our classifier was ~ 99%.
[:None,
Eagas Luger
outpuk:
['Noac, 32, 32, 13
oone
CamBllock
ontguti
Input:
[None, .8 16, 54)
CouvBlunk
input:
(Nurw, 64)
outpct:
(Nouc,, 8, 8, 120)
Imput:
(Nom,, 8, 1200)
conyy comy_block_e
Cocvtioos
outyyst
(Nomo, 9, 5, 258
copv bockk
ConyBoock
impast:
(Nuun, 4, 4, 236
ftatten
Mratten
Inport
[Nonn, 2, (891)
onteput
NNou,, 2010)
drepout
Drapout
nutput:
DNone, 20LB)
Impur:
[Nonn, 2046)
octput:
[Neac, 10)
Figure 5: Classifier model
B APPENDIX GENERATOR AND DISCRIMINATOR MODELS
We implemented very simple Generator and Discriminator. The exact models are presented in Fig. 7a and Fig. 7b Each conv block is exactly the same as the one implemented for the Classifier, except the use of Layer normalization instead of Batch normalization, as recommended Arjovsky et al. (2017). Each upsample block consists of 2D Upsampling layer with factor of 2x2 followed by a convolutional layer with filter size 3x3 and stride 1x1. These operations are followed by LeakyReLU activation with 0.2. Batch normalization is applied after the activation. The final apsample block also uses dropout with drop prob 0.3. The final upsample block uses the tanh activation function to squash the values between 1 and 1.
In the model size experiments (Section 5.1.1 we extend the generator model by one extra dense layer of size 16, 384 the structure above. The final model can be seen in fig
We also replaced the above mentioned generator with a generator that uses residual blocks in the generation. The full architecture can be seen in fig

---

Under review as a conference paper at ICLR 2023
accuracy
0.20
0.98
0.96
0.10
0.94
0.00
20
40
80
100
20
60
80
100
spochs
apochs
(a) Training loss
(b) Training accuracy
Figure Classifier convergence graphs
Ingwi_2
InputLayer
Input:
[[Nome 1281]]
Inputl
Inpar::
[NNnn, 32, 32, 131
outpat:
output:
(Namw, 128)
block
CoavBlock
(Nonu, 32, 32,
matipat:
(Nonn, 4086)
oubput:
(None, 16, 16, 64)
(None, 4(96))
(None, 16, 16,
BescchNormalizatioo
ontggut (Nonc, 1096)
conv_block.
output.
(Nono, 8, 6, 120)
Iaaly
outprtt:
inpul:
(Nouo, 4096)
sonv.
oulput:
imput:
(None,
None, 4, 4, 256)
rechapo
Rechapo
Inpot:
conv block 3
ConvBlack
Input:
output:
(Nobo, 1, 4, 256)
output.
(Nono, 2, 2, 512)
uposmple block
U Block
Inpat:
(Nurw,, 4,
Batten
Flatien
imput:
(Nonw, 2,
outpui:
(Nobe, 8, 8, 128)
output:
(Nono, 2048)
upoample 1
Upsamplellliokk
Inpot:
(Neno, 18, 16,
(Nona, 8, 8, 128)
dropout 2
Dropout
input:
(None, 2048)
(Nono, 204B)
upoample block
UpoempleBBooc
Iupal:
(Nouc: 16, 16, 64)
Disc
Dense
impul:
(Nosee, 2048)
Generator model
Diseriminator model
Figure 7: Models
C APPENDIX OTHER EXPERIMENTS
Setting 7 and of the batch concatenation (b.c.) method as described i to four generators instead of two. The convergence graphs show that all 4 generators converge the same pace (the lines overlap) and the discriminator loss converges to 0 (Fig. TO). Our method improves the state-of-the-an with an FID 2. 2.9439 implying that high quality images are generated that match the dataset distribution. Table 3 and Fig. [ show the average class probabilities and correspondingly generated samples. This method of training results an average pairwise TVD = 0.9633.
the setup until convergence. This method is extended version
In Table 3 the values in bold represent which generator gives the maximum probability over that class. The number is generated by generator 1, 2 and and the "workload" required for this digit being distributed among the three of them. We can see that for O, the TVD metric goes down as expected, but we can still obtain a comparable FID to what we can achieve using the TVD loss component.
12

---

Under review as a conference paper at ICLR 2023
Inpor
Imgutlagee
Inper:
[iKens 1.81
[[Nenn,
demee
leputt
(Krws, 1:80)
16:30
donas
Elense
lngra:
Doase 38:8111
(Naano
butel cnneddcation
(Nooe, a0190
(Nieew 4000
upsample
UpwansteBllck
supput
(Noca,,
uposenpls_Hock3
UpeampleBlock
tiput
("oee, 32, 32,
Figure 8: Deep generator model
Table 3: Average class probabilities(?> 0, n
Generator number
0
4
5
6
7
8
9
generator
00.02
00.15
00.04
13.00
06.91
04.92
36.91
00.11
33.14
04.79
generator
38.99
00.00
11.32
22.84
00.01
24.13
00.03
00.00
02.33
00.33
generator
00.15
45.49
15.88
00.14
31.99
00.68
02.65
00.86
00.84
01.31
generator
00.36
00.29
22.05
02,46
00.72
00.95
00.00
41.96
01.94
29.24
13

---

Under review as a conference paper at ICLR 2023
Figure 9: Deep generator model
14

---

Under review as a conference paper at ICLR 2023
generators convergence
40
g0. jososs
g1. loss
g2 loss
20
g3 loss
d_ d_loss
10
-10
20
30
-40
20
40
60
80
100
epochs
Figure 10: Convergence graph for generators
3 8 8
O 2
241
7
2
9 2
6
6
3
0 3
4
9
7
3
3
5 503
2 242
7 727
5
6
5
124 4
9
(a) Samples generated by gen0
(b) Samples generated by gen1
(c) Samples generated by gen2
(d) Samples generated by gen'
Figure 11: Samples generated by the 4 generators
15


Reviews:
[{"review_id": "f4Sx-6frCI", "reviewer": "Reviewer_YBt6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the possibility of using multiple generators in a GAN setting. The idea is to let each generator specialize to a certain set of modes -- this \"assignment\" is currently assisted by a classifier, and encouraged via a novel total variation distance (TVD) loss. This idea may well have some merit, although the use of a classifier seems quite limiting. Unfortunately, the paper has pretty low information density and reads like a course work rather than an ICLR paper. The only dataset is MNIST, which is woefully inadequate in 2022. Realistically the multimodality problem should not exist in MNIST, as even in ImageNet StyleGAN-XL has SOTA results implying that modern GANs scale pretty far. So, while I agree that the problem still exists in the limit of highly diverse datasets, it surely should not exist in MNIST (using a StyleGAN family, for example). ", "summary_of_the_review": "Basically the scientific novelty, tested datasets and thus the breadth of conclusions, as well as the information density are all clearly below ICLR standards. A resubmission to a workshop might be viable option. ", "strength_and_weaknesses": "Pros:\n- TVD loss may be a novel idea, seems promising.\n- Clear exposition.\n\nCons:\n- The existence of the problem is not defended adequately.\n- MNIST is not enough. I don't believe it actually needs multiple generators.\n- SOTA GANs not used, conclusions may be incorrect.\n- Low information density for an ICLR paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_qn6VH-5SN6", "reviewer": "Reviewer_f1XE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents collaborative GAN, which aims to better balance the workload between individual generators. Specifically, a pre-trained classifier is introduced, and the training strategy is modified by adding a total variation distance based on WGAN-GP. Some experiments were conducted on MNIST dataset.\n ", "summary_of_the_review": "The novelty and experiments of this paper are in doubt.", "strength_and_weaknesses": "1) The novelty of the proposed method is very limited. It simply adds additional TVD loss on the existing WGAN-GP. And it is not clear why adding such TVD loss to the WGAN-GP can optimize the workload of the individual generators.\n\n2) The organization of this paper is poor. The paper spends too much space on introducing existing models and works, while the introduction of the proposed model is too concise.\n\n3) The experiments are only conducted on the simple MNIST datasets. The results are not convincing at all.  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "-Cjvkc6vov", "reviewer": "Reviewer_WzHB", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors target to learn a set of collaborative generators and a single discriminator in the generative adversarial network (GAN) framework. They are motivated by the belief that collaboration among multiple skill-specific generators can reach general and complex performance. They prove the concept on the MNIST dataset with the help of Total Variation Distance (TVD).", "summary_of_the_review": "See above.", "strength_and_weaknesses": "*************************\nStrengths\n*************************\n+ Clear writing, except for the equations and the algorithm.\n+ Easy to reproduce.\n\n*************************\nWeaknesses\n*************************\n\n- The name “CoGAN” has been used in [r1].\n\n- The motivation is meaningless.\n  - It is only applicable to data with finite discrete modes. And the complexity (number of generators) is proportional to the number of modes. This is one of the major reasons their experiments only involve the toy-like MNIST dataset, and impossible for Stacked MNIST [r2] with 1000 modes nor any other natural images like FFHQ human faces.\n  - The reviewer suggests authors stop thinking along this direction, which is deviated from the standard prototypes of research, has zero practical usage and zero new theoretical thoughts.\n\n- Technical details are vague and inaccurate.\n  - In Eq. 6, what are the definitions of P and Q? What is the distance metric used to measure |P(x) - Q(x)|? It has a summation over a set of x and, for each summation term, P and Q take the same x. This differs a lot from the formulation of delta, where x_i and x_j are different single samples not a set. Please correct and articulate your formulations.\n  - In Algorithm 1, k is not reflected in the summation terms when calculating L_g.\n\n- Experiments are far from convincing.\n  - The dataset includes only the toy-like MNIST dataset. Actually the method itself cannot be applicable to other realistic datasets.\n  - The WGAN-GP backbone is out-of-date. Experiment with multiple backbones including the recent SOTAs like StyleGAN-XL [r3]. Please also discuss the recent progress in GANs, to convince readers that the authors are knowledgeably ready to research in the GAN regime.\n\n[r1] Liu, Ming-Yu, and Oncel Tuzel. \"Coupled generative adversarial networks.\" NeurIPS 2016.\n[r2] Metz, Luke, et al. \"Unrolled generative adversarial networks.\" NeurIPS 2017.\n[r3] Sauer, Axel, Katja Schwarz, and Andreas Geiger. \"Stylegan-xl: Scaling stylegan to large diverse datasets.\" SIGGRAPH 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject