Title: INCREMENTAL TRAINING OF MULTI-GENERATIVE AD-VERSARIAL NETWORKS

Abstract: Generative neural networks map a standard, probability distribution to a complex high-dimensional distribution, which represents the real world data set. However, a determinate input distribution as well as a specific architecture of neural networks may impose limitations on capturing the diversity in the high dimensional target space. To resolve this difficulty, we propose a training framework that greedily produce a series of generative adversarial networks that incrementally capture the diversity of the target space. We show theoretically and empirically that our training algorithm converges to the theoretically optimal distribution, the projection of the real distribution onto the convex hull of the network's distribution space.

1. INTRODUCTION:

Generative Adversarial Nets (GAN) Goodfellow et al. (2014) is a framework of estimating generative models. The main idea Goodfellow (2017) is to train two target network models simultaneously, in which one, called the generator, aims to generate samples that resemble those from the data distribution, while the other, called the discriminator, aims to distinguish the samples by the generator from the real data. Naturally, this type of training framework admits a nice interpretation as a twoperson zero-sum game and interesting game theoretical properties, such as uniqueness of the optimal solution, have been derived Goodfellow et al. (2014) . It is further proved that such adversarial process minimizes certain divergences, such as Shannon divergence, between the generated distribution and the data distribution.

Simply put, the goal of training a GAN is to search for a distribution in the range of the generator that best approximates the data distribution. The range is often defined by the input latent variable z and its specific architecture, i.e., Π = {G(z, θ), θ ∈ Θ}. When the range is general enough, one could possibly find the real data distribution. However, in practice, the range is usually insufficient to perfectly describe the real data, which is typically of high dimension. As a result, what we search for is in fact the I-projection Csiszár & Shields (2004) of the real data distribution on Π.

Consider two cases::

1. The range of the generator Π is convex (see figure 1(a) ), or it is not convex but the projection of the real data distribution on Π's convex hull (CONV Π) is in Π (see figure 1(b) ). 2. The range of the generator is non-convex and the projection of the real data distribution in CONV Π is not in the range Π (see figure 1(c) ). In case 1, one can find the optimal distribution in Π to approximate real data set in CONV Π. But in case 2, using standard GANs with a single generator, one can only find the distribution in Π that is nearest to the projection. It then makes sense to train multiple generators and use a convex combination of them to better approximate the data distribution (than using a single generator in the non-convex case (see figure 1 

(c))).

The above argument is based on the assumption that one could achieve global optimality by training, while this is not the case in general. When reaching a local optimal distribution, in order to improve performance, do we need to add more generators and restart training? In this paper, we put forward a sequential training procedure that adds generators one by one to improve the performance, without retraining the previously added generators. Our contributions can be summarized as follows.

• We derive an objective function tailored for such a incremental training process. The objective function takes both the real data distribution and the pre-learned distribution into consideration. We show that with this new objective, we actually maximize marginal contribution when adding a new generator. We also put forward an incremental training algorithm based on the new objective function. • We prove that our algorithm always converges to the projection of real data distribution to the convex hull of the ranges of generators, which is the optimal solution with multiple generators. This property continues to hold in online settings where target distribution changes dynamically.

• Our experiments show that our algorithm can overcome the local optimal issue mentioned above. We perform experiments on a synthetic dataset as well as two real world datasets, e.g., CelebA and MNIST, and conclude that our algorithm could improve the mixture distribution even in the case where the range is not sufficient enough. • Experiments also show that, compared with previous methods, our algorithm is fast and stable in reducing the divergence between mixture distribution and the real data.

1.1. RELATED WORKS:

Recently, there have been intensive researches on improving the performance of generative adversarial neural networks. Two lines of works are closely related to our paper. They focus mainly on improving the discriminator and the generator respectively. Arora et al. (2017) introduces a new metric on distributions and proposes a MIX+GAN to search for an equilibrium. But all these methods need to train the multiple generators simultaneously, and none of them can deal with the case when the training process reaches a local optima. Also, these models lack flexibility, in the sense that when one tries to change the number of generators, all the generators need to be retrained.

Another closely related work is Tolstikhin et al. (2017) , in which the authors propose a method called AdaGAN, which is based on a robust reweighting scheme on the data set inspired from boosting. The idea is that the new generators should focus more on the previous bad training data. But AdaGAN and other boosting-like algorithms are based on the assumption that one generator could catch some modes precisely, which may not be reasonable since the generator always learns to generate the average samples among the real data set in order to obtain low divergence, especially when the generator's range is under condition of Figure 1 (c). In Section 5, we compare our algorithm with AdaGAN with different dataset.

2. PRELIMINARIES:

A GAN Goodfellow et al. (2014) takes samples (a.k.a. latent variables z) from a simple and standard distribution as its input and generates samples in a high dimensional space to approximate the target distribution. This is done by training a generative neural network and an auxiliary discriminative neural network alternatively. An f- GAN Nowozin et al. (2016) generalizes the adversarial training as minimizing the f-divergence between the real data distribution and the generated distribution, defined as

D f (p q) = x q(x)f p(x) q(x)

dx. A GAN is a special f-GAN that minimizes the Jensen-Shannon divergence. The general objective function of an f-GAN can be defined as follows:

min θ max ξ F (θ, ξ) = E x∼p data [T ξ (x)] + E x∼p θ [-f * (T ξ (x))].

Here f * is the conjugate function of f in f-divergence; T represents a neural network regarded as the corresponding discriminator; finally, θ and ξ denote the parameters of the generator and the discriminator, respectively.

2.1. GENERATOR GROUP:

The adversarial training method proposed by Goodfellow et al. (2014) is playing a minimax game between the generator and the discriminator. Such a method can be caught in local optima and thus is undesirable (e.g., mode collapse).

In this paper we propose a novel framework to train multiple generators sequentially: We maintain a group of generators (empty at the beginning) as well as their corresponding weights, then add new generators into the group one by one and rebalance the weights. In particular, only the newly added generator at each step is trained. The purpose here is to augment the capacity of the group of generators and mitigate the local optima issue.

Define the distribution range of a generator as Π = {p | p = G(z, θ), θ ∈ Θ}, i.e., the set of distributions that the generator can produce with different parameter θ. The distribution range is determined by the distribution of input z and the architecture of the generative network. Define a generator group as G = {G 1 , G 2 , , . . . , G n } , where G i is the generator added in step i. We associate each generator with a weight ω i > 0. Then the mixed distribution of the group is:

p pre (x) = 1 Φn n i=1 ω i p i (x), where p i (x) = G i (z) and Φ n = n i=1 ω i is the sum of weights.

When a new generator G n+1 joins in the group G, the group becomes G = G ∪ {G n+1 } and the mixed distribution becomes

p now (x) = 1 Φ n+1 n+1 i=1 ω i p i (x) = 1 Φ n+1 (Φ n • p pre (x) + ω n+1 • p n+1 (x)) .

3. AN INCREMENTAL TRAINING FRAMEWORK:

In this section, we describe how we use a generator group to improve the performance and tackle the local optima issue mentioned previously. To train such a generator group, we propose an incremental training algorithm (algorithm 1) adding generators to the group sequentially. In algorithm 1, we use

Algorithm 1 Incremental Training Algorithm Predetermined: p real , p z , G i (z), i ∈ {1, 2, . . . , N } ω i , i ∈ {1, 2, . . . , N + 1} Input p real , p z , ω i , i > 0.

Output generator group G.

Initialize i ← 1. repeat Build and initialize generator G i using the same network structure. Set target distribution for G i to be

p target = Φi•p real -j<i ωj pj ωi . Train G i to minimize D(p target , p i ). i ← i + 1. until Convergence D(•,

•) to denote the "distance" between two distributions, which can be any divergence (e.g., fdivergence or Wasserstein distance) or a general norm.

The key step in algorithm 1 is the choice of the target distribution for training G i . Ideally, if D(p target , p i ) = 0, then we have p i = p target . In this case, p real = 1 Φi i j=1 ω j p j and after adding G i , the generator group G can perfectly produce the desired distribution p real . However, in general, we have D(p target , p i ) = 0 and our algorithm proceeds in a greedy fashion, i.e., it always maximizes the marginal contribution of G i to the generator group G. We devote the rest of this section to proving the above statement.

3.1. MARGINAL CONTRIBUTION MAXIMIZATION AT EACH ROUND:

In algorithm 1, we use different loss functions for each generators. The marginal contribution of the (N + 1)-th generator is as follows when we adopt f-divergence as the distance measure:

EQUATION

To get a better approximation to the real distribution, we fix the existing generators in the group and tune the parameters of the new generator to minimize the distance between the new group and the real distribution. In fact, this is equivalent to maximizing the marginal contribution of the new generator

V (G N +1 ) by selecting G N +1 = arg max G N +1 V (G N +1 ). Formally,

Proposition 1. For any distribution p real and any existing generator group G = (G 1 , G 2 , . . . , G N ), the optimal target distribution p * N +1 for a new generator G N +1 to join the group G is

p * N +1 (x) = Φ N +1 • p real (x) -Φ N • p pre (x) ω N +1 + , where[•] + = max{•, 0}

.

To show this, we first introduce the χ 2 -divergence. Definition 1 (χ 2 -divergence). The χ 2 -divergence between distribution p and q is:

D χ 2 (p||q) = x (p(x)-q(x)) 2 q(x) dx. Note that χ 2 -divergence is a special case of the f-divergence: D χ 2 (p||q) = D f (p||q), when f (u) = (u -1) 2

In fact, with some mild assumptions on f , the f -divergence is well-approximated by χ 2 -divergence when p and q are close. The following lemma can be obtained via Taylor expansion Csiszár & Shields (2004) . Lemma 1. For any f-divergence with f (u), if f (u) is twice differentiable at u = 1 and f (1) > 0, then for any q and p close to q we have:

D f (p||q) ∼ f (1) 2 D χ 2 (p||q).

Proof of proposition 1. We rewrite the objective function equation 1 for χ 2 -divergence:

V (G N +1 ) = x p real f p pre p real -f p now p real dx = x (p real -p pre ) 2 -(p real -p now ) 2 p real dx. Algorithm 2 Training G N +1 by Gradient De- scent Input: p real , p z , η, ω N +1 and G i (•, θ i ), ω i , i ∈ {1, 2, . . . , N } repeat Sample {x j real } from p real Sample {z j } from p z for i = 1 to N do Obtain samples by G i (z j , θ i ) as {x j Gi } in Group G end for Generate samples {x j gen } by G N +1 Update ξ t+1 = ξ t + η∇ ξ F (θ t N +1 , ξ t ) Update θ t+1 N +1 = θ t N +1 -η∇ θ F (θ t N +1 , ξ t ) until Convergence Output: G N +1 (•, θ N +1 ) Figure 2: A framework for training G N +1

Based on the former definition, we obtain

V (G N +1 ) = x ω N +1 •[-ω N +1 •p 2 N +1 +A•p N +1 +B] (Φ N +1 ) 2 p real dx, where A = 2(Φ N +1 • p real -Φ N • p pre ) and B = (Φ N +1 + Φ N )p 2 pre -2Φ N +1 • p pre • p real . To maximize the quadratic function V (G N +1 ), we have p * N +1 = [A/2ω N +1

] + , which concludes the proof.

3.2. ALGORITHMS TO TRAIN G i:

According to algorithm 1, in each round, a new generator G N +1 is added and the loss function is set to be D(p target , p N +1 ). Therefore, when training each generator G i , the target distribution only depends on the real distribution and the previous generators in G. In particular, both of them are already known (figure 2 ).

To minimize D(p target , p G N +1 ), we conduct adversarial training by using an auxiliary discriminator T:

F (θ, ξ) = E x∼ptarget [T ξ (x)] + E x∼p θ [-f * (T ξ (x))],

where by the linearity of expectation:

E x∼ptarget [T ξ (x)] = {Φ N +1 E x∼p real [T ξ (x)] -Φ N E x∼ppre [T ξ (x)]} ω N +1 = {Φ N +1 E x∼p real [T ξ (x)] - N i=1 ω i • E x∼pi [T ξ (x)]} ω N +1 .

Based on these, we propose an incremental training algorithm for G N +1 as algorithm 2.

4. THEORETICAL ANALYSES:

In this section, we show that although our framework which trains each generator in a greedy way, the output distribution of the generator group will always converge. Furthermore, the converged distribution is the closest one to the target distribution among the set of all possible distributions that a group of generators can produce (i.e., the optimal one within the distribution range of the group of generators).

Recall our notation that the distribution range of a generator is Π. By taking a convex combination of multiple generators (with the same architecture), the set of all possible output distributions becomes the convex hull of Π:

CONV Π = {p(x) | p(x) = 1 Φ N N i=1 ω i p i (x)},

where ω i > 0 is the weight of the i-th generator, p i (x) = G i (z), 1 ≤ i ≤ N , and G i represent the generative neural networks with the same architecture but different parameters. One can consider the parameters ωi Φ N as the probability of choosing the i-th generator as the final output. 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 )/2 and we can easily extend it to χ 2 -divergence by D KL (p||q) ≤ D χ 2 (p||q) + 0.42. In other words, while greedily optimizing each G N +1 , the distance between p target and p G N +1 is also approximately minimized. Hence it is reasonable to assume that for each G N +1 , its distance to p target is approximately minimized with some tolerance ≥ 0, i.e., p G N +1 -p target ≤ inf p -p target + . Under such an assumption, our algorithm approximately converges to the the optimal distribution in CONV Π: Proposition 2. For any Π that is connected and bounded, algorithm 2 approximately converges to the optimal distribution within the closure of the convex hull CONV Π of Π.

To simplify the argument, we fix each ω i to be 1 and embed the discrete probability distributions into a Hilbert space. In this case, each G N +1 approximately minimizes the distance to p target = (N + 1)p real -N i=1 p Gi can be formalized as:

p G N +1 -p target ≤ inf p∈Π p -p target + ,

and our algorithm approximately converges to the optimal distribution in CONV Π if as N → ∞,

1 N N i=1 p G N -p real ≤ inf p∈CONV Π p -p real + .

Then proposition 2 is implied by the following lemma. Lemma 2. Consider a connected and bounded subset Π of a Hilbert space H and any target ρ ∈ H. Let {p * n } ∞ n=1 be a sequence of points in Π such that for

ρ target = (n + 1)ρ -nT n , p * n+1 -ρ target ≤ inf p∈Π p -ρ target + , ( )

where

T n = 1 n n i=1 p * n and ≥ 0 is a constant. Then for any δ > 0, there exists N > 0 such that ∀n > N, T n -ρ ≤ inf p∈CONV Π p -ρ + (1 + δ) .

Corollary 1. With the finite change of target distribution, algorithm 2 can converge to the new optimal distribution within CONV Π.

Due to the space limit, we send the proof to the appendix.

Based on corollary 1, regardless the change of target distribution, as long as it is an finite variation, algorithm 2 can converge to the projection of new target distribution. Due to the sequential nature and the above theoretical guarantee, our algorithm naturally generalizes the dynamic online settings.

5. EXPERIMENTS:

We test our algorithm on a synthesized Gaussian distribution dataset and two well-known real world datasets: CelebA and MNIST, which are the complex high dimensional real world distributions. We design the experiment to test our sequential training algorithm. The main purpose is not to demonstrate high quality results, e.g., high definition pictures, but to show that our algorithm can search for an appropriate distribution that significantly improved the performance of mixture distributions as the number of generators increase, especially when the generator's range is rather limited.

In all experiments, we use the Adam optimizer Kingma & Ba (2014) with learning rate of 5 × 10 -5 , and β 1 = 0.5, β 2 = 0.9. Finally, we set weights ω i = 1 for convenience.

Metric. As the method mentioned in Nowozin et al. (2016) , when we fix the generator, we can train an auxiliary neural network to maximize the derived lower bound to measure the divergence between the generated data and the real data, i.e., D f (P ||Q). Based on these theories, we import an auxiliary neural network to measure the performance of different methods. The architecture of the auxiliary neural network is the same as the discriminator used in each experiment. We train it for 50 epoches, which is enough to measure the differences. Then we take the mean value of the last 100 iterations as the final output.

Synthesized data. In this part, we design some experiments in R 2 space. The dataset is sampled from 8 independent two-dimensional Gaussian distributions (i.e., the blue points in figure 3 ). The model is previously proposed by Metz et al. (2016) .

Firstly, following the experiment designed in Metz et al. (2016) and Hoang et al. (2017) , we choose the latent variable z in a high dimensional space as z ∼ N (0, I 256 ), i.e., the distribution projection is likely to be in the generator's range, which meets the condition of figure 1(a) or figure 1(b). In figure 3 , the blue points are the real data while the corresponding colored number represents the data points generated by each generator respectively. As figure 3 shows, we train up to 4 generators to approximate the data distribution and the first generator tends to catch the data with high probability around the centre of each Gaussian. As the number of generators increasing, generated data tends to cover the data away from the centre in order to be complementary to previous mixture distributions and thus gains a considerable marginal profit. These results demonstrate our marginal maximization algorithm can promote the mixture distributions to cover the data with low probabilities.

Secondly, we reduce the dimension of z to 1, i.e., z ∼ N (0, 1) and simplify the corresponding network architecture, so that the condition of figure 1(c ) is likely met. In this part, we compare our algorithm with the state of the art incremental training method AdaGAN Tolstikhin et al. ( 2017) and the baseline method Orignal GAN. 1 We train up to 20 generators in each experiment with the same starting generator (i.e., identical first generator for each method), then measure the D χ 2 (p||q) between real distribution and the generated mixed distribution. We repeat the experiment for 30 times to reduce the effect of random noises. Figure 5 and figure 6 illustrate the average and the best performance with different numbers of generators, respectively. According to the result, our algorithm approaches to p real faster than the other two methods and achieves the best performance among all three methods.

In summary, our algorithm outperforms the other two both in terms of the speed of converging to the real distribution and the quality of the final result under the case of figure 1(c).

MNIST. In this experiment, we run our algorithm on the MNIST dataset LeCun et al. (1998) . We design this experiment to measure the performance of our algorithm for a more complex data distribution. We choose the latent variable as z ∼ N (0, 1) to limit the corresponding generator range and Then we train up to 22 generators to approximate the real distribution and the result is showed in figure 4 . Our algorithm outperforms the Original GAN but is inferior to the AdaGAN with the first 8 generators. As the number of generators increases, AdaGAN seems to run into a bottleneck while both our algorithm and the Original GAN gradually approximate to the real data distribution.

In order to analysis the convergence, we further train up to 100 generators with both our algorithm and the original GAN. In figure 7 , the horizontal dash lines represent the minimum value of the Wasserstein distance for the two method respectively. As showed in figure, the distance gradually decrease with the number of generators increasing, and our algorithm is much faster to reduce the distance and can even obtain a better performance. More over, as we tends to investigate the property of each generator in the generators' group, we measure the Wasserstein distance between distribution G(z, θ) and the real data, the experiment result is showed in figure 8 and the dash lines in figure 8 represent the mean value of the 100 generators. Interestingly, the result shows that, in each generator, original GAN tends to search a distribution in the distribution range that is closer to the real data distribution, while our algorithm is searching for a distribution that is complementary to the generators' group (i.e., a huge decrease in the mixture condition in figure 7 ) even if its own performance is poor (i.e., a high distance in the in figure 8 ).

CelebA. We also conduct our experiment on the CelebA dataset Liu et al. (2015) . As shown in figure 1 , we start with an identical generator and train up to 6 generators using different methods. The measured Wasserstein distance between mixed distribution of Group G and the real-data distribution is showed in figure 1 . In this experiment, we use the training method WGAN-GP proposed by Gulrajani et al. (2017) . The experiment results indicates that our algorithm outperforms the other two methods after the second generator. It demonstrates the potential of our algorithm applying to real world datasets. 

:

The only difference is the target distribution, for Original GAN, we train each generator with the real data distribution, i.e., p real , then make a convex combination of them.

FUTURE WORKWe intend to further investigate the problem of how to accelerate the convergence via optimal generator weights, i.e., ω i . Moreover, by measuring the relativity between the current group G and the dynamically changing target distribution, we intend to apply our algorithm to online learning.

This is a similar method to the metric defined previously, the only difference is that we use WGAN-GP method instead of f-GAN.

Figures and Tables:

Figure 1: Different cases of the distribution projection on Π's convex hull

Figure 3: We train 4 generators to catch the modes (i.e., 8 Gaussian distributions).

Figure 4: Wasserstein distance between p real and p now as the number of generators increases.

Figure 5: The mean log(D χ 2 (p real ||p now )) of 30 repetition experiments .

Figure 7: W-distance between mixture distribution and the real data.

Figure 8: W-distance between distribution G i (z, θ) and the real data .use the Wasserstein distance to measure the gap between mixed distribution and the real dataset. 2 Then we train up to 22 generators to approximate the real distribution and the result is showed in figure4. Our algorithm outperforms the Original GAN but is inferior to the AdaGAN with the first 8 generators. As the number of generators increases, AdaGAN seems to run into a bottleneck while both our algorithm and the Original GAN gradually approximate to the real data distribution.

Wasserstein distance (WD) between Group G and the CelebA dataset with different numbers of generators (i.e., |G|). Yaxing Wang, Lichao Zhang, and Joost van de Weijer. Ensembles of generative adversarial networks. CoRR, abs/1612.00991, 2016. David Warde-Farley and Yoshua Bengio. Improving generative adversarial networks with denoising feature matching. 2016.

Table content: <table><tr><td>Method</td><td>|G|=1</td><td>|G|=2</td><td>|G|=3</td><td>|G|=4</td><td>|G|=5</td><td>|G|=6</td></tr><tr><td>Original GAN</td><td/><td colspan="5">588.482 337.859 307.117 189.149 146.062</td></tr><tr><td>AdaGAN</td><td>789.655</td><td colspan="5">607.619 569.948 322.936 206.326 248.529</td></tr><tr><td>Incremental GAN</td><td/><td colspan="5">704.004 277.993 154.544 206.693 52.9456</td></tr></table>

Peer review (iclr2019):
 In this work, the authors propose to use multiple generators to estimate the target distribution. Especially, it assumes the case that the range of generators is non-convex and the target distribution doesn*t fall into it. To solve this issue, the multiple generators are convex combined to do better approximation and an incremental training process is proposed to train multiple generators one by one. 1) Using multiple generators seems reasonable based on the authors assumption (non-convex of the range of generators), but is this assumption based on having a perfect discriminator? Could you assume a similar case for the discriminator? 2) In figure 3, it is shown each generator tries to improve the estimated target distribution. However, it is not clear what generator generates what samples. It would be better to use different colors for different generators. If I assume that the red samples are from the first generator, why the second image (top right) shows slightly shifted samples compared to the first image (top left). As far as I understand, the first generator is fixed after it is converged. 3) It is shown that the (convex) weights for generators are fixed to 1, is there any reason to fixed it? 4) On page 3, the equation in section 2.1 looks like missing , could you confirm this? 5) is the Original GAN exactly the Ian*s original GAN or WGAN? 6) Have you tried this approach using small sized generator (having small number of parameters)?