A Greedy Approach to Max-Sliced Wasserstein GANs

1 INTRODUCTION . Generative Adversarial Networks ( GANs ) were first introduced in Goodfellow et al . ( 2014 ) , where instead of the application of a mathematically well-established loss function an other differentiable neural network , a discriminator was applied to approximate the distance between two distributions . These methods are popularly applied in data generation and has significantly improved the modelling capabilities of neural networks . It was demonstrated in various use cases that these approaches can approximate complex high-dimensional distributions in practice Karras et al . ( 2017 ) , Yu et al . ( 2017 ) , Brock et al . ( 2018 ) . Apart from the theoretical advantage of GANs and applying a discriminator network instead of a distance metric ( e.g . ` 1 or ` 2 loss ) , modelling high-dimensional distributions with GANs often proves to be problematic in practice . The two most common problems are mode collapse , where the generator gets stuck in a state where only a small portion of the whole distribution is modeled and convergence problems , where either the generator or the discriminator solves his task almost perfectly , providing low or no gradients for training for the other network . Convergence problems were improved , by introducing the Wasserstein distance Gulrajani et al . ( 2017 ) Arjovsky et al . ( 2017 ) , which instead of a point-wise distance calculation ( e.g . cross-entropy or ` 1 distance ) calculates a minimal transportation distance ( earth mover ’ s distance ) between the two distributions . The approximation and calculation of the Wasserstein distance is complex and difficult in highdimensions , since in case of a large sample size calculation and minimization of the transport becomes exponentially complex , also distance can have various magnitudes in the different dimensions . In Deshpande et al . ( 2018 ) it was demonstrated that high-dimensional distributions can be approximated by using a high number of one dimensional projections . For a selected projection the minimal transport between the one dimensional samples can be calculated by sorting both the real and the fake samples and assigning them according to their sorted indices correspondingly . As an additional advantage , it was also demonstrated in Deshpande et al . ( 2018 ) that instead of the regular mini-max game of adversarial training , the distribution of the real samples could be approximated directly by the generator only , omitting the discriminator and turning training into a simple and more stable minimization problem . The theory of this novel method is well described and it was demonstrated that it works in practice , but unfortunately for complex , high-dimensional distributions a large amount of projections are needed . In Deshpande et al . ( 2019 ) it was demonstrated how the high number of random projections could be substituted by a single continuously optimized plane . The parameters of this projection are optimized in an adversarial manner selecting the ” worst ” projection , which maximizes separation between the real and fake samples using a surrogate function . This modification brought the regular adversarial training back and created a mini-max game again , where the generator creates samples which resemble well to the original distribution according to the selected plane and the discriminator tries to find a projection , which separates the real and fake samples from each other . The essence of Sliced Wasserstein distances is how they provide a method to calculate minimal transportation between the projected samples in one-dimension with ease , which approximates the Wasserstein distance in the original high-dimension . In theory this approach is sound Nadjahi et al . ( 2019 ) and works well in practise . It was proven in Kolouri et al . ( 2019 ) that the sliced Wasserstein distance satisfies the properties of non-negativity , identity of indiscernibles , symmetry , and triangle inequality , this way forming a true metric . However it approximates high-dimensional distributions well , we would like to demonstrate in this paper that the assignment of real and fake samples by sorting them in one dimension also has its flaws and a greedy assignment approach can perform better on commonly applied datasets . We would also argue regarding the application of the Wasserstein distance . We will demonstrate that in many cases various assignments can result the same minimal transportation during training and calculation of the Wasserstein distance with sorting can alter the distribution of perfectly modeled samples even when only a single sample differs from the approximated distribution . 2 WASSERSTEIN DISTANCE FOR COMPARING DISTRIBUTIONS . Generative adversarial networks can be described by a generator ( G ) , whose task is to generate a fake distribution ( PF ) , which resembles a distribution of real samples ( PR ) and a discriminator , whose task is to distinguish between PF and PR samples . Formally the following min-max objective is iteratively optimized : min G max D V ( PF , PR ) ( 1 ) Where V is a distance measure or divergence between the two distributions . In Arjovsky & Bottou ( 2017 ) the Wasserstein-p distance was proposed to improve the staibility of GANs , which can be defined as : WP ( PF , PR ) = inf γ∈ ∏ ( PF , PR ) ( E ( x , y ) vγ ‖x− y‖ p ) 1 p ( 2 ) where p is a parameter of the distance , and ∏ ( PF , PR ) defines all the possible joint distributions over PF and PR . The number of possible joint distributions increases factorially with the number of samples and the calculation of the minimal transport can be difficult . The instability and high complexity of wasserstein GANs were further improved by the introduction of the sliced Wassersstein distance in Deshpande et al . ( 2018 ) , which can be defined as : WP ( PF , PR ) = [ ∫ w∈Ω W pp ( PwF , PwR ) dw ] p ( 3 ) where PwF , PwR are one-dimensional projections of the high dimensional real and fake samples to w and Ω denotes a sufficiently high number of projections on the unit sphere . In this setting the Wasserstein distance can be calculated by sorting both the real and fake samples and assigning them by their indices , instead of checking all the possible joint distributions in ∏ ( PwF , PwR ) . Max-sliced Wasserstein GANs can be introduced by selecting the worst projected distribution from Ω noted by wmax and since the projection can be implemented by a fully connected layer , one can re-introduce the mini-max game used in regular GAN training : min G max D ( w∈Ω ) WP ( PF , PR ) ( 4 ) In this setup the distance in the single , one-dimensional projected space can be calculated by sorting the samples in the similar manner as in case of the sliced Wasserstein distance . 2.1 A CRITICISM OF THE WASSERSTEIN DISTANCE . In case of high sample number and complex high-dimensional distributions the optimal transport can often be calculated using various assignments between two sets of samples . For example if a sample containing n elements from PF can be defined by the series of Fw1 ... n after the projection , and a sample from PR is defined by Rw1 ... n and we know that for every i and j ( i , j ∈ 1 ... n ) : Fwi < R w j ( 5 ) Which means that the projected samples are well separated from each other ( all the projected fake samples are smaller than any of the projected real samples ) , for p = 1 all possible assignments of the i , j pairs will result the same distance , which is the minimal transportation of the samples , although the pairwise differences can be very different . One can also easily see , that the minimal transport calculated with sorting might not assign identical samples to each other , which is depicted on Fig . 1 . As it can be seen from the figure , for example in case of two Gaussian distributions with the same variance , but different mean values using sorting to calculate the minimal transport ( along arbitrary dimensions ) , will results the shift of one of the distributions ( in this case green ) . This transformation will also affect those samples which are at the intersection of the two distributions . This means that there might be identical pairs in ( PF and PR ) generating an error , which is not correct . One could assume that if a generator produces a sample identical to one of the samples of PR it should not generate an error and no gradients should be invoked by this sample pair . In this case the assignment will not pair these identical samples for comparison . Instead of sorting the samples we propose a method which assigns similar samples to each other . First we would like to remove the intersection of the two distributions and calculate the minimal transport for the remaining disjunct regions . By this calculation the value of the minimal transport will be the same ( in case of a sufficiently large sample number ) , but more similar samples will be assigned to each other . The previous example reveals a caveat of the Wasserstein distance , that it optimizes the global transport between two distributions , but ignores identical sample pairs or intersecting regions of density functions in them . Unfortunately identical samples will be extremely rarely found in two distributions , but closer sample pairs should also be assigned to each other . In training one typically uses mini-batches and will not see the whole distribution , certain parts or elements might not be present in each mini-batch if they probability in the original distribution is lower than one over the minibatch size . This can cause further problems in the calculation of the minimal transport using sorting , since the appearance of one wrongly generated sample at the first or last position of the projected samples can result a completely different assignment . This problem is depicted on Fig . 2 . We have to emphasize that although the assignment and distance calculation happen in one dimension , these distances are results of the projection of high dimensional embeddings , where there can exist dimensions which can decrease the distance between the two samples without significantly changing the projected position of all the other samples . 2.2 A GREEDY APPROACH FOR SAMPLE ASSIGNMENT . We introduce greedy training of max sliced Wasserstein GANs , where instead of sorting the samples of the projections we iteratively select the most similar pairs of them for loss calculation . Our training algorithm as a pseudocode is defined in Algorithm 1 . The training process is really similar to the approach introduced in Deshpande et al . ( 2019 ) . The main and only alteration is between line ten and sixteen , where instead of the sorting operation of the original approach we first generate a matrix which determines the distances for all possible sample pairs in one-dimension . First we select the smallest element in this matrix and remove its row and column and iterate this process until the matrix contains only one element which will be the distance between the last two , least similar sample pairs . We have to note that our approach requiresO ( n3 ) steps compared to the originalO ( nlog ( n ) ) operations of sorting . We have to implement nminimum selections in the distance matrix with size n×n . We also have to note that this increase in complexity refers for training only and has no effect ton the inference complexity of the generator , also n is the sample or mini-batch size during training , which is usually a relatively low number . In our experiences , using batches of 16 to 512 samples , this step did not result a significant increase in computation time during training .

Summary: This paper proposes two alternative approaches to max-sliced Wasserstein GANs. They are based on the authors’ claim that there is a “flaw” in the Wasserstein-1 distance between probability distributions on the one-dimensional space. Briefly, the authors’ argument says that the “flaw” is that the optimal transport may not be unique, some of which are better for network learning than others. One proposal, described in Section 2.2, is to find a plausible transport plan in a greedy manner. The other proposal, described in Section2.3, is a hybrid of the greedy approach in Section 2.2 and the original sliced Wasserstein distance.