A Greedy Approach to Max-Sliced Wasserstein GANs
Authors: András Horváth
Venue:  (2020)
Keywords: GEnerative Adversarial Networks
GANs
Wasserstein distances
Sliced Wasserstein Distance
Max-sliced Wasserstein distance

Abstract: Generative Adversarial Networks have made data generation possible in various use cases, but in case of complex, high-dimensional distributions it can be difficult to train them, because of convergence problems and the appearance of mode collapse.
Sliced Wasserstein GANs and especially the application of the Max-Sliced Wasserstein distance made it possible to approximate Wasserstein distance during training in an efficient and stable way and helped ease convergence problems of these architectures.

This method transforms sample assignment and distance calculation into sorting the one-dimensional projection of the samples, which results a sufficient approximation of the high-dimensional Wasserstein distance. 

In this paper we will demonstrate that the approximation of the Wasserstein distance by sorting the samples is not always the optimal approach and the greedy assignment of the real and fake samples can result faster convergence and better approximation of the original distribution.

Under review as a conference paper at ICLR 2020
A GREEDY APPROACH TO MAX-SLICED WASSER- STEIN GANS
Anonymous authors Paper under double-blind review
ABSTRACT
Generative Adversarial Networks have made data generation possible in various use cases, but in case of complex, high-dimensional distributions it can be difficult to train them, because of convergence problems and the appearance mode col- lapse. Sliced Wasserstein GANs and especially the application of the Max-Sliced Wasserstein distance made it possible to approximate Wasserstein distance during training in an efficient and stable way and helped case convergence problems of these architectures.
This method transforms sample assignment and distance calculation into sorting the one-dimensional projection of the samples, which results sufficient approxi- mation of the high-dinge -ige--iinnsional Wasserstein distance.
In this paper we will demonstrate that the approximation of the Wasserstein dis- tance by sorting the samples not always the optimal approach and the greedy assignment of the real and fake samples can result faster convergence and better approximation of the original distribution.
INTRODUCTION
Generative Adversarial Networks (GANs) were first introduced in Goodfellow et al. (2014), where instead of the application of a mathematically well-established loss function an other differentiable neural network, a discriminator was applied to approximate the distance between two distributions. These methods are popularly applied in data generation and has significantly improved the modelling capabilities of neural networks. was demonstrated various use cases that these approaches can approximate complex high-dimensional distributions in practice Karras et al. (0117). Yu et al. 2017 Brack et al. (2018).
Apart from the theoretical advantage of GANs and applying discriminator network instead of distance metric (e.g. El or f2 loss), modelling high-dimensional distributions with GANs often proves be problematic in practice. The two the generator gets stuck in state where only a small portion of the whole distribution is modeled and convergence problems, where either the generator or the discriminator solves his task almost perfectly, providing low or no gradients for training for the other network.
common problems are mode collapse, where
Convergence problems were improved, by introducing the Wasserstein distance Guirajani et al. Arjovsky et al. (2017). which instead of a point-wise distance calculation (e.g. cross-entropy or distance) calculates minimal transportation distance (earth mover's distance) between the two distributions.
The approximation and calculation of the Wasserstein distance is complex and difficult in high- dimensions, since in case of a large sample size calculation and minimization of the transport be- comes exponentially complex, also distance can have various magnitudes in the different dimen- sions.
In Deshpande et al. (2018) it was demonstrated that high-dimensional distributions can be approxi- mated by using high number of one dimensional projections. For selected projection the minimal transport between the one dimensional samples can calculated by sorting both the real and the fake samples and assigning them according to their sorted indices correspondingly. As additional advantage, it was also demonstrated in Deshpande et al. (2018) that instead of the regular mini-max game of adversarial training, the distribution of the real samples could be approximated directly by

---

Under review as a conference paper at ICLR 2020
the generator only, omitting the discriminator and turning training into a simple and more stable min- imization problem. The theory of this novel method is well described and it was demonstrated that it works in practice, but unfortunately for complex, high-dimensional distributions large amount of projections are needed.
In Deshpande al. (2019) it demonstrated how the high number of random projections could be substituted by a single continuously optimized plane. The parameters of this projection are optimized in adversarial manner selecting the "worst" projection, which maximizes separation between the real and fake samples using surrogate function. This modification brought the regular adversarial training back and created a mini-max game again, where the generator creates samples which resemble well to the original distribution according to the selected plane and the discriminator tries to find projection, which separates the real and fake samples from each other.
The essence of Sliced Wasserstein distances is how they provide method to calculate minimal transportation between the projected samples in one-dimension with ease, which approximates the Wasserstein distance in the original high-dimension. In theory this approach is sound[Nadjall et al.) (2019) and works well practise. was proven in Kolouri et al. (2019) that the sliced Wasserstein distance satisfies the properties of non-negativity, identity of indiscernibles, symmetry, and triangle inequality, this way forming a true metric. However it approximates high- nigh-dimensional distributions well, we would like to demonstrate in this paper that the assignment of real and fake samples by sorting them in one dimension also has its flaws and a greedy assignment approach can perform better commonly applied datasets. We would also argue regarding the application of the Wasser- stein distance. We will demonstrate that in many cases various assignments can result the same minimal transportation during training and calculation of the Wasserstein distance with sorting can alter the distribution of perfectly modeled samples even when only a single sample differs from the approximated distribution.
2 WASSERSTEI DISTANCE FOR COMPARING DISTRIBUTIONS
Generative adversarial networks can be described by generator (G), whose task is to generate a fake distribution (PP)), which resembles a distribution of real samples (PR)) and a discriminator, whose task is to distinguish between Pr and Pn samples. Formally the following min-max objective iteratively optimized:
min max ((FFPPt PR) D Where V a distance measure or divergence between the two distributions. In Arjossyy Bottou 2017) the Wasserstein-p distance was proposed to improve the staibility of GANs, which can be defined as:
(PP, = inf (E (a. 1| y
TEIT(Pr Pn) where p is a parameter of the distance, and [IPP PR) defines all the possible joint distributions over PF and PR- The number of possible joint distributions increases factorially with the number of samples and the calculation of the minimal transport can be difficult.
The instability and high complexity of wasserstein GANs were further improved by the introduction of the sliced Wassersstein distance in Deshpande al. 2018). which can be defined as:
(P(.r.PR) = // WPPPF, (UPPF Fd) 1"
(3)
WEn
where Py, Py are one-dimensional projections of the high dimensional real and fake samples to w and 5 denotes a sufficiently high number of projections on the unit sphere. In this setting the Wasserstein distance can be calculated by sorting both the real and fake samples and assigning them by their indices, instead of checking all the possible joint distributions in (PF,P))
Max sliced Wasserstein GANs can be introduced by selecting the worst projected distribution from noted by Wmaz and since the projection can be implemented by a fully connected layer, one can re- re-introduce the mini-mat game used in regular GAN training:
min max (TP IPr))
(4)
G D(wE()
2

---

Under review as a conference paper at ICLR 2020
In this setup the distance in the single, one-dimensional projected space can be calculated by sorting the samples the similar manner as in case of the sliced Wasserstein distance.
2.1 A CRITICISM OF THE WASSERSTEIN DISTANCE
In case of high sample number and complex high-dimensional distributions the optimal transport can often be calculated using various assignments between two sets of samples. For example if sample containing elements from PF can be defined by the series of FT... after the projection, and sample from PR is defined by RT.... and we know that for every and j (i,j 1...n):
F RT
(5)
Which means that the projected samples are well separated from each other (all the projected fake samples are smaller than any of the projected real samples), for p = 1 all possible assignments the i, pairs will result the same distance, which is the minimal transportation of the samples, although the pairwise differences can be very different.
One can also easily see, that the minimal transport calculated with sorting might not assign identical samples to each other, which depicted on Fig. [ As can be seen from the figure, for example in case of two Gaussian distributions with the same variance, but different mean values using sorting to calculate the minimal transport (along arbitrary dimensions), will results the shift of one of the distributions (in this case green). This transformation will also affect those samples which intersection of the two distributions. This means that there might be identical pairs (Pr and PR) generating
the
which is not correct. One could assume that if generator produces sample identical to one of the samples of PR it should not generate an eπrо and no gradients should be invoked by this sample pair. In this case the assignment will not pair these identical samples for comparison.
Figure I: This figure depicts flaw of the Wasserstein distance. In these figures a blue (desired out- put distribution and green, generated fake distribution can be seen), the assignmentss/esiiee trans- ports are illustrated by red arrows. Wasserstein distance calculates minimal transportation between all the samples. many cases this results shifting all the samples to cover the whole distribution correctly, this is depicted are at the intersection of the two distributions, meaning that for these generated samples an identical pair could be found in the real samples, will also be altered and shifted towards other part of the real distribution. Instead method. This will ensure that that identical (or similar) sample pairs will be selected first, and after this the transportation between the disjunct regions will be calculated, which is depicted on the right subfigure.
the left subfigure. Unfortunately this means that those samples which
this we propose the calculation of the assignment using a greedy
Instead of sorting the samples we propose method which assigns similar samples to each other. First we would like to remove the intersection of the two distributions and calculate the minimal transport for the remaining disjunct regions. By this calculation the value of the minimal transport will be the same (in case of sufficiently large sample number), but more similar samples will be assigned each other.
The previous example reveals a caveat the Wasserstein distance, that it optimizes the global trans- port between distributions, but ignores identical sample pairs or intersecting regions of density

---

Under review as a conference paper at ICLR 2020
functions in them. Unfortunately identical samples will be extremely rarely found in two distribu- tions, but closer sample pairs should also be assigned to each other. In training one typically uses mini-batches and will not the whole distribution, certain parts elements might not be present in each mini-batch if they probability in the original distribution is lower than one over the mini- batch size. This can cause further problems in the calculation of the minimal transport using sorting, since the appearance of one wrongly generated sample at the first or last position of the projected samples can result a completely different assignment. This problem is depicted on Fig. 2 We have to emphasize that although the assignment and distance calculation happen in one dimension, these distances are results of the projection of high dimensional embeddings, where there can exist di- mensions which can decrease the distance between the two samples without significantly changing the projected position of all the other samples.
Figure 2: In this figure we would like to demonstrate different approaches for minimal transportation in mee-iimension using mini-batcher Real samples are noted by blue and fake samples are plotted by red stars. The position of the samples were shifted vertically for better display, but only their horizontal position matters. On both figures the same constellation can be seen: four similar sample pairs and one which is very different. The left subfigure depicts sample assignment by sorting, meanwhile the right figure depicts assignment by a greedy approach. The summed distances are the same in both cases for = 1. We would like to argue that the lower assignment can be better for network training.
2.2 A GREEDY APPROACH FOR SAMPLE ASSIGNMENT
We introduce greedy training of max sliced Wasserstein GANs, where instead of sorting the samples of the projections we iteratively select the most similar pairs of them for loss calculation. Our training algorithm as a pseudocode is defined in Algorithm The training process is really similar to the approach introduced in Deshpande et al. (2019). The main and only alteration is between line ten and sixteen, where instead of the sorting operation of the original approach we first generate matrix which determines the distances for all possible sample pairs in one-dimension. First we select the smallest element in this matrix and remove its row and column and iterate this process until the matrix contains only one element which will be the distance between the last two, least similar sample pairs.
We have note that our approach requires O(n²) steps compared the original ((nlog(n)) opera- tions of sorting. We have to implement n minimum selections in the distance matrix with size n x n. We also have to note that this increase in complexity refers for training only and has effect ton the inference complexity of the generator, also n is the sample or mini-batch size during training, which is usually relatively low number. In our experiences, using batches of 16 to 512 samples, this step did not result a significant increase in computation time during training.
approach
2.3 HYBRID APPROACH FOR SAMPLE ASSIGNMENT
In the previous section flaws of the Wasserstein distance using sorting in one-dimension were pre- sented and greedy sample assignment was suggested as a possible method fixing these defects. On the other hand one has to admit that in certain cases greedy approach does not ensure minimal transportation, transportation resulted by sorting will always be lower or equal to the assignment calculated by the greedy approach. We also have to state that distance calculation with the greedy assignment does not form proper metric since it does not satisfy the triangle inequality. It can also easily be seen that one could generate one-dimensional cases where the greedy assignment is arbitrarily larger than sorting. To ensure that these cases can not occur during training we have in- troduced a hybrid approach where we first sort the samples and calculate their Wasserstein distance (noted as W and after this step we also caleulate the distance using greedy assignment (noted as
4

---

Under review as a conference paper at ICLR 2020
Algorithm Training the Greedy Max-Sliced Wasserstein Generator
Given: Generator parameters gg. discriminator parameters Od was sample size n, learning rate a
1: while Og not converged do
2: for i 1 + do
3: Sample data {DD') PR generated samples {1r 1" i=1
PF
4:
compute surrogate loss swwDi, wil
5: return L S(wDD wIF ).
6:
(w, 0a) + (0, 02) 2.60
7:
end for
8:
compute max-sliced Wasserstein Distance mar - WW(WDD ,WFb
Sample data {ID' )== PR generated samples
Fb 1" i=1 PF
9:
10:
L
Create distance matrix: Ma | wIf b,
11:
12:
for k = + n do Find min value of M: min(M) m,,t
14:
L < L + Ma,t
Remove Row and Column from M
16:
end for
17:
0g < 0g - a VooL
18:
end while
W In this case parameter (v) can be set, determining limit and if the difference between the two distances is larger than this value the sorted distance will be used. This way Wasserstein distance with the hybrid assignment (WF) can be defined as:
WS W
: (WC W)) > uw
wH =
(6)
otherwise
In case of v = the greedy assignment will only be used in those cases, where also results the minimal transportation. If is set to one, the greedy assignment will be used if the distance calculated this way is less than twice the minimal distance. We were using = 1 in our experiments. also worth note, that sorting the samples before creating the distance matrix (M) can help finding the minimal element, since in this case all series starting from the minimal element in selected row or column will form a monotonically increasing series.
COMPARISON OF THE DIFFERENT APPROACHES AND RESULTS
3.1 ONE-DIMENSIONAL GAUSSIAN MIXTURE MODEL
For the first tests we have generated a one-dimensional toy problem using Gaussian Mixture Model with five modes, each of them with variance of 0.15 and expected values of 0, 2, 4, 6 and 8 ac- cordingly. We have trained four layered fully connected network containing 128, 256, 512 and neurons in the corresponding layers for 5000 iterations using Adam optimizer, with learning rate of 0.001, with batches of 64. No discriminator was applied in this case, the position of the generated samples were optimized directly by the generator and distance calculation was done directly on the output of the generator. setup used the greedy approach introduced in Algorithm I For further details regarding the hy- perparameters of all simulations and for the sake of reproducibility our code can be found on github, containing all details of the deseribed simulations: the link removed from the text but for the sake of anonymity, but a link pointing to the codes was used during submission
one setup loss was calculated by sorting the samples, the other
After training have used the generator to generate 16000 samples and have compared it to 16000 samples generated by the Gaussian Mixtures Model. We have calculated the Kullback- Leibler divergence and the Pearson correlation coefficient between these two distributions, repeated
5

---

Under review as a conference paper at ICLR 2020
all experiments ten times and averaged the results. The results be seen in Table | seen training the network with greedy assignment resulted lower Kullback-Leible divergence and higher correlation between the generated and the original distributions, which signals the the greedy approach could approximate the original distribution of the samples better. For the calculation the Kullback-Leible divergence histogram calculation was done using 100 bins uniformly distributed between -0.5 and 8.5. An image plotting the histograms using these parameters can be seen on Fig.
As it can be
3
Table 1: KL divergence, and Pearson correlation between the original and generated distribution using sorting and a greedy approach.
Method
KL DIV
Pearson Corr
Sorted
0.91
0.41
Greedy
0.68
0.74
685
Greedy Sorted
130
Figure 3: Histograms of the one-dimensional distributions. The blue curve plots the original dis- tribution of the Gaussian Mixture Model, the green curve depicts the generated distribution using sorting for sample assignment, meanwhile the red curve display the result of the greedy approach.
3.2 INO-DIMENSIONAL GAUSSIAN MIXTURE MODEL
We have repeated the simulations described in section 3.1 using two-dimensional Gaussian Mixture models. Nine modes were generated forming a small 3 x grid, each with variance of 0.1 and the distance between neighbouring modes 3.5 in the grid. In this case five-layered network with 128, 256, 512, 1024 and neurons was applied and training was executed for 500.000 iterations. All other parameters of the simulation were kept the same. The Kullback-Leibler divergence and Pearson correlation coefficients these simulations comparing the greedy and sorting approaches can be seen Table 2 meanwhile random samples are depicted on Fig. ??. for qulaitative compar- ison. In this experiment two-dimensional histogram was used for the Kullback-Leibler divergence calculation, where uniform grid was formed between -2 and forming 100 bins.
3.3 EXPERIMENTS ON THE MNIST DATASET
We have evaluated our approach on the MNIST dataset (LeCun (1998) well, where we have used the DCGAN architecture Radford et al. (005)) for image generation. Images were resized to 64 x 64 to match the input dimension of the architecture, but single channel images were used. We were using batches 128 and 16 samples and Adam optimizer for training both the generator and the discriminator (which this case 4096) distributions is complex, we have binarized the images (thresholded half of the maximum intensity) and calculated the KI. divergence between the distribution of white and black
single projection). Since the comparison of high-dimensional (in

---

Under review as a conference paper at ICLR 2020
Table 2: KI. divergence, and Pearson correlation between the original and generated distribution
using sorting and a greedy approach for the two-dimensional Gaussian Mixture Model.
Method
KL DIV
Pearson Corr
Sorted
1.21
0.55
Greedy
0.33
0.80
Figure 4: This Figure depicts the real (blue) and the generated (red) samples for two-dimensional Gaussian Mixture Models. The upper rows were generated using sorting, meanwhile the lower samples were produced using greedy assignment. The subfigures from left to right display results at 3, 4 and five hundred thousand iterations.
values each pixel for the original distribution and 60000 generated fake samples. The results can be seen in Table meanwhile qualitative comparison of randomly selected samples can be seen on Fig.
6
7
7
5
8
0
8
8
3
1
4
3
9
9
J 1
3
3
{
2
7
0
L
2
7
9
8
4 6 3
9
4
9
8
9
9
9
8
/
9
7 4
2
8
0
1
1
4
8
4
5
8
3
6
3
6
0
6
6
8 4
8
0
8
6
1
Figure 5: This Figure displays randomly selected samples generated with the same network archi- tecture and training parameters the MNIST dataset using batches of 16. The samples the left were generated using sorting assignment with Max-sliced Wasserstein distance, the samples in the middle were generated using greedy sample assignment and the samples on the right were generated using the hybrid approach. All samples were generated after 20 epochs of training.
3.4 EXPERIMENTS ON THE CELEBA DATASET
We have also conducted experiments on the resized CelebA dataset (Liu et al. (2018)), in our case images were downscaled to 64 64. We have used the DCGAN architecture and compared three different approaches for sample assignment for distance calculation. We did not use any special normalization initialization methods during training. We used batches of 16 for training.
We have selected 10.000 samples randomly from the dataset and generated 10.000 random pro- jection and used sliced Wasserstein distance with sorting to compare the distributions along these

---

Under review as a conference paper at ICLR 2020
Table 3: KI. divergence between the original and generated distributions using sorting, greedy and a hybrid approach and different batch sizes (16 and 128) on the MNIST dataset.
Method
KL DIV
6
0.0927
Greedy 16
0,0233
Hybrid16
0.0147
S2rted128
0.0489
Greedy 128
0.0172
128
0.009
Table The sliced Wasserstein distance on the generated distributions on the CelebA dataset using sorting, greedy method and hybrid approach for sample assignment.
Method
Sliced Wass.
Sorted
3.472 x 10-3
Greedy
1.294 x 10-3
Hybrid
0.783 x 10-3
lines. The results can be seen in Table meanwhile qualitative comparison of randomly selected samples can seen on Fig.
Figure 6: This Figure depicts random samples generated with the same network architecture and training parameters the CelebA dataset using batches of 16. All samples were generated using Max-sliced Wasserstein distance, the samples on the left were generated using sorting for sample assignment, in the middle the greedy approach used, meanwhile the samples the right were generated using the hybrid approach. All samples were generated after 20 epochs of training.
CONCLUSION
this paper we have introduced greedy sample assignment for Max-Sliced Wasserstein GANs. We have shown that using one-dimensional samples, in many cases multiple assignments can result optimal transportation and in most cases sorting changes all the samples, meanwhile those parts of the distribution which are at "good" position should not generate error.
We proposed greedy assignment as a possible solution, where samples will be assigned to their most similar counterparts. We have also introduced how the combination of the two methods can be applied resulting hybrid approach in which it can automatically selected based on the difference of the two measures- which assignment will be used.
We have demonstrated on simple toy datasets that greedy assignment performs better than sorting the samples and we have evaluated both the greedy and the hybrid methods commonly investigated
8

---

Under review as a conference paper at ICLR 2020
datasets (MNIST and CelebA). With all datasets the greedy assignment resulted lower Kullback- Leibler divergence and higher correlation than the traditional approach.
We have used the Max-Sliced Wasserstein distance for the base of our comparison, since this is the most recent version, which also results the best performance, but all the approaches can be exploited in case of regular Sliced Wasserstein distances as well. Also our approach changes the distance calculation only and can be applied together with various other improved techniques and architectures which are used in GAN training.
REFERENCES
Martin Arjovsky and Léon Bottou. Towards principled methods for training generative adversarial networks. arxiv. 2017.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pp. 214-223, 2017.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1096, 2018.
Ishan Deshpande, Ziyu Zhang, and Alexander G Schwing. Generative modeling using the sliced wasserstein distance. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3483-3491, 2018.
Ishan Deshpande, Tuan-Ting Hu, Ruoyu Sun, Ayis Pyrros, Nasir Siddiqui, Sanmi Koyejo, Zhizhen Zhao, David Forsyth, and Alexander Schwing. Max-sliced wasserstein distance and its use for gans. arXiv preprint xrXiv:1904008877. 2019.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Im- proved training of wasserstein gans. Advances neural information processing systems, pp- 5767-5777, 2017.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. arXiv preprint arXiv: 10196, 2017.
Soheil Kolouri, Kimia Nadjahi, Umut Simsekli, Roland Badeau, and Gustavo K Rohde. Generalized sliced wasserstein distances. arXiv preprint arXiv: vrXiv:1902.000 1902.004444 2019.
Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. con/exdb/mmist// 1998.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Large-scale celebfaces attributes (celeba) dataset. Retrieved August, 15:2018, 2018.
Kimia Nadjahi, Alain Durmus, Umut Şimşekli, and Roland Badeau. Asymptotic guaran- tees for learning generative models with the wasser-was distance. arXiv preprint S5t::/9006045// 2019.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint 1rX1v:1511.06434, 2015.
Yang Yu, Zhiqiang Gong, Ping Zhong, and Jiaxin Shan. Unsupervised representation learning with deep convolutional neural network for remote sensing images. In International Conference on Image and Graphics, pp. 97-108. Springer, 2017.
9


Reviews:
[{"review_id": "rygE9UQgqS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes two alternative approaches to max-sliced Wasserstein GANs. They are based on the authors’ claim that there is a “flaw” in the Wasserstein-1 distance between probability distributions on the one-dimensional space. Briefly, the authors’ argument says that the “flaw” is that the optimal transport may not be unique, some of which are better for network learning than others. One proposal, described in Section 2.2, is to find a plausible transport plan in a greedy manner. The other proposal, described in Section2.3, is a hybrid of the greedy approach in Section 2.2 and the original sliced Wasserstein distance.\n\nThe working hypothesis in this paper, that the above “flaw” is indeed problematic in learning in max-sliced Wasserstein GANs, has not been confirmed in any sense in this paper. Algorithm 1 is meant to explain one of the proposal, the greedy approach, but I found that several undefined symbols are used there, so that it seems hard to understand it. Numerical experiments in Section 3 are not well described. Because of these, I would not be able to recommend acceptance of this paper.\n\nAlgorithm 1 seems to heavily rely on Algorithm 1 in Deshpande et al., 2019. This paper does not provide any explanation about why one should sample n data for each i running from 1 to n (line 3), what the “surrogate loss” is (line 4), what ¥omega is (line 4), why one should care about the surrogate loss between ith data and ith generated sample (line 4), as well as what D^i_k means (line 11).\n\nI do not understand how the Pearson correlation coefficient between the generator (or fake) distribution and the real distribution in Section 3. As far as my understanding, fake samples and real samples are sampled independently, so that correlation coefficient should ideally vanish in any case. Also, the KL divergence is not symmetric, so that whether KL(P_F,P_R) or KL(P_R,P_F) was evaluated has to be explicitly stated. Furthermore, recalling that the Wasserstein distance has originally been introduced to the GAN literature in order to alleviate the problems associated with the KL-based divergence (Jensen-Shannon), I do not understand either why the authors chose to use the KL divergence in their performance comparison.\n\nThe “flaw” argued in this paper does not apply to the Wasserstein distance in general, but specifically to the Wasserstein-1 distance between one-dimensional distributions. This fact should be stated clearly.\n\nPage 2, equation (2): The subscript ¥mathbb{P} should read p.\nPage 2, two lines below equation (3): w should be italicized.\nPage 4, line 4: if the(y->ir) probability\nPage 4, line 24: has no effect (t)on the inference complexity\nPage 5, Algorithm 1, line 17: g of ¥theta g should be a subscript of ¥theta.\nPage 6, line 3: which signals th(e->at) the\nPage 6, line 12: was executed for (500.000->500,000) iterations.\nPage 6, line 15: Figure number is missing.\nPage 7, lines 9-10: (10.000->10,000) random projection(s) and used\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rkxm6Fp2tr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a variant of the max-sliced Wasserstein distance, where instead of sorting, a greedy assignment is performed. As no theory is provided, the paper is purely of experimental nature. \n\nConsidering the above, the experimental evaluation is way too preliminary:\n\n- Looking at the generated images, much better results can be achieved with a Vanilla-GAN + GP regularization, so it is completely unclear to me why the proposed GAN should be used, as it is seems more complicated to implement.\n\n- The KL divergence evaluation seems non-standard, and it is not explained why this metric is chosen over the standard ones (FID, Inception Score). However, I think a evaluation with respect to standard metrics is a must for an experimental GAN paper. \n\n- I would have liked to see a comparison to using the exact Wasserstein distance, as it also scales roughly like n^3. For example, the recent paper \"Wasserstein GAN with Quadratic Transport Cost\" computes the exact distance using linear programming, and there it is shown to yield good results w.r.t. FID.  \n\nMinor comments I noticed during reading (no impact on my rating):\n\n- In Eq. 1, the maximization over D is confusing given that in the following Eq. 2 the primal form of the Wasserstein distance is shown.\n\n- The number of possible joint distributions is a continuum, and not a discrete quantity that increases factorially. Anyway if both distributions are discrete, then the problem can be reduced to a discrete optimization problem with factorially many candidate solutions, but this is very misleading as there are polynomial time algorithms. \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BylRM78J9S", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper suggests a new way to train max slides Wasserstein GANs. I find that the paper has too few innovation in comparison with the approach introduced in Deshpande et al. (2019) (reference in the paper). The authors themselves described that the difference is ‘instead of sorting the samples of the projections we iteratively select the most similar pairs of them for loss calculation’. I think it is not enough for a publication. \n\nMinor suggestions: \n* Page 3. ‘As it can be seen from the figure, for example…’ I think ‘for example’ here is redundant.  In the same sentence ‘will results’ -> ‘will result’. \n* Section 2.2. ‘First we select the smallest element…’ I would remove ‘first’ because it was used in the previous sentence. \n* Section 2.2. has no effect ton-> on \n* Equation (6). There must be a comma before ‘otherwise’. And maybe it would look better to write the equation as a system (with one left braсket).\n* Section 3.2. There is ‘Fig. ??’", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a variant of the max-sliced Wasserstein distance, where instead of sorting, a greedy assignment is performed. As no theory is provided, the paper is purely of experimental nature. 

Unfortunately the work is too preliminary to warrant publication at this time, and would need further experimental or theoretical strengthening to be of general interest to the ICLR community.
Decision: Reject