Do WGANs succeed because they minimize the Wasserstein Distance? Lessons from Discrete Generators
Authors: Ariel Elnekave
Yair Weiss
Venue: ICLR 2025 Poster (2025)
Keywords: Generative
GAN
Wasserstein Distance
Primary area: generative models
TL;DR: We show that WGANs succeed because they minimize Wasserstein distance over patches, not images.

Abstract: Since WGANs were first introduced, there has been considerable debate whether their success in generating realistic images can be attributed to minimizing the Wasserstein distance between the distribution of  generated images and the training distribution. In this paper we present theoretical and experimental results that show that successful WGANs {\em do} minimize the Wasserstein distance but the form of the distance that is minimized depends highly on the discriminator architecture and its inductive biases. Specifically, we show that when the discriminator is convolutional,  WGANs minimize the Wasserstein distance between {\em patches} in the generated images and the training images,  not the Wasserstein distance between images.
Our results are obtained by considering {\em discrete} generators for which the Wasserstein distance between the generator distribution and the training distribution can be computed exactly and the minimum can be characterized analytically.   We present experimental results with discrete GANs that generate  realistic fake images (comparable in quality to their continuous counterparts) and present evidence that they are minimizing the Wasserstein distance between real and fake patches and not the distance between  real and fake images.

Published as a conference paper at ICLR 2025
Do WGANS SUCCEED BECAUSE THEY MINIMIZE THE WASSERSTEIN DISTANCE? LESSONS FROM DISCRETE GENERATORS
Ariel Elnekave, Yair Weiss
The Hebrew University of Jerusalem (Ariel. Einnkkv,, Kair.Weiss Wwi.ss httammiiiiijiiaa.a www..uijiii....... ac.
ABSTRACT
Since WGANs were first introduced, there has been considerable debate on whether their success in generating realistic images can be attributed to minimiz- ing the Wasserstein distance between the distribution of generated images and the training distribution. In this paper, present theoretical and experimental re- sults that show that WGANs do minimize the Wasserstein distance but the form of the distance that is minimized depends highly the discriminator architec- ture and its inductive biases. Specifically, we show that when the discriminator is convolutional, WGANs minimize the Wasserstein distance between patches in the generated images and the training images, not the Wasserstein distance between images. Our results leverage the advantages of discrete generators for which the Wasserstein distance between the generator distribution and the training distribu- tion can be computed exactly and the minima can characterized analytically. We present experimental results with discrete GANs that generate realistic fake images (comparable in quality to their continuous counterparts) and present ev- idence that they are minimizing the Wasserstein distance between real and fake patches and not the distance between real and fake images. Our code is available at hnttps://githul wwt. http://rreee111eee...eeeeeeeeeee
INTRODUCTION
In a seminal paper, (Arjovsky et al. 2017) showed the relationship between generative adversarial networks (GANs) and the Wasserstein distance (W1) between two distributions. They argued that when the data lies on low dimensional manifold, the Wasserstein distance is a more sensible optimization criterion compared to the KL divergence and showed that the Wasserstein distance can be approximately optimized using an adversarial game between two neural networks: generator network and critic network. The key difference between their method, the Wasserstein GAN (WGAN), and previous GANs is that the critic is regularized to be I-Lipshitz, and great deal of subsequent research has focused on improved regularization techniques (Gulrajani et al.) 2017 Miyato et al. 2018 Anil et al. 2019). WGANs have been used in many applications and can provide excellent sample quality in different challenging image datasets (Radford al. 2015 Isola et al.
2017 Brock et al. 2018 Karras et al. 2020 Sauer et al. 2022 Pan et al. 2023).
In recent years, however, the connection between the success of GANs and the Wasserstein distance has been questioned (Stanczuk et al. 2021: Fedus er al.) 2018) Mallasto et al., 2019 Korotin et al.) 2022 Milne & Nachman (002)). The first criticism is the extent to which WGANs do minimize the Wasserstein distance. Several authors have shown that approximately minimizing W, using WGANs can yield a poor approximation (Pinetz et al., 2019 Mallasto al. 2019 Stanezuk al.) 2021 Korotin et al. 2021). A second criticism is whether minimizing the Wasserstein distance be- tween two distributions is sensible optimization criterion for generative models of images. Figure shows a result from Mallasto et al. (2019): model that does much better job of minimizing an empirical estimate of the Wasserstein distance actually produces results of much lower visual qual- ity. This has led to an alternative view whereby "GANs succeed because they fail to approximate the Wasserstein distance" (Stanczuk et al. (202)) and that GANs should not be seen as minimizing a loss function (Goodfellow et al. 2020 Fedus et al. 2017). Many papers have completely abandoned

---

Published as a conference paper at ICLR 2025
Gradient Clipping
c-transform
Poor Batch Wasserstein Distance
Good Batch Wasserstein Distance
Figure I: When a WGAN generator is trained using gradient clipping, the approximation to Wasser- stein distance is poor and batch of generated faces have a higher Wasserstein distance to batch from the training set compared to when the "c-transform" method is used (Mallasto et al. 2019). Nevertheless, the visual quality higher with the poor approximation.
the distribution- matching approach and focused instead on analyzing the adversarial game and its equilibrium (Sidheckh et al. 2021 Farnia & Ozdaglar 2020 Schäter et al. 2019 Qin et al. 2020).
seemingly simple approach to answering the question of whether WGANs succeed because they minimize W1 is to measure the Wasserstein distance between Po, the distribution over fake images defined by the GAN, and Paotas the true distribution, and to compare that distance with alternative methods that minimize Wi (P, P.nta). Unfortunately, are continuous distributions can only done for a limited class of distributions.
exact calculation of Wi (P, Q) where P, Q previous works (e.g. figure I. empirical approximation to W1 (P, Q) was used, but this approximation is known to be poor for -ige-dimensional data such as images (Weed & Bach 2019). Thus as long as we use continuous, non-parametric distributions for the data and the generated images, it is impossible to give rigorous answer to whether WGANs minimize W1.
In this paper, present alternative approach that allows to give rigorous answer. We leverage the advantages of discrete GANS. These GANS are identical to the standard GANs in which a noise vector = passed through neural network Sa generate image. But in discrete GANs, the noise vector z is sampled uniformly from M possible fixed noise vectors and thus the generator can generate at most M possible images. Our work was motivated by our initial findings that when M is sufficiently large, discrete GANs generate images that are of comparable quality to that of standard GANs with the same architecture. Figure shows images generated by variant of FastGAN [ we trained as discrete GAN with M 70, 000 fixed noise vectors. The results are comparable in quality to training the same architecture with continuous noise vectors and similar results are obtained with other values of M.
By using discrete GAN we obtain the following advantages:
We can exactly compute the Wasserstein distance between the GAN distribution Pe and the empirical distribution Patto
We can analytically characterize the optimal distribution Pa that minimizes W, (Po, Pdata) for different values of M.
We can directly optimize Wi Paata) and compare these (locally) optimal solutions to the ones found by WGANs.
In this paper, we leverage these advantages of discrete GANs to provide theoretical and experi- mental evidence that successful WGANs do minimize the Wasserstein distance but the form of the distance that is minimized depends highly on the discriminator architecture and its inductive biases. Specifically, we show that when the discriminator is convolutional, WGANs minimize the Wasser- stein distance between patches in the generated images and the training images, not the Wasserstein distance between images.
See appendix
B.2
2

---

Published as a conference paper at ICLR 2025
Fake
Data NN
Figure 2: (top:) Images generated by a discrete version of FastGAN trained on 128x128 images from FFHQ with M = 70, 000. Even though the generator is discrete the visual quality is high and comparable that of the continuous generator. Similar results are obtained with different values of M. (botttom:) the closest training example to each generated image.
2 EXACT COMPUTATION OF W1 IN DISCRETE SETTING AND CHARACTERIZATION OF THE MINIMUM.
We start by reviewing the connection between Wasserstein distance and WGANs. The Wasserstein distance Wi Q) between distributions is defined as:
W.(P. Q)
inf *EI((P.))
Euw yll
(1)
where II(P, Q) denotes the set of joint distributions whose marginal probabilities are P, Q. The connection GANs is more evident the dual form:
W1 Q) = max Ep(f) - Eq(f)
(2)
JeP
where F, is the class of 1-Lipschitz bounded functions. Thus if we denote by P the distribution over images defined by the generator and Q the training distribution, the minimization of Wi (P, Q) can be performed using an adversarial game in which the generator attempts to decrease Ep(f) - and the discriminator, critic attempts increase (fp(f) Ef(f).
As mentioned in the introduction, the Wasserstein distance between arbitrary P, Q cannot be com- puted efficiently and in this paper we leverage the advantages of discrete distributions for which exact computation and optimization is possible.
Definition 2.1. (Discrete distribution) Given set of N points {x:) we denote the discrete distribution defined by these points by P(ax) (x) = &/4 6(z= 2;), where ö is the Dirac delta function.
Definition 2.2.
(Discrete Wi): Given a of M points {x;} and second of N points {wr} the Wasserstein distance between the discrete distributions defined by the two sets is given by:
Wr(P = min Σ Tullys x,
(3)
1.
with mij M N matrix whose elements are in [0, 1] and satisfies Σ, Rij 1 -E, Tiz Ti
Unlike the continuous case, the solution for the optimal transport matrix æ can done in polynomial time so that W1 (P(((u), P(u)) can be computed exactly. The connection to discrete GANs is described in the following definition:
Definition 2.3. (Discrete W, optimization problem). Given a set of N points {y:} the discrete Wi optimization problem is find set of M points (æy) such that Wi (P(w): P(y)) ) is minimal.
3

---

Published as a conference paper at ICLR 2025
M N
M N
M N
g
Figure 3: In the discrete Wasserstein minimization problem, we approximate training distribution consisting of N examples (blue circles) with a generated distribution consisting of M examples (red asterisk). In this paper present an algorithm (OT-means) for solving this problem and we characterize the optimal solution. Specifically, we show that when M < the optimal solution generates images that are linear combinations of training examples, while for M N and M > N it copies the training examples.
Figure 3 Billustrates this problem. The training training distribution has N examples (blue circles) and we try to approximate it with a generated distribution with M examples (red asterisks). The figure shows numerical solutions to this problem using an iterative algorithm that we call "OT-Means"..
Algorithm (OT-Means). Repeat until convergence:
Given the current estimate of the generated points {a,} set to be the optimal transport matrix betwen { and the training set {
Given æ minimize:
T = arg min
Tay
æ|
This minimization the geometric median problem and can be performed using iteratively reweighted least squares (Weiszfeld 1937).
It is easy to show that this algorithm decreases W ((tu)),(ut)) at each iteration.
Figure 3 shows the output OT-means on the same toy dataset with different values of M. It can be seen that when M < N the optimal solution generates samples that are linear combinations of training examples, while for M = N and M > it copies the training examples. The following theorem characterizes the solutions to the problem.
Theorem 2.4. For M N or M N, M = k optimal solution the discrete W, optimization problem is for the generator to copy the training examples. For M < N, at any local minimum of the problem, each generated sample is linear combination of at least N/M training examples.
Proof. (sketch) The result for M N M k:N follows from the fact that W1 2 0 and copying the examples yields W, 0. The result for M N follows by differentiating W, (P(z)- + Piv) with respect to a specific zy and setting the gradient to zero. See appendix A.I for full proof.
Given these results, we can now rigorously answer the following question: is the success of the discrete GAN shown in figure 2 due to minimizing W1 This GAN was trained with M N 70, 000 thus the optimal solution is to simply copy the training examples. But as can be seen, in figure 2 the discrete GAN is not copying the training images and when we compute the exact W, for this problem we see that is far from the optimal value of zero. As another example, consider the discrete GAN shown figure 4 here M = 10, 000 and N = 70, 000 so at any local minimum the generated images should be linear combination of at least 7 training images. Indeed when run OT-means with these values of M, N we obtain images of low quality (shown in the bottom of the figure) even though the exact W1 is better. This is reminiscent of the Mallasto et al. (2019) result in figure Tutt note that here we are using the exact W, between two discrete distributions and avoiding the intractable problem of approximating W, between continuous distributions.
4

---

Published as a conference paper at ICLR 2025
Discrete FastGAN (M 10K) (W, 1.82e04):
OT-Means (M 10K) (W1 R 7.6 7.6900)))
Figure 4: Top: results of discrete fastGAN with M 10K so that M N. The results are still sharp and realistic. Bottom: OT-means with the same M, N. This produces blurred images and exact W1 is lower.
3 THE IMPORTANCE OF THE DISCRIMINATOR ARCHITECTURE
Given the dual form of W1 (equation 2). how do explain the fact that WGANs do such poor job of minimizing W1? ? One possibility is that in the standard training of WGANs, the discriminator is not optimized convergence: the common practice is iterate few iterations of gradient descent for the generator and then a few iterations for the discriminator. Perhaps this use of iterative gradient updates precludes the WGANs from optimizing W1 this section, we present evidence that this is not the case. In fact, even with iterative gradient updates, WGANs can do an excellent job of optimizing the Wasserstein distance, but the specific form of the distance that being optimized is heavily influenced by the architecture of the discriminator.
Figures 5 show experiments with fully connected discriminators. Fully connected discriminators are interesting because they satisfy the universal approximation property Hornik et al. (989)): by using a sufficiently wide fully connected network with the gradient penalty method WGANs, the discriminator should be able to implement any 1-Lipshitz function. In these figures, we considered three toy datasets of N = 1000 images: (1) white squares on a black background (2) MNIST and (3) face thumbnails (of size 64 64). We trained discrete WGAN with fully connected discriminator on these datasets with different values of M. The generator network was also a fully connected network and used mini-batches of size 64. We also ran OT-Means on the same data and for the same values of M. We found that for range of values of M, the discrete WGAN did an excellent job of minimizing Wi.
Figure5]showe the results for M = 64. Recall that given theorem 2.4 any local minimum of the loss should generate images that are linear combinations of at least 15 different training images. Indeed when we look at the results of OT-Means (in the middle column) we find that the generated images blurred, as expected. When we look the results of the DiscreteWGANT we see that they are also blurred. Unlike the typical published WGAN generated images, where the results are sharp and contain high-resolution detail (e.g. figure D.. now the results are visually similar to the results of OT-Means. Perhaps most convincingly, when we measure the exact W1 between the generated images and the training set, we see that the DiscreteWGAN-FR
(right column)
samples achieve nearly the same Wi as the OT-Mcans result (and significantly better than the W1 obtained by randomly choosing a batch of M examples from the data as the generated images).
Figure 6 shows similar pattern of results for the case M N. According to theorem 2.4 in this setting the optimal W1 is obtained when the generator simply copies the training examples. Indeed such solution is found rapidly using OT-Means. But more surprisingly, this solution is also found using the Discrete GAN: even though the generator never has direct access to the training set (only through noisy gradient signal given by the discriminator) and is trained with mini-bate mini-batches, it
5

---

Published as a conference paper at ICLR 2025
Data samples
OTMaans
DisAreteWGAN~ PCC
OITIORAL
Vesertte
9
(
9
8
7
1
4
9
9
/
/
7
Oftesuro
3
7
6
5
2
_
7
6
0
5
7
8
3
7
6
9
9
4
7
2
/
8
4
9
20.21
0
6
3
c
5
5
3
/
5
2
0
Ofreaane
Figure 5: Using a fully connected discriminator minimizes W1 almost as well as OT-Means. The plots on the right show the exact W1 for OT-Means (green), WGAN (blue), and randomly choosing a batch of training images (red).
Pale
Real NN
Reel XON
Fake
Beal DOX
Figure 6: Using Fully connected discriminator and M = N makes WGANs copy the data. Again, this is almost as good as minimizing WI with OT-Means.
manages copy the training examples and achieves a W1 value that is close the optimal value of zero. Figure 20 in the appendix shows more results for the case M = N: all images generated by the WGAN are copies of a training image.
Taken together, these results show that WGANs can do an excellent job of minimizing W1, even with iterative gradient updates. How then do we explain the failure of W2 minimization in successful WGANs such as those shown in figures 22 discriminators.
As we now show, this is because they use convolutional
3.1 CONVOLUTIONAL DISCRIMINATORS
The key assumption in the connection between WGANs and W1 is that the discriminator can ap- proximate any 1-Lipshitz function. But what happens if the discriminator architecture has a strong inductive bias? Almost all discriminators used in practical GANs convolutional. The following theorems show that for such discriminators, WGANs no longer minimize W, between images but rather between smaller patges.

---

Published as a conference paper at ICLR 2025
Global patch W1
Local patch W1
Denct
FFHQ
8
7
Mnist
C
3
Squares
Figure 7: Using convolutional discriminators with the same generator as above learns either global patch statistics (with CNN-GAP) or local patch statistics (with CNN-FC). In all cases, the patch W1 that it finds is similar to directly optimizing the appropriate patch W1.
We start by formally defining two convolutional architectures (see detailed description in the ap- pendix B.I).
Definition 3.1. (CNN-GAP) We denote by CNN-GAP(S) CNN in which all layers except the last two are convolutional, followed by global average pooling (GAP) and a fully connected layer. The receptive field of units in the layer before the global average pool is S.
Definition 3.2. (CNN-FC) We denote by CNN-FC(S) CNN in which all layers except the last layer are convolutional followed by fully connected layer and the receptive field of units in the layer before last is S.
These two CNNs are abstractions of CNNs that are used in successful WGANs: CNN-GAP is an abstraction of the patchGAN discriminator (Isola et al. (017) and CNN-FC is an abstraction of the DCGAN discriminator (Radford et al. 2015). More details are in Appendix [B.l
Theorem 3.3. Training WGAN with CNN-GAP discriminator is equivalent minimizing
Wi (Po, Patta) where Po, Pauta the distribution over all patches of size S in the generated im- ages and training images respectively.
Proof. (sketch) For this discriminator the output be written patches and by linearity of expectation Eo( can be written as sum of expectations over patch distributions. Constraining the image discriminator be I-Lipshitz also constrains the patch discriminator. Full proof in A.2
of discriminators
This proof generalizes the connection between convolutional discriminators and patch distributions that was first presented in Isola et al. (2017). We note also that most of the proof (which relies on the linearity of expectations) also holds for other forms of GANs such as Cramer GANs (Bellemare) et al. 2017). Sobolev GANs (Mroueh et al.) (017)). and MMD-GANs (Li et al.) 2017) For the non-saturating GAN with a convolutional discriminator, the discriminator output cannot be written as linear sum of patch outputs and so the proof does not hold although we observe empirically that such NS-GANS behave qualitatively similar to WGANs (figure 17). Figure 19 shows the influence of the receptive field of the discriminator the patch distribution minimized by the WGAN: increase the receptive field of the discriminator, the images increasingly capture more the global structure of the training images.

---

Published as a conference paper at ICLR 2025
1999
Figure 8: Top column: 20 randomly selected fake patches from single location from discrete GAN with M = N 70K' Second column: the most similar true patches from the same location in the training set. Even though the generator never has direct access to the training images, it does good job of copying the training patches. The mean cosine distance to the nearest neighbor is 0.99. Bottom: comparisons of histograms of random projections of fake and true patches at the same location. The histograms almost perfectly aligned. The histogram of LSUNS patches at the same location are shown for comparison.
Theorem 3.4. Training a WGAN with a CNN-FC discriminator is equivalent to minimizing an .Patt)) , where p p are the distributions over all patches of size S in the generated images and training images respectively location (i.j). This bound holds for any location (i.j).
upper bound on W1(P')
Proof. (sketch) For this discriminator, the output can be written weighted sum of discriminators over patches at a specific location. By choosing the weights to be one only at one location and zero everywhere else, obtain bound on the Wasserstein distance between fake and generated patches at that location. Full proof in A.3
Figure (Ist and 3rd left columns) shows the dramatic influence of the discriminator architecture on the toy datasets that we showed in figure Even though we are using exactly the same value of M (64) and exactly the same training set and generator architecture, the Discrete GANs with CNN discriminators generate completely different images compared to OT-Meens (Fig. 5
In order to directly optimize patch W1 patches must satisfy the constraint that they are taken from M images with overlap. As an alterna- tive, we use SGD to train the same generator network used by the discrete GANs but replace the WGAN training loss with an estimate of the appropriate patch Wi.
longer use OT-Means because the generated
As mentioned in the introduction empirical estimates of W1. For image W1 we need to compute an optimal transport between M = 64 generated images and training images and this is still feasible but for global patch W1 need to compute optimal transport between all patches in 64 images and all patches in the training images and this is infeasible. We therefore used Sliced Wasserstein distance (SWD) Pitie al. (2005); Rabin et al. (2011); Bonneel et al. (2015)
the discrete generators avoid using minibatch-based
cheaper proxy.
The sliced Wasserstein distance is defined the expected W1 value between 1-d projections of two distributions into random directions. An unbiased approximation of this distance is computed from random directions. The efficiency of SWD stems from the fact that for one-dimensional data, the W1 can be computed by simply sorting the samples and the distance between two samples of size A can be computed in O(M log M). While in general SW D(P, P Q) and (P,,QQ may be different, SWD shares with W1 the property that SW D(P, Q) 0 if and only if P = Q. Thus by minimizing patch SWD D we are encouraging the patch distribution in the fake images to match the patch distributions in the true images.
Figure 7 (2nd and 4th columns from the left) shows the results of direct optimization of patch SW D on the same datasets and using the values of M. Unlike the OT-Means results, matching patch
8

---

Published as a conference paper at ICLR 2025
Figure 9: Evidence that the discrete DCGAN (2 leftmost plots) and FastGAN (2 rightmost plots) match the local statistics in the generated image and the training images. Even though each image is quite far from a training image, the local histograms of random projections of patches from the true images and the generated images are almost identical. This is true for various values of M including a continuous prior.
distributions yields images that preserve high-frequency details and lack global structure. Clearly, discrete GANs that use convolutional discriminators generate images similar to those obtained by approximately minimizing patch distributions, not by minimizing image W1.
To summarize, our proof's show that theoretically WGANs with convolutional discriminators should minimize patch Wi and not image W1 and our experiments with toy data show that this is indeed the case, even if we use iterative gradient training. We now ask, can the minimization of patch W1 explain the success of WGANs in realistic datasets?
DO SUCCESSFUL. DISCRETE GANS MINIMIZE PATCH WASSERSTEIN DISTANCE?
Model
DCGAN
FastGAN
OT-Means
AFHQ
Imagenet 70K
M
1OK
70K
IOK
70K
IOK
15K
Local patch swd
0.044
0.040
0.058
0.056
0.045
0.033
0.099
0.145
0.181
Table 1: The local SWD between real FFHQ (N=70K) image patches and fake image patches for different successful DiscreteWGANs (averaged over locations) with different M values. In all cases, the average local SWD is below 0.05. For reference comparison, the average local SWD for OT- Means (M=10K) is around 0.1 and > 0.14 for images from AFHQ and ImageNet
We trained discrete versions of FastGAN and DCGAN on N = 70, 000 FFHQ images of size 128 x 128. We varied the number of fixed noise vectors A and found that for both architectures, the discrete GANs generate comparable results to the continuous counterparts (figures See also figure 14 in the appendix.
The preceding theory and experiments have told us what expect if these successful discrete GANs are minimizing Wasserstein distance. The easiest case is when M = N. For such a case, we expect discrete GAN copy images from the training set if it is minimizing image Wi and to copy patches from the training set it is minimizing patch Wi Figure [8 (top) shows fake patches from single location generated by the discrete FastGAN from figure 2 and the bottom row shows the closest match from the same location in the training set. Even though the generator never has direct access to the training images, does a good job of copying the training patches. The mean cosine distance to the nearest neighbor is 0.99.
If the discrete GAN minimizing patch W, is not enough for it to copy patches from the training set: it should also maintain the same patch distribution. Consider for example a location where there are only two possible patches in the training set: 60% are blue patches and 40% are green patches. If the fake patches are 99% blue and 1% green, then the patch W1 would still be high (Elnekave & Weiss 2022). Only if the fake patches were also 60% blue and 40% green is the patch Wi minimal. To visualize how well the discrete GAN matches the local patch distribution we considered histograms of random projections of the patches. As can be seen in three plots at the bottom of figure 8 the histograms for fake patches and true patches are almost perfectly aligned. Thus the discrete GAN is not only copying patches but also matching the local patch distribution. Even when consider the histograms corresponding to patches in the edges of the image for true and fake
9

---

Published as a conference paper at ICLR 2025
images, the two histograms align quite well (see figure 13 in the appendix). Thus even for locations that often correspond to the background, the discrete GAN matches the local patch distribution in the true and fake images.
When M < NV then we cannot expect the discrete GANs to copy training patches but rather its output should look like the output of OT-Means on patches from single location (see appendix D): each generated patch is a linear combination of training patches and the local histograms of true and fake patches location should match. Figure / shows that the local distribution is matched very well for a discrete DCGAN with different values of M. Importantly, the matching also holds with the continuous prior.
We summarize all of our experiments in table I for the two GAN architectures and for different values of .,, the local distributions at a particular location are well matched between the true images and the generated images. As a reference we also compute the distance to batch of 10K images from ImageNet and a batch from AFHQ (Choi et al. (0200:: the average local SWD between AFHQ and FFHQ is about 0.15 and 0.18 while the different WGANs consistently achieve local SWD that about 0.04.
5 LIMITATIONS AND EXTENSIONS
Our theoretical results are based on simplifications of convolutional discriminators used in practice. One major simplification is that we assume that the receptive field sizes are at most S and this value of S determines the patch size used in patch W1. For practical CNNs, the theoretical limit on the receptive field may be very large, but the effective receptive field size is still small (Brendel & Bethge 2019) and the CNN can be well-approximated as if the receptive field size was much smaller than the theoretical limit. A second simplification is that we implicitly assumed that the CNNs do not use any padding when performing convolutions (and this was also the case in the GANs that we trained for the toy data). When padding is used, patches at different image locations actually be distinguished even with a patch-based discriminator (e.g. (Isola et al., 2017).(Shaham et al., 2019)) so even CNN-GAP discriminator will optimize patch distance that is location-dependent
Even though our use of the discrete setting allows us to exactly measure W1 between true and fake images, it still leaves open the question of how small W1 needs be in order for a method to be described as "successful" in minimizing W,. For this reason, we compare the results of the WGANs to direct optimization of W1 and also visualize the histograms and measure their distance using SWD.
The most significant limitation of our paper, of course, is our focus on discrete GANs while SOTA GANs use continuous prior. We note that most of our theoretical results do not require the GANs to be discrete: working with discrete GANs allows us to empirically measure W, and rigorously de- termine whether the predictions hold. In practice, we find that continuous GANs with convolutional discriminators behave similarly to their discrete counterparts (with large M).
6 DISCUSSION
The question that motivated this paper is whether WGANs succeed because they optimize the Wasserstein distance. We leveraged the discrete setting that allows us to compute the Wasserstein distance exactly, to characterize its optima, and to optimize it with alternative algorithms. Our results indicate that the answer is "yes" but the form of W1 that is being optimized depends on the architec- ture of the discriminator. Specifically, we have shown that when the discriminator is convolutional, what is being minimized is the patch Wi and not the image W1.
A major advantage of having a well-defined loss function that is being optimized is the ability to monitor learning algorithms and check for their convergence. We hope that our results will yield WGAN learning algorithms that are considerably more stable than the current methods and require less parameter tuning.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Cem Anil, James Lucas, and Roger Grosse. Sorting out Lipschitz function approximation. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 291-301. PMLR, 09-15 Jun 2019. URL httppp:://ppooecc.gg.g......... ani 11a1 html.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pp. 214-223. PMLR, 2017.
Mare G Bellemare, Ivo Danihelka, Will Dabney, Shakir Mohamed, Balaji Lakshminarayanan, Stephan Hoyer, and Rémi Munos. The cramer distance as solution to biased wasserstein gradi- 2017.
ents. arXiv preprint
Nicolas Bonneel, Julien Rabin, Gabriel Peyré, and Hanspeter Pfister. Sliced and radon wasserstein barycenters of measures. Journal of Mathematical Imaging and Vision, 51(1):22-45 2015.
Wieland Brendel and Matthias Bethge. Approximating cnns with bag-of-local-fennee works surprisingly well imagenet. arXiv preprint wiXiv:190400099 2019.
models
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1096, 2018.
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung- Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pp. 8188-8199 8197, 2020.
Ariel Elnekave and Yair Weiss. Generating natural images with direct patch distributions matching. In European Conference on Computer Vision, pp. 544-560. Springer. 2022.
Farzan Farnia and Asuman Ozdaglar. Do gans always have nash equilibria? In International
Conference on Machine Learning, pp. 3029-30399 PMLR, 2020.
William Fedus, Mihaela Rosca, Balaji Lakshminarayanan, Andrew M Dai, Shakir Mohamed, and Ian Goodfellow. Many paths to equilibrium: Gans do not need to decrease a divergence at every step. arXiv preprint 17100077 2017.
William Fedus, Mihaela Rosca, Balaji Lakshminarayanan, Andrew Dai, Shakir Mohamed, and Ian Goodfellow. Many paths to equilibrium: Gans do not need to decrease divergence at every step. In International Conference on Learning Representations, 2018
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(1 13:39944944 2020.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Im- proved training of wasserstein gans. Advances in neural information processing systems, 30, 2017.
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are uni- versal approximators. Neural networks, 2(5):359-366, 1989.
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1 1125-1134, 2017.
Tero Karras, Samuli Laine, Miik Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pp. 81 10-811110 19,, 2020.
Alexander Korotin, Lingxiao Li, Aude Genevay, Justin M Solomon, Alexander Filippov, and Evgeny Burnaev. Do neural optimal transport solvers work? a continuous wasserstein-2 benchmark. Advances in Neural Information Processing Systems, 34:14593-14605, 2021.

---

Published as a conference paper at ICLR 2025
Alexander Korotin, Alexander Kolesov, and Evgeny Burnaev. Kantorovich strikes back! wasserstein gans are not optimal transport? Advances in Neural Information Processing Systems, 35:13933- 13946, 2022.
Chun-Liang Li. Wei-Cheng Chang. Yu Cheng, Yiming Yang, and Barnabás Póczos. Mmd gan: Towards deeper understanding of moment matching network, 2017. URL https://arxiv. org/abs 1705 08584
Bingchen Liu, Yizhe Zhu, Kunpeng Song, and Ahmed Elgammal. Towards faster and stabilized gan training for high-fidelity few-shot image synthesis. In International Conference on Learning Representations, 2020.
Anton Mallasto, Guido Montúfar, and Augusto Gerolin. How well do wgans estimate the wasserstein metric? arXiv preprint arXiv:1900.mm ptt...9910...55 2019.
Tristan Milne and Adrian I Nachman. Wasserstein gans with gradient penalty compute congested transport. In Conference on Learning Theory, pp. 103-129. PMLR, 2022.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint
2018.
Youssef Mroueh, Chun-Liang Li, Tom Sercu, Anant Raj, and Yu Cheng. Sobolev gan, 2017. URL https://arxiv: /tttp:://ammiv.orr//b////// 04899
Xingang Pan, Ayush Tewari, Thomas Leimküihler Lingjie Liu, Abhimitra Meka, and Christian Theobalt. Drag your gan: Interactive point- based manipulation on the generative image mani- fold. In ACM SIGGRAPH 2023 Conference Proceedings, pp. 1-11, 2023.
Thomas Pinetz, Daniel Soukup, and Thomas Pock. On the estimation of the wasserstein distance in generative models. In Pattern Recognition: 41st DAGM German Conference, DAGM GCPR 2019, Dortmund, Germany, September 10-13, 2019, Proceedings 41, pp. 156-170. Springer, 2019.
Francois Pitie, Anil C Kokaram, and Rozenn Dahyot. N-dimensional probability density function transfer and its application to color transfer. In Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1, volume 2, pp. 1434-1439, IEEE, 2005.
Chongli Qin, Yan Wu, Jost Tobias Springenberg, Andy Brock, Jeff Donahue, Timothy Lillicrap, and Pushmeet Kohli. Training generative adversarial networks by solving ordinary differential equations. Advances in Neural Information Processing Systems, 33:5599-5609, 2020.
Julien Rabin, Gabriel Peyré, Julie Delon, and Marc Bemot. Wasserstein barycenter and its applica- tion to texture mixing. In International Conference on Scale Space and Variational Methods in Computer Vision, pp. 435-446. Springer, 2011.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv: 11155.111.444 2015.
Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. In ACM SIGGRAPH 2022 conference proceedings, pp. 1-10, 2022.
Florian Schäfer, Hongkai Zheng, and Anima Anandkumar. Implicit competitive regularization in gans. arXiv preprint arXiv: 1910.0582, 2019.
Tamar Rott Shaham, Tali Dekel, and Tomer Michaeli. Singan:
Learning a generative model from single natural image. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4570-4580, 2019.
Sahil Sidheekh, Aroof Aimen, and Narayanan C Krishnan. On characterizing gan convergence through proximal duality gap. In International Conference on Machine Learning, pp. 9660-9670. PMLR. 2021.
Jan Stanezuk, Christian Etmann, Lisa Maria Kreusser, and Carola-Bibian Schönlieb. Wasser- stein gans work because they fail (to approximate the wasserstein distance). arXiv preprint 103.0:2103.01668, 2021.
12

---

Published as a conference paper at ICLR 2025
Jonathan Weed and Francis Bach. Sharp asymptotic and tinie-ssmple rates of convergence of em- pirical measures in wasserstein distance. 2019.
Endre Weiszfeld. Sur le point pour lequel la somme des distances de points donnés minimum. Tohoku Mathematical Journal, First Series, 43:355-386, 1937.
A PROOFS
A. PROOF OF THEOREM 2.4
To show that for M < N at any local minimum, the generator must be linear combination of at least N/M training examples, we differentiate equation with respect generated image 2; and setting the gradient to zero yields that Fa must satisfy:
K2 Σ Wijlij
with wij a vector of weights that sums to one and wa X mallæz -; 1-11-11---- Now since Tij is solution the optimal transport between M generated images and N training images, is a -eggattae matrix that satisfies Σ, Tiy N and E, Waj M This implies that for each i there must be at least N/M different indices j for which mij >> 0 and this shows that must be linear combination of at least N/M training images.
A.2 PROOF OF THEOREM 3.3
We can write the output of the critic for an input x as:
f(x) = WT Σ h(pe))
n2
where p,r extracts the ith patch in image x, are the weights in the final layer, and n is the number of patches in the penultimate feature map.
Moving W into the above sum and defining g(x) = WTh(x) where æ is patch we can rewrite the image critic, f (I)) as:
f(x) = - E 9(Pxz)
Recall that the critic / attempts to maximize Er(s) Ee(S), By the linearity the expectation Ef(f) equal to Eg(g) and likewise Ep, (f) = Ep, (g) where P, Q are the distributions over patches in the true and fake images. Denoting by GAP; the class of I-Lipshitz functions that can be implemented by CNN-GAP architecture, this means that:
maz
(p, - Ee(s) max Ep, (g) - Eg(g) gEG,
jEGAP
G1 is the class of functions that operate on S patches and can implemented by the units in the layer before the GAP.
Constraining the image discriminator to be I-Lipshitz means that for any two images Iı, ø2 such that ||, Ø2ll < d, [f(z)) f(z)] < d. In particular, this holds when choose æ1, so that they differ only in a single patch and this means that must also be 1-Lipshitz.
A.3 PROOF OF THBOREM 3.4
We can write the output of the critic for an input x as:
f(x) = ΣΣ wief
(4)
13

---

Published as a conference paper at ICLR 2025
where P.t. extracts the ith patch in image 2, C' is the number of channels and Wie the weights in the final layer.
Now define the function class CNN as all functions that can be implemented using equation $ and the subelass CN as the set of functions that can be implemented by equation where Wie
vc, Vj # k. Since CNN; CNN we have
Ep, (f) E
JECNN, Ep, Eo(f)
JECNN
and since for f E CNN,, fs only a function of the ith patch we can write:
max JECNN;
(P) - -E(lf)
max Ex (gi) - (9))))
iEL.9IEG
max wpatches, (Po. Q) SET
where again (x) is critic for the ith patch: 9,(x) = Eo wie.o(P.z).
B ARCHITECTURES
B.1
TOY CNNS
We bring here further details about CNN-GAP and CNN-FC architectures used in the experiment section. Both architectures start with 3 convolutional layers with kernel size 3, stride 2 and no padding each followed by a RELU layer. These layers transform a 64x64x3 image into 7x7x256 feature map. The receptive field at this point, i.e., the size of the patch in the original images that affect each pixel this feature map is 15x15. CNN-GAP applies global average pooling to this feature map that transforms it into 1x1x256 layer that is later linearly projected into scalar. CNN-FC reshapes the feature map into a long vector of 7#7 that is linearly projected into a scalar. Figure illustraate the two architectures with their common backbone.
B.2 FASTGAN VARIANT
We experimented with FastGAN Liu et al. (0200) performs comparably to SOTA GANs. our experiments, we simplified its architecture branching off this implementation PI We removed the auto-encoding branches of the discriminator so that both the generator and the discriminator are simple feed-forward networks with skip connections.
easy. fast-to-converge GAN model that
C OTMEANS WITH DIFFERENT VALUES OF M
In the main paper we showed results for small M (in which case the generated images are blurred) and M N (in which case the generated images are copies of the training set). In figure [] we show intermediate values of M as well. As can be seen, as M increases, the images are increasingly sharper until they appear to copy the training set.
D OTMEANS ON PATCHES
One reason to expect better results with the minimization of patch W, is related to theorem 2.4 minimizing image Wi causes the generator to either copy the training set or to generate images that are linear combinations of several training images and will therefore be of poor visual quality. But if we minimize patch W1 we can still generate images that are very different from the training images by copying patches from the training set and combining them in novel ways. We can also generate patches that are linear combinations of training patches, but since the distribution of patches is simpler in some sense than that of images, linear combinations of training patches do not necessarily
14

---

Published as a conference paper at ICLR 2025
CNN-GAP
Linear
Conv2d+Relu
256
Real/Fake
Roml
Roll
RF:
RF: 15
7x7x256
15x15x128
31x3164
Linear
64xG413
77 TT'298
Real/Fake
Figure 10: An illustration of the toy CNN used the experiments in figures 5-7. 3 Convolutional layers are followed by two types of linear heads that project the feature map into a scalar. The annotation RF stands for the receptive field the previous layer.
M=100
M=200
M=400
M=600
M=800
Figure 11: Results of OTmeans with N 1000 and different values M. As M increases, the images increasingly sharper until they appear copy the training set.
15

---

Published as a conference paper at ICLR 2025
real:
OTMeans Image:
OTMeans Patch:
3000
1200
Tue
OT Means Image
brue
2500
GT Means Palth
1000
0TM Oans Patch
01M
2000
800
1500
600
1000
400
200
-10000
intensity
intensity
Figure 12: The difference between minimizing image W and patch W,. When we minimize image Wi and then extract small (20 20) patches (middle row) the extracted patches look very different from the training patches. When we minimize patch W1, i.e. the Wasserstein distance between 20 20 patches that location in the true and fake images, the generated patches (bottom row) look similar to real patches. The success in capturing the local patch distribution can be visualized by comparing the histograms of random projections of the patches.
have to be blurred. Figure [2 illustrates this point. We minimized patch W1 by running OTmeans with M 10, 000 only on small 20x20 patch in location that corresponds to the left part of the mouth. We then compared the generated patches to those obtained by running OTmeans on the full image (hence minimizing image Wi) and then extracting the patches. As be seen in the figure, the patches obtained by minimizing patch W1 are high-contrast and sharp, unlike the patches obtained by minimizing image Wi.
We can visualize how well the generated patches match the distribution of training patches by plot- ting the histogram of random projections of the patches. As shown in the bottom of figure [2, when we minimize image Wi the histograms of projections of generated patches are visibly different from those of training patches the same location. When minimize the local patch Wi, the histograms match very well.
E HISTOGRAMS AT DIFFERENT LOCATIONS
Figure 13 visualize how well the discrete GAN matches the local patch distribution we considered histograms of random projections of the patches additional 3 locations.
F RESULTS WITH DISCRETE SOTA ARCHITECTURES
We bring here the results of training discrete- WGANs with more capable architectures. As can be seen, for both DCGAN Radford al. 22015) and FastGAN Liu et al. (2020) the results with two different discrete priors (M=10K,70K, are comparable to the results with normal prior (M=oo). Each second row shows the data nearest neighhor to show there is copying involved.
16

---

Published as a conference paper at ICLR 2025
locadon
lecalior:
2004
Figure 13: Histograms the same random projection for and fake patches different locations. Note that the histograms are different between different locations but highly similar for true and fake patches. This true even for locations at the edge of the image.
F.1 RESULTS WITH DISCRETE DCGAN
See generated images and data nearest neighbors in figure 14
F.2 RESULTS WITH DISCRETE FASTGAN
See generated images and data nearest neighbors figure
G ADDITIONAL ABLATION EXPERIMENTS
G.1 RESULTS WITH CONVOL.TTIONAL GENERATOR
In all of the experiments in the paper, used the same FC generator. We repeat here the results from figures 5 and 8 from the paper with a convolutional generator (DCGANRadiord et al. (2015). As can be seen in figure 16 the WGAN and direct optimization show similar behavior.
G.2 RESULTS WITH DIFFERENT GAN LOSSES
Our paper deals with WGANs. While the same can be done for all IPMs like Sobolev GANs and MMD GANs may not be directly applied to some other losses like the original or non-saturating GAN losses. However, our experiments with Non-saturating GANs show (figure 17p qualitatively similar results.
G.3 RESULTS WITH DIFFERENT PATCH SIZES
We have conducted experiments from figures 5,7 with different receptive field sizes for the discrim- inator. We used convolutional layers instead of 3 in the discriminator to get RF size=8 and the same number of layers (3) with kernel size 4 (instead of to get RF size of 22. As be seen in figure 19 as the receptive field increases the WGAN generated images preserve statistics of patches of larger sizes.
G.4 ADDITIONAL RESULTS WITH M=N
We bring here in figure 20 more results from the experiment of figure 6
17

---

Published as a conference paper at ICLR 2025
M = 10, 000:
RROIP
M 70, 000:
M = 00
Figure 14: Images generated with Discrete DCGAN for different values of M. For each value of M we show five generated images and below them the closest image from the training set for each generated image.
18

---

Published as a conference paper at ICLR 2025
M = 10, 000:
M 70, 000:
M :
Figure 15: Images generated with Discrete FastGAN variant for different values of M. For each value of M we show five generated images and below them the closest image from the training set for each generated image.
19

---

Published as a conference paper at ICLR 2025
Q
3
3
3
1
2
5
4
3
W
FFHQ
(b) Mnist
(c) Squares
Figure 16: Comparing direct optimization in image/patch level with WGAN with FC/Convolutional discriminator when the generator is convolutional
7 1
122
(a) FFHQ
(b) Mnist
(c) Squares
Figure 17: Results of training NS-GAN with FC/Convolutional discriminators. As can be seen, we the same trend with WGAN where the discriminator architecture controls whether the statistic being preserved model trained with WGAN loss and of direct optimization for reference. The NS-GAN is not as good as the WGAN at minimizing the appropriate W, but the results are qualitatively similar.
the image or the patch level. The graphs also show metrics taken from the same
20

---

Published as a conference paper at ICLR 2025
WCANLoss
WGANLoss CNN-GAP 15
WGANLoss CNN- GAP 15
WGAALoss CNN-
7
WEANLoss CNN-GAP 27
WGANLoss CNN GAP 22
WGANLoss CNN-GAP 22
2
8
6 7
Figure 18: CNN-GAP with different receptive fields
Figure 19: Results of DiscreteWGAN trained with Convolutional discriminator with GAP but with different receptive fields. As can be seen, as the patch size grows the statistics of bigger patches are preserved.
21

---

Published as a conference paper at ICLR 2025
B
Batch2:
Batch3:
Batch4:
Figure 20: Additional 4 batches of results from the model from figure 6 trained with M=N=1K on FFHQ each batch's top row shows fake images and the bottom row shows data nearest neighbors. All generated images are copies of training images.
22


Reviews:
[{"review_id": "ajq7sS9TER", "reviewer": "Reviewer_pwLS", "summary": "This paper presents theoretical and experimental results demonstrating that WGAN actually minimizes the Wasserstein distance (W1), though this depends on the architecture of the discriminator. Using a convolutional discriminator as an example, the paper shows that the W1 distance is minimized over batches rather than individual images.\n\nSpecifically, the authors begin with a discrete GAN, where the latent vector z is uniformly sampled from M fixed noise vectors, using this setup as a tool to investigate the Wasserstein distance, as it allows for a more exact computation of the W1 distance. They introduce an iterative algorithm, OTmeans, to compute the W1 distance, serving as a baseline for investigating W1 distance minimization in WGAN. Using this tool, the authors present two main findings in the paper:\n\nFinding 1: Using a 2D discrete GAN to motivate the theorem, the authors observe that when M≥N (where M is the number of noise samples and N is the number of training samples), the discrete GAN reproduces the training examples. Otherwise, it generates outputs as linear combinations of the training examples. They also study cases with images where M=64<N=1000, resulting in blurry images that look similar between WGAN and the OTMeans algorithm. When M=N, there are copied training example.\n\nFinding 2: The authors design two discriminator architectures to demonstrate that optimizes the W1 distance over batches with the convolutional discriminator. The first architecture is a standard convolutional model with convolutional layers followed by a fully connected (FC) layer that operates on batches. The second architecture inserts Global Average Pooling (GAP) between the convolutional layers and the FC layer, making it act on entire images. This setup is then compared with the OTMeans approach on images, where OTMeans uses SGD and Sliced Wasserstein Distance (SWD) to make computations tractable. The authors show that the CNN with GAP behaves similarly to global patch W1, while the other approach resembles local patch W1 distances. They also provide evidence from histograms that demonstrate how GANs learn local patch statistics similar to those in the training set.", "questions": "I have a few questions and also some suggestions for further experimenents:\n\n1. Regarding the case where M=N, do all generated examples match the data examples, or are only a few of them exact matches by chance? And what GANs might behave when M>N or M>>N, e.g., do it generate by copying one training sample for more than different fixed noise inputs? Could the authors please provide quantitative results on the percentage of exact matches when M=N and test and report results for cases where M>N and M>>N? \n\n2. WGAN’s learning behavior might also depend on the ratio of M and N. The GANs bahavior could be smoother than just for the concrete cases of M<N, N=N and M>N. Do the authors have any deeper analysis on this aspect? One way to investigate e.g., could the authors conduct experiments with a range of M/N ratios and plot key metrics (e.g., FID score, exact match percentage) as a function of this ratio to visualize if any smooth transitions in behavior.\n\n3. The study is based on a small CNN with two configurations: CNN-GAP and CNN-FC. Is it not clear what method is used to regularize the Lipschitz constraint while training these models. The results might vary considerably depending on the regularization strength and hyperparameters used. If strong regularization were applied, the results could be quite different. Could the authors explicitly provide details of the Lipschitz constraint regularization method used, also the hyperparameters, and conduct an ablation study showing how different regularization strengths affect the results.\n\n4. The study focuses on the W1 distance but DCGAN used in some studies, which also shows similar patch-local distribution behavior. Does this suggest that GANs learn on batches when the discriminator is convolutional, regardless of the GAN loss function used? I’m curious why the authors didn’t use WGAN-GP, given that the study is about the W1 distance. Could the authors include experiments with WGAN-GP for direct comparison, and if possible extending the analysis to other GAN variants to test the generality of the findings across different loss functions?\n\n5. What are Direct_Patch_SWD vs. Direct_LocalPatch_SWD? Can the authors explain how SWD is computed for Direct_Patch_SWD vs. Direct_LocalPatch_SWD?\n\n6. Can the authors explain how to histograms on patches in experiments are generated?\n\n7. Are the generators identical across the two convolutional designs? What was the architecture of the generator used? Could the authors include the details of architectures used in the studies?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This is an important finding that sheds light on how WGANs learn, making a valuable contribution to the GAN research community. Overall, it is a well-written paper with interesting results.", "weaknesses": "The experiments are limited to simple CNN architectures, and there is no exploration of regularization techniques to enforce the Lipschitz constraint, which might behave differently depending on the hyperparameters used.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I5hcLgoTGw", "reviewer": "Reviewer_naBZ", "summary": "The paper tries to prove that when the discriminator is convolutional, WGANs minimize the Wasserstein distance between patches in the generated images and the training images, not the Wasserstein distance between images. Yet no solid proof is provided.", "questions": "- Please see above.\n- The paper seems unfinished since there are empty sections in the appendix.", "limitations": "", "rating": 1, "confidence": 5, "soundness": 1, "presentation": 1, "contribution": 1, "strengths": "The motivation seems to be good.", "weaknesses": "- The symbols used in the paper is chaos, what's $x$, is it the input of the generator or its output?\n- The paper is generally not well organized and hard to follow. \n- The proofs are not rigorously proven. For example, in theory 3.2, how to justify the so called $N/M$? It is wrong, for example, $M=3, N=5$, while one $x_i$ and $y_j$ overlaps to each other. To prove the theorem, I think the discrete metric need to be introduced.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zAW252P0jZ", "reviewer": "Reviewer_bEHd", "summary": "The Wasserstein GAN (WGAN) is a generative model that attempts to minimize the Wasserstein distance between the distribution of generated samples and the distribution of the training data. Several papers have pointed out that successful WGANs don't seem to minimize the Wasserstein distance. These papers further claimed that this is actually a good thing, as minimizing the Wasserstein distance would lead to blurry generated images.\nThis paper presents more rigorous experiments that support the conclusion from previous works. This is done by training discrete WGANs - models that are constrained to generate one out of M images - which allows to precisely compute the Wasserstein distance between the distribution of the GAN outputs and the empirical distribution of the training set.\nA key observation in the paper is that when the discriminator is a convolutional network, WGANs minimize the Wasserstein distance between the distribution of patches of generated images and the distribution of patches of the training images. This is while when the discriminator is fully connected, WGANs do minimize the Wasserstein distance over whole images.", "questions": "Regarding the first weakness stated above, I would be happy to hear the authors' thoughts about the following. As opposed to [2,3], which explicitly attempt to minimize a loss that is the sum of GAN losses over patches, standard GAN training applies the GAN loss after the pooling in the discriminator. In that case, the loss can be expressed as a sum of GAN losses over patches (i.e. swapped with the pooling operation) only if the loss is linear. This is the case for WGANs but not for other types of GANs. Therefore, other types of GANs with a convolutional discriminator don't directly minimize distances between patch distributions. This is while WGANs do. Is that correct?\n\nIf this statement is correct, then the paper would benefit a lot from emphasizing and discussing it. Otherwise, as stated above, it seems that the main claim in the paper is well known and not specific to WGANs (namely, it seems that all types of GANs with a convolutional discriminator minimize distances between patch distributions).", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The systematic analysis using discrete distributions is clever and makes a lot of sense.\n\n- The theoretical characterization of the solution minimizing the Wasserstein distance in the the discrete case is nice.\n\n- The paper does a pretty good job of empirically demonstrating the main claim, which is that WGANs with convolutional discriminators don't minimize the Wasserstein distance between distributions of whole images, but rather between distributions of patches.\n\n- The paper is well written. Arguments are easy to follow.", "weaknesses": "- The main takeaway message that the paper highlights is the fact that WGANs with convolutional discriminators minimize the Wasserstein distance between patch distributions. The paper doesn't provide sufficient context and discussion about why this observation is novel. As stated in the paper, Isola et al. (2017) called GANs with a convolutional discriminator \"patch GANs\", suggesting they minimize distances between patch distributions. This was also explicitly stated by Rott-Shaham et al. (2019). In fact, the origins of these GANs can be traced back to [1] which called them \"Markovian GANs\", and to [2,3] which called them \"spatial GANs\". The latter two papers explicitly expressed the loss as a sum of GAN losses over patches of the size of the receptive field, which implies that these GANs attempt to minimize distances between patch distributions and not whole images. From that standpoint it could seem to the readers that this fact is well known and also that it is not unique to WGANs, but rather applies to any GAN. Nevertheless, I believe that this might not be a material weakness, but rather a weakness in the exposition in the paper (see Questions section below).\n\n- The topic is not timely. The popularity of GANs has constantly decreased over the last few years, as diffusion models gained popularity. I consider this to be a minor weakness but I do believe that it affects the potential impact of the paper.\n\n[1] C. Li and M. Wand, \"Precomputed real-time texture synthesis with Markovian generative adversarial networks\", ECCV`16.\n\n[2] N. Jetchev, U. Bergmann, and R. Vollgraf \"Texture synthesis with spatial generative adversarial networks\", NIPS 2016 adversarial learning workshop.\n\n[3] N. Jetchev, U. Bergmann, and R. Vollgraf, \"Learning texture manifolds with the periodic spatial GAN\", ICML`17.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "34pkUEaGk1", "reviewer": "Reviewer_1sSp", "summary": "In this paper, the authors propose a framework to analyzing Wasserstein GANs, and in particular, understanding if the model truly minimizes the W1 loss while training. The insights derived help explain the visual quality of the images generated by certain GAN architectures, and also pave the way for similar analysis of other GAN variants.", "questions": "See Weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The presentation of the paper is good and the writing is easy to follow\n- The overall results derived make sense, and correlate well with the observed performance of existing patchGAN and DCGAN-style architectures. \n- The theoretical formulation is sound, to the best of my reading. The results also make intuitive sense. \n- I am also able to corroborate when I read, with the empirical evidence I’ve seen while training GANs, which, in my view, suggests that the paper succeeds at what it does", "weaknesses": "- While the analysis carried out is good, I do find it to be somewhat lacking in breadth. It might be good to see how this theory could be applied outside of the Lipschitz-constraint-based W1 loss. For example, could one draw insights into the various gradient regularization strategies that people have used to approximate W1 (i.e., the Sobolev spaces). E.g., [1,2,3] (just to name a few), where we observe very similar artifacting to the ones derived and analyzed in this paper. Would analyzing such other discriminator architectures/losses yield a more holistic view of the space of WGANs?\n\n- The ablation concerning the Convolutional GANs seemed lacking. Given such a strong correlation between the receptive field of the convolution layers and the patch-based W1 minimization, some insights into how we control or interpret this value would be useful for future design. While I understand from Section 5 that estimating S is not practical for DCGAN-style architectures, maybe toy experiments involving a single convolution layer might show links between the filter size of the convolution and S. \n\n- There have many other works that propose and analyze well-defined loss function that allow one to monitor learning algorithms and check for their convergence (See [4,5,6]). It might be worth discussing such approaches too in the related works. \n\n- Overall, I still feel that the insights developed by this paper outweigh the weaknesses mentioned, and some of them, such as generalizing to other losses could be discussed now, but more thoroughly explored in future works. I am therefore inclined towards an accept. \n\n======\n\n[1] “Banach WGANs,” Adler and Lunz,\n\n[2] “Demystifying MMD GANs,” Binkowski et al.\n\n[3] “Coulomb GANs,” Unterthiner et al.\n\n[4] “Euler-Lagrange Analysis of GANs,” Asokan and Seelamantula\n\n[5] “Sobolev GANs,” Mroueh et al. \n\n[6] “How Well Generative Adversarial Networks Learn Distributions,” Liang", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)