Kolmogorov–Smirnov GAN

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-09 12:07:28

Abstract: We propose a novel deep generative model, the Kolmogorov-Smirnov Generative Adversarial Network (KSGAN). Unlike existing approaches, KSGAN formulates the learning process as a minimization of the Kolmogorov-Smirnov (KS) distance, generalized to handle multivariate distributions. This distance is calculated using the quantile function, which acts as the critic in the adversarial training process. We formally demonstrate that minimizing the KS distance leads to the trained approximate distribution aligning with the target distribution. We propose an efficient implementation and evaluate its effectiveness through experiments. The results show that KSGAN performs on par with existing adversarial methods, exhibiting stability during training, resistance to mode dropping and collapse, and tolerance to variations in hyperparameter settings. Additionally, we review the literature on the Generalized KS test and discuss the connections between KSGAN and existing adversarial generative models.

Main paper: 
 introduction: 
 Generative modeling is about fitting a model to a target distribution, usually the data. A fundamental taxonomy of models assigns them into prescribed and implicit statistical models [9] , with partial overlap between the two classes. Prescribed models directly parameterize the distribution's probability Submitted to 38th Conference on Neural Information Processing Systems (NeurIPS 2024). Do not distribute.density function, while implicit models parameterize the generator that allows samples to be drawn from the distribution. The ultimate application of the model primarily dictates the choice between the two approaches. It does, however, have consequences regarding the available types of divergences that we can minimize when fitting the model. The divergences differ in the stability of optimization and computational efficiency, as well as statistical efficiency, which all affect the final performance of the model.The natural approach for fitting a prescribed model is maximum likelihood estimation (MLE), equivalently formulated as minimization of Kullback-Leibler divergence. Likelihood evaluation for normalized models is straightforward. In non-normalized models, density evaluation is expensive; in this context, Hyvärinen [22] proposed the score matching objective, which can be interpreted as the Fisher divergence [30] . This approach is very effective for simulation-free training of ODE [7] /SDE [42, 19] -based models which are state-of-the-art in multiple domains today.The principle driving the fitting of implicit statistical models is to push the model to generate samples that are indistinguishable from the target. An inflection point for this family of models came with the Generative Adversarial Network (GAN) [13] , which took the principle literally and introduced an auxiliary classifier trained in an adversarial process to discriminate between the two distributions.The classification error given an optimal classifier relates to the Jensen-Shannon divergence between generator and the target. Initial work in this area involved applying heuristic tricks to deal with learning problems, namely vanishing gradients, unstable training, and mode dropping or collapse.Further advancements focused on using other distances based on the principle of adversarial learning of auxiliary models, which were supposed to have certain favorable properties with respect to the original GAN.The Bayesian inference community has been reluctant to adopt adversarial methods [8] , and the attempts to apply them in this context [40] indicate a credibility problem. A significant drawback of approximate methods is the excessive reduction of diversity in the distribution [17] , the extremes of which lead to mode dropping [1] . In this work, we consider another distance for training implicit statistical models, i.e., the Kolmogorov-Smirnov (KS) distance, which, to the best of our knowledge, has not been used in this context before. The distinctive feature of the KS distance is that it directly measures the coverage discrepancy of each other's credibility regions by the distributions under analysis at all confidence levels. Thus, its minimization straightforwardly leads to the correct spread of the probability mass, avoiding mode dropping, overconfidence, and mode collapse when applied with a sufficient sampling budget.We term the proposed model as Kolmogorov-Smirnov Generative Adversarial Network (KSGAN).We show how to generalize the standard KS distance to higher dimensions based on Polonik [38] in section 2, allowing our method to be used for multidimensional distributions. Next, in section 3, we show how to efficiently leverage the distance in an adversarial training process and show formally that the proposed algorithm leads to an alignment of the approximate and target distributions. We support the theoretical findings with empirical results presented in section 6. 
generalized kolmogorov-smirnov distance: 
 We generalize the Kolmogorov-Smirnov (KS) distance (sometimes called simply Kolmogorov distance) between continuous probability distributions on one-dimensional spaces to multidimensional spaces and show that it is a metric. The test statistic of the KS test is a KS distance between empirical and target distributions (or two empirical in the case of the two-sample case). For this reason, our proposal is directly inspired by the generalization of the test introduced in Polonik [38] .Let us consider two probability measures P F and P G on a measurable space (X , A), where the sample space X is a vector space such as IR d and A is the corresponding event space;F : X → [0, 1]and G : X → [0, 1] are the cumulative distribution functions (CDFs) of P F and P G respectively. 1We say thatEQUATIONIn the multivariate case, the problem with using the KS distance as is is that on a d-dimensional space, there are 2 d -1 ways of defining a CDF. The distance has to be independent of the particular definition and thus should be the largest across all the possibilities [35] . This, however, becomes prohibitive for any d > 2. In other words, the challenge comes from a multidimensional vector space not being a partially ordered set. Everything that follows in this section consists of proposing a partial order, showing that, under certain conditions, a probability distribution can be uniquely determined on its basis and operationalizing it in an optimization problem.We begin by bringing the classical result thatEQUATIONwhere G -1 : [0, 1] → X is the inverse CDF also called the quantile function. Einmahl and Mason [10] show that there exists a natural generalization of the quantile function to multivariate distribution, which we restate below.Definition 1 (Generalized Quantile Function). Let v : A → IR + be a measure, and C ⊂ A an arbitrary subset of the event space, then a functionEQUATIONis called the generalized quantile function in C for P with respect to v2 .The generalized quantile function evaluated at level α yields a minimum-volume set [36] whose probability is at least α, and it is the smallest with respect to v such set in C, thus the name. For the remainder of this paper, we assume that v is the Lebesgue measure.It may seem that it is enough to plug C P G ,C (α) in place of G -1 (α) and P F in place of F in eq. ( 2)to establish the Generalized KS distance but it turns out that such a distance does not satisfy the positivity condition D KS (P F , P G ) > 0 if P F ̸ = P G as the example below shows.Example 1 (Polonik [38] ). Let P F be the probability measure of a chi distribution with one degree of freedom χ 2 1 which has support on IR + and P G the probability measure of a standard Gaussian distribution N (0, 1) which has support on the whole IR. Given C = A we haveEQUATIONwhile clearly P F ̸ = P G . The statement in eq. ( 4) is easy to show by observing that ∀x ∈ [0, ∞) the density of P F is twice the density of P G and C P G ,C (α) are intervals centered at 0. Instead, a solution based on the quantile functions of both distributions is needed, which we present in definition 2.Definition 2 (Generalized Kolmogorov-Smirnov distance). Let the Generalized Kolmogorov-Smirnov distance be formulated as follows:EQUATIONSuch distance is symmetric, satisfying the triangle inequality as shown in appendix A.1. For the remainder of this section, we will show that the Generalized KS distance in eq. ( 5) meets the necessary D GKS (P, P) = 0 and sufficient D GKS (P F , P G ) > 0 if P F ̸ = P G conditions to consider it a metric.In the proof, we will rely on the probability density function of P with respect to a reference measure v, which we denote with p :EQUATIONdenote the density level set of p at level λ ⩾ 0 (also called the highest density region [21] ), and letΠ p := {Γ p (λ) : λ ⩾ 0}.The following observations about level sets will introduce the fundamental tools to prove the necessary and sufficient conditions for the generalized KS distance.Remark 1 (The silhouette [37] ). For any density p, the following holdsEQUATIONwhere 1 C denotes the indicator function of a set C. The RHS of eq. ( 7) is called the silhouette.An immediate consequence of remark 1 is that Π p ordered with respect to λ ⩾ 0 fully characterizes P, because p does. Graphically, the silhouette is a multidimensional stack of level sets.Remark 2. Density level sets are minimum-volume sets [38] The quantity P(C)λ v(C) is maximized over A by Γ p (λ), and thus if Γ p (λ) ∈ C, then Γ p (λ) = C P,C (α)3 at levelα = P(Γ p (λ)) = p(x)1 [λ,∞) (p(x))dx.Below, we present the fundamental theoretical result behind the proposed method, which restates Lemma 1.2. of Polonik [38] .Theorem 1 (Necessary and sufficient conditions). Let v be a measure on (X , A). Suppose that P F and P G are probability measures on (X , A) with densities (with reference measure v) f and g respectively. Assuming thatA.1 Π f ∪ Π g ⊂ C;A.2 C P F ,C (α) and C P G ,C (α) are uniquely determined 4 in C with respect to v the following two statements are equivalent:S.1 P F = P G ; S.2 D GKS (P F , P G ) = 0.See proof in Appendix A .Meeting assumption A.1 is a demanding challenge, almost equivalent to learning the target distribution.Below, we propose a relaxation of it, which we will use to show the validity of our method.Theorem 2 (Relaxation of assumption A.1). Theorem 1 holds if assumption A.1 is relaxed to the case that C contains sets that are uniquely determined with density level sets of P F and P G up to a set C such thatEQUATIONand let r := P F (C) = P G (C), then the supremum in statement S.2 is restricted to [0, 1r].See proof in Appendix A . 
kolmogorov-smirnov gan: 
 For the remainder of the paper, we will consider P F as the target distribution represented by a dataset {x F }, and P G as the approximate distribution that we want to train by minimizing the Generalized KS distance in eq. ( 5) with Stochastic Gradient Descent. We model P G as a pushforward g θ # P Z of a simple (e.g., Gaussian, or Uniform) latent distribution P Z supported on Z, with a neural network g θ : Z → X , parameterized with θ, which we call the generator.The major challenge in utilizing eq. ( 5) is the necessity of finding the C P,C (α) terms which is an optimization problem on its own. The idea that we propose in this work is to amortize the procedure by modeling the generalized quantile functions C P F ,C (α) and C P G ,C (α) with additional neural networks which have to be trained in parallel to the generator g θ . Therefore, our method is based on adversarial training [13] , where optimization proceeds in alternating phases of minimization and maximization for different sets of parameters. Hence the name of the proposed method, the Kolmogorov-Smirnov Generative Adversarial Network.The generalized quantile function defined in definition 1 is an infinite-dimensional vector functionC P,C : [0, 1] → C ∈ C.Such objects do not have an expressive, explicit representation that allows for gradient-based optimization. Therefore, we use an implicit representation inspired by density level sets in eq. ( 6). We propose to use neural level sets defined in definition 3 that are modeled by a neural network c : X → IR, which we will refer to as the critic.Definition 3 (Neural level set). Given a neural network c : X → IR, the neural level set at level λ is defined as 5Γ c (λ) := {x : c(x) ⩽ λ}, and let Π c := {Γ c (λ) : λ ∈ IR}.Neural level sets are used, for example, in image segmentation [6, 20] and surface reconstruction from point clouds [3] . They fit our application because for computing the Generalized KS distance in eq. ( 5), the explicit materialization of generalized quantiles is not required as long as the probability measure can be efficiently evaluated on the implicitly specified sets. We set C = Π c , and thus C P,Πc (α) = Γ c (λ α ), with λ α = arg min λ∈I R {λ : P(Γ c (λ)) ⩾ α}. For a probability measure P ′ the following holds:EQUATIONwhich shows that the terms in eq. ( 5) under neural level sets can be Monte-Carlo estimated given samples from the respective distributions. Assumption A.2 is satisfied by neural level sets by construction.The formulation of the Generalized KS distance in eq. ( 5) includes two generalized quantile functions C P F ,C (α) corresponding to target distribution P F and C P G ,C (α) corresponding to the approximate distribution P G . Both have to be modeled with the respective neural networks c ϕ F and c ϕ G , where we use ϕ = {ϕ F , ϕ G } to denote the joint set of their parameters. In section 3.3, we show how to parameterize both critics with a single neural network. We setC = Π c ϕ F ∪ Π c ϕ G .The Generalized KS distance in eq. ( 5) is a supremum over a unit interval and two functions; thus, it can be upper-bounded asEQUATIONNext, we plug in 11) and use eq. ( 10) to get generator's objective:C = Π c ϕ F ∪ Π c ϕ G to eq. (EQUATIONIn practice, the expectations in eq. ( 12) are estimated on finite samples from the two distributions, i.e. {x F } mentioned before, and {x G } sampled from the approximate distribution P G using the reparametrization trick to facilitate backpropagation of gradients. Therefore, the two terms become step functions in λ, and the supremum is located on one of the steps. That way, a line search on IR reduces to a maximum over a finite set. To preserve the differentiability of the cost function calculated in this way, we apply Straight-through Estimator [4] in place of indication function 1. A schematic depiction of the process for a single critic is shown in fig. 1 .By optimizing critics' parameters ϕ, we want to satisfy assumption A.1 so that Generalized KS distance becomes a metric. For the problem posed in such a way, we lack supervision, i.e., we do not know the target sets' shapes. However, we can reformulate the problem as an estimation of the density functions of the two considered measures P F and P G and use the obtained approximate density models to build level sets. We can constitute an optimization problem for such a task based solely on finite sets of samples, which we have for P F and can arbitrarily generate from P G . As the estimator, we propose to use the Energy-based model (EBM) [43] , which, thanks to the lack of constraints in the choice of architecture, can be very expressive while having favorable computational complexity at inference. To carry out EMB training effectively, we will introduce a new min-max game, the "min phase" of which will turn out to be the initial objective in eq. ( 5), and in this way, we will close the adversarial cycle.Let the critic c ϕ F (x) serve as the energy function. The density given by the EBM is then p c ϕ F (x) = exp(-c ϕ F (x))/Z c ϕ F , where Z c ϕ F = exp(-c ϕ F (x))dxEQUATIONUpdate ϕ by using ∂(Lc-βRc) ∂ϕ to maximize L c -βR c ;7 for i = 1 to k θ do 8Draw batch {x} ∼ P F and {z} ∼ P Z ; // generator's inner loop9 {c F } ← {c ϕ (x) : {x}} and {c G } ← {c ϕ (g θ (z)) : {z}}; 10 {λ} ← {c F } ∪ {c G }; 11 L g,F ← max {λ} 1 |{z}| {c G } 1 (-∞,λ] (c G ) -1 |{x}| {c F } 1 (-∞,λ] (c F ) ; 12 L g,G ← max {λ} 1 |{x}| {c F } 1 (-∞,-λ] (-c F ) -1 |{z}| {c G } 1 (-∞,-λ] (-c G ) ; 13 L g ← L g,F + L g,G ;14 Update θ by using ∂Lg ∂θ to minimize L g ; 15 until not converged; 16 return g θ # P Z function. The standard technique for learning the model given target data distribution P F is MLE, where the likelihoodEQUATIONis maximized wrt ϕ F . An unbiased estimate of the gradient of the second term can be obtained with samples from the EBM itself, typically achieved with MCMC sampling. Many approaches to avoid this expensive procedure have been described in the literature [43] , and among them, the one based on adversarial training [23] is the most appealing to us. It introduces an auxiliary distribution P aux(F ) , such that the gradient of eq. ( 13) wrt ϕ F is approximated with the gradient ofEQUATIONConsequently, an additional objective L aux(F ) must be introduced, the optimization of which will lead to the alignment of P aux(F ) and P c ϕ F , where P c ϕ F denotes the probability distribution with density p c ϕ F (x). We take an analogous approach to estimate c ϕ G (x).When we (i) set c ϕ G (x) := -c ϕ F (x), and (ii) repurpose P G as P aux(F ) and P F as P aux(G) , we show in appendix A.2 that the MLE objectives for the critics -now, denoted as c ϕ -simplify asL c = E x∼P G [c ϕ (x)] -E x∼P F [c ϕ (x)],which is then maximized in an adversarial game against the Generalized KS distance in eq. ( 5).The standard approach for aligning the auxiliary distributions with their targets is to use the Kullback-Leibler divergence. We propose using the Generalized KS distance instead. We set L aux(F ) = D GKS P G , P c ϕ and L aux(P G ) = D GKS P F , P -c ϕ . By analyzing these objectives in the fashion of section 3.2, we note that L aux(P G ) is the same as our original objective D GKS (P F , P G ) -which is symmetric -when we approximate sampling from P c ϕ with the target distribution P F . Analogously for L aux(P G ) where sampling from P -c ϕ is approximated with P G . Therefore, we have shown that the auxiliary objectives are already integrated into the adversarial game.In practice, we find the score penalty regularizer of Kumar et al. [26] , derived from the score matching objective, helpful to stabilize training. Therefore, we subtract it from L c weighted by a hyperparameter β. In this way, we get a critic that is smoother and, therefore, generates regular level sets that facilitate optimization. We summarize the proposed training procedure in algorithm 1.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes] Justification: All assumptions are explicitly mentioned, we believe our proofs are correct.In addition, in the theoretical part of our work, we rely on previously published results by other authors, which we always cite as a reference.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.• All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.• Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.• Theorems and Lemmas that the proof relies upon should be properly referenced. 
discussion: 
 In section 3.3, where we justify the choice of the critic's objective function, we refer to methods for training EBMs, which are approximate density distribution models. Thus, the reader can expect that our proposed critic c ϕ in the limit of convergence of the algorithm will become a source of information about the density distribution of the target distribution P F accompanying the model that generates samples P G . However, this does not happen as a consequence of the design choice (i), that is, the setup of c ϕ F = -c ϕ G = c ϕ . An EBM can only be equivalent to its inverse in the case of a uniform distribution. In addition, because of design choice (ii), during training, the critic is not evaluated outside of the support of P F and P G and, therefore, can reach arbitrary values there.Despite these observations, the Generalized KS distance present in our algorithm exposes sufficient conditions because of theorem 2.The feature distinguishing KSGAN from other adversarial generative modeling approaches is that regardless of the outcome of the critic's inner problem, minimizing eq. ( 5) is justified because Generalized KS distance, despite not meeting assumption A.1, is a pseudo-metric [38] . For comparison, the dual representation of Wasserstein distance, used in WGAN [2] requires attaining the supremum in the inner problem.The distances used for training generative models all fall into either the category of f -divergencesD f (P F , P G ) = A f (dP F /dP G ) dP G or integral probability metrics (IPMs) D F (P F , P G ) = sup f ∈F |E x∼P F f (x) -E x∼P G f (x)|. The classical one-dimensional KS distance is an instance of IPM with F = {1 (-∞,t] |t ∈ IR} or F = {1 G -1 (α) |α ∈ [0, 1]} when having access to the inverse CDF of one of the distributions based on eq. ( 2). One can see the Generalized KS distance from the perspective of IPM withF = {1 C(α) |α ∈ [0, 1] & C ∈ {C P F ,C , C P G ,C }}. Assuming direct access to C P F ,Cand C P G ,C , for example when both P F and P G are Normalizing Flows [24, 34] , measuring the distance comes down to a line search.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: Experiments (including the evaluation protocol) are described in detail in the main text and completed with more information in the appendix. In addition, we include a link to the repository containing the code that was used to conduct the experiments.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.• If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.• Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.• While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
related work: 
 The need to generalize the KS test, and therefore distance, to multiple dimensions arose naturally from the side of practitioners who collected such data and wished to test related hypotheses. It was first addressed by Peacock [35] , where a two-dimensional test for applications in astronomy was proposed. It involves considering all possible orders in this space and using the one that maximizes the distance between the distributions. A modification of this procedure has been proposed by Fasanoand Franceschini [11] where only four candidate CDFs have to be considered, causing the test to be applicable in three dimensions, with eight candidates, under similar computational constraints.Chronologically, the following approach was the one on which we base our work, proposed in Polonik [38] but made possible by the author's earlier work [36, 37] . To the best of our knowledge, the first work that practically uses the theory developed by Polonik is Glazer et al. [12] , which we recommend as an introduction to our work. It proposes applying the Generalized KS test based on the support vector machines for detecting distribution shifts in data streams.As an instance of the adversarial generative modeling family, our work is related to all the countless GAN [13] follow-ups. We highlight those that study the learning process from the perspective of the distance being minimized. The work of Arjovsky and Bottou [1] provides a formal analysis of the heuristic tricks used for stabilizing the training of GANs. The f -GAN [33] proposes a unified training framework targeting f -divergences, which relies on a variational lower bound of the objective that results in the adversarial process. Approaches relying on the integral probability metric include FisherGAN [32] , the Generative Moment Matching Networks [29] based on MMD, just like the later, more sophisticated MMD GAN [28] , and finally the Wasserstein GAN (WGAN) [2] with the WGAN-GP follow-up [16] which shares common features with our work. Our maximum likelihood approach to fitting the critic results in the same functional form of the loss as WGAN(-GP) uses. In addition, the score penalty we use is similar to the gradient penalty of WGAN-GP.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes]Justification: We include a link to the repository containing the code that was used to conduct the experiments. We use only publicly available data.Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details.• While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).• The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more details.• The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.• The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.• At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).• Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experiments: 
 We evaluate the proposed method on eight synthetic 2D distributions (see appendix B.1 for details) and two image datasets, i.e. MNIST [27] and CIFAR-10 [25] . We compare against other adversarial In all KSGAN experiments, we relax the maximum in line 11 and line 12 of algorithm 1 with sample average. In all experiments, we re-use the last batch of samples from the latent distribution (and target distribution in the case of KSGAN) from the critic's optimization inner loop as the first batch for the generator's optimization inner loop.Analyzing adversarial methods on synthetic, low-dimensional distributions is not popular. However, we conduct such an experiment because we are interested in whether the model generates samples from the support of the target distribution and how accurately it approximates the distribution.Working with small-dimensional distributions, we do not have to be as concerned about the curse of dimensionality when calculating sample-based distances, and we can visually compare the resulting histograms.In table 1, we report the squared population MMD [15] between target and approximate distributions, computed with Gaussian kernel on 65536 samples from each distribution. Details about how we chose the kernel's bandwidth can be found in appendix B.1. GAN and WGAN-GP fail to converge with k ϕ = k θ = 1 (we do not report the results to economize on space); thus, we set k θ = 5 for them.The proposed KSGAN with k θ = 1 performs at a similar level to WGAN-GP, the better of the two former, despite using five times less training budget. We present additional results on the synthetic datasets in appendix D.1, which include performance with different training dataset sizes, non-default hyper-parameter setups for KSGAN, and histograms of the samples for qualitative comparison.We use the 50000 training instances to train the models, and based on visual inspection of the generated samples (reported in appendix D.2), we conclude that all the methods achieve comparable, high samples quality. To assess the quality of the distribution approximation, we use a pre-trained classifier on the same data as the generative models (details in appendix B.2). We run the same experiment on 3StackedMNIST [44] , which has 1000 modes. We report the results in table 2.In this experiment, we set the training budget for all methods to k ϕ = 1, k θ = 1 for a fair comparison.We find that all methods always recover all the modes with the standard MNIST target. However, GAN fails to distribute the probability mass uniformly between the digits. As the number of modes increases with the 3StackedMNIST target, GAN demonstrates its inferiority to other methods by losing 198 modes on average (four initialization cover approx. 985 modes, and one fails to converge, achieving only 98 modes). WGAN-GP and KSGAN consistently recover all the modes while being on par regarding KL divergence, which differs little between networks' initialization.Table 2 : The number of captured modes and Kullback-Leibler divergence between the distribution of sampled digits and target uniform distribution averaged over five random initializations with the standard deviation calculated with Bessel's correction in the parentheses. All the methods were trained with the same budget k ϕ = 1, k θ = 1. WGAN-GP and KSGAN cover all the modes in all experiments while demonstrating low KL divergence.Method (k ϕ , k θ ) # modes ↑ KL ↓ # modes ↑ KL ↓ GAN (1,We use the 50000 training instances to train the models and report the generated samples in appendix D.3. We train the models in a fully unconditional manner, i.e., not using the class information at all -contrary to many unconditional models that use class information in normalization layers.We quantify the quality of fitted models by computing the Inception Score (IS) [41] and Fréchet inception distance (FID) [18] from the test set and report the results in table 3 based on five random initializations. For reference, in the table, we include the IS of the training dataset and the FID between the training and test sets.In this experiment, we set the training budget for all methods to k ϕ = 1, k θ = 1 for a fair comparison.All models fail to accurately approximate the target distribution, which is evident from a quantitative comparison in table 3 and a qualitative one in appendix D.3. KSGAN is characterized by the lowest variance between initializations among the methods considered.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: Experiments (including the evaluation protocol) are described in detail in the main text and completed with more information in the appendix. In addition, we include a link to the repository containing the code that was used to conduct the experiments.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.• The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusions and future work: 
 In this work, we investigated the use of Generalized Kolmogorov-Smirnov distance for training deep implicit statistical models, i.e., generative networks. We proposed an efficient way to compute the distance and termed the resulting model Kolmogorov-Smirnov Generative Adversarial Network because it uses adversarial learning. Based on the empirical evaluation of the proposed model, the results of which we report, we conclude that it can be considered as an alternative to existing models in its class. At the same time, we point out that many properties of KSGAN have not been studied, and we leave this as a future work direction.Interesting aspects to explore are the characteristics of learning dynamics with the number of generator updates exceeding the number of critic updates, alternative ways to train the critic, and alternative representations of generalized quantile sets. The natural scaling of the Generalized KS distance may also prove beneficial regarding the interpretability of learning curves, learning rate scheduling, or early stopping. In addition, we hope that our work will draw the attention of the machine learning community to the Generalized KS distance, applications of which remain to be explored.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: All the evaluation metrics were computed on five random initializations. We report the average scores with standard deviation computed with Bessel's correction.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: In the appendix, we include information about the resources needed to reproduce results reported in the paper, and give an estimate of resources spent on preliminary experiments not reported in the paper.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: None of the datasets used in the paper has been deprecated. We do not identify any concerns regarding societal impact and potential harmful consequences of our work.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: Our work is foundational research, and thus does not have a direct positive or negative societal impact. We disclaim responsibility for the malicious use of our work.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]  
: 
 the quantile function, which acts as the critic in the adversarial training process.We formally demonstrate that minimizing the KS distance leads to the trained approximate distribution aligning with the target distribution. We propose an efficient implementation and evaluate its effectiveness through experiments. The results show that KSGAN performs on par with existing adversarial methods, exhibiting stability during training, resistance to mode dropping and collapse, and tolerance to variations in hyperparameter settings. Additionally, we review the literature on the Generalized KS test and discuss the connections between KSGAN and existing adversarial generative models. The λ threshold moves from -∞ to +∞ establishing a stack of level sets. At each level, the fraction of datapoints (• and •) below the threshold is calculated for each distribution independently. This produces the P F Γ c ϕ (λ) and P G Γ c ϕ (λ) curves. The Generalized KS distance is the largest absolute difference between the curves shown as ↕ ↕ ↕ in the right figure. Best viewed in color. 
 
 Appendix: Theorem 1 (Necessary and sufficient conditions). Let v be a measure on (X , A). Suppose that P F and P G are probability measures on (X , A) with densities (with reference measure v) f and g respectively. Assuming thatA.1 Π f ∪ Π g ⊂ C;A.2 C P F ,C (α) and C P G ,C (α) are uniquely determined6 in C with respect to v the following two statements are equivalent:S.1 P F = P G ; S.2 D GKS (P F , P G ) = 0.Proof of Theorem 1. The S.1 =⇒ S.2 direction is trivial to show and works without satisfying the assumptions [38] . Therefore, we focus on showing that S.2 =⇒ S.1. LetEQUATIONand denote with Γ(λ) the level set of density of P as defined in eq. ( 6), and let Π := {Γ(λ) : λ ⩾ 0}.Further, let SC denote the least concave majorant [5] to S C (P), that is, the smallest concave function there is a level λ for which C = Γ(λ), and it is equal the left-hand derivative of SC in the point v(C).From remark 1, we have that the silhouette fully characterizes P, and therefore ∂ SC (P) does it as well.Eventually, we conclude the proof with the observation that given S.2, under Lemma 2.1 of Polonik [38] (where A.2 is utilized) we have that the extremal points of the convex hulls of S C (P F ) and S C (P G ) are the same points, thus ∂ SP F (P) = ∂ SP G (P), and finally P F = P G .Theorem 2 (Relaxation of assumption A.1). Theorem 1 holds if assumption A.1 is relaxed to the case that C contains sets that are uniquely determined with density level sets of P F and P G up to a set C such thatEQUATIONand let r := P F (C) = P G (C), then the supremum in statement S.2 is restricted to [0, 1 -r].Proof of Theorem 2. The statement in eq. ( 8) is equivalent to saying that P F = P G on (C, 2 C ).Analogously to the proof of theorem 1 we can show that P F = P G on (X \ C, 2 X \C ). By observing that probability measures are σ-additive, we conclude that P F = P G on (X , A), and thus the result of theorem 1 holds.Let us consider three probability measures P F , P G , and P H on a measurable space (X , A).D GKS (P F , P H ) + D GKS (P H , P G ) = sup α∈[0,1] C∈{C P F ,C ,C P H ,C } [|P F (C(α)) -P H (C(α))|] + sup α∈[0,1] C∈{C P H ,C ,C P G ,C } [|P H (C(α)) -P G (C(α))|] (i) = sup α∈[0,1] C∈{C P F ,C ,C P H ,C ,C P G ,C } [|P F (C(α)) -P H (C(α))|] + sup α∈[0,1] C∈{C P H ,C ,C P G ,C ,C P F ,C } [|P H (C(α)) -P G (C(α))|] = sup α∈[0,1] C∈{C P F ,C ,C P H ,C ,C P G ,C } [|P F (C(α)) -P H (C(α))|] + [|P H (C(α)) -P G (C(α))|] (ii) ⩾ sup α∈[0,1] C∈{C P F ,C ,C P H ,C ,C P G ,C } [|P F (C(α)) -P G (C(α))|] = sup α∈[0,1] C∈{C P G ,C ,C P F ,C } [|P F (C(α)) -P G (C(α))|] = D GKS (P F , P G )In (i), we use the fact that the supremum of absolute difference in distribution coverage is maximized with the generalized quantile function of one of them. In (ii), we apply triangle inequality for absolute value. Thus we have shown that D GKS (P F , P H ) + D GKS (P H , P G ) ⩾ D GKS (P F , P G ) which is the triangle inequality for the Generalized KS distance.Given two adversarial maximum likelihood objectives from Kim and Bengio [23] , we (i) set c ϕ G (x) := -c ϕ F (x), and (ii) repurpose P G as P aux(F ) and P F as P aux(G) , and show that:1 2 (E x∼P F [-c ϕ F (x)] -E x∼P aux(F ) [-c ϕ F (x)]) + 1 2 (E x∼P G [-c ϕ G (x)] -E x∼P aux(G) [-c ϕ G (x)]) = 1 2 (E x∼P F [-c ϕ (x)] -E x∼P G [-c ϕ (x)] + E x∼P G [c ϕ (x)] -E x∼P F [c ϕ (x)]) = E x∼P G [c ϕ (x)] -E x∼P F [c ϕ (x)].In this section, we provide additional details about experiments conducted in the paper that did not fit in the main text. All the models reported in the paper were trained under 12 hours on a single Nvidia GeForce GTX TITAN X GPU (12GB vRAM) with 32GB of RAM and 2 CPU cores. We report results based on 645 models trained, which amounts to 7740 GPU hours at most. We estimate that about three times as much computing time was used for preliminary experiments not reported in the paper.The synthetic 2D distributions are adopted from the official code of Grathwohl et al. [14] https: //github.com/rtqichen/ffjord. We randomly generate 65536 training and 65536 test instances from each distribution. In appendix D.1, we report the results of training the models with fewer instances but evaluated using the entire test set.We choose the bandwidth of the Gaussian filter in squared population MMD as the median L2 distance between two samples, of 32768 instances each, from the simulator. The resulting values can be found in the code we provide with the paper. To detect the modes in the (3Stacked)MNIST experiments, we use a pre-trained classifier from PyTorch examples, trained for 14 epochs of the train set of the original MNIST dataset. We expect to find 10 and 1000 modes for the MNIST and 3StackedMNIST, respectively. We measure the KL divergence between the classifier's output and discrete uniform distribution for both distributions.We compute the Inception Score using the implementation from https://github.com/sbarratt/ inception-score-pytorch. We compute the Fréchet inception distance using the implementation from https://github.com/mseitzer/pytorch-fid.For all of the methods and distributions, we use the same architecture, described in table 4 , with spectral normalization [31] on linear layers for GAN. In all cases, we train the generator and critic with Adam(β 1 = 0.5, β 2 = 0.9) optimizer with a constant learning rate of 0.0001, without L2 regularization or weight decay, for 128000 generator updates with batch size equal to 512. We use the standard loss for GAN, enforcing class 1 for real samples and 0 for generated samples. In WGAN-GP, we use 0.1 weight on gradient penalty (identified as a good value in preliminary experiments, which we do not report), and in KSGAN β = 1.0 as the weight for score penalty.For the MNIST experiments, we use the DCGAN [39] architecture, without batch normalization layers, with 128-dimensional latent Gaussian distribution. For the 3StackedMNIST distribution, we increase the number of input and output channels for the critic and generator, respectively. We train the generator and critic with Adam(β 1 = 0.5, β 2 = 0.9) optimizer with a constant learning rate of 0.0001, without L2 regularization or weight decay, for 200000 generator updates with batch size equal to 50. In the case of GAN for 3StackedMNIST, we use a learning rate of 0.001 (identified as a good value in preliminary experiments, which we do not report). We use the flipped loss for GAN, enforcing class 0 for real samples and 1 for generated samples. In WGAN-GP, we use 10.0 weight on gradient penalty (identified as a good value in preliminary experiments, which we do not report), and in KSGAN β = 1.0 as the weight for score penalty. For the CIFAR-10 experiments, we use ResNet architecture from Gulrajani et al. [16] . We train the generator and critic with Adam(β 1 = 0.0, β 2 = 0.9) optimizer with a constant learning rate of 0.0001, without L2 regularization or weight decay, for 199936 generator updates with batch size equal to 64. We use the flipped loss for GAN, enforcing class 0 for real samples and 1 for generated samples. In WGAN-GP, we use 10.0 weight on gradient penalty (identified as a good value in preliminary experiments, which we do not report), and in KSGAN β = 1.0 as the weight for score penalty.In this section, we report additional experiment results that did not fit in the main text. This includes materials allowing a qualitative comparison of the trained models.In fig. 2 , we report, extended relative to table 2 in the main text, a study of the quality of trained models as measured by the squared population MMD. Solid lines denote the average over five random initializations, and the shaded area represents the two-σ interval. KSGAN performs on par with WGAN-GP while being trained with a five times less training budget. In fig. 3 , we show the histograms of 65536 samples from the models (a single random initialization), with a histogram of test data in the first column for reference. For KSGAN, in addition to the configurations included in table 2, we include one with a training budget matching that of GAN and WGAN-GP, and one with a training budget reduced by two, where the critic is updated only every second update of the generator.In fig. 4 , we show samples from one of the random initializations reported in table 2 in the main text.All models demonstrate similar sample quality, while for GAN, the digit "1" is over-represented, which corresponds with the high KL in table 2. In fig. 5 , we show samples from one of the random initializations reported in table 3 in the main text.All models demonstrate similar, low sample quality.play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.In what follows we will use PF for the true data distribution and PG for the learnt oneIn the general case, CP,C(α) at any given level α is not uniquely determined, i.e. there may exist several sets C, C ′ ∈ C s.t. C ̸ = C ′ that satisfy the condition in eq. (3). For simplicity, we will call all such sets the (generalized) quantile sets at level α and write CP,C(α) = C and CP,C(α) = C ′ for all of them.There may be other sets C = CP,C(α) but Γp(λ) will certainly be one of them.In the sense defined in Polonik[38]Please note that the direction of the inequality in eq. (9) is opposite of the one in eq. (6) which is a convention that aligns the critic with the energy function of Energy-Based models.In the sense defined in Polonik[38] 


Decision: The paper proposes a deep generative model to train a GAN using the Kolmogorov-Smirnov (KS) distance as opposed to other distances like IPM.

After the rebuttal and discussion periods, 3 out of the 4 reviewers find that the paper is under the acceptance bar. The main criticism is that there is no clear incentive to use the KS distance in this case (its experimental performance isn't superior and there aren't clear advantages supporting this choice). Moreover, the experimental results are also somewhat limited (cf. Reviewer BXzn).

The authors argue that theoretical works are also welcome at NeurIPS. This is certainly the case, however it is not clear whether this paper should be judged as a theoretical contribution when its main contribution is to introduce an algorithm. Quoting Reviewer L1Da: "a good theory paper usually solves or makes progress towards a problem that is of interest to the community (for instance, improving on a learning bound on a known scenario, or making a connection that could draw tools from other fields to solve something). I don't see that case happening here [...] I don't really see here that the goal is to understand something better (nor how this paper would do such a thing), as theory papers are usually meant to do".

Reviews: review: paper_summary: paper_summary: The paper proposed a new kind of GAN training method called KS-GAN. The method is based on minimizing the Kolmogorov–Smirnov distance. The KSGAN updates the generator by minimizing an upper bound of the generalized KS distance. It updates the discriminator (or the critic network) by using energy-based model training with regularization terms. The KSGAN is a novel attempt to explore new approaches to train generative adversarial networks.
strength_weakness: strength_weakness: My main concern about the paper is its weak evaluation baselines and questionable practical usage:

* (1) Though the idea of using new objectives for training GAN generators is attractive, the practical usage of KSGAN seems questionable, especially for high-dimensional data. For instance, in the CIFAR10 generation experiment, the author compares KSGAN with WGAN-GP and Vanilla GAN, which have shown weak empirical performances. However, it is well-known that, for CIFAR10 data, the StyleGAN2-ADA[1] model is a strong baseline GAN model. I think it would strengthen the paper a lot if the authors could somehow show strong performances of KSGAN using StyleGAN2's architectures and implementation techniques. However, I do admit that such a requirement may be too tough for new methods.

* (2) The KSGAN's critic function is constructed with EBMs. However, even with regularization terms, energy-based models are well-known for poor scaling ability to high-dimensional data. This may prevent the practical usage of KSGAN for real-world high-dimensional data.
questions: questions: Please see the weakness part.
limitations: limitations: The author has addressed the limitations.
paper_summary: paper_summary: The authors introduce a generalized KS distance applicable to high dimensional spaces, formulate the corresponding dual problem, and use adversarial training to construct a generative model that minimizes the GKS between data and generated distributions.

The paper is well presented and appears technically correct through what I've seen, though I didn't check the proofs in details.

The main problem is that there's no clear motivation for why using the GKS is beneficial at all (either theoretically or practically). As such, despite being novel, I don't see any clear impact from the paper. Furthermore, the final algorithm is quite complicated and the results are fairly underwhelming, so at the end of the day the cons dramatically outweigh the pros of the newly introduced algorithm.
strength_weakness: strength_weakness: The main problem I have with the paper is that I don't see any clear advantages of using the KS distance (gan) as a replacement of other distances like the Wasserstein one, or their GAN equivalent. The only mention of this, which should arguably be the most important thing in a paper introducing a new GAN, is in lines 224-228 of page 7. The authors claim there that they don't need to maximize the supremum in (5) which is false depending on how to interpret it, if you just take any set C in (5) you end up with |P_F(C) - P_G(C)| which is just measuring one moment for a given characteristic function, and far from being anything meaningful (and the same holding true for most IPMs). The results are also not particularly interesting to merit the claim that there's anything particularly different or benefitial on using this new formulation.
questions: questions: The main suggestions I have is the following:
1) Take some time thinking why using the GKS is a better idea than using other distances between probability distributions.
2) Validate these claims with targeted experiments (for instance, on tractable distributions, without adversarial training).
3) Show that these properties translate to the adversarial setup, either theoretically or with experiments.

At the end of the day you're trying to convince readers that what you did is worth trying, so instead of focusing so much on mathematical details, consider more why a busy reader should care about this problem in the first place. What are the fundamental limitations of existing techniques you're trying to adress?
limitations: limitations: No clear motivation or benefit from using their algorithm.
paper_summary: paper_summary: This paper proposes a novel variant of the generative adversarial network that uses the Kolmogorov-Smirnov distance to align the generated distribution with the target distribution. This distance is calculated using the quantile function, which acts as the critic in the adversarial training process. Experiments are conducted on synthetic distributions and small image datasets to show that the proposed KSGAN performs on par with the existing adversarial methods.
strength_weakness: strength_weakness: 1. It is still unclear why we need another adversarial design based on KS distance. The vanilla GAN paper shows that the designed bi-level optimization process can already be seen as optimizing the distance between the generated and the target distribution. Then, what are the specific advantages KS distance can bring within the adversarial framework?

2. The experiments are merely conducted on synthetic datasets and small image datasets. It is unclear whether the proposed method can be adapted to larger-scale datasets or incorporated into more advanced frameworks like StyleGAN. Moreover, the compared baselines are limited to early works, and the experimental results of KSGAN are worse than those of WGAN-GP. Thus, I do not see many advantages of KSGAN in terms of the presented experiments.
questions: questions: 1. Could the authors explain or provide more evidence about line 40, "The Bayesian inference community has been reluctant to adopt adversarial methods"?

2. How can we guarantee the optimality of the learned quantile functions such that the estimated KS distance is reliable?

3. Again, with adversarial training, the original gan is already optimizing the distance between the two distributions. What are the benefits of optimizing in this way by KSGAN?

4. In Table 3, the performance of the models with IS around 2.4 and FID around 40 does not seem very successful in generative training. And the improvements seem minor.
limitations: limitations: Please see the discussions above.
paper_summary: paper_summary: This paper first generalizes the Kolmogorov–Smirnov (KS) distance from one-dimensional spaces to multidimensional spaces and proposes the Kolmogorov-Smirnov GAN, which formulates the generative model by minimizing the Kolmogorov-Smirnov (KS) distance. Theoretical results are also given in this paper and the experiments also show the superiority of stability during training, resistance to mode dropping and collapse, and tolerance to variations in hyperparameter settings.
strength_weakness: strength_weakness: 1. I'm skeptical about some parts of the theory. (See questions for details)
2. It seems that the advantages of KS distance over JS divergence and Wasserstein Distance are not explained.
3. The idea of reformulating a distance between distributions to a GAN model seems to be old and is now unlikely to attract readers' interest.
4. The experimental setup is relatively simple, only comparing with vanilla GAN and WGAN on Synthetic, MNIST, and CIFAR10 datasets
5. According to the experimental results, the advantages of KSGAN lie in the stability of training and resistance to mode dropping. A significant issue is that with current network architectures and training techniques, these two problems are rarely encountered.
questions: questions: Q1. In Line 66-67, what's the meaning there are $2^d-1$ ways of defining a CDF on a d-dimensional space?

Q2. Definition 1 seems strange. What is the measure v? Is it a determined measure or any arbitrary measure? Is P the known CDF in Eq.3? What is its relationship with v?

Q3. Based on the definition of mapping, given function f: A->B, for each element a in A, there is always a unique element b in B corresponding to it. Then for $G^{-1}(\alpha)$ in Eq.2 and $C_{P,C}(\alpha)$, how is the uniqueness of the set guaranteed given $\alpha\in [0,1]$?
limitations: limitations: The authors have discussed the limitations.

Comments: title: Regarding comparisons and performance
Thanks to the authors' rebuttal.
comment: A few more details:

0) I did not suggest establishing learning bounds. Learning bounds can often be beside the point unless they are substantially different from one algorithm to another (or from one bound to another) in a way that's illuminating. If you do decide to go that route, try to think exactly where that difference may come from and why it's important. 
1) My point of arguing why using KS instead of other IPMs is that if you introduce a new algorithm, in order for that to be a significant contribution, it's important to argue well why that algorithm has benefits that other algorithms do not.
2) I do understand that, though as other reviewers suggested, perhaps the evidence to that is not enough at this point.
We want to thank the reviewer for this comment. We will work on providing results of using the proposed KS distance in advanced SOTA training pipelines like StyleGAN for future revisions of the papers.
We want to thank the reviewer for this comment. We will try to establish learning bounds for GAN, WGAN-GP, and the proposed KSGAN for future paper revisions. As well as study other theoretical properties of the aforementioned methods. 

However, there are two things that we want to clarify:
1. We do not suggest using KS instead of IPMs. We show that KS is an IPM.
2. In terms of empirical results we do show improvements over the existing methods in terms of performance or by reducing computational demand.
Thank you for the response. Unfortunately, my concerns remain unaddressed. The authors claim that KSGAN can better approximate the target distribution based on experimental results. However, the results presented are basic and limited. Evaluating only on a basic dataset is quite restrictive, and I have not observed any significant performance improvement over state-of-the-art models. For real-world datasets, the authors only compared GAN and WGAN on CIFAR-10, but these methods are now somehow outdated. In the context of the current development of generative models, it is essential for papers in leading conferences to include SOTA algorithms and multiple large-scale datasets in their experiments. Regrettably, I cannot raise the score for this paper at this time.
Thanks for your rebuttal. I am still concerned about the weaknesses listed in my original comment and thus I maintain my score.
One comment I want to make is that personally I don't think the main drawback is the lack of better experimental performance. In fact, I definitely welcome theoretical papers. However, I don't think theory is always useful and we as reviewers are also entrusted with the task of evaluating a theoretical contribution. To me a good theory paper usually solves or makes progress towards a problem that is of interest to the community (for instance, improving on a learning bound on a known scenario, or making a connection that could draw tools from other fields to solve something). I don't see that case happening here, there is little incentive (at least argumented in the paper) for using the KS distance instead of IPMs or other ones. Furthermore, the paper is from the abstract claiming that they want to introduce an algorithm to do something, so it makes sense for reviewers to judge said algorithm on the performance on doing said something. I don't really see here that the goal is to understand something better (nor how this paper would do such a thing), as theory papers are usually meant to do.
I thank the authors' efforts in rebuttal. I would like to raise my score to 6 (weak accept) to encourage authors on their innovation of using KS distance for training GAN generators.