Title: Kolmogorov–Smirnov GAN
Venue: NeurIPS 2024
Accepted: no
Keywords: generative models

generative adversarial networks

adversarial training
Abstract: We propose a novel deep generative model, the Kolmogorov-Smirnov Generative Adversarial Network (KSGAN). Unlike existing approaches, KSGAN formulates the learning process as a minimization of the Kolmogorov-Smirnov (KS) distance, generalized to handle multivariate distributions. This distance is calculated using the quantile function, which acts as the critic in the adversarial training process. We formally demonstrate that minimizing the KS distance leads to the trained approximate distribution aligning with the target distribution. We propose an efficient implementation and evaluate its effectiveness through experiments. The results show that KSGAN performs on par with existing adversarial methods, exhibiting stability during training, resistance to mode dropping and collapse, and tolerance to variations in hyperparameter settings. Additionally, we review the literature on the Generalized KS test and discuss the connections between KSGAN and existing adversarial generative models.

Abstract

We propose a novel deep generative model, the Kolmogorov-Smirnov Generative Adversarial Network (KSGAN). Unlike existing approaches, KSGAN formulates the learning process as a minimization of the Kolmogorov-Smirnov (KS) distance, generalized to handle multivariate distributions. This distance is calculated using the quantile function, which acts as the critic in the adversarial training process. We formally demonstrate that minimizing the KS distance leads to the trained approximate distribution aligning with the target distribution. We propose an efficient implementation and evaluate its effectiveness through experiments. The results show that KSGAN performs on par with existing adversarial methods, exhibiting stability during training, resistance to mode dropping and collapse, and tolerance to variations in hyperparameter settings. Additionally, we review the literature on the Generalized KS test and discuss the connections between KSGAN and existing adversarial generative models.

p

F

(

x

)

p

G

(

x

)

c

φ

(

x

)

λ

x

F

∼

P

F

x

G

∼

P

G

c

φ

(

x

F

)

c

φ

(

x

G

)

λ

P

(

Γ

c

φ

(

λ

)

)

%

%

D

KS

P

F

(

Γ

c

φ

(

λ

)

)

P

G

(

Γ

c

φ

(

λ

)

)

1 Introduction 14

Generative modeling is about fitting a model to a target distribution, usually the data. A fundamental 15 taxonomy of models assigns them into prescribed and implicit statistical models [9], with partial 16 overlap between the two classes. Prescribed models directly parameterize the distribution's probability 17

density function, while implicit models parameterize the generator that allows samples to be drawn 18 from the distribution. The ultimate application of the model primarily dictates the choice between the 19 two approaches. It does, however, have consequences regarding the available types of divergences 20 that we can minimize when fitting the model. The divergences differ in the stability of optimization 21 and computational efficiency, as well as statistical efficiency, which all affect the final performance of 22 the model. 23

24

25

26

27

28

29

The natural approach for fitting a prescribed model is maximum likelihood estimation (MLE), equivalently formulated as minimization of Kullback-Leibler divergence. Likelihood evaluation for normalized models is straightforward. In non-normalized models, density evaluation is expensive; in this context, Hyvärinen [22] proposed the score matching objective, which can be interpreted as the Fisher divergence [30]. This approach is very effective for simulation-free training of ODE[7]/SDE[42, 19]-based models which are state-of-the-art in multiple domains today.

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

The principle driving the fitting of implicit statistical models is to push the model to generate samples that are indistinguishable from the target. An inflection point for this family of models came with the Generative Adversarial Network (GAN) [13], which took the principle literally and introduced an auxiliary classifier trained in an adversarial process to discriminate between the two distributions. The classification error given an optimal classifier relates to the Jensen-Shannon divergence between generator and the target. Initial work in this area involved applying heuristic tricks to deal with learning problems, namely vanishing gradients, unstable training, and mode dropping or collapse. Further advancements focused on using other distances based on the principle of adversarial learning of auxiliary models, which were supposed to have certain favorable properties with respect to the original GAN.

The Bayesian inference community has been reluctant to adopt adversarial methods [8], and the attempts to apply them in this context [40] indicate a credibility problem. A significant drawback of approximate methods is the excessive reduction of diversity in the distribution [17], the extremes of which lead to mode dropping [1]. In this work, we consider another distance for training implicit statistical models, i.e., the Kolmogorov-Smirnov (KS) distance, which, to the best of our knowledge, has not been used in this context before. The distinctive feature of the KS distance is that it directly measures the coverage discrepancy of each other's credibility regions by the distributions under analysis at all confidence levels. Thus, its minimization straightforwardly leads to the correct spread of the probability mass, avoiding mode dropping, overconfidence, and mode collapse when applied with a sufficient sampling budget.

We term the proposed model as Kolmogorov-Smirnov Generative Adversarial Network (KSGAN). We show how to generalize the standard KS distance to higher dimensions based on Polonik [38] in section 2, allowing our method to be used for multidimensional distributions. Next, in section 3, we show how to efficiently leverage the distance in an adversarial training process and show formally that the proposed algorithm leads to an alignment of the approximate and target distributions. We support the theoretical findings with empirical results presented in section 6.

2 Generalized Kolmogorov-Smirnov distance

We generalize the Kolmogorov-Smirnov (KS) distance (sometimes called simply Kolmogorov distance) between continuous probability distributions on one-dimensional spaces to multidimensional spaces and show that it is a metric. The test statistic of the KS test is a KS distance between empirical and target distributions (or two empirical in the case of the two-sample case). For this reason, our proposal is directly inspired by the generalization of the test introduced in Polonik [38].

Let us consider two probability measures P F and P G on a measurable space ( X , A ) , where the sample space X is a vector space such as I R d and A is the corresponding event space; F : X → [0 , 1] and G : X → [0 , 1] are the cumulative distribution functions (CDFs) of P F and P G respectively. 1 We say that P F = P G iff ∀ A ∈ A , P F ( A ) = P G ( A ) . When dim( X ) = 1 then the KS distance is P P

In the multivariate case, the problem with using the KS distance as is is that on a d -dimensional 66 space, there are 2 d -1 ways of defining a CDF. The distance has to be independent of the particular 67

definition and thus should be the largest across all the possibilities [35]. This, however, becomes 68 prohibitive for any d > 2 . In other words, the challenge comes from a multidimensional vector space 69 not being a partially ordered set. Everything that follows in this section consists of proposing a partial 70 order, showing that, under certain conditions, a probability distribution can be uniquely determined 71 on its basis and operationalizing it in an optimization problem. 72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

We begin by bringing the classical result that

where G -1 : [0 , 1] →X is the inverse CDF also called the quantile function. Einmahl and Mason [10] show that there exists a natural generalization of the quantile function to multivariate distribution, which we restate below.

Definition 1 (Generalized Quantile Function) . Let v : A → I R + be a measure, and C ⊂ A an arbitrary subset of the event space, then a function C P , C ( α ) : [0 , 1] →C such that

is called the generalized quantile function in C for P with respect to v 2 .

The generalized quantile function evaluated at level α yields a minimum-volume set [36] whose probability is at least α , and it is the smallest with respect to v such set in C , thus the name. For the remainder of this paper, we assume that v is the Lebesgue measure.

It may seem that it is enough to plug C P G , C ( α ) in place of G -1 ( α ) and P F in place of F in eq. (2) to establish the Generalized KS distance but it turns out that such a distance does not satisfy the positivity condition D KS ( P F , P G ) > 0 if P F = P G as the example below shows.

Example 1 (Polonik [38]) . Let P F be the probability measure of a chi distribution with one degree of freedom √ χ 2 1 which has support on I R + and P G the probability measure of a standard Gaussian distribution N (0 , 1) which has support on the whole I R . Given C = A we have

̸

̸

while clearly P F = P G . The statement in eq. (4) is easy to show by observing that ∀ x ∈ [0 , ∞ ) the density of P F is twice the density of P G and C P G , C ( α ) are intervals centered at 0.

Instead, a solution based on the quantile functions of both distributions is needed, which we present in definition 2.

Definition 2 (Generalized Kolmogorov-Smirnov distance) . Let the Generalized Kolmogorov-Smirnov distance be formulated as follows:

̸

Such distance is symmetric, satisfying the triangle inequality as shown in appendix A.1. For the 95 remainder of this section, we will show that the Generalized KS distance in eq. (5) meets the necessary 96 D GKS ( P , P ) = 0 and sufficient D GKS ( P F , P G ) > 0 if P F = P G conditions to consider it a metric. 97 In the proof, we will rely on the probability density function of P with respect to a reference measure 98 v , which we denote with p : X → [0 , ∞ ) . Let 99

100

101

102

denote the density level set of p at level λ ⩾ 0 (also called the highest density region [21]), and let Π p := { Γ p ( λ ) : λ ⩾ 0 } . The following observations about level sets will introduce the fundamental tools to prove the necessary and sufficient conditions for the generalized KS distance.

103

104

Remark 1 (The silhouette [37]) . For any density p , the following holds where 1 C denotes the indicator function of a set C . The RHS of eq. (7) is called the silhouette .

̸

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

An immediate consequence of remark 1 is that Π p ordered with respect to λ ⩾ 0 fully characterizes P , because p does. Graphically, the silhouette is a multidimensional stack of level sets.

Remark 2. Density level sets are minimum-volume sets [38] The quantity P ( C ) -λ v( C ) is maximized over A by Γ p ( λ ) , and thus if Γ p ( λ ) ∈ C , then Γ p ( λ ) = C P , C ( α ) 3 at level α = P (Γ p ( λ )) = ∫ p ( x ) 1 [ λ, ∞ ) ( p ( x ))d x .

Below, we present the fundamental theoretical result behind the proposed method, which restates Lemma 1.2. of Polonik [38].

Theorem 1 (Necessary and sufficient conditions) . Let v be a measure on ( X , A ) . Suppose that P F and P G are probability measures on ( X , A ) with densities (with reference measure v ) f and g respectively. Assuming that

the following two statements are equivalent:

See proof in Appendix A .

Meeting assumption A.1 is a demanding challenge, almost equivalent to learning the target distribution. Below, we propose a relaxation of it, which we will use to show the validity of our method.

Theorem 2 (Relaxation of assumption A.1 ) . Theorem 1 holds if assumption A.1 is relaxed to the case that C contains sets that are uniquely determined with density level sets of P F and P G up to a set C such that

and let r := P F ( C ) = P G ( C ) , then the supremum in statement S.2 is restricted to [0 , 1 -r ] .

See proof in Appendix A .

3 Kolmogorov-Smirnov GAN

For the remainder of the paper, we will consider P F as the target distribution represented by a dataset { x F } , and P G as the approximate distribution that we want to train by minimizing the Generalized KS distance in eq. (5) with Stochastic Gradient Descent. We model P G as a pushforward g θ # P Z of a simple (e.g., Gaussian, or Uniform) latent distribution P Z supported on Z , with a neural network g θ : Z → X , parameterized with θ , which we call the generator .

The major challenge in utilizing eq. (5) is the necessity of finding the C P , C ( α ) terms which is an optimization problem on its own. The idea that we propose in this work is to amortize the procedure by modeling the generalized quantile functions C P F , C ( α ) and C P G , C ( α ) with additional neural networks which have to be trained in parallel to the generator g θ . Therefore, our method is based on adversarial training [13], where optimization proceeds in alternating phases of minimization and maximization for different sets of parameters. Hence the name of the proposed method, the Kolmogorov-Smirnov Generative Adversarial Network .

3.1 Neural Quantile Function

The generalized quantile function defined in definition 1 is an infinite-dimensional vector function C P , C : [0 , 1] → C ∈ C . Such objects do not have an expressive, explicit representation that allows for gradient-based optimization. Therefore, we use an implicit representation inspired by density level sets in eq. (6). We propose to use neural level sets defined in definition 3 that are modeled by a neural network c : X → I R , which we will refer to as the critic .

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

Definition 3 (Neural level set) . Given a neural network c : X → I R , the neural level set at level λ is defined as 5

Neural level sets are used, for example, in image segmentation [6, 20] and surface reconstruction from point clouds [3]. They fit our application because for computing the Generalized KS distance in eq. (5), the explicit materialization of generalized quantiles is not required as long as the probability measure can be efficiently evaluated on the implicitly specified sets. We set C = Π c , and thus C P , Π c ( α ) = Γ c ( λ α ) , with λ α = arg min λ ∈ I R { λ : P (Γ c ( λ )) ⩾ α } . For a probability measure P ′ the following holds:

P ′ ( C P , Π c ( α )) = E x ∼ P ′ [ 1 ( -∞ ,λ α ] ( c ( x )) ] , (10) which shows that the terms in eq. (5) under neural level sets can be Monte-Carlo estimated given samples from the respective distributions. Assumption A.2 is satisfied by neural level sets by construction.

The formulation of the Generalized KS distance in eq. (5) includes two generalized quantile functions C P F , C ( α ) corresponding to target distribution P F and C P G , C ( α ) corresponding to the approximate distribution P G . Both have to be modeled with the respective neural networks c ϕ F and c ϕ G , where we use ϕ = { ϕ F , ϕ G } to denote the joint set of their parameters. In section 3.3, we show how to parameterize both critics with a single neural network. We set C = Π c ϕ F ∪ Π c ϕ G .

3.2 Optimizing generator's parameters θ

The Generalized KS distance in eq. (5) is a supremum over a unit interval and two functions; thus, it can be upper-bounded as

Next, we plug in C = Π c ϕ F ∪ Π c ϕ G to eq. (11) and use eq. (10) to get generator's objective:

In practice, the expectations in eq. (12) are estimated on finite samples from the two distributions, i.e. { x F } mentioned before, and { x G } sampled from the approximate distribution P G using the reparametrization trick to facilitate backpropagation of gradients. Therefore, the two terms become step functions in λ , and the supremum is located on one of the steps. That way, a line search on I R reduces to a maximum over a finite set. To preserve the differentiability of the cost function calculated in this way, we apply Straight-through Estimator [4] in place of indication function 1 . A schematic depiction of the process for a single critic is shown in fig. 1.

3.3 Optimizing critics' parameters ϕ

By optimizing critics' parameters ϕ , we want to satisfy assumption A.1 so that Generalized KS distance becomes a metric. For the problem posed in such a way, we lack supervision, i.e., we do not know the target sets' shapes. However, we can reformulate the problem as an estimation of the density functions of the two considered measures P F and P G and use the obtained approximate density models to build level sets. We can constitute an optimization problem for such a task based solely on finite sets of samples, which we have for P F and can arbitrarily generate from P G . As the estimator, we propose to use the Energy-based model (EBM) [43], which, thanks to the lack of constraints in the choice of architecture, can be very expressive while having favorable computational complexity at inference. To carry out EMB training effectively, we will introduce a new min-max game, the 'min phase' of which will turn out to be the initial objective in eq. (5), and in this way, we will close the adversarial cycle.

Let the critic c ϕ F ( x ) serve as the energy function. The density given by the EBM is then p c ϕ F ( x ) = exp( -c ϕ F ( x )) /Z c ϕ F , where Z c ϕ F = ∫ exp( -c ϕ F ( x ))d x is the normalizing constant called partition

Input : Target distribution P F ; latent distribution P Z ; generator network g θ ; critic network c ϕ ; number of critic updates k ϕ ; number of generator updates k θ ; score penalty weight β ; Output: Trained model P G approximating P F ;

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

function. The standard technique for learning the model given target data distribution P F is MLE, where the likelihood

is maximized wrt ϕ F . An unbiased estimate of the gradient of the second term can be obtained with samples from the EBM itself, typically achieved with MCMC sampling. Many approaches to avoid this expensive procedure have been described in the literature [43], and among them, the one based on adversarial training [23] is the most appealing to us. It introduces an auxiliary distribution P aux ( F ) , such that the gradient of eq. (13) wrt ϕ F is approximated with the gradient of

Consequently, an additional objective L aux ( F ) must be introduced, the optimization of which will lead to the alignment of P aux ( F ) and P c ϕ F , where P c ϕ F denotes the probability distribution with density p c ϕ F ( x ) . We take an analogous approach to estimate c ϕ G ( x ) .

When we (i) set c ϕ G ( x ) := -c ϕ F ( x ) , and (ii) repurpose P G as P aux ( F ) and P F as P aux ( G ) , we show in appendix A.2 that the MLE objectives for the critics - now, denoted as c ϕ - simplify as L c = E x ∼ P G [ c ϕ ( x )] -E x ∼ P F [ c ϕ ( x )] , which is then maximized in an adversarial game against the Generalized KS distance in eq. (5).

The standard approach for aligning the auxiliary distributions with their targets is to use the KullbackLeibler divergence. We propose using the Generalized KS distance instead. We set L aux ( F ) = D GKS ( P G , P c ϕ ) and L aux ( P G ) = D GKS ( P F , P -c ϕ ) . By analyzing these objectives in the fashion of section 3.2, we note that L aux ( P G ) is the same as our original objective D GKS ( P F , P G ) - which is symmetric - when we approximate sampling from P c ϕ with the target distribution P F . Analogously for L aux ( P G ) where sampling from P -c ϕ is approximated with P G . Therefore, we have shown that the auxiliary objectives are already integrated into the adversarial game.

In practice, we find the score penalty regularizer of Kumar et al. [26], derived from the score matching objective, helpful to stabilize training. Therefore, we subtract it from L c weighted by a hyperparameter β . In this way, we get a critic that is smoother and, therefore, generates regular level sets that facilitate optimization. We summarize the proposed training procedure in algorithm 1.

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

4 Discussion

In section 3.3, where we justify the choice of the critic's objective function, we refer to methods for training EBMs, which are approximate density distribution models. Thus, the reader can expect that our proposed critic c ϕ in the limit of convergence of the algorithm will become a source of information about the density distribution of the target distribution P F accompanying the model that generates samples P G . However, this does not happen as a consequence of the design choice (i), that is, the setup of c ϕ F = -c ϕ G = c ϕ . An EBM can only be equivalent to its inverse in the case of a uniform distribution. In addition, because of design choice (ii), during training, the critic is not evaluated outside of the support of P F and P G and, therefore, can reach arbitrary values there. Despite these observations, the Generalized KS distance present in our algorithm exposes sufficient conditions because of theorem 2.

The feature distinguishing KSGAN from other adversarial generative modeling approaches is that regardless of the outcome of the critic's inner problem, minimizing eq. (5) is justified because Generalized KS distance, despite not meeting assumption A.1 , is a pseudo-metric [38]. For comparison, the dual representation of Wasserstein distance, used in WGAN [2] requires attaining the supremum in the inner problem.

The distances used for training generative models all fall into either the category of f -divergences D f ( P F , P G ) = ∫ A f (d P F / d P G ) d P G or integral probability metrics (IPMs) D F ( P F , P G ) = sup f ∈F | E x ∼ P F f ( x ) -E x ∼ P G f ( x ) | . The classical one-dimensional KS distance is an instance of IPM with F = { 1 ( -∞ ,t ] | t ∈ I R } or F = { 1 G -1 ( α ) | α ∈ [0 , 1] } when having access to the inverse CDF of one of the distributions based on eq. (2). One can see the Generalized KS distance from the perspective of IPM with F = { 1 C ( α ) | α ∈ [0 , 1] & C ∈ { C P F , C , C P G , C }} . Assuming direct access to C P F , C and C P G , C , for example when both P F and P G are Normalizing Flows [24, 34], measuring the distance comes down to a line search.

5 Related work

The need to generalize the KS test, and therefore distance, to multiple dimensions arose naturally from the side of practitioners who collected such data and wished to test related hypotheses. It was first addressed by Peacock [35], where a two-dimensional test for applications in astronomy was proposed. It involves considering all possible orders in this space and using the one that maximizes the distance between the distributions. A modification of this procedure has been proposed by Fasano and Franceschini [11] where only four candidate CDFs have to be considered, causing the test to be applicable in three dimensions, with eight candidates, under similar computational constraints. Chronologically, the following approach was the one on which we base our work, proposed in Polonik [38] but made possible by the author's earlier work [36, 37]. To the best of our knowledge, the first work that practically uses the theory developed by Polonik is Glazer et al. [12], which we recommend as an introduction to our work. It proposes applying the Generalized KS test based on the support vector machines for detecting distribution shifts in data streams.

As an instance of the adversarial generative modeling family, our work is related to all the countless GAN [13] follow-ups. We highlight those that study the learning process from the perspective of the distance being minimized. The work of Arjovsky and Bottou [1] provides a formal analysis of the heuristic tricks used for stabilizing the training of GANs. The f -GAN [33] proposes a unified training framework targeting f -divergences, which relies on a variational lower bound of the objective that results in the adversarial process. Approaches relying on the integral probability metric include FisherGAN [32], the Generative Moment Matching Networks [29] based on MMD, just like the later, more sophisticated MMD GAN [28], and finally the Wasserstein GAN (WGAN) [2] with the WGAN-GP follow-up [16] which shares common features with our work. Our maximum likelihood approach to fitting the critic results in the same functional form of the loss as WGAN(-GP) uses. In addition, the score penalty we use is similar to the gradient penalty of WGAN-GP.

6 Experiments

We evaluate the proposed method on eight synthetic 2D distributions (see appendix B.1 for details) and two image datasets, i.e. MNIST [27] and CIFAR-10 [25]. We compare against other adversarial

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

methods, GAN and WGAN-GP, using the same neural network architectures and training hyperparameters unless specified otherwise (see appendix C for details). All the quantitative results are presented based on five random initializations of the models. The source code for all the experiments is provided in anonymous code repository.

In all KSGAN experiments, we relax the maximum in line 11 and line 12 of algorithm 1 with sample average. In all experiments, we re-use the last batch of samples from the latent distribution (and target distribution in the case of KSGAN) from the critic's optimization inner loop as the first batch for the generator's optimization inner loop.

6.1 Synthetic distributions

Analyzing adversarial methods on synthetic, low-dimensional distributions is not popular. However, we conduct such an experiment because we are interested in whether the model generates samples from the support of the target distribution and how accurately it approximates the distribution. Working with small-dimensional distributions, we do not have to be as concerned about the curse of dimensionality when calculating sample-based distances, and we can visually compare the resulting histograms.

In table 1, we report the squared population MMD [15] between target and approximate distributions, computed with Gaussian kernel on 65536 samples from each distribution. Details about how we chose the kernel's bandwidth can be found in appendix B.1. GAN and WGAN-GP fail to converge with k ϕ = k θ = 1 (we do not report the results to economize on space); thus, we set k θ = 5 for them. The proposed KSGAN with k θ = 1 performs at a similar level to WGAN-GP, the better of the two former, despite using five times less training budget. We present additional results on the synthetic datasets in appendix D.1, which include performance with different training dataset sizes, non-default hyper-parameter setups for KSGAN, and histograms of the samples for qualitative comparison.

6.2 MNIST

We use the 50000 training instances to train the models, and based on visual inspection of the generated samples (reported in appendix D.2), we conclude that all the methods achieve comparable, high samples quality. To assess the quality of the distribution approximation, we use a pre-trained classifier on the same data as the generative models (details in appendix B.2). We run the same experiment on 3StackedMNIST [44], which has 1000 modes. We report the results in table 2.

In this experiment, we set the training budget for all methods to k ϕ = 1 , k θ = 1 for a fair comparison. We find that all methods always recover all the modes with the standard MNIST target. However, GAN fails to distribute the probability mass uniformly between the digits. As the number of modes increases with the 3StackedMNIST target, GAN demonstrates its inferiority to other methods by losing 198 modes on average (four initialization cover approx. 985 modes, and one fails to converge, achieving only 98 modes). WGAN-GP and KSGAN consistently recover all the modes while being on par regarding KL divergence, which differs little between networks' initialization.

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

6.3 CIFAR-10

We use the 50000 training instances to train the models and report the generated samples in appendix D.3. We train the models in a fully unconditional manner, i.e., not using the class information at all - contrary to many unconditional models that use class information in normalization layers. We quantify the quality of fitted models by computing the Inception Score (IS) [41] and Fréchet inception distance (FID) [18] from the test set and report the results in table 3 based on five random initializations. For reference, in the table, we include the IS of the training dataset and the FID between the training and test sets.

In this experiment, we set the training budget for all methods to k ϕ = 1 , k θ = 1 for a fair comparison. All models fail to accurately approximate the target distribution, which is evident from a quantitative comparison in table 3 and a qualitative one in appendix D.3. KSGAN is characterized by the lowest variance between initializations among the methods considered.

7 Conclusions and future work

In this work, we investigated the use of Generalized Kolmogorov-Smirnov distance for training deep implicit statistical models, i.e., generative networks. We proposed an efficient way to compute the distance and termed the resulting model Kolmogorov-Smirnov Generative Adversarial Network because it uses adversarial learning. Based on the empirical evaluation of the proposed model, the results of which we report, we conclude that it can be considered as an alternative to existing models in its class. At the same time, we point out that many properties of KSGAN have not been studied, and we leave this as a future work direction.

Interesting aspects to explore are the characteristics of learning dynamics with the number of generator updates exceeding the number of critic updates, alternative ways to train the critic, and alternative representations of generalized quantile sets. The natural scaling of the Generalized KS distance may also prove beneficial regarding the interpretability of learning curves, learning rate scheduling, or early stopping. In addition, we hope that our work will draw the attention of the machine learning community to the Generalized KS distance, applications of which remain to be explored.

326

327

328

329

330

References

M. Arjovsky and L. Bottou. Towards principled methods for training generative adversarial networks. In International Conference on Learning Representations , 2017.

M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein generative adversarial networks. In International conference on machine learning , pages 214-223. PMLR, 2017.

M. Atzmon, N. Haim, L. Yariv, O. Israelov, H. Maron, and Y. Lipman. Controlling neural level sets. 331 Advances in Neural Information Processing Systems , 32(NeurIPS), 2019. 332

Y. Bengio, N. Léonard, and A. Courville. Estimating or propagating gradients through stochastic neurons 333 for conditional computation. arXiv preprint arXiv:1308.3432 , 2013. 334

C. A. Carolan. The least concave majorant of the empirical distribution function. The Canadian Journal of 335 Statistics / La Revue Canadienne de Statistique , 30(2):317-328, 2002. 336

G. Chen, Z. Yu, H. Liu, Y. Ma, and B. Yu. DevelSet: Deep Neural Level Set for Instant Mask Optimization. 337 IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems , 42(12):5020-5033, 338 2023. 339

R. T. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud. Neural ordinary differential equations. 340 Advances in neural information processing systems , 31, 2018. 341

K. Cranmer, J. Brehmer, and G. Louppe. The frontier of simulation-based inference. Proceedings of the 342 National Academy of Sciences , 117(48):30055-30062, 2020. 343

P. J. Diggle and R. J. Gratton. Monte carlo methods of inference for implicit statistical models. Journal of 344 the Royal Statistical Society. Series B (Methodological) , 46(2):193-227, 1984. 345

J. H. J. Einmahl and D. M. Mason. Generalized Quantile Processes. The Annals of Statistics , 20(2), jun 346 1992. 347

G. Fasano and A. Franceschini. A multidimensional version of the Kolmogorov-Smirnov test. Monthly 348 Notices of the Royal Astronomical Society , 225(1):155-170, mar 1987. 349

A. Glazer, M. Lindenbaoum, and S. Markovitch. Learning high-density regions for a generalized 350 kolmogorov-smirnov test in high-dimensional data. Advances in Neural Information Processing Sys351 tems , 1:728-736, 2012. 352

I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. 353 Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Weinberger, 354 editors, Advances in Neural Information Processing Systems , volume 27. Curran Associates, Inc., 2014. 355

W. Grathwohl, R. T. Chen, J. Bettencourt, I. Sutskever, and D. Duvenaud. Ffjord: Free-form continuous 356 dynamics for scalable reversible generative models. arXiv preprint arXiv:1810.01367 , 2018. 357

A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola. A kernel two-sample test. Journal 358 of Machine Learning Research , 13(25):723-773, 2012. 359

I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville. Improved training of wasserstein 360 gans. Advances in neural information processing systems , 30, 2017. 361

J. Hermans, A. Delaunoy, F. Rozet, A. Wehenkel, V. Begy, and G. Louppe. A crisis in simulation-based 362 inference? beware, your posterior approximations can be unfaithful. Transactions on Machine Learning 363 Research , 2022. 364

M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two time-scale 365 update rule converge to a local nash equilibrium. Advances in neural information processing systems , 30, 366 2017. 367

J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in neural information 368 processing systems , 33:6840-6851, 2020. 369

P. Hu, B. Shuai, J. Liu, and G. Wang. Deep level sets for salient object detection. Proceedings - 30th IEEE 370 Conference on Computer Vision and Pattern Recognition, CVPR 2017 , 2017-Janua:540-549, 2017. 371

R. J. Hyndman. Computing and graphing highest density regions. The American Statistician , 50(2): 372 120-126, 1996. 373

A. Hyvärinen. Estimation of non-normalized statistical models by score matching. Journal of Machine 374 Learning Research , 6(24):695-709, 2005. 375

T. Kim and Y. Bengio. Deep directed generative models with energy-based probability estimation. arXiv 376 preprint arXiv:1606.03439 , 2016. 377

I. Kobyzev, S. J. Prince, and M. A. Brubaker. Normalizing flows: An introduction and review of current 378 methods. IEEE transactions on pattern analysis and machine intelligence , 43(11):3964-3979, 2020. 379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

A. Krizhevsky. Learning multiple layers of features from tiny images. 2009.

R. Kumar, S. Ozair, A. Goyal, A. Courville, and Y. Bengio. Maximum entropy generators for energy-based models. arXiv preprint arXiv:1901.08508 , 2019.

Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

C.-L. Li, W.-C. Chang, Y. Cheng, Y. Yang, and B. Póczos. Mmd gan: Towards deeper understanding of moment matching network. Advances in neural information processing systems , 30, 2017.

Y. Li, K. Swersky, and R. Zemel. Generative moment matching networks. In International conference on machine learning , pages 1718-1727. PMLR, 2015.

S. Lyu. Interpretation and generalization of score matching. In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence , pages 359-366, 2009.

T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations , 2018.

Y. Mroueh and T. Sercu. Fisher gan. Advances in neural information processing systems , 30, 2017.

S. Nowozin, B. Cseke, and R. Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 29. Curran Associates, Inc., 2016.

G. Papamakarios, E. Nalisnick, D. J. Rezende, S. Mohamed, and B. Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research , 22(57):1-64, 2021.

J. A. Peacock. Two-dimensional goodness-of-fit testing in astronomy. Monthly Notices of the Royal Astronomical Society , 202(3):615-627, mar 1983.

W. Polonik. Minimum volume sets in statistics: Recent developments. In R. Klar and O. Opitz, editors, Classification and Knowledge Organization , pages 187-194, Berlin, Heidelberg, 1997. Springer Berlin Heidelberg.

W. Polonik. The silhouette, concentration functions and ml-density estimation under order restrictions. The Annals of Statistics , 26(5):1857-1877, 1998.

W. Polonik. Concentration and goodness-of-fit in higher dimensions: (Asymptotically) distribution-free methods. Annals of Statistics , 27(4):1210-1229, 1999.

A. Radford, L. Metz, and S. Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In Y. Bengio and Y. LeCun, editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings , 2016.

P. Ramesh, J.-M. Lueckmann, J. Boelts, Á. Tejero-Cantero, D. S. Greenberg, P. J. Goncalves, and 412 J. H. Macke. GATSBI: Generative adversarial training for simulation-based inference. In International 413 Conference on Learning Representations , 2022. 414

T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen. Improved techniques for 415 training gans. Advances in neural information processing systems , 29, 2016. 416

Y. Song and S. Ermon. Generative modeling by estimating gradients of the data distribution. Advances in 417 neural information processing systems , 32, 2019. 418

Y. Song and D. P. Kingma. How to train your energy-based models. arXiv preprint arXiv:2101.03288 , 419 2021. 420

A. Srivastava, L. Valkov, C. Russell, M. U. Gutmann, and C. Sutton. Veegan: Reducing mode collapse in 421 gans using implicit variational learning. Advances in neural information processing systems , 30, 2017. 422

A Proofs 423

424

425

426

Theorem 1 (Necessary and sufficient conditions) . Let v be a measure on ( X , A ) . Suppose that P F and P G are probability measures on ( X , A ) with densities (with reference measure v ) f and g respectively. Assuming that

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

A.1 Π f ∪ Π g ⊂ C ;

A.2 C P F , C ( α ) and C P G , C ( α ) are uniquely determined 6 in C with respect to v the following two statements are equivalent:

S.1 P F = P G ;

S.2 D GKS ( P F , P G ) = 0 .

Proof of Theorem 1. The S.1 = ⇒ S.2 direction is trivial to show and works without satisfying the assumptions [38]. Therefore, we focus on showing that S.2 = ⇒ S.1 . Let

and denote with Γ( λ ) the level set of density of P as defined in eq. (6), and let Π := { Γ( λ ) : λ ⩾ 0 } . Further, let ˜ S C denote the least concave majorant [5] to S C ( P ) , that is, the smallest concave function from I R + to [0 , 1] lying above S C ( P ) . ˜ S C is supported on the generalized quantiles of P in C , i.e. on the points (v( C P , C ( α )) , P ( C P , C ( α ))) . Finally, let ∂ ˜ S C ( P ) be the intersection of the extremal points of the convex hull of S C ( P ) with the graph of ˜ S C . Given Π ⊂ C which we assume in A.1 for P F and P G , and in the light of remark 2 we have that for any set C such that (v( C ) , P ( C )) ∈ ∂ ˜ S C ( P ) there is a level λ for which C = Γ( λ ) , and it is equal the left-hand derivative of ˜ S C in the point v( C ) . From remark 1, we have that the silhouette fully characterizes P , and therefore ∂ ˜ S C ( P ) does it as well. , under Lemma 2.1 of Polonik

Eventually, we conclude the proof with the observation that given S.2 [38] (where A.2 is utilized) we have that the extremal points of the convex hulls of S C ( P F ) and S C ( P G ) are the same points, thus ∂ ˜ S P F ( P ) = ∂ ˜ S P G ( P ) , and finally P F = P G .

Theorem 2 (Relaxation of assumption A.1 ) . Theorem 1 holds if assumption A.1 is relaxed to the case that C contains sets that are uniquely determined with density level sets of P F and P G up to a set C such that

and let r := P F ( C ) = P G ( C ) , then the supremum in statement S.2 is restricted to [0 , 1 -r ] . 449

Proof of Theorem 2. The statement in eq. (8) is equivalent to saying that P F = P G on ( C, 2 C ) . 450 Analogously to the proof of theorem 1 we can show that P F = P G on ( X \ C, 2 X\ C ) . By observing 451 that probability measures are σ -additive, we conclude that P F = P G on ( X , A ) , and thus the result 452 of theorem 1 holds. 453

A.1 Generalized KS distance satisfies triangle inequality 454

Let us consider three probability measures P F , P G , and P H on a measurable space ( X , A ) . 455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

In (i), we use the fact that the supremum of absolute difference in distribution coverage is maximized with the generalized quantile function of one of them. In (ii), we apply triangle inequality for absolute value. Thus we have shown that D GKS ( P F , P H ) + D GKS ( P H , P G ) ⩾ D GKS ( P F , P G ) which is the triangle inequality for the Generalized KS distance.

A.2 Objective for the critic

Given two adversarial maximum likelihood objectives from Kim and Bengio [23], we (i) set c ϕ G ( x ) := -c ϕ F ( x ) , and (ii) repurpose P G as P aux ( F ) and P F as P aux ( G ) , and show that:

B Experiments details

In this section, we provide additional details about experiments conducted in the paper that did not fit in the main text. All the models reported in the paper were trained under 12 hours on a single Nvidia GeForce GTX TITAN X GPU (12GB vRAM) with 32GB of RAM and 2 CPU cores. We report results based on 645 models trained, which amounts to 7740 GPU hours at most. We estimate that about three times as much computing time was used for preliminary experiments not reported in the paper.

B.1 Synthetic

The synthetic 2D distributions are adopted from the official code of Grathwohl et al. [14] https: //github.com/rtqichen/ffjord . We randomly generate 65536 training and 65536 test instances from each distribution. In appendix D.1, we report the results of training the models with fewer instances but evaluated using the entire test set.

We choose the bandwidth of the Gaussian filter in squared population MMD as the median L2 475 distance between two samples, of 32768 instances each, from the simulator. The resulting values can 476 be found in the code we provide with the paper. 477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

B.2 MNIST

To detect the modes in the (3Stacked)MNIST experiments, we use a pre-trained classifier from PyTorch examples, trained for 14 epochs of the train set of the original MNIST dataset. We expect to find 10 and 1000 modes for the MNIST and 3StackedMNIST, respectively. We measure the KL divergence between the classifier's output and discrete uniform distribution for both distributions.

B.3 CIFAR-10

We compute the Inception Score using the implementation from https://github.com/sbarratt/ inception-score-pytorch . We compute the Fréchet inception distance using the implementation from https://github.com/mseitzer/pytorch-fid .

C Architectures and hyper-parameters

C.1 Synthetic

For all of the methods and distributions, we use the same architecture, described in table 4, with spectral normalization [31] on linear layers for GAN. In all cases, we train the generator and critic with Adam( β 1 = 0 . 5 , β 2 = 0 . 9 ) optimizer with a constant learning rate of 0.0001, without L2 regularization or weight decay, for 128000 generator updates with batch size equal to 512. We use the standard loss for GAN, enforcing class 1 for real samples and 0 for generated samples. In WGAN-GP, we use 0.1 weight on gradient penalty (identified as a good value in preliminary experiments, which we do not report), and in KSGAN β = 1 . 0 as the weight for score penalty.

C.2 MNIST

For the MNIST experiments, we use the DCGAN [39] architecture, without batch normalization layers, with 128-dimensional latent Gaussian distribution. For the 3StackedMNIST distribution, we increase the number of input and output channels for the critic and generator, respectively. We train the generator and critic with Adam( β 1 = 0 . 5 , β 2 = 0 . 9 ) optimizer with a constant learning rate of 0.0001, without L2 regularization or weight decay, for 200000 generator updates with batch size equal to 50. In the case of GAN for 3StackedMNIST, we use a learning rate of 0.001 (identified as a good value in preliminary experiments, which we do not report). We use the flipped loss for GAN, enforcing class 0 for real samples and 1 for generated samples. In WGAN-GP,

we use 10.0 weight on gradient penalty (identified as a good value in preliminary experiments, which we do not report), and in KSGAN

β

= 1

.

0

as the weight for score penalty.

-

0

.

002

0

.

000

0

.

002

0

.

004

0

.

006

0

.

008

Squared population MMD

swissroll

circles

rings

moons

10

3

10

4

# training instances

-

0

.

002

0

.

000

0

.

002

0

.

004

0

.

006

0

.

008

Squared population MMD

8gaussians

10

3

10

4

# training instances

pinwheel

10

3

10

4

# training instances

2spirals

10

3

10

4

# training instances

checkerboard

GAN (5, 1)

WGAN-GP (5, 1)

KSGAN (1, 1)

C.3 CIFAR-10 507

For the CIFAR-10 experiments, we use ResNet architecture from Gulrajani et al. [16]. We train 508 the generator and critic with Adam( β 1 = 0 . 0 , β 2 = 0 . 9 ) optimizer with a constant learning rate of 509 0.0001, without L2 regularization or weight decay, for 199936 generator updates with batch size 510 equal to 64. We use the 511

flipped loss for GAN, enforcing class 0 for real samples and 1 for generated samples. In WGAN-GP, 512 we use 10.0 weight on gradient penalty (identified as a good value in preliminary experiments, which 513 we do not report), and in KSGAN β = 1 . 0 as the weight for score penalty. 514

515

516

517

518

D Extended results

In this section, we report additional experiment results that did not fit in the main text. This includes materials allowing a qualitative comparison of the trained models.

D.1 Synthetic data

In fig. 2, we report, extended relative to table 2 in the main text, a study of the quality of trained 519 models as measured by the squared population MMD. Solid lines denote the average over five 520 random initializations, and the shaded area represents the twoσ interval. KSGAN performs on par 521 with WGAN-GP while being trained with a five times less training budget. In fig. 3, we show the 522 histograms of 65536 samples from the models (a single random initialization), with a histogram of 523 test data in the first column for reference. For KSGAN, in addition to the configurations included in 524 table 2, we include one with a training budget matching that of GAN and WGAN-GP, and one with a 525 training budget reduced by two, where the critic is updated only every second update of the generator. 526

D.2 MNIST 527

In fig. 4, we show samples from one of the random initializations reported in table 2 in the main text. 528 All models demonstrate similar sample quality, while for GAN, the digit '1' is over-represented, 529 which corresponds with the high KL in table 2. 530

Real data

GAN (5, 1)

WGAN-GP (5,1)

KSGAN (1, 1)

KSGAN (5, 1)

KSGAN (1, 2)

swissroll

circles

O

O

O

O

O

O

O

moons

pinwheel

S

S

S

S

S

S

2spirals

?

5

G

checkerboard

2

(a) GAN (1, 1)

2

2572432162714

69

S

1/48373131032

S

力

816389518386

0

8

27

2

hh

D

9772

h

53

6

6

72722674839264

?

3

$240

47867358

(b) WGAN-GP (1, 1)

352202

5×2a

5

2

140474

1343188

433

827

5

4

003$3862

y

4

314

2

h十

024

0

51

4

25//

6

S

B

4

9073/9

J

32

4

33624

8

6

22

126

0

G

2

8

又

5

D.3 CIFAR-10 531

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We provide a formal derivation of the proposed method, with all the steps described and justified. Claims about the empirical behavior of the proposed method are supported by the results of experiments reported in the main text and appendix.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]

Justification: We explicitly specify all the assumptions made regarding the theoretical part. In the main text, we admit that the empirical evaluation does not explore all the properties of the proposed method. We propose further lines of work that we consider promising based on our experience with the method.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency

play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All assumptions are explicitly mentioned, we believe our proofs are correct. In addition, in the theoretical part of our work, we rely on previously published results by other authors, which we always cite as a reference.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Experiments (including the evaluation protocol) are described in detail in the main text and completed with more information in the appendix. In addition, we include a link to the repository containing the code that was used to conduct the experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We include a link to the repository containing the code that was used to conduct the experiments. We use only publicly available data.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https://nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Experiments (including the evaluation protocol) are described in detail in the main text and completed with more information in the appendix. In addition, we include a link to the repository containing the code that was used to conduct the experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: All the evaluation metrics were computed on five random initializations. We report the average scores with standard deviation computed with Bessel's correction.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: In the appendix, we include information about the resources needed to reproduce results reported in the paper, and give an estimate of resources spent on preliminary experiments not reported in the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: None of the datasets used in the paper has been deprecated. We do not identify any concerns regarding societal impact and potential harmful consequences of our work.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our work is foundational research, and thus does not have a direct positive or negative societal impact. We disclaim responsibility for the malicious use of our work.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All the creators of assets are properly credited in the paper and the code. Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

Werecognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper first generalizes the Kolmogorov–Smirnov (KS) distance from one-dimensional spaces to multidimensional spaces and proposes the Kolmogorov-Smirnov GAN, which formulates the generative model by minimizing the Kolmogorov-Smirnov (KS) distance. Theoretical results are also given in this paper and the experiments also show the superiority of stability during training, resistance to mode dropping and collapse, and tolerance to variations in hyperparameter settings.\n\nSTRENGTHS:\nI like the idea of generalizing the one-dimensional KS distance to multi-dimensional, which I had thought about before but failed to achieve. The motivation and writing are good, and the experiments of KSGAN seem to have achieved good results.\n\nWEAKNESSES:\n1. I'm skeptical about some parts of the theory. (See questions for details)\n2. It seems that the advantages of KS distance over JS divergence and Wasserstein Distance are not explained.\n3. The idea of reformulating a distance between distributions to a GAN model seems to be old and is now unlikely to attract readers' interest.\n4. The experimental setup is relatively simple, only comparing with vanilla GAN and WGAN on Synthetic, MNIST, and CIFAR10 datasets\n5. According to the experimental results, the advantages of KSGAN lie in the stability of training and resistance to mode dropping. A significant issue is that with current network architectures and training techniques, these two problems are rarely encountered.\n\nQUESTIONS:\nQ1. In Line 66-67, what's the meaning there are $2^d-1$ ways of defining a CDF on a d-dimensional space?\n\nQ2. Definition 1 seems strange. What is the measure v? Is it a determined measure or any arbitrary measure? Is P the known CDF in Eq.3? What is its relationship with v?\n\nQ3. Based on the definition of mapping, given function f: A->B, for each element a in A, there is always a unique element b in B corresponding to it. Then for $G^{-1}(\\alpha)$ in Eq.2 and $C_{P,C}(\\alpha)$, how is the uniqueness of the set guaranteed given $\\alpha\\in [0,1]$?\n\nLIMITATIONS:\nThe authors have discussed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel variant of the generative adversarial network that uses the Kolmogorov-Smirnov distance to align the generated distribution with the target distribution. This distance is calculated using the quantile function, which acts as the critic in the adversarial training process. Experiments are conducted on synthetic distributions and small image datasets to show that the proposed KSGAN performs on par with the existing adversarial methods.\n\nSTRENGTHS:\n1. The paper is well-presented and easy to follow.\n\n2. The claims and methodology designs are well supported by theoretical analysis.\n\nWEAKNESSES:\n1. It is still unclear why we need another adversarial design based on KS distance. The vanilla GAN paper shows that the designed bi-level optimization process can already be seen as optimizing the distance between the generated and the target distribution. Then, what are the specific advantages KS distance can bring within the adversarial framework?\n\n2. The experiments are merely conducted on synthetic datasets and small image datasets. It is unclear whether the proposed method can be adapted to larger-scale datasets or incorporated into more advanced frameworks like StyleGAN. Moreover, the compared baselines are limited to early works, and the experimental results of KSGAN are worse than those of WGAN-GP. Thus, I do not see many advantages of KSGAN in terms of the presented experiments.\n\nQUESTIONS:\n1. Could the authors explain or provide more evidence about line 40, \"The Bayesian inference community has been reluctant to adopt adversarial methods\"?\n\n2. How can we guarantee the optimality of the learned quantile functions such that the estimated KS distance is reliable?\n\n3. Again, with adversarial training, the original gan is already optimizing the distance between the two distributions. What are the benefits of optimizing in this way by KSGAN?\n\n4. In Table 3, the performance of the models with IS around 2.4 and FID around 40 does not seem very successful in generative training. And the improvements seem minor.\n\nLIMITATIONS:\nPlease see the discussions above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce a generalized KS distance applicable to high dimensional spaces, formulate the corresponding dual problem, and use adversarial training to construct a generative model that minimizes the GKS between data and generated distributions.\n\nThe paper is well presented and appears technically correct through what I've seen, though I didn't check the proofs in details.\n\nThe main problem is that there's no clear motivation for why using the GKS is beneficial at all (either theoretically or practically). As such, despite being novel, I don't see any clear impact from the paper. Furthermore, the final algorithm is quite complicated and the results are fairly underwhelming, so at the end of the day the cons dramatically outweigh the pros of the newly introduced algorithm.\n\nSTRENGTHS:\nThe paper is well written. It reads easily and it is clear what they want to do. The contribution of a generalized KS distance to multidimensional spaces and the algorithm to approximate it are to the best of my knowledge, novel.\n\nWEAKNESSES:\nThe main problem I have with the paper is that I don't see any clear advantages of using the KS distance (gan) as a replacement of other distances like the Wasserstein one, or their GAN equivalent. The only mention of this, which should arguably be the most important thing in a paper introducing a new GAN, is in lines 224-228 of page 7. The authors claim there that they don't need to maximize the supremum in (5) which is false depending on how to interpret it, if you just take any set C in (5) you end up with |P_F(C) - P_G(C)| which is just measuring one moment for a given characteristic function, and far from being anything meaningful (and the same holding true for most IPMs). The results are also not particularly interesting to merit the claim that there's anything particularly different or benefitial on using this new formulation.\n\nQUESTIONS:\nThe main suggestions I have is the following:\n1) Take some time thinking why using the GKS is a better idea than using other distances between probability distributions.\n2) Validate these claims with targeted experiments (for instance, on tractable distributions, without adversarial training).\n3) Show that these properties translate to the adversarial setup, either theoretically or with experiments.\n\nAt the end of the day you're trying to convince readers that what you did is worth trying, so instead of focusing so much on mathematical details, consider more why a busy reader should care about this problem in the first place. What are the fundamental limitations of existing techniques you're trying to adress?\n\nLIMITATIONS:\nNo clear motivation or benefit from using their algorithm.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposed a new kind of GAN training method called KS-GAN. The method is based on minimizing the Kolmogorov–Smirnov distance. The KSGAN updates the generator by minimizing an upper bound of the generalized KS distance. It updates the discriminator (or the critic network) by using energy-based model training with regularization terms. The KSGAN is a novel attempt to explore new approaches to train generative adversarial networks.\n\nSTRENGTHS:\n* (1) The paper studied using generalized KS distance to train GAN generators is a novel attempt to extend the GAN literature. \n\n* (2) Some theoretical arguments about implementing the empirical KS distance using neural networks is novel yet constructive.\n\nWEAKNESSES:\nMy main concern about the paper is its weak evaluation baselines and questionable practical usage:\n\n* (1) Though the idea of using new objectives for training GAN generators is attractive, the practical usage of KSGAN seems questionable, especially for high-dimensional data. For instance, in the CIFAR10 generation experiment, the author compares KSGAN with WGAN-GP and Vanilla GAN, which have shown weak empirical performances. However, it is well-known that, for CIFAR10 data, the StyleGAN2-ADA[1] model is a strong baseline GAN model. I think it would strengthen the paper a lot if the authors could somehow show strong performances of KSGAN using StyleGAN2's architectures and implementation techniques. However, I do admit that such a requirement may be too tough for new methods.\n\n* (2) The KSGAN's critic function is constructed with EBMs. However, even with regularization terms, energy-based models are well-known for poor scaling ability to high-dimensional data. This may prevent the practical usage of KSGAN for real-world high-dimensional data.\n\nQUESTIONS:\nPlease see the weakness part.\n\nLIMITATIONS:\nThe author has addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}