Bayesian Nonparametric Learning using the Maximum Mean Discrepancy Measure for Synthetic Data Generation

Authors: forough fazeli asl
michael minyi zhang
lizhen  lin
Venue: NeurIPS_cc_2024_Workshop_BDU
Published: 2024-09-05 11:36:33

Abstract: We introduce a Bayesian estimator for maximum mean discrepancy (MMD), enabling a novel approach to measure-based data generation. To demonstrate the adaptability of our method, we embed this estimator within a generative adversarial network (GAN) framework. This integration offers a powerful avenue for Bayesian nonparametric (BNP) learning, showcasing the estimator's broad applicability. Our BNP-driven GAN not only enhances sample diversity but also improves inferential accuracy, surpassing the performance of traditional methods. Further theoretical properties, proofs, and experiments are given by the Appendix.

Main paper: 
 introduction: 
 Data augmentation is the technique of generating synthetic data, often to train machine learning models when the data are scarce or the model is non-robust to perturbations in the data. When the likelihood is intractable to compute, evaluating the model's fit can be difficult. Maximum mean discrepancy (MMD) addresses this problem by enabling comparisons between distributions without explicit likelihoods through its feature matching properties. This property ensures that generated data matches the features of real data, making MMD an effective tool for evaluating and developing deep generative models. Bayesian nonparametric methods are a powerful tool with strong theoretical justifications but they have seen limited applications in MMD estimation. A key advantage of the Bayesian approach is that it incorporates expert knowledge through a prior distribution, offering regularization by introducing uncertainty in the sampling distribution via the Dirichlet process (DP). The absence of such methods in MMD estimation restricts statisticians who prefer Bayesian frameworks without making strong assumptions. This paper addresses this gap.We propose a BNP estimator to accurately estimate the MMD between a parametric model and an unknown distribution by placing a DP prior on the unknown distribution. We extend the bootstrap method from [1] beyond posterior parameter inference, applying our estimator to training a generative adversarial network (GAN). Our approach uses the MMD estimator as a robust discriminator, combining MMD measurement with BNP inference to enhance GAN training, reduce mode collapse, and improve generator performance compared to frequentist (FNP) methods. 
previous work: 
 Previous work on simulation-based inference has mainly used discrepancy measures from a frequentist nonparametric (FNP) perspective. Notably, GANs have been extensively explored in data augmentation and medical data synthesis where the fake images created by the GAN are used to supplement the training data [2] [3] [4] [5] [6] . A standard GAN features two neural networks: the generator {G ω } ω∈W and the discriminator {D θ } θ∈Θ [7] . The generator tries to fool the discriminator into misidentifying generated samples as real. However, these models are expanded beyond the classic loss function, which could potentially introduce challenges such as mode collapse-memorizing certain modes of data distribution while overlooking other diversities-and training instability.A Bayesian approach, known as approximate Bayesian computation (ABC), estimates model parameters through simulation by comparing summary statistics of simulated and observed data [8] . ABC faces challenges in selecting informative summary statistics, which affects the accuracy of posterior inference [9, 10] . One particularly attractive choice of statistic is to use the MMD [11] . As the threshold decreases, ABC converges to the standard Bayesian posterior, which can be sensitive to model misspecification and lacks robustness [1] .To improve robustness, generalized Bayesian inference (GBI) replaces the likelihood with a robust loss function [12] . Example applications of GBI include using MMD in pseudo-likelihood approaches [13] or stochastic gradient MCMC for posterior inference [14] . Despite these advancements, GBI's sensitivity to hyperparameters and the computational demands of MCMC remain challenges [1] . To address these issues, an MMD posterior bootstrap method has been developed, offering a more efficient alternative [1, [15] [16] [17] . 
dirichlet process: 
 To perform Bayesian nonparametric learning (BNPL), we first must take samples from the Dirichlet process. The DP is an infinite generalization of the Dirichlet distribution that is considered on the sample space denoted as X, which possesses a σ-algebra A comprising subsets of X [18] . F follows a DP with parameters (a, H) with the notation F Pri := (F ∼ DP (a, H)), if for any measurable partition A 1 , . . . , A k of X with k ≥ 2, the joint distribution of the vector (F (A 1 ), . . . , F (A k )) follows a Dirichlet distribution characterized by parameters (aH(A 1 ), . . . , aH(A k )). Moreover, it is assumed that H(A j ) = 0 implies F (A j ) = 0 with probability one. The base measure H captures the prior knowledge regarding the data distribution, while a signifies the strength or intensity of this knowledge.As a conjugate prior, the posterior distribution of F also follows a DP, denoted by F Pos := (F |X 1:n ∼ DP(a + n, H * )), for n independent and identically distributed (IID) draws, X 1:n ∈ R d , from the random probability measure F where H * = a(a + n) -1 H + n(a + n) -1 F X1:n , and F X1:n represents the empirical cumulative distribution function of the sample X 1:n .To sample from the DP posterior, we use a finite approximation devised by Ishwaran and Zarepour [19] , which allows for convenient simulation. In the context of posterior inference, this approximation is given byEQUATIONwhere J Pos ), the space of probability measures on R d endowed with the topology of weak convergence [19] . Although the stick-breaking representation is a commonly employed series representation for DP inference [20] , it lacks the necessary normalization terms to convert it into a probability measure [21] . Additionally, simulating from an infinite series is only feasible through using a random truncation approach to handle the terms within the series. In the subsequent sections, we investigate the efficacy of this approximation within a regularization method in a BNP generative model. 
dpmmd-gan: a bayesian nonparametric learning in data generation: 
 In our BNPL method, we define a DP prior on F , leading to a DP posterior on F given the data. The key idea is that any posterior on the generator's parameter space W can be derived by mapping F pos approximately through the push-forward measureEQUATIONwhere for a given sample Y 1:m ∼ F Gω the posterior-based MMD estimator is defined byEQUATIONIn this context, the discriminator D can be viewed as a black box that uses the MMD estimator to differentiate between the real and fake data, reducing the computational cost compared to a neural network-based discriminator. We discuss some of the properties of estimator (3) in the Appendix. Let ω * be the optimal parameter of G ω that minimizes MMD BNP (F pos N , F Gω,m ). Since MMD BNP (F pos N , F Gω,m ) serves as a BNP estimation of ( 5), it is crucial to evaluate this estimation's accuracy, focusing on the GAN's ability to generate realistic samples (generalization error) and handle outliers (robustness). Lemma 2 in the appendix addresses these aspects. While the previous statements provide upper bounds for the MMD estimator's expectation, the next lemma offers stochastic bounds on the estimation error to assess posterior consistency.Lemma 1 Building upon the general assumptions stated in Lemma 2, for a given sample X 1 , . . . , X n from distribution F in the probability space (X, A, Pr) and any ϵ > 0, i.Pr|MMD(F pos N , F G ω * ,m ) -MMD(F, F G ω ′ )| ≥ h(n, m, K, ϵ) + |∆ 1 | + |∆ 2 | ≤ 2 exp -ϵ 2 nm 2K(n+m) , ii. Pr (MMD(F, F G ω * ) > ϵ) ≤ 1 ϵ MMD(F, F G ω ′ ) + 2K √ n + 4aK a + n + 2 (a + n + N )K (a + n + 1)N . , where, h(n, m, K, ϵ) = 2 √ K( √ n + √ m)/ √ nm + ϵ, ∆ 1 = MMD(F pos N , F G ω * ) - MMD(F n , F G ω ′ ,m ), and ∆ 2 = MMD(F, F G ω * ) -MMD(F, F G ω ′ ).A direct consequence of Lemma 1(ii) is that for a fixed value of a, Pr(MMD(F, F G ω * ) ≥ ϵ) → 0, as n → ∞ and N → ∞, for any ϵ > 0, when MMD(F, F G ω ′ ) = 0 (well-specified case). This implies F G ω * converges in probability to the data distribution F as the sample size increases in well-specified cases. A detailed guide on choosing DP hyperparameters and kernel settings is provided in the Appendix. 
experimental results: 
 We consider the MNIST dataset including handwritten digits with 10 modes, bone marrow biopsy (BMB) histopathology, Labeled Faces in the Wild (LFW) Dataset, and brain MRI images to analyze the model performance. All data description are given by the Appendix. Following the design choices of [22] , we use the Gaussian neural network for the generator with four hidden layers each having rectified linear units activation function and a sigmoid function for the output layer. We also set mini-batch sizes to be n mb = 1, 000 and use a mixture of six Gaussian kernels corresponding to the bandwidth parameters 2, 5, 10, 20, 40, and 80 to train networks in 40, 000 iterations. We generate samples from the trained BNP GAN using Algorithm 2 from the Appendix, as depicted in Figure 1 -Row 2. The results of [22] are also presented by Figure 1 -Row 3, as the frequentist counterpart of our BNP procedure. Based on these preliminary results, we can see that our generated images can, at least, replicate the results of [22] and in some cases produce sharper images. This result can also be deduced from the presented scores of MMD, Kernel Inception Distance (KID) [23] and Frechet Inception Distance (FID) [24] in Table 1 . The low scores suggest better performance, as smaller values indicate closer similarity to real images. Conversely, our results show that the BNP GAN with a mixture of Gaussian kernels outperforms the single Gaussian kernel approach. To explore this further, Figure 2 in the Appendix presents samples from the trained generator using various σ values and the median heuristic σ M H . Note that σ M H is updated in each iteration, so no specific value is reported. Although higher σ increases image diversity, the resolution remains below that achieved with the mixture kernel.To assess whether the proposed discriminator used in the BNP GAN leads to faster or better convergence of the generated samples compared to the baseline proposed by [22] , we consider the synthetic distribution 1 2 N (-1 d , I d ) + 1 2 N (1 d , I d ) as the true distribution and provide the corresponding MMD values for both models over 20,000 iterations in the data generation process, as shown in Figure 3 of Appendix. Our proposed GAN clearly displays a higher speed of convergence for the corresponding cost function to zero, and thus better performance compared to the baseline.  
concluding remarks: 
 Our BNP approach effectively estimates the MMD between an unknown and an intractable parametric distribution, showing promise in training GANs by using the estimator as a discriminator to induce a posterior on the generator's parameters. The stick-breaking representation, however, lacks normalization and shows stochastic decrease, making it inefficient for simulating from a DP [21] .Exploring alternative DP approximations for MMD estimation is a promising direction for future research. Future work will also focus on generating 3D medical images to further improve results.For a given data space X, consider the random variables X and Y, drawn from distributions F 1 and F 2 respectively. Here, F 1 and F 2 belong to B(X), which represents the set of Borel probability distributions on X. We consider the discrepancy d : B(X) × B(X) → [0, ∞) through the integral pseudo-probability metric (IPM) [25] , defined as shown in (4) . The class of functions F is designed to be rich enough to distinguish between F 1 and F 2 , and restrictive enough to provide accurate estimates based on a finite sample.EQUATION)The MMD is then defined by considering F = {h ∈ H k | ||h|| H k ≤ 1},X ∈ X, h(X) = ⟨h, k(X, •)⟩ H k , where ⟨•, •⟩ H k represents the inner product in H k . Consider function µ F1 (•) = E F1 [k(X, •)] ∈ H k ,which is defined as the kernel mean embedding of the distribution F 1 in [26] . Then, for given X, Theorem 5] . In practice, distributions F 1 and F 2 are not accessible, and then the biased, empirical estimator of (5) (V-statistic) is calculated using empirical distributions F 1,n and F 2,m asX ′ i.i.d. ∼ F 1 , Y, Y ′ i.i.d. ∼ F 2 , if E F ( k(X, X)) < ∞ for all F ∈ B(X), the MMD is given by MMD 2 (F 1 , F 2 ) = ||µ F1 -µ F2 || 2 H k = E F1 [k(X, X ′ )] -2E F1,F2 [k(X, Y)] + E F2 [k(Y, Y ′ )]. (5) Note that MMD 2 (F 1 , F 2 ) = 0 if and only if F 1 = F 2 , when H k is a universal RKHS defined on a compact metric space X and k(•, •) is continuous [26,EQUATIONwhere X 1 , . . . , X n is a sample from F 1 and Y 1 , . . . , Y m is a sample generated from F 2 .In the context of approximating the posterior on the parameter space, the prior choice for F and determining the strength of belief becomes challenging. We consider a small value for a as a non-informative prior, following the suggestion by [1] , thanks to its broad ability to characterize uncertainty [27] . However, it's important to note that setting a = 0 as done by [1] is not always well-defined mathematically, as the DP is only defined for a > 0. Therefore, we opt for a = 10 -6 . In this case, the DP posterior remains invariant to the choice of H.In our method, we choose to use the standard radial basis function (RBF) kernel as its feature space corresponds to a universal RKHS. [22, 28] and [29] used the Gaussian kernel in training MMD-GANs because of its simplicity and good performance. [28] also evaluated some other RBF kernels such as the Laplacian and rational quadratic kernels to compare the results of the MMD-GANs with those obtained based on using Gaussian kernels. They found the best performance by applying the Gaussian kernel in the MMD cost function.Hence, we consider the Gaussian kernel function in our proposed procedure. To choose the bandwidth parameter σ, we follow the idea of considering a set of fixed values of σ's such as {σ 1 , . . . , σ T }, then compute the mixture of Gaussian kernels k(•,•) = T t=1 k Gσ t (•, •), to consider in (3). For each σ(t), 0 ≤ k Gσ t (•, •) ≤ 1; hence, 0 ≤ k(•, •) ≤ T ,which satisfies the theoretical results presented in the paper. As it is mentioned in [22] , this choice reflects a good performance in training MMD-GANs.The construction of MMD-based procedures is proposed based on considering a kernel function with feature space corresponding to a universal RKHS. The radial basis function (RBF) kernel is the most well-known kernel family satisfying the above situation. For two vectors X, Y ∈ R d , the RBF kernel is represented byk(X, Y) = h(||X -Y||/σ),where, h is a function from the positive real numbers R + to R + , || • || represents the L2 -norm, and σ is the bandwidth parameter that indicates the kernel size. There are many functions assigned to h, for example, the Gaussian, exponential, rational quadratic kernels, and Matern, represented byEQUATIONrespectively; where, α in h 3 is a positive-valued scale-mixture parameter, and the ν in h 4 is a parameter that controls the smoothness of the kernel results [30, 31] .One of the simplest kernel functions above is the Gaussian kernel, which is mostly used in machine learning problems and only depends on bandwidth parameter σ. The Gaussian kernel tends to 0 and 1 when σ → 0 and σ → ∞, respectively. Both situations lead to MMD 2 being zero. Hence, the choice of the parameter σ has a crucial effect on the performance of this kernel. Numerous methods are proposed to choose the value of σ, however, there is no definitive optimization method for this problem. The median heuristic is one of the first methods used in choosing σ empirically and will be denoted in our experimental results by σ M H . More precisely, for two samples [32] . Selecting σ based on maximizing the power of two-sample problems is another strategy considered by [33] . The selection of the MMD bandwidth on held-out data to maximize power was first proposed by [34] for linear-time estimates and by [35] for quadratic-time estimates. Recently, bandwidth selection without data splitting has been proposed for quadratic [36] and linear [37] MMD estimates. Regarding the choice of σ in kernel-based GANs, a common idea is assigning several fixed values to σ and then considering the mixture of their corresponding Gaussian kernel. This strategy has received much attention and shown an acceptable performance in training GANs 2 . Generate a sample of noise vector {u ℓ } n mb ℓ=1 from uniform distribution U (-{X i } n i=1 and {Y i } m i=1 , the σ M H is considered as the median of {||X i -Y j || 2 : 1 ≤ i ≤ n, 1 ≤ j ≤ m}, which is mostly used in kernel-based testsGenerate a sample fromF Gω i , distribution of G ωi , as {y ℓ = G ωi (u ℓ )} n mb ℓ=1 9:Generate a sample of size N fromF pos = F |{x mb ℓ } n mb ℓ=1 using N i=1 J * i,N δ v * i . 10:Use generated samples in steps 9 and 10 to compute MMD 2 BNP (F pos N , F Gω i ,N ).Compute the gradient:∂MMDBNP(F pos N , F Gω i ,m ) ∂ωi = 1 2 MMD 2 BNP (F pos N , F Gω ,m ) ∂MMD 2 BNP (F pos N , F Gω ,m ) ∂ω .12:Use backpropagation for calculating partial derivatives∂Gω i (u ℓ ) ∂ωiin the previous step to update parameter ω i . 13: end for 14: return ω * ▷ An optimized parameter for G ω that minimizes the cost function.Proof of Theorem 1 For samples {V ℓ } N ℓ=1 and {Y ℓ } m ℓ=1 , respectively, from H and F 2 , the triangle inequality impliesMMD 2 BNP (F pri 1,N , F 2,m ) -MMD 2 (H N , F 2,m ) ≤ K N ℓ,t=1 J ℓ,N J t,N - 1 N 2 + 2 m N ℓ=1 m t=1 J ℓ,N - 1 N .By Proposition 1, which provides some theoretical properties of the DP approximation given in (7) , the right-hand side of the above inequality converges almost surely to 0 as a → ∞ for fixed N . This convergence immediately concludes the proof of (i). To prove (ii), sinceEQUATIONNow, it is sufficient to compute the following conditional expectation,EQUATIONSince sets {V i } N i=1 and {Y i } m i=1 include i.i.d. random variables, separately, replacing (8) in expectation (9) implies:EQUATIONThe proof of (ii) is concluded by letting a → ∞, N → ∞, and m → ∞ in the above equation. Lastly, since 1 m < 1, m-1 m < 1, a(N -1) (a+1)N < 1, and a+N (a+1)N < 2, then, for any N, m ∈ N and a ∈ R + , (10) < E H [k(V 1 , V 2 )] -2E H,F2 [k(V 1 , Y 1 )] + E F2 [k(Y 1 , Y 2 )] + 3K,which concludes the proof of (iii).Lemma 2 Let W be the parameter space for G ω and ω * ∈ W be the value that optimizes the objective function (2) and ω ′ be the true value that minimizes MMD(F, F Gω ). Assume that F ∼ DP (a, H) and let k(•, •) be any continuous kernel function with feature space corresponding to a universal RKHS defined on a compact metric space X such that |k(z, z ′ )| < K, for any z, z ′ ∈ R d . For a given sample X 1 , . . . , X n from distribution F : i. Generalization error:E (MMD(F, F G ω * )) ≤ MMD(F, F G ω ′ ) + 2K √ n + 4aK a + n + 2 (a + n + N )K (a + n + 1)N .ii. Robustness: Suppose there exist outliers in the sample data, which arise from a noise distribution Q. Consider the Hüber's contamination model [38, 39] , given by F = (1 -ϵ)F 0 + ϵQ, where ϵ ∈ (0, ∼ F 0 (cleaned data) if Z i = 0; otherwise, X i i.i.d. ∼ Q. Then, E (MMD(F 0 , F G ω * )) ≤ min ω∈W MMD(F 0 , F Gω ) + 4ϵ + 2K √ n + 4aK a + n + 2 (a + n + N )K (a + n + 1)N .The proof of Lemma 2(i) relies on the proof given in [1, Theorem 9] which is expanded for infinite stick-breaking representation, while we consider the finite DP approximation given in (7) . By employing a similar technique as in the previously mentioned theorem, we haveE (MMD(F, F G ω * )) = E F (E F pos MMD(F, F G ω * )|X 1:n ) ≤ min ω∈W MMD(F, F Gω ) + 2E F (MMD(F n , F )) + 2E F pos (MMD(F pos N , H * )) + 2E F (E H (MMD(F n , H * )|X 1:n )) .Building on the results of [1, Lemma 7] , we can establish thatE F pos MMD 2 (F pos N , H * ) ≤ N ℓ=1 E F pos [J * 2 ℓ,N ]E H * [k(V * ℓ , V * ℓ )] ≤ (a + n + N )K (a + n + 1)N ,where the right-hand side of the above inequality follows from the fact that k(•, •) ≤ K andEQUATIONa+n+1)N 2 . Now, the Jensen's inequality impliesE F pos (MMD(F pos N , H * )) ≤ (a + n + N )K (a + n + 1)N .On the other hand, [39, Lemma 7.1] and [1, Lemma 8] , respectively, imply thatE F (MMD(F n , F )) ≤ K √ n , E F (E H (MMD(F n , H * )|X 1:n )) ≤ 2aK a + n ,which concludes the proof of (i). To establish (ii), we adopt the approach used in the proof of [ E (MMD(F 0 , F G ω * )) ≤ 2ϵ + E (MMD(F, F G ω * )) .Applying the result in (i) to the right-hand side of the above inequality implies:E (MMD(F 0 , F G ω * )) ≤ 2ϵ + min ω∈W MMD(F, F Gω ) + 2K √ n + 4aK a + n + 2 (a + n + N )K (a + n + 1)N .Finally, we employ [39, Lemma 3.3] once again, but this time to bound MMD(F, F Gω ) by 2ϵ + MMD(F 0 , F Gω ) for any ω ∈ W, thereby completing the proof of (ii).Let L BNP (ω) = MMD(F pos N , F Gω ), L n,m (ω) = MMD(F n , F Gω,m ), and L(ω) = MMD(F, F Gω ). Then, for ω * ∈ W, [26, Theorem 7] impliesEQUATIONHence, with a probability at leastEQUATIONOn the other hand, the triangle inequality impliesEQUATIONFinally, the proof of (i) is concluded by considering inequality (12) in (13) . To prove (ii), Markov's inequality impliesPr (MMD(F, F G ω * ) ≥ ϵ) ≤ E (MMD(F, F G ω * )) ϵ .The result follows by substituting the bounds from Lemma 4(i) into the right-hand side of the above inequality. The MNIST dataset includes 60,000 handwritten digits of 10 numbers from 0 to 9 each having 784 (28 × 28) dimensions. This dataset is split into 50000 training and 10000 testing images and is a good example to demonstrate the performance of the method in dealing with the mode collapse problem. We use the training set to train the network.F.1.2 BMB [40] :The bone marrow biopsy (BMB) dataset is a collection of histopathology of BMB images corresponding to 16 patients with some types of blood cancer and anemia: 10 patients for training, 3 for testing, and 3 for validation. This dataset contains 10,800 images in the size of 28 × 28 pixels, 6,800 of which are considered for the training set. The rest of the images have been divided into two sets of equal size for testing and validation. The whole dataset can be found at https://github.com/jmtomczak/ vae_householder_flow/tree/master/datasets/histopathologyGray. The results based on 6800 training images are presented.The labeled faces in the wild dataset (LFD) include 13,000 facial image samples with 1,024 (32 × 32) dimensions. The dataset is available at https://conradsanderson.id.au/lfwcrop/.The sensitivity to the bandwidth parameter of the Gaussian kernel is given by Figure 2 . The learning rate of BNPMMD GAN versus FNP Counterpart is given by Figure 3 . For further details, see[22] and[29]. 
 
 Appendix:  



Reviews: review: title: title: Integration of a Bayesian nonparametric (BNP) estimator for the maximum mean discrepancy (MMD) into generative adversarial networks (GANs), with reasonable theoretical support but limited empirical evidence.
main_review: main_review: The paper proposes integrating a Bayesian nonparametric (BNP) estimator for the Maximum Mean Discrepancy (MMD) into the training of generative adversarial networks (GANs) by using it as a discriminator, which is the novelty of this work. The motivation behind this is to address typical challenges with GANs, such as reducing mode collapse and simplifying optimization. The paper provides original theoretical guarantees on the consistency, generalization, and robustness of their estimator, though these may not fully apply in practice due to the complexities of GAN training.

For evaluation, the paper provides preliminary empirical results on datasets such as MNIST. However, the empirical results are limited and do not sufficiently support the advantages of the proposed method compared to frequentist MMD GANs or even regular GANs (see cons below). This raises questions about the significance of the paper, as the empirical results may not support the motivation behind BNP MMD GANs.

## Pros:

- The paper proposes a novel idea of incorporating Bayesian uncertainty into Maximum Mean Discrepancy for GANs. The idea itself, independent of any specific application or use case, is an interesting problem to tackle in Bayesian statistics. The theoretical arguments are sound (though with some concerns raised in the cons) and support the theoretical claims.
- The paper effectively identifies the issue that different data distributions have different uncertainties which cannot be handled by frequentist MMD GAN formulations. It is important to take that into account through a Bayesian formulation.
- The paper is clear and effectively conveys the main idea.
- **L112:** It is commendable that the paper acknowledges that the provided empirical results are preliminary (line 112).

## Cons:

- The empirical evaluation is limited and does not support the claims. In particular:
  - In Table 1, the paper should include the performance of state-of-the-art neural network-based discriminator GANs. It appears that the authors assume MMD GANs outperform such GANs in metrics like FID and KID. However, [1] shows that neural network-based discriminator GANs perform quite well (see Table 3 in Appendix F of [1]). If MMD GANs cannot outperform these models, the only advantage to discuss would be the optimization dynamics of MMD GANs being easier than regular GANs, and the claims on reducing mode collapse may not hold.

  - Regarding the comparison to frequentist MMD methods in Figure 1 and Table 1, the details about the frequentist MMD (FNP) are unclear. Are these results reproduced from prior works? What are the details? It seems that competing methods might perform better than what is shown in the paper based on the literature. For instance, in [2], their MMD GAN generates MNIST samples (see Figure 5.1(c) in [2]) that are visually better compared to the BNP samples in this paper (Figure 1, row 3). This is crucial because if this is the case, then the motivation for improving MMD GANs by incorporating a Bayesian posterior is weakened, and the claims that BNP MMD GANs reduce mode collapse are challenged.

  - FID and KID are random measurements. The paper lacks confidence intervals for their measurements.

  - It would be beneficial to showcase the method on the CIFAR dataset to demonstrate its effectiveness on more complex data.

- While a fine-grained study of the optimization dynamics may be out of scope, the discussion on "better" convergence in **L122-126** is not convincing. First, the experiment is on a mixture of Gaussian distributions. In such a scenario, does a regular GAN have problems with convergence? (or is it difficult to train a regular GAN on such data?) Second, how generalizable is this argument? Would we see the same behavior on MNIST or more challenging datasets? Third, in Figure 3, it appears that both FNP and BNP converge to their local minima at almost the same speed, so it's unclear why the paper claims otherwise. Lastly, since the cost functions of FNP and BNP are different (though both are MMDs), the authors should clarify why comparing different cost functions is informative here.

- In the Lemma 1.ii, the argument is made for $ w^* $. However, in practice, for GANs there is no guarantee of finding the optimal weights due to the non-convexity of the cost function. Therefore, the lemma may not have practical application in the GAN context.

- The theoretical proofs in the appendix feel disjointed. The propositions and theorems are presented without sufficient references or explanations. The whole section is almost an orphan in the main paper, making it difficult to read and connect to the lemmas in the main body. The notation is poorly explained or missing. For instance, what does \( w' \) refer to in **L92**? Moreover, there are issues like referring to Algorithm 2 in **line 110**, while there is no Algorithm 2 in the paper, or referring to Lemma 4 in **line 394**, while there is no Lemma 4.

## References

[1] M. Bińkowski, D. J. Sutherland, M. Arbel, and A. Gretton, “Demystifying MMD GANs” in International Conference on Learning Representations, 2018.  
[2] C.-L. Li, W.-C. Chang, Y. Cheng, Y. Yang, and B. Póczos, “MMD-GAN: Towards deeper understanding of moment matching network” Advances in Neural Information Processing Systems, vol. 30, 2017.
title: title: Comments of paper #62
main_review: main_review: This paper proposes a Bayesian estimator for maximum mean discrepancy (MMD) and embeds it in a generative adversarial network (GAN), enabling a novel approach to Bayesian nonparametric (BNP) learning. BNP-GAN improves sample diversity and inferential accuracy, outperforming traditional methods. 

Pros: 
1. A novel scheme for MMD. This scheme embeds a GAN to learn the Bayesian nonparametric networks, which is novel for me. 
2. Good writing. This paper is well-written and easy to understand.

Cons:
1. Lacking of sufficient citations for better introduction of the background. For instance, it is suggested to add a related citation within the first statement in introduction “Data augmentation is the technique of generating synthetic data, often to train machine learning models when the data are scarce or the model is non-robust to perturbations in the data.”
2. Unclear statement of dataset selection. In the experimental section, the authors choose several tiny datasets like MNIST and LFW. While I understand that these tiny samples can better fit the Bayesian theoretical proofs, I would like to see more analysis about the relationship between the experimental results from larger data samples like ImageNet and Bayesian lemmas, or theorems. It may be more convincing when proposing the scheme.