Probabilistic Matching of Real and Generated Data Statistics in Generative Adversarial Networks
Authors: Philipp Pilar
Niklas Wahlström
Venue: Accepted by TMLR ()

Abstract: Generative adversarial networks constitute a powerful approach to generative modeling. While generated samples often are indistinguishable from real data, there is no guarantee that they will follow the true data distribution. For scientific applications in particular, it is essential that the true distribution is well captured by the generated distribution. In this work, we propose a method to ensure that the distributions of certain generated data statistics coincide with the respective distributions of the real data. In order to achieve this, we add a new loss term to the generator loss function, which quantifies the difference between these distributions via suitable $f$-divergences. Kernel density estimation is employed to obtain representations of the true distributions, and to estimate the corresponding generated distributions from minibatch values at each iteration. When compared to other methods, our approach has the advantage that the complete shapes of the distributions are taken into account. We evaluate the method on a synthetic dataset and a real-world dataset and demonstrate improved performance of our approach.

Published in Transactions on Machine Learning Research (09/2024)
Probabilistic Matching of Real and Generated Data Statistics in Generative Adversarial Networks
Philipp Pilar
philipp-pilardee pilar su.se
Deportment of Information Technology
Uppsala University
Niklas Wahlström
miklas. www.wwwwwwww........ ut..e
Department of Information Technology
Uppsala University
Reviewed on OpenReview:
net/ pert/formm?eee id= oloetB/uSu
Abstract
Generative adversarial networks constitute powerful approach to generalive modeling. While generated samples often indistinguishable from real data, there is guarantee that they will follow the true data distribution. For scientific applications in particular, it is essential that the true distribution is well captured by the generated distribution. In this work, we propose a method to ensure that the distributions of certain generated data statistics coincide with the respective distributions of the real data. In order to achieve this, we add new loss term the generator loss function, which quantifies the difference between these distributions via suitable f-divergences. Kernel density estimation employed to obtain representations of the true distributions, and to estimate the corresponding generated distributions from minibatch values at each iteration. When compared to other methods, our approach has the advantage that the complete shapes of the distributions are taken into account. We evaluate the method on a synthetic dataset and a real-world dataset and demonstrate improved performance of our approach.
1 Introduction
Generative adversarial networks (GANs) GGoodellow et al. 2014) comprise a generator and a discriminator network trained adversarially until the generator manages to produce samples realistic enough to fool the discriminator. Since their conception, GANs have become a popular tool for generative modeling (Hong et al., 2019 Gui et al. h The GAN framework is generally applicable and it is probably best known for its
successes in image generation (Reed et al. 2016 Mathieu et. al. 2016 Isola et al. 2017 Ledig et al. 2017
Although GANs have proven powerful, challenges such as mode collapse and non-convergence remain Saxena and Cao 2021). It often the case that the generated samples, while realistic, stem from only a subspace of the true data distribution, or do not reflect the relative frequencies with which they occur accurately. For scientific applications particular, such as in cosmology (Rodriguez et al. 2018 Villacscusa-Navarro et al. 2021 | high-energy physics (Paganini et al., 20188] Alanazi al. 2021). where GANs may serve differentiable surrogate models for expensive but highly accurate numerical simulations, having a good match between the distributions is essential (Kansal et al. 2023
It is this latter aspect Ihat we tackle in this work, by matching properties of the generated distribution with those of the real data distribution. In particular, we consider statistics of the dataset such as the power spectrum components, and match their distributions. We incorporate these requirements in the form of probabilistic constraints since it not propertics of individual samples that are enforced, but collective characteristies of the dataset. The approach is chiefly aimed at applications in science, where suitable statistics

---

Published in Transactions on Machine Learning Research (09/2024)
to be matched can be chosen through domain knowledge. The only requirement the statistics is that they need to be differentiable.
The main ingredients of our approach are the following: we approximate both the distributions of the real data and the generated data statistics efficiently via kernel density estimation (KDE) (Silverman 1986.. In each iteration, the mismatch between true and generated distributions then calculated through suitable f-divergences and added as an additional term to the generator loss. That way, we end up with constrained generated distribution. Using f-divergences, opposed to e.g. low-order moments of the distributions, has the advantage that the full shapes of the distributions are taken into account. In the following, we refer to our method as probabilistcally constrained GAN (peGAN).
2 Related Work
The field of plysics-informed machine learning, where prior knowledge is introduced into the ML model, has been an active area of research in recent years (Karniadakis et al., 2021 Cuomo et al., 2022). In the context of GANs, two main approaches for including prior knowledge in the model exist.
In the first approach, the constrained values can be fed as additional inputs into the discriminator, such that it can explicitly use constraint fulfillment as means to distinguish between real and generated data. In Stinis et al. (2019), GANs are employed for interpolation and extrapolation of trajectories following known governing equations. The generated trajectories are constrained to fulfill these equations by passing the constraint residuals as additional inputs to the discriminator; in order to prevent the discriminator from becoming too strong, some noise added to the residuals of the real data, which might otherwise be very close to zero. When extrapolating, the GAN is applicd itcratively from some initial condition; in order train stably, it learns to predict the correct trajectory from slightly incorrect positions of the previous step.
In Yang et al. (001)), a physically-informed GAN (PI-GAN) is developed to model groundwater flow. They make use of the same basic idea as physics-informet neural networks (Raissi et al.) 2019) and employ automatic differentiation in order to obtain a partial differential equation (PDE) residual on the GAN output, which is in turn fed into the discriminator. By evaluating the GAN prediction at many different points and comparing to an equivalent ensemble of true values of the corresponding physical field, the GAN is constrained to adhere to stochastic PDE.
In the second approach, prior knowlodge may be taken into account via additional loss terms in either discriminator generator loss: in Khattak et al. (2018 2019), GANs are employed to simulate detector signals for high-energy physics particle showers. Here, physical constraints such as the particle energy are taken into account via additional generator loss terms.
In Yang et al. p.21p, the incorporation of imprecise deterministic constraints into the GAN is investigated; e.g. the case where the GAN output is supposed to follow a PDE, but where the PDE parameters are not known accurately could be formulated as an imprecise constraint. In first step, deterministic constraints can be included by adding the constraint residuals an additional loss term to the generator loss; they argue that it is better to add such terms to the generator since this strengthens the weaker party in the adversarial game, instead of giving an even larger advantage to the discriminator. In order to make the constraint imprecise, they do not require that the residuals go to zero, but instead only include residuals above a certain threshold value e² in the loss.
The work closest in aim to ours is probably that by Wu et al. 2020). where a statistical constrained GAN is introduced. They add an additional term to the generator loss function in order to constrain the covariance structure of the generated data to that of the true data, This additional term is measure of similarity between the covariances, and they concluded that the Frobenius norm was the best choice for this purpose. They use their method to obtain better solutions for PDE-governed systems.
Similar to Wu et al. 2020), our method also imposes probabilistic constraints via an additional term to the generator loss. However, there are significant differences: firstly, our method does not consider the covariance structure of the dataset in particular, but instead allows to constrain on arbitrary statistics of the data. Secondly, our method uses f-divergences to match the distributions of true and generated data statisties
2

---

Published in Transactions on Machine Learning Research (09/2024)
explicitly and takes the complete shapes of the distributions into account, instead of only the second-order moments.
3 Background
The basic idea of generative adversarial networks (GANs) (Goodfellow et al. 2014) is to train a generator to generate samples of a given distribution and a discriminator (or critic) to distinguish between real and generated data. During the training, both networks are pitted against each other in a minimax game with value function
min max V(D,) G) Ex-passs (x) [log [xx)] + (z) [log(1 - D(G(z))).
(1)
Here, D denotes the discriminator, G the generator, x samples drawn from the real data and randomly generated latent space vectors serving as input to the generator; Pdaza and ps denote the real data distribution and the latent vector distribution, respectively. Discriminator and generator are then trained alternatingly (with m 2 1 discriminator updates between cach generator update); in (Goodfellow et al. 2014), it is shown that stable equilibrium to the minimax problem [ exists and that the optimal solution lies in the generator producing samples from the true data distribution.
The standard GAN can be very difficult to train and often suffers from mode collapse. In Arjovsky et al. (2017), the Wasserstein GAN (WGAN) was introduced, where they suggest the carth-mover (EM) distance new loss for the GAN. They show that the discriminator and generator losses can then be expressed
D = ((rgen) - D(Frrue), La = D(Igen),
(2a)
(2b)
under the condition that the discriminator is Lipschitz continuous. Rather crudely, this is enforced by clipping the weights of the discriminator. In the end, the terms in are approximated as expectations over
minibatches.
With this loss function, the discriminator can be interpreted as critic that assigns scores to both true and generated samples. These scores are not constrained any specific range and can therefore give meaningful feedback the generator also when the discriminator is outporforming. Advantages of the WGAN include improved learning stability as well as meaningful loss curves (Gui et al. 2021
In this work, we also consider two other common variants of the GAN: firstly, the WGAN with gradient penalty (WGAN-GP) /Gulrajani et al.) 2017), where the aforementioned weight clipping is avoided by instead imposing a penalty on the discriminator that is supposed to enforee Lipschitz continuity. Secondly, the spectrally normalized GAN (SNGAN) (Miyato et al., 2018), where Lipschitz continuity ensured by constraining the spectral norm of each layer of the discriminator explicitely.
4 Method
The aim of our method is to consider the distributions of N, differentiable statistics z of the true dataset, such e.g. components of the power spectrum (compare Appendix C.I), and to ensure that the same statistics, when extracted from the generated data, are distributed equally.
In order to match true (Porue) and generated (pgen) distributions, we modify the generator loss 2b as follows:
N.
CS = La + Σ Xeh(pere (2a))
(3)
The function h is an f-divergence that quantifies the mismatch between Ptrne and Pgen. À is a global weighting factor for the constraints, and the A. for which Σ. d = 1, allow to weight the constraints individually.
Three important choices remain be made: how to choose the function h, how to obtain suitable functional representations for Ptrue and Pgen; and how to adequately weight the different loss terms.
3

---

Published in Transactions on Machine Learning Research (09/2024)
Prrae
Poen (bs 64)
(optimal Ø)
P
0.05
0.05
0.25
0.05
0.51
b2-32
bs- twith history)
4,06
0.04
0.04
10.25
0.04
b56
0.04
0.92
0.02
0.02
0.02
0.01
0.01
0.61
0.01
10
20
30
4.0
50
Figure 1: The various representations involved in matching the statistic are depicted. The histogram in the background shows the true data distribution. Left:: Representation of the true data distribution. Middle left: Representation of the generated data distribution with batch size 64 for different choices of in the kernel. Middle right: Representation of the generated data distribution for various batch sizes with optimal choice for (as determined via Algorithm g in the appendix). Right: Taking the recent minibatch history into account (here with 0.9) smoothen out fluctuations and lead In this figure, perfectly trained generator has been assumed, i.e. the minibatches have been sampled from real data.
accurate representation.
4.1 Quantifying the Mismatch
Let p, q be arbitrary probability density functions (PDFs). For f-divergences h, it holds that h(p, g) with equality if and only if p These properties justify the use of f-divergences for the function h in (3) A major advantage of using f-divergenoes, as opposed to e.g. the Wasserstein distance, is that they are efficient to calculate.
> 0,
The Kullback-Leibler (KL) divergence constitutes a straightforward choice for and is defined
Mp. = KI(pplq) = / p(x) log
000
p(z) dr.
(4)
q(x)
The KL divergence is asymmetric and we consider the forward KL, also known as zcro-avoiding, in order to ensure a complete overlap of areas with non-zero probability of the distributions; in case of the reverse, or zero-foreing, KI,, the loss term would typically tend to match g to one of the peaks of p and hence fail to match the distributions in a way suitable for our purposes.
The Jeffreys divergence (JD), which can be thought of a symmetrized version of the KL divergence, as well the total variation (TV) distance constitute further options:
h(p.g) J(pllq) (KL(p)la) KL(g)p)),
(5)
h(p.g) = V(p
(p(x) g(z)]dr.
(6)
An advantage of the latter choice is that no divisions by zero can occur, which may cause problems with the other two options.
As an alternative to using f-divergences, also discuss the maximum mean discrepancy (MMD) possible loss function in Appendix B.2] We show that there are drawbacks to using the MMD loss and that the method performs better when using f-divergences.
4.2 Obtaining Representations
In order to evaluate the loss termss in 3 means of extracting representations for both the true and generated PDFs are required. We denote these representations as Ptrue and Pgen- Note that Perue will need to be determined only once, in advance of the GAN training, since remains constant. In contrast to the true
4

---

Published in Transactions on Machine Learning Research (09/2024)
distribution, the generated distribution changes dur- ing GAN training, and hence Pgen also needs to be determined anew after each generator update.
Algorithm 1 High-level algorithm
Step 1: obtain Phrte via (7)
Kernel density estimation (KDE) [Silverman 1986) has proven effective for obtaining these representa- tions. For the true distributions, we then get
Step 2: determine the optimal values f² in
(sce Algorithm 3 in the appendix)
Step 3: train the pcGAN (see Algorithm
N
(irue(2s)
Zo
Zzj
A Algorithm 2 Training the probabilistieall constrained GAN (pcGAN)
where N denotes the number of datapoints, K the kernel function and oz the bandwidth of the ker- (Exz)) has
Input: Untrained D and G; Peruei data fotrue);
nel. The choice o. = 200 (Inax;(2x))
h; X
proven to give accurate representations for the true distributions (compare e.g. the leftmost plots in Figs. 2 and 5), as typically have N > 1000 and can afford to choose such small value. Throughout the paper, we use Gaussian kernels with K (x) (2m)-1/2g-="))) We evaluate different kernel choices in Appendix B.1
Output: Trained D and G for i i= to Nú do for k = to m m-1 do
sample Strue
generate Igen LD = (Dcan(((rgen)) (fttrue))
update D clip weights end for
We also approximate the generated distributions at each iteration using KDE, using the constraint val- ues as obtained from the current minibatch samples. That is, we obtain the approximate generated PDFs
generate Igen C = - mean (Fggen) for s = 1 to N₂ do calculate statistics { Pbatch from Zgen
Plaaac
2B
2aj
determine Vz (2.) according to [10
Pgea (2a)
K
(8)
Tbatch
4, h1 (2x), P (z+))
end for
where Tbaach denotes the batch size.
= - min(l) + 0.1 min()))
For (zs), choosing a, adequately is crucial and requires more thought than in the case of Ptrue- This is due to the fact that there are much fewer samples available in the minibatches. The bandwidths a, are chosen separately for each constraint 2s, under the criterion that Pren as obtained from minibatches drawn from the true data should have mismatch as small as possible with Ptrue- Since the optimal values of a, would be expected to depend both on the range of value 2, in the true dataset and the batch size, we paramcterized them as
A, = V/- En T/
C. X xe Aole
LG Lo + LG
update G
end for
(T(ath)) std(zz)/ (Natth))
A detailed description of how to determine the optimal values for fö given in Appendix A
In order to improve the accuracy of Pgen (assuming that Pgen does not change drastically between subsequent. iterations), can include information from the preceding minibatches via exponentially decaying historical average:
gen (2.) = (1 Pee (z,) + (2,),
(z.),
(10)
where the parameter defines the strength of the decay and i denotes the current iteration. In this way, the potentially strong fluctuations between minibatches are smoothened out, allowing for a more accurate representation of Pgen-
With representation 110 for the generated distribution and 7 for the true distribution, the one-dimensional integrals required for evaluating h(Prrue: Pgen) in (3) can be carried out numerically. In Fig. the various representations are illustrated.
5

---

Published in Transactions on Machine Learning Research (09/2024)
4.3 Weighting the Constraints
The following heuristic scheme has proven effective in weighling the constraints according to how big their mismatches are relative to each other: first, we calculate m min + (.1(max(1) - min()), where i is a vector with components I, = (2)t Pgen Then we assign X, = % Σ. n.- The first term in the definition of quantifies the constraint fulfillment relative to the best-fulfiledd one and the second term prevents already well-matched constraints from ceasing to be included. The global weighting factor À needs to be tuned separately.
A high-level overview of the method is given in Algorithm a the PcGAN training is detailed in Algorithm 2
Note that, while our training algorithm is based on the WGAN, the modified generator loss 3 more general and can be used for other types of GANs well.
5 Results
In this section, we present the results obtained with our model. We introduce a set of evaluation metrics and consider a synthetic example and a real-world dataset from physics. The evaluation metrics are chosen to cover different aspects of the generated distribution and evaluate the GAN performance both the sample space and in owerrdimensional space of high-level features, as is common practice (Kansal et al. 2023). We compare the peGAN the unconstrained WGAN, WGAN-GP, SNGAN, and the statistical constrained GAN from Wu et al. (2020). We investigate the impact that training parameters have the model performance and we combine the probabilistic constraint with the different GAN variants to evaluate its potential for improving their performance.
More results are given in the appendices. In Appendix B.1 we investigate the impact that the choice of kernel for the KDE has on the model performance. In Appendix B.2, we evaluate how well the MMD loss would perform instead of the f-divergences for matching the constraints. A discussion the training time required for the different models is given in B.4] Additional information on the datasets, the training parameters, and the high-level features is given in Appendix
5.1 Evaluation Metrics
To compare the different models, we consider four evaluation metries:
The Fréchet distance in the sample space, as an alternative to the widely used Fréchet Inception distance Heusel et al. 2017). It quantifies the agreement of the first and second-order moments of the real and generated distribution and calculated via.
& = Iu 1212 Tr EE --E'--VEE'
(11)
where μ, Σ correspond to the true distribution and μt, E' to the generated distribution.
The F1-score in the space of high-level features, which is defined as the harmonic mean between precision (P) and recall (R):
PR
F 2 P+R
(12)
In the context of generative modeling, the precision is the fraction of generated samples that lie in the real data manifold and the recall gives the fraction of real data samples that lie in the generated data manifold
Sajjadi et al. 20118). They are calculated suggested in Kynkäänniemi et al. 20199) with choice k 10 for the k-ncarest noighbor.
The agreement of the distributions of the constrained statistios, by calculating the average of the total variations of the differences between their histograms:
V.
1 N
Σ
V
hist (pbiat (2e) )
Pggn (ze)),
hist
(13)
TTe code for the project is available on GitHub:
https:/ / /ttg//gtthb.ooo httpaa/coooo/ppppapp/////// PpNN

---

Published in Transactions on Machine Learning Research (09/2022)
WSSN
WGAN
SNGAN
0.04
KBC
0.04
0.04
0.04
0.04
0.03
0.00
0.03
0.03
0.03
0.03
0.72
0.02
0.22
0.02
0.02
0.02
0.01
0.00
0.01
0.01
0.01
0.01
80
[1]
[4]
ps
[5]
20
[A]
0.5
0.5
0.5
0.4
044-
0.4-
0.4-
0.4-
0.4
0.3-
0.3
6.3-
0.1
0.1
0.3
5.0
0.0
p<(15)
px]35)
[15]]
p=]15]
2.0
2.0
2.0
2.-
2.0
1.5
15-
15
15-
15-
3.5
0.5
0.2
0.6
0.6
0.0
0.2
0.6 0.6
0.0
02
0.4
0.5
0.0
0.2
0.4
0.5
0.0 0.2
0.4
0.6
0.0
0.2
[105]
0.6
Figure (Synthetic example) The distributions of three different power spectrum components ps as obtained by the different models are depicted, where the orange lines show the true distribution as obtained via KDE (7). From left to right, the histograms correspond to the real data, the pcGAN, the method of Wu et al (2020), WGAN, WGAN-GP, and SNGAN. For the histograms, 20000 generated samples have been considered (or the full dataset, in case of the real distribution). Parameters for the PeGAN: bs 256,
A = 500, = 0.9, = KL.
where Perue and pla are given by the outline of the histograms in c.g. Fig. 2 Here, the histograms have been chosen instead of KDE, in order to use a quantity that is not directly constrained (and hence without the danger of overfitting to).
The agreement between the distributions of the Ny high-level features (here denoted as r). We proceed in the same way as for the constrained statistics:
1 Nr N₁
V
Σ
V (p (2,) Pgen (Ir)).
(14)
To get an idea of the complexity of each metric, it helps consider them in the following way: the Fl score takes the full shape of the data distribution into account, dr the first two moments, and V. and Vy the marginal distributions of the constrained statistics and the chosen set of high-level features, respectively,
5.2 Synthetic Example
For our first experiment, we consider a superposition of sine waves. Fach wave consists of two sine waves, x = 1 Σ sin(wat), with angular frequencics sampled randomly from W; [1 1)), and we generate 200 equally-spaced measurements in the interval E [0, 20). In total, create 100 samples of size 200 to training data. We perform the Fourier transform for real-valued inputs for each time series in the datasct and we use the square roots of the power spectrum components (i.c. the absolute values of the Fourier coefficients) as the statistics to constrain when training the GAN; that is, we have 101 separate constraints (compare Appendix C.I
In Figure 2 results for the different GAN variants are depicted. The data generated by the PeGAN matches the true distributions very well. The method of Wu al. (2020) comes in second, managing cover the correct range of constraint values, but failing to adhere to the precise shapes of the PDFs. The unconstrained
7

---

Published in Transactions on Machine Learning Research (09/2024)
WGAN
WGAN pe (peGAN)
Wu al.
Wu et + pe
WGAN-GP
WGAN-GP
SNGAN
SNGAN + +.
+ pc
(L)
48.94166.65
12.32422.99
3.8112.13
2.49:10.75
29.3845.26
3.5020.76
49.6745.76
12.6743.51
P (t)
0.15+0006
0.18:10.05
0.16+0003
0.20:1006
0.16:0.00
0.19:0.05
0.16:0.09
0.18+0.08
V. (L)
1.110.100
0,20+0.02
0.47+0.22
0.17+0.03
1.07
0.20=0.01
1.40#0.12
0.34+0.04
V (1)
1.7010.10
0.805
0.95110112
0.75.10.05
1.61:0.155
0.70100.5
1.6310.11
0.8810.08
Table 1: (Synthetic example) The different GAN variants and their combinations with the probabilistic constraint are evaluated via different performance metrics, defined in Section 5.1 the Fréchet distance dp, the F1 score, the agreement of the constraint distributions Ve, and the agreement of the distributions of a selection of high-level features Vy. The arrows indicate whether high or low values are better. Ten runs have boen conducted per model, and the mean values plus-or-minus one standard deviation are given. Bold Tont highlights best performance. Parameters: bs = 256, e 0.9, h KL, and À = [500, 500, 500, 2500), respectively, from left to right for the constrained variants.
d7 (4)
11
(c(4)
Vy (.(4)
5.200
16.
4200
3300
0.8
14-
3.200
0.7
2.355
0.350
2300
6.325
6.3-
62-
1+10
A-300
19500
200
1-120
A-5000
A-100
A-300
4-500
Aw5000
A+300
A-500
1.20
1200
Figure 3: (Synthetic example) Different values of the weighting coefficient À are considered (with bs 256,
= 0.9, h = KT.). Ten runs have been conducted per model, and the mean values plus-or-minus one standard deviation are depicted.
WGAN, WGAN-GP and SNGAN are distinctly worse and tend to assign too much weight to the highest peak of the distribution.
In Table [ we evaluate the performance of the different models in terms of the evaluation metrics defined in Section 5. The results for the constraint distributions are well reflected here and the unconstrained versions of the GAN tend to perform worse than both the PcGAN and the method of Wu et al. (2020) on metrics other than the F1 score. When considering the Fl score, peGAN is ahead of the unconstrained models. Between the peGAN and Wu al. (2020), pcGAN outperforms Wu et al. (2020) in all metrics apart from dp, where the method of Wu et al. (2020) is slightly better. This makes sense since de only evaluates agreement of the first and second-order moments of the distribution; the latter are precisely what the method of Wu et al. (2020) constrains.
In addition to the PeGAN, results for combinations of the other GAN variants with the probabilistic constraint are also given in the table. It is apparent that adding the constraint also leads to improved performance of the other models. In Appendix B.3 a plot visualizing the constraint fulfillment equivalent Fig. given for the different constrained GANs.
In Fig. 3 the impact of the global weighting factor X is investigated. A clear improvement with increasing À is visible for most of the metries, up to À ~ 100. Overall, the value X = 500 appears to be good choice. The fact that de and Fj also improve indicates that the constraints help to produce a better diversity of samples.
In Fig. 4 we consider the impact that the batch size and the historical averaging have on the results. Both de and constraint fulfillment improve with increasing batch size, although we observe diminishing returns for batch sizes larger than 256. The inclusion of historical averaging improves dr, with higher values of yielding larger improvements, whereas constraint. fulfillment V. is only weakly affected by the choice of and the
8

---

Published in Transactions on Machine Learning Research (09/2024)
df
V. (4)
(,(4)
3500
2.15
5:50
0.50
3200
5.255
6.4-
8.200
0.00
033
2300-
2.150
1500
2.325
62
0.30
1300
2.300
0.05
0.1
be
2e-512, m0
13-3024 20320
bo
br be-226.
u- ceds
x m055 52.
se 51552 512,0002
bosk
be
be
Figure 4: (Synthetic example) Different batch sizes with and without historical averaging are considered
(with À = 500, h = KL). The different colors indicate which points belong the same batch size. Ten runs have been conducted per model, and the mean values plus-or-minus one standard deviation are depicted.
h=TV, c00.5
h=TV, e=0.9 e00.9
h=KL, c=0.5
h=KL, e=0.9
h=JD, c=0.5
h=JD, c=0.9
(L)
0.2110.06
0.15:10.03
0.15:10.0
0.1210.02
0.15110.05
0.1410.04
100
Fi (T)
0.15+00.04
0.13+0.03
0.17+0.05
0.18+0.04
0.18+0000
0.16+0.06
V. (L)
0.29:10.05
0.25:10.06
0.19:0.02
0.19::000
0.14::.001
0,16::0.02
VG (1)
0.81+00.07
0.80:10.08
0.76:10.05
0.79:t0.06
0.75++0.07
0.86+0.08
Table 2: (Synthctic example) Different choices for the f-divergence quantifying the mismatch between Ptrue and Pgon in 33 are considered (with bs 256, À 500, h KL.. Ten runs have been conducted per model, and the mean values plus-or-minus one standard deviation are given. Bold font highlights best performance.
metric Vy is negatively affected. Fi seems to be largely unaffected by the batch size and somewhat negatively affected by large values of e. The larger the batch size, the smaller the impact of historical averaging.
In Table 2 the different options for the f-divergence used for matching the statistics are evaluated. The results indicate that the Jeffreys divergence performs slightly better than the KL divergence, and the total variation is nolably worse than the other two options. Furthermore, we observe that increasing the factor for the historical averaging tends to improve the results for the total variation and the KT, divergence, but slightly decreases the performance in case of Jeffreys divergence.
We conclude that the probabilistic constraint holds promise for improving the performance of many different GAN variants. When training the pcGAN, larger batch sizes are advantagoous. For smaller batch sizes, the historical averaging can yield improvements. When choosing the f-divergence for matching the constraints, the KL divergence the Jeffreys divergence should be selected rather than the total variation. The weighting parameter À is essential to consider when tuning the PcGAN.
The architectures used for the discriminator and generator were inspired by the DCGAN architecture and the ADAM optimizer [Kingma and Ba 2015) was used for optimization. A discussion on the runtime of the different models is given in Appendix B.4 A detailed description of the architecture, settings for the training procedure, and samples as obtained from the different models can be found in Appendix C.2
5.3 IceCube- Gen2 Radio Signals
The TeeCube neutrino observatory Aartson et al. 2017 and its planned successor IceCube-Gen2 (Artssn et al. 2021) are located at the South Pole and make use of the huge ice masses present there in order to detect astrophysical high-energy neutrinos. Deep learning methodology has already been employed to extract information such as shower energy or neutrino dircction from radio-detector signals (Glascr et al.) 2023 Holmberg 2022). Holmberg (2022) also investigatod the use of GANs to simulate detector signals. We are going to consider the filtered Askaryan radio signals from Holmberg (2022), which were generated using
9

---

Published in Transactions on Machine Learning Research (09/2024)
WGAN
WGAN
SNGAN
KBE
20-
20-
20-
23-
23-
20-
10-
1.0
1.0
17a
max
This
Tix
20
20
20
20
20
10-
12-
10+
12-
13-
10-
0.
-2.0
-0. -0.6
-2.0
-03
-0.6
-1.2
-0.8
-0.6
-0.8
-06
0-
-1.0
-0.
-0.6
0
-1.0
-0
-0.6
Figure 5: (IceCube-Gen2) The distributions of minimum and maximum values as obtained by different models are compared, where the orange lines show the true distribution as obtained via KDE 7 From left right, the histograms correspond the real data, the pcGAN, the method of Wu et al. (2020) WGAN, WGAN-GP, and SNGAN. For the histograms, 20 000 generated samples have been considered (or the full dataset, in case of the real distribution). Parameters for the PcGAN: bs 256, À 2, 0.9, h KL.
WGAN
WGAN + pe (pcGAN)
Wu et al.
Wu et al. pc
WGAN-GP
WGAN-GP pc
SNGAN
SNGAN pe
de (L)
25.34=12.85
16.71+9.52
15.82=8.15
14.21+6.20
5.36+6.09
6.88+44.80
9.02=7.007
7.09+3.17
P (t)
0.35.10.17
0.3810.12
0.36.10.08
0.36.10.06
0.4810008
0.45.00.66
0.46.10.04
0.46.10.06
V. (L)
0.97+0.11
0.30=:0.06
0.99
0.30++005
0.71+0.11
0.16:t0.04
1.01
0.17+0.04
Vy (L)
0.76.10.09
0.70:10.08
0.75:10.05
0.68:10007
0.66:10.06
0.65:10.07
0.65-10.05
0.6310.04
Table 3: (ceeube-Gen2) The different GAN variants and their combinations with the probabilistic constraint are evaluated via different performance metrics, defined in Section 5.1 the Fréchet distance de, the F1 score, the agrcement of the constraint distributions Ve, and the agrecment of the distributions of a selection of high-level features Vy. The arrows indicate whether high or low values are better. Ten runs have been conducted per model, and the mean values plus-or-mimus one standard deviation are given. Bold font. highlights best performance. Parameters: bs = 256, À = 2, E 0.9, h KL.
the NuRadioMC code (Glaser et al. 2020) according to the ARZ algorithm Sllvarez-Muing et al. 2010 These signals take the form of ID waveforms and in our experiments we want to focus solely on the shape of these waves, not their absolute amplitudes; this is achieved by normalizing each signal to its maximum absolute value. We use the pcGAN to constrain the generated data on the distributions of the minimum and maximum values of the signals.
The results depicted in Fig. 5 The peGAN matches the characteristies of both minimum and maximum distribution well. In particular, it manages to match the spikes at -1 and 1 more accurately than any of the other models. The distributions as obtained via the other models also exhibit the two peaks in each of the distributions but do not reproduce their precise shapes correctly. Out of the remaining models, WGAN-GP matches the distributions best, with only slightly less pronounedd spikes at -1 and 1 than the peGAN. plot showing the constraint fulfilment. for the different constrained GANs given in Appendix B.3
In Table the evaluation metrics are given for the different GAN variants together with their constrained versions. The constraints are matched well for all of the constrained models. In terms of the remaining metrics, adding the constraint yields improvements for WGAN, Wn et al. and SNGAN. For WGAN-GP, on the other hand, a slight decrease in performance can be observed. While WGAN-GP performs best on df. and F,, WGAN-GP + pe is the best choice for overall performance when taking constraint fulfillment into account.
The network architecture used for the GANs is based on that from Holmberg 2022. 0. More details on the
training procedure, well plots of generated samples, given in Appendix
10

---

Published in Transactions on Machine Learning Research (09/2024)
6 Conclusions and Future Work
We have presented the probabilistically constrained GAN (pcGAN), method to incorporate probabilistic constraints into GANs. The method is expected to be particularly useful for scientific applications, where it especially important for generated samples to represent the true distribution accurately and where suitable statistics be matched can be identified through domain knowledge. For a given statistic 2, this is achieved by adding the mismatch between the corresponding true and generated distribution as quantified via suitable f-divergence to the generator loss. Kernel density estimation is employed to obtain representations for the distributions of the statistic. By adequately weighting the different loss terms, a large number of statistics can be matched simultaneously.
We have evaluated our method using two different datasets. Our experiments clearly demonstrate that the probabilistic constraint. effective at matching the chosen dataset statistics. In terms of the evaluation metrics, the pcGAN constitutes a significant improvement over the standard WGAN. Depending on the dataset under consideration, it can also outperform WGAN-GP, SNGAN, and the method of (Wu et al.) 2020). Combining the probabilistic constraint with GAN variants other than the WGAN also improves the respective models in most cases.
For future work, it would be interesting to extend the method to consider the joint distribution of different. statistics, in order to also include correlations between them in the constraint. Furthermore, it would be important to find a way to make the method compatible with conditional GANs in order widen the range of possible applications. Finding automated ways for obtaining suitable statisties to match, e-g- by using features of classifier networks, could improve the approach and would allow for its application to situations where insufficient domain knowledge is available. In principle, the probabilistic constraint could also be added to other types of generative models. The main requirements would be that new samples are generated during each iteration of the training procedure and that a suitable spot for adding the loss term can be identified. Investigating the applicability of the approach to other generative models, such as autoencoders (Kingma) and Welling 2014) or denoising diffusion probabilistic models (Ho et al. 2020), therefore constitutes another promising avenue for future research.
Acknowledgements
The work is financially supported by the Swedish Research Council (VR) via the project Pryysics-informed machine learning (registration number: 2021-04321). We thank the anonymous reviewers from TMLR for their constructive feedback, and Jens Sjölund for comments on an early manuscript. Our special thanks also go to Christian Glaser, for interesting discussions and for providing access to the IceCube data generator.
References
Ian Goodfellow, Jean Pouget-Abadie Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, oditors, Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
Yongjun Hong, Uiwon Hwang, Jaeyoon Yoo, and Sungroh Yoon. How generative adversarial networks and their variants work: An overview. ACM Computing Surveys (CSUR), 52(1):1-43, 2019.
Jie Gui, Zhenan Sun, Yonggang Wen, Dacheng Tao, and Jieping Ye. A review on generative adversarial networks: Algorithms, theory, and applications. IEEE transactions on knowledge and data engineering, 2021.
Scott Read, Zeynop Akata, Xinchen Yan, Lajamugen Logeswaran, Bernt Schicle, and Honglak Loc. Generative adversarial text to image synthesis. In International conference on machine learning, pages 1060 1069. PMLR, 2016.
Michael Mathien, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square error. 2016. 4th International Conference on Learning Representations, ICL. 2016.
11

---

Published in Transactions on Machine Learning Research (09/2024)
Phillip Isola, Jun- Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference compuler vision and paltern recognation, pages 1125 1134, 2017.
Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, al. Photo-realistic single image super-resolution using generative adversarial network. In Proceedings of the IEEE conference on compuler vision und pattern recognition, pages 4681-4690, 2017.
Divya Saxena and Jiannong Cao. Generative adversarial networks (GANs) challenges, solutions, and future directions. ACM Computing Surveys (CSUR), 54(3):1-42, 2021.
Andres C Rodriguez, Tomasz Kacprzak, Aurelien Lucchi, Adam Amara, Raphaël Sgier, Janis Fluri, Thomas Hofmann, and Alexandre Réfrégier. Fast cosmic web simulations with generative adversarial networks. Computational Astrophysics and Cosmology, 5(1):1-11, 2018.
Francisco Villaescusa-Navarre Daniel Anglés-Alcázan Shy Genel, David N Spergel, Rachel S Somerville, Romeel Dave, Annalisa Pillepich, Lars Hernquist, Dylan Nelson, Paul Torrey, et al. The CAMELS project: Cosmology and astrophysics with machine-learning simulations. The Astrophysical Journal, 915(1):71,
2021.
Michela Paganini, Luke de Oliveira, and Benjamin Nachman. Accelerating science with generative adversarial networks: an application to 3d particle showers in multilayer calorimeters. Physical review letters, 120(4): 042003, 2018.
Yasir Alanazi, Nobuo Sato, Tianbo Liu, Wally Melnitchouk, Pawel Ambrozewicz, Florian Hauenstein, Michelle P. Kuchera, Evan Pritchard, Michael Robertson, Ryan Strauss, Luisa Velasco, and Yaohang Li. Simulation of clectron-proton scattering events by a feature-augmenneed and transformed generative adversarial network (FAT-GAN). In Zhi-Hua Zhou, editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAT-21, pages 2126-2132, 2021.
Raghav Kansal, Anni Li, Javier Duarte, Nadezda Chernyavskaya, Maurizio Pierini, Breno Orzari, and Thiago Tomei. Evaluating generative models in high energy physics. Physical Review D, 107(():076017, 2023.
Bernard W Silverman. Density estimation for statistics and data analysis, volume 26. CRC press, 1986.
George Em Karniadakis, Ioannis Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang. Physics- informed machine learning. Nature Reviews Physics, 3(6):422-440, 2021.
Salvatore Cuomo, Vincenzo Schiano Di Cola, Fabio Giampaolo, Gianluigi Rozza, Maziar Raissi, and Francesco Piccialli. Scientific machine learning through physics informed neural networks: where we are and what's next. Journal of Scientific Computing, (3) 2022.
Panos Stinis, Tobias Hagge, Alexandre M Tartakovsky, and Enoch Yeung. Enforcing constraints for interpo- lat and extrapolation in generative adversarial networks. Journal of Compulational Physsiss, 397:108844,, 2019.
Liu Yang, Sean Treichler, Thorsten Kurth, Keno Fischer, David Barajas-Solano, Josh Romero, Valentin Churavy, Alexandre Tartakovsky, Michael Houston, Mr Prabhat, et al. Highly-scalable, physics-informed GANs for learning solutions of stochastic PDEs. In 2019 IEEE/ACM Third Workshop on Deep Learning on Supercomputers (DLS), pages 1-11. IEEE, 2019.
Maziar Raissi, Paris Perdikaris, and George Karniadakis. Physics- neural networks: A deep
learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686 707, 2019.
Gul Rukh Khattak, Sofia. Vallecorsa, and Foderico Carminati. Three dimensional energy parametrized generative adversarial networks for clectromagnetic shower simulation. In 2018 25th IEEE International Conference on Image Processing (ICIP), pages
2018
12

---

Published in Transactions on Machine Learning Research (09/2024)
Gul Rukh Khattak, Sofia Vallecorsa, Federico Carminati, and Gul Muhammad Khan. Particle detector simulation using generative adversarial networks with domain related constraints. In 2019 18th IEFF International Conference On Machine Learning And Applications (ICMLA), pages 28 33, 2019.
Zeng Yang, Jin-Long Wu, and Heng Xiao. Enforcing imprecise constraints on generative adversarial networks for emulating physical systems. Communications in Computational Physics, 30(3):635-6633 2021.
Jin-Long Wu, Karthik Kashinath, Adrian Albert, Dragos Chirila, Heng Xiao, et al. Enforcing statistical constraints in generative adversarial networks for modcling chaotic dynamical systems. Journal of Compulalional Physics, 406:109209, 2020.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generalive adversarial networks. In International conference on machine learning, pages 214-223. PMLR, 2017.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of Wasserstein gans. Advances in neural information processing systems, 30, 2017.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv: 05957, 2018
Martin Housel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. Advances in neural information processing systems, 30, 2017.
Mehdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall. Advances in newral information processing systems, 31, 2018.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems, 32, 2019.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of International Conference on Learning Representations (ICLR), 2015.
Mark G Aartsen, M Ackermann, J Adams, JA Aguilar, M Ahlers, M Ahrens, D Altmann, K Andeen, T Anderson, I Ansseau, et al. The IceCube neutrino observatory: instrumentation and online systems. Journal of Instramentation, P0000)) 2017.
Mark G Aartsen, R Abbasi, M Ackermann, J Adams, JA Aguilar, M Ahlers, M Ahrens, C Alispach, P Allison, NM Amin, et al. Lcenube-Gen2: the window to the extreme universe. Journal of Physics G: Nuclear and Particle Physics, 1866)0, 2021.
Christian Glaser, S McAleer, Sigfrid Stjärnholm, P Baldi, and SW Barwick. Deep-learning- based reconstruction of the neutrino direction and energy for in-ice radio detectors. Astroparticle Physics, 145:102781, 2023.
Anton Holmberg. Fast simulations of radio neutrino detectors: Using generative adversarial networks and artificial neural nerworks, 2022.
Christian Glaser, Daniel Garcia-Fornándes Anna Nolles, Jaime Alvaroz-Muñiz, Steven W Barwick, Dave Z Besson, Brian Clark, Amy Connolly, Cosmin Deaconu, KD de Vries, et al. NuRadioMC: Simulating the radio emission of neutrinos from interaction to detector. The European Physical Journal C, 80:1-35, 2020.
Jaime Alvarez-Muñiz Andrés Romero-Wodd Wolf, and Enrique Zas. Čerenkov radio pulses from electromagnetic showers in the time domain. Physical Review D, 81 (12):123300, 2010.
Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. In 2nd International Conference Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advonces in Neural Information Processing Systems, 33:6840-6857 2020.
13

---

Published in Transactions on Machine Learning Research (09/2024)
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-777 -73,, 2012.
Yujia Li, Kevin Swersky, and Rich Zemel. Generative moment matching networks. In International conference machine learning, pages 1718-1777 17277 PMLR, 2015.
Gintare Karolina Dziugaite, Daniel M Roy, and Zoubin Ghahramani. Training generative neural networks via maximum mean discrepancy optimization. arXie preprint orXiv:1505.00 039006 2015.
Chun-Liang Li, Wei-Cheng Chang, Yu Cheng, Yiming Yang, and Barnabás Póezos. MMD GAN: Towards deeper understanding of moment matching network. Advonces in neural information processing systems, 30, 2017.
Tommaso Dorigo, Andrea Giammanco, Pictro Vischia, Max Achle, Mateusz Bawaj, Alexey Boldyrev, Pablo de Castro Manzano, Denis Derkach, Julien Donini, Auralee Edelen, et al. Toward the end-to-end optimization of particle physics instruments with differentiable programming. Reviews in Physics, 10: 100085, 2023.
14

---

Published in Transactions on Machine Learning Research (09/2024)
A Algorithm to Determine fö
We propose method to determine optimal values for fö that is based on the assumption that data sampled from the true distribution should on aver- age have the best possible match between the true distribution and the KDE approximation obtained via the minibatches.
Algorithm 3 Determining the optimal value of fe
Input: true data {2s}; (truo(=s), h, Navg Output: fo sld(z.)
f, = logspaccee -1, 200) for in E [0, Navg); i E (o, 1;); do
The procedure is summarized in Algorithm 3 In order to determine the optimal value of fa for given constraint 2g in (9), we perform grid search over possible values for Introducing the standard deviation of zs into the definition of vía e helps to narrow the range in which the optimal values fä lie. The grid is defined in the array afs. For each value of for we evaluate the mismatch between the true distribution Perue
sample minibatch {2a}
determine Pgen(Ze) according to
S using 0
f. ix] = h(Perue (2e)) Pgen
end for
f. min (mean (H, dim 1)) fo a l...]
7 and the generated distribution Pgen (2a) S via the f-divergence h.
The minibatches are sampled from the true data since the aim is to obtain a mismatch as small as possible for true data. The obtained values for the mismatch are then averaged over Navg minibatches. Subsequently, the value fo corresponding to the minimum mean value is determined; this value is the desired optimal value Sa. Figures and ralustrate the grid search.
B Additional Results
In this appendix, we present additional experiments and results.
B.1 The Choice of Kernel
Here, we investigate the impact that the specific choice of kernel for approximating the distributions in (3) has on the performance the PeGAN. We consider the following kernels (compare Table 4] Gaussian (G), uniform (u), Epanechnikov (epa), triweight (tri), and cosine (cos). The kernel bandwidths are obtained via g and Algorithm
kernel
G
K(x)
/2x
1
K(x)
7
u
The results are depicted in Table 5
opa
K(x)
[ - (1²)
x
W
0
The choice of kernel does not have a big impact on the model performance. Overall, the Gaussian kernel seems to the best choice as it consistently performs well for all the metrics. A potential reason why the Gaussian kernel is superior can be found its unbounded support. This means that there will always be some overlap between real and generated distributions, as obtained via KDE, enabling more informative gradients.
tri
K(x)
32
35
(1 2233
W
0
cos
COs
K(x)
Table 4: Different choices of kernel.
B.2 Using Maximum Mean Discrepancy to Match Statistics
The maximum mean discrepancy (MMD) is a kernel-based statistical test. that can be employed to determine whether two distributions are the same (Gretton et al. 2012). It has been used as a loss function to establish generative moment matching networks, a distinct class generative models (Li et al. 2015 Dziugaite et al., 2015). While an adversarial approach has been suggested to improve MMD networks by learning more suitable kernels [Li et al.) 2017), they constitute their model class and not extension of the GAN. Tn
15

---

Published in Transactions on Machine Learning Research (09/2024)
=
K
K epa
K tri
4 (4)
13.39-3.08
19.73:3.57
15.33.14.60
15.1515.770
11.6911883
(T)
0.17+00.06
0.13:00.04
0.20:t0.05
0.14+00.33
0.18:00.04
V. (
0.20=0.02
0.2140.02
0.31+0.08
0.25:1006
0,23+0.03
V (
0.8010.07
0.8910006
0.84+00.4
0.84:0.00
0.82:10.04
Table 5: (Synthetic example) The peGAN with different choices of kernel K is evaluated via different performance melrics, defined in Section 5.1 the Fréchet distance dr, the F1 score, the agreement of the constraint distributions Ve, and the agreement of the distributions of a selection of high-level features Vy. The arrows indicate whether high or low values are better. Ten runs have been conducted per model, and the mean values aais-o one standard deviation are given. Bold font highlights best performance. Parameters: bs 256, À = 500, e 0.9, KT.
O0 = 1
an =1 A=
dg = 1
do =1
On 0.5
do = X= 0.5
do = 5
= [1,2,
A 1.0
5.0
0.5
x= 0.5
(L)
21.52:3..97
38.42=3.03
13.49
23.26:12.93
19.52:12. 12
17.27::413
18.16:5..63
17.63:+3.47
(1)
0.14+0.05
0.11=0004
0.11+0.03
0.10+0.07
0.13#0.04
0.17=00.06
0.18+0.04
0.13+0.04
V. (L)
0.56.10.05
0.7110.09
0.76.10.10
0.96100006
0.50.10.03
0.90110.12
1.05
0.77100005
VJ (4)
1.20+0.11
1.13=0.06
1.0700007
1.01+0.06
1.08#0.06
1.10#0.08
1.10=0.13
1.07=0.05
Table 6: (Synthetic example) The pcGAN with MMD loss is evaluated for different weighting factors À and kernel widths dg via different performance metrics, defined in Section 5.1 the Fréchet distance de, the Fl score, the agreement of the constraint distributions V., and the agreement of the distributions of a selection of high-level features Vy. The arrows indicate whether high or low values are better. Ten runs have been conducted per model, and the mean values plus-or- minus one standard deviation are given. Bold font highlights best performance.
this appendix, we do not consider MMD networks but explore instead the effectiveness of using MMD as the loss function for matching the high-level statistics. That is, we use the MMD loss instead of f-divergences
(compare Section 4.1 in
The kernel maximum mean discrepancy between two distributions is defined as
MMD = E K X
X
x (x-x)= + Y -
2K
(15)
where X denotes real data and Y generated data. This leads to the following loss function, where estimate the expectations over minibatches and omit constant terms (i.e. terms that do not contain Y):
M
N
Σ K Um
Iim
2
ym
Fn
LMDD = M(M - 1)0
K
(16)
MNo
magémm
==
n=1
where M is the number of generated samples and N the number of real samples in the current minibatches.
One drawback of this approach is the mixed loss term in equation [6]; would be too computationally costly to take into account the entire dataset at each iteration wherefore we also need to batch the real data. When using f-divergences in loss (3) of our approach, similar problems can be circumvented by evaluating Ptrue once on fixed grid in advance of the training. Here, the same trick does not work, since the statistics extracted from the generated data determine the points at which the kernel K needs to be evaluated.
The results for the MMD loss are given in Table 6 We consider different weighting factors for the loss term, as well as Gaussian kernels with different bandwidth. The bandwidths of the Gaussian kernels for the different constraints are given by the values a, in (9) times the factors do given the figure. When multiple factors are given, then the sum of the corresponding Gaussian kernels usod. Both in terms of matching the constraints and terms of the performance metrics, the method performs better than the standard WGAN, but worse than the peGAN (compare Table. []
16

---

Published in Transactions on Machine Learning Research (09/2024)
WGAN
WGAN-
SNGAN
0.04
3.04
0.04
0.04
0.04
0.0
0.0
0.02
0.02
0.03
0.02
3.02
0.92
0.02
0.02
0.01
0.01
0.01
0.01
0.61
0.00
20
64
0.00
20
pa [A]
0.00
25
pa [113
0.00
20
0.00
20
0.4-
0.4
D4-
94-
0.2
0.2
6.1
6.1
0.1
0.1
6.0
0.0
ps 115]
ps [151
ps [15)
pC [15]
pd (15)
2.5
2.5
2.5
2.5
2.5-
2.0
2.0
2.0
2.0
1.5
1.5
15
15
15
0.5
0.5
05-
0.5
0.5
[30]
[30]
[30]
[30)
Figure (Synthetic example) The distributions of three different power spectrum components ps as obtained with the different GAN variants when combined with the probabilistic constraint are depicted, where the orange lines show the true distribution as obtained via KDE 7 For the histograms, 20000 generated samples have been considered (or the full dataset, in case of the real distribution). Parameters: bs 256,
0.9, T=KL, and À = [500, 500, 500, 2500], respectively, from left. to right.
B.3 Additional plots
In Fig. plots of the constraint fulfilment for the different GAN variants combined with the probabilistic constraint are given. It is apparent that all the constrained models reproduce the constraint distributions well, with WGAN-GP pe giving the smoothest fit. The distributions obtained with SNGAN + pe are bit more jagged than the other ones.
In Fig. equivalent plot is given for the CceCube-Gen2 dataset. Again, all of the constrained GANs match the distributions very well.
B.4 Runtime and parameter tuning
Table 7: Runtime (in hours) for one run of 100 iterations.
WGAN
WGAN pc (peGAN)
Wu al.
Wu al. + pc
WGAN-GP
WGAN-GP pc
SNGAN
SNGAN
pc
Synthetic 5.2
1.20
1.95
1.38
2.06
1.91 0.51
2.68
1.40
2.07
IceCube-Gent Gen2 3.3
0.37
0.46
0.45
0.46
0.56
0.60
0.67
The runtime required for one of 100000 iterations for the various models and for the different datasets is given in Table 7 The time required to extract the representations Ptrue for all of the constrained statistics combined is negligible in comparison: for the synthetic dataset, it takes 66 seconds, and for the CoCCube-Gen2 dataset 0.05 seconds. These numbers have been obtained on a system with NVIDIA RTX 3060 Ti SGB GPU, Intel Core i7 7700-K 4.2GHz CPU, and 16GB RAM. Overall, adding the probabilistic constraint increases the runtimes of the corresponding base GANs by around 40% 60% case of the synthetic dataset, where 101 statistics are matched. For the IceCube-Gen) dataset, where only 2 statistics are matched, the increases in runtime significantly lower at less than 30%.
17

---

Published in Transactions on Machine Learning Research (09/2024)
real
WGAN
al.
WGAN-GP
SNGAN
40
40
40
40
30
30-
30
30-
30-
20
20
20
20
20
10
10-
10
10-
10
0-
0.9
0.7
0.9
0.7
0.5
1.0
0.8
1.0
0.9
max
20
20
20
20
20
1.5
15-
15-
15-
15-
min
min
min
Figure (IceCube-Gen2) The distributions of minimum and maximum values as obtained with the different GAN variants when combined with the probabilistic constraint depicted, where the orange lines show the true distribution obtained via KDE (7). For the histograms, 20 000 generated samples have been considered (or the full dalaset, in case of the real distribution). Parameters: bs 256, 0.9, h=KL, A = 2.
Apart from the increased runtime for individual runs, new hyperparameters get introduced with the proba- bilistic constraint which will cause additional upfront cost when tuning the model. They include the global weighting coefficient X, the parameter determining the amount of historical averaging, and the function h to quantify the mismatch between the distributions. In principle, the heuristic choices in the formula for X can also be finetuned. The values used throughout the paper, e.g. those given in Tables [ and 3 should serve as good starting points for these hyperparameters. The most important parameter to tune individually for each dataset the global weighting coefficient. (3.. Assuming that the underlying unconstrained GAN has already been well-tuned, obtaining good results with the probabilistically constrained GAN should be possible with 5-10 additional runs.
The fine-tuning process of the peGAN can add sizable amount of time to the model development. In cases where the simulation would take minutes or hours to generate a single sample, and where thousands or millions samples need to be genorated, the pcGAN can still provide speedups. Apart from that, GANs have the advantage of being differentiable, which allows for their use end-to-end optimization pipelines, e.g. for detector design (Dorigo et al., 2023). Hence, being faster than the traditional simulation not always essential for GANs to be useful.
C Details on the Experiments
In this appendix, we give additional information on the experiments conducted in Sections 5.2 5.3 in particular on the network architectures and the training parameters. The code for the project is available at https://githuu..omm httpluu.coooopppllaa//aaaaaaa ppppppppppcee /pNAN note, however, that only the data for the synthetic example is available there.
C.1 Constraints and High-level Features
We start by giving an overview of the different quantities that have been employed either as constraints or performance metrics.
For the ID signals x of length N2 = 200 in Sections 5.2 and 5.3 we used the minimum and maximum values, min min(z ..i............................. .N- max max(#o.... ............. - mean values, mean N; N x, the mean absolute values, mean(abs) N. N [as]l, the number zero crossings, Nze, and the number of maxima, Nmax, of the curves.
18

---

Published in Transactions on Machine Learning Research (09/2024)
The discrete Fourier transform for real-valued inputs (as implemented in torch.fttrfft) was utilized to obtain the complex Fourier coefficients for positive frequencies & E [0, Ns
below the Nyquist frequency,
N₂ X = Σ
Kn EnE
(17)
and the corresponding power spectrun components are obtained as Sa N, |X&l². The total spectral energy AF J++ Ei= k=0
is then calculated as S = spectrum components directly, but instead on ps(k: = Sk.
Sg. When employed as constraints, we did not constrain on the power
For the different experiments, we considered the following set of high-level features: mean, mean(abs), max-min, E, 2e,, and Naax. For the IceCube-Gen2 experiment, we omitted the mean, since it did not exhibit interesting structure in its distribution.
C.2 Synthetic Example (Section 5.2
The synthetic dataset consists of 000 samples of size 200, generated as described in Section 5.2 For this example, we employed convolutional networks for both the discriminator and generator; details on the corresponding network architectures are given in Tables g and [0] respectively. In layers where both batch normalization and activation function listed in the column 'Activation', batch normalization is applied to the input of the layer whereas the activation function is applied to the output. Padding is employed in each layer such that the given output sizes are obtained; reflection padding is utilized.
Table 8: Hyperparameters used for the experiments.
Experiment
Navg
N₂
fached
itached
31
Bg
clamping
Syuthetic (5.2)
50
100 000
2e-4
0.5
70000
0
0.9
[0, 0.005]
CroCube-Gen22 (5.3)
50
100
000
5o-4
0.5
40000
0
0.9
0.10
In Figure 8 the search for the best values fö in ( is illustrated for KL. It is apparent, that there is a clear, batch size-dependent minimum of the KT.-divereence for each constraint, with larger batch sizes tending towards larger values of fā; this is due to the fact that more samples in the minibatch allow for a more fine-grained approximation of the generated distribution. In the top right plot, optimal values of fe are depicted for all components psji] of the power spectrum. The spike around Rs 10 is the result of some outliers in the values of the power spectrum components; they lead to high standard deviation of the true distribution, which in turn requires a large fü in order to obtain small enough standard deviations for the KDE to resolve the narrow peak well.
In the bottom row, approximations of the generated distributions as obtained the minibatches are depicted. It apparent that the mixtures of Gaussians approximate them reasonably well, with larger batch sizes typically giving better results.
In Figure 10 samples from the true distribution well generated samples from the different GANs are depicted. All of the GANs produce reasonable-looking results, although upon closer inspection it becomes apparent that they do not necessarily constitute superposition of two sine waves. Only the WGAN seems to have a tendency to produce rugged curves.
C.3 IceCube-Gen2 (Section 5.3
For this example, we considered 50 000 IceCubo-Gen2 radio-detoctor signals of size 200 (generated using the NuRadioMC code (Glaser et al. 2020 according to the ARZ algorithm (Alvarez-Muñiz et al. 2010)), ) normalized to their respective maximum absolute values. The networks employed are a mixture of convolutional and fully connected networks, which have been based on the architectures used Holmberg (2022); details on discriminator and generator architectures are given in Tables 11 and 12] respectively. For the discriminator, the input to the network is first fed through four convolutional layers in parallel, the outputs of which are
19

---

Published in Transactions on Machine Learning Research (09/202))
p5
[30)
to=32 towd4 Is-255
b5+32
bo
Id+32
10
10
bS-64
bS-256
12
bi 6d
256
26
5vegemoe
10-
20
1.2
12
10-1
16-
10-1
0.00
1om32
bp
1.4
0.25
0.4-
1.2
0.24
0.03
0.2
0.6
26-32
0.2
0.4
1o-64
0.00
20
50
0.0
0.0
02
0.4
0.
05
1.0 222
(0)
(20)
Figure 8: (Synthetic example) Determining optimal values f for h = KT. Top left: The three plots on the top left depict the dependency of the KL divergence on the factor fo for different power spectrum components; the curves have been averaged over 50 minibatches sampled from the original dataset. Bottom left: The first three plots in the bottom row depict the distribution of the constraint values together with (8) from minibatches of different size (not averaged);
their KDE representation, as well as curves obtained it is between them that the KL divergences in the top row have been calculated. Top right: Optimal values of the factor are depicted for different batch sizes, where the index gives the respective component of the power spectrun.
subsequently concatenated into one long array. The LeakyReLU activation function with factor 0.2 is applied. During training, we also check for a rare failure mode where the GAN generates only horizontal lines; if this happens, the training is restarted.
In Figure 9 a plot on the process of determining optimal values for fö shown the example KL. Same as for the synthetic example (compare Fig. the KT. divergences function of exhibit clear minima that depend on the batch size.
In Figure 12 samples from the true distribution as well as generated samples from the different GANs are depicted. Altogether, most of the generated samples look good, with none the models clearly outperforming the others.
C.4 Training Parameters
Here, we summarize the training parameters used for the different experiments. Ni. gives the number of training iterations, Ir the learning rate, and À the weighting factor for the constraints in (3). In the column 'clamping', the range is given to which network parameters of the discriminator were clamped order enforce the Lipschitz constraint in WGANs (Arjovsky al.) 2017). The ADAM optimizer (Kingmaa and Ba 2015) was used for training the networks, with hyperparameter values 31 and A2; scheduler employed that reduced the learning rate by factor of feched after itsched iterations. The weighting factor for the statistical constraint from Wu et al. (2020) was chosen as Awn 1. The weighting factor for the gradient penalty in WGAN-GP was chosen as AGP 10. The parameter m, which gives the nunber of discriminator updates per generator update, was chosen as 1.
20

---

Published in Transactions on Machine Learning Research (09/2024)
D5-236
RLdiargene
1.0
10a
10a4
18
10a4
80
30-
52
141
70
B56
25-
256
KDD
0.7 0.8
0.0
1.5
-2.5
-0.9
-0.8
-0.7
-06
-25
max
mn
Figure 9: (IceCube-Gen2) Determining optimal values f for h KL. Top left: The two plots the top left depict the dependency of the KT, divergence on the factor So; the curves have been averaged over 50 minibatches sampled from the original dataset. Bottom left: The first two plots in the bottom row depict the distributions of the constraint values together with their KDE representation, as well as curves obtained via (8) from minibatches of different size (not averaged); is between them that the KL divergences in the top row have been calculated. Top right: Optimal values of the factor f are depicted for different batch sizes, where the index gives the respective constraint.
Table 9: Discriminator architecture for the synthetic example.
Layer
Output Size
Kernel Size
Stride
Activation
Input
1 x 200
Conv
32 x 99
BatchNorm, ReLU
Conv
32 x 99
BatchNorm, ReLU
Conv
32 99
ReLU
Conv
64 48
BatchNorm, ReLU
Conv
64 48
3
1
BatchNorm, ReLU
Conv
64 x 48
ReLU
Conv
128 x 23
3
2
ReLU
Conv
128 x 23
ReLU
Conv
128 x 23
1
ReLU
Conv
256 x 10
ReLU
Conv
256 x 10
ReLU
Conv
256 x 10
ReLU
Flatten
2560
Linear
21

---

Published in Transactions on Machine Learning Research (09/2024)
Table 10: Generator architecture for the synthetic example.
Layer
Output Size
Kernel Size
Stride
Activation
Input
X 5
BatchNorm
Conv Transpp
256 x 25
16
BatchNorm, Tanh
Conv
256 x
BatchNorm, Tanh
Conv
256 x
BatchNorm, Tanh
ConvTransp
128 x
BatchNorm, Tanh BatchNorm, Tanh
Conv
128 x 50
Conv
128 x 50
BatchNorm, Tanh
Conv Transp
64 100
2
BatchNorm, Tanh
Conv
64 X 100
BatchNorm, Tanh
Conv
64 x 100
BatchNorm, Tanh BarchNorm, Tanh BatchNorm, Tanh Tanh
ConvTraasp
32 200
Conv
32 200
Conv
32 200
Conv
1 200
Tanh
Table 11: Discriminator architecture for the IceCube-Gen2 data. Conv01-Conv04 in parallel and the outputs are subsequently concatenated into one long array.
The input is first fed through the layers
Layer
Output Shape
Kernel Size
Stride
Activation
Input
x 200
Conv0l
32 x 49
4
LeakyReLU
Conv02
32 x 47
15
4
LeakyReLU
Conv03
32 x 44
25
LeakyReLU
Conv04
32 x 42
LeakyReLU
Concatenate
32 x 182
Conv
182
LeakyReLU
Linear
92
LeakyReLU
Linear
45
LeakyReLU
Linear
20
LeakyReLU
Lincar
1
Table 12: Generator architecture for the IceCube-Gen2 data.
Layer
Output Size
Kernel Size
Stride
Activation
Input
Linear
24
ReLU
Conv
48 x 24
3
1
ReLU
Conv
48 x 24
3
1
ReLU
ConnTransp
24 49
3
2
ReLU
Conv
24 49
3
1
ReLU
ConnTransp
12 99
3
2
ReLU
Conv
12 99
3
1
ReLU
ConvTransy
6 x 199
3
2
ReLU
Conv
1 x 200
4
1
22

---

Published in Transactions on Machine Learning Research (09/2024)
real
PCGAN
Wu ot al.
WGAN
WGAN GP
SNGAN
1.0
1.0
-9.1
M
0.5
0.5
0.5-
-0.10-
M
-0.2
0.0
-0.3
0.20
-0.5
0.5
-0.5
100
100
0.30
100
/M/0
0.1-
hm
M
-0.1-
W
9.0-
1
V
0.0
0.0-
0.2-
0.5
0.3-
-0.5
0.4
-2.4
1.0
M
0.5
-0.3
0.5
0.0
0.00
0:25
-2.3
-0.50
-02
2.4
-2.3
1.0
100
2:0
100
250
100
260
160
200
100
200
100
200
1.0.
-0
M
0.5
M
-0.55
0.0-
0.0-
0.0
-0.1
-0 2.20
-0.20
-0.2
0.30
100
200
100
200
1.00
200
1.00
200
200
200
200
200
1.0
0.2
M
M
M
0.5
0.0
0.5
0.0
W
0.0
0.2
0.5
5.4
1.00
202
202
5.0
0.5-
0.5
0.6-
9.9
-2.1
0.4-
0.0
0.0
0.2
-10
.3.05
0.5
0.5
0.5
0.50
0.25
-03-
0.0
0.0
0.0
-0.15
-025
3.3
0:25
-0.5
-1.0
-..0
100
1.00
1.00
100
200
Figure 10: Samples for the synthetic example as obtained from the different models.
23

---

Published in Transactions on Machine Learning Research (09/2024)
real
WGAN (pcGAN)
Wu al.
WGAN-GP
SNGAN
0.3
1.0
M
0.5
M
0.0
0.0
0.0
-0.5
-0.5
-0.1
-0.5
-1.0
-1.0
-0.2
-1.0
100
200
100
200
0
100
200
100
200
100
200
0.50
0.25
0.5
W
M
0.5
0.00
0.0
-0.25
-0.50
-0.5
-0.5
-1.0
-0.75
1.0
100
200
100
200
100
2000
100
200
100
200
0.5
MM
0.5
MM
N
0.2
W
0.5
M
M
M
M
N
0.0
0.0
-0.5
-0.5
-0.5
0.2
-1.0
1.0
-1.0
1.0
1.0
0.5
0.0
W
0.0
0.0
0.0
-0.5
-0.5
-1.0
-1.0
100
200
100
200
100
200
100
200
100
200
1.0
1.0
1.0
MM
W
M
0.0
0.0
-0.5
-0.5
00.5
-0.5
-1.0
-1.0
1.00
200
100
200
100
200
100
200
100
200
1.0
1.0
1.0
M
0.75
M
0.0
0.0
0.25
-0.5
0.00
-0.5
0.5
00.5
-1.0
-0.25
-1.0
-1.0
1000
200
100
200
100
200
0
100
200
100
200
1.00
0.75
0.5
W
W
0.0
0.25
-0.5
-0.5
0.00
-1.0
-1.0
100
200
100
200
100
200
100
200
100
200
Figure 11: Samples for the synthetic example as obtained from the different constrained models.
24

---

Published in Transactions on Machine Learning Research (09/2024)
real
PCGAN
Wu ct al.
WGAN
WGAN GP
SNGAN
10a
10-
1.0
1.0
1.0
0.5
0.5-
0.5
0.5
0.0-
0.0
0.0
0.3
-0.5
-05
-0.5
-0.5
-0.5
102
102
1.0.
1.0.
1.0
0.5
0.5
0.5
0.5
0.5
9.0
-0.5
-0.5
-9.5
0.5
-3.5
-10
0.5
0.5
0.5-
0.0-
0.0
-0.5
-0.5
-1.0
1.0
1.0
1.0
1.5
102
200
100
100
200
1.00
200
100
300
200
0.5
0.5
0.5
0.5
0.5
9.5
0.0
0.0
0.0
0.0
0.0
-0.5
-0.5
-0.5
-0.5
-0.5
-0.5
-1.0
-1.0
-1.0
-1.0
100
250
100
200
100
200
100
200
100
200
100
200
1.0
1.0
0.5
0.5
0.5
5.0
-3.0
100
202
202
1.0
1.0
1.0
0.5
0.5
0.5
0.5
555
0.0
0.0
0.9
-35
-0.5
-0.5
-1.0
0.5
0.5
0.5
0.5
0.0
0.0
0.0
8.0
-05
-10
-0.5
-2.5
100
100
100
200
200
Figure 12: Samples for the IceCube-Gen2 dataset as obtained from the different models.
25

---

Published in Transactions on Machine Learning Research (09/2024)
real
WGAN (pcGAN)
Wu
WGAN-GP
SNGAN
1.0
1.0
1.0
1.0
0.5
0.5
0.0
0.0
-0.5
-0.5
-1.0
-0.3
0..5
100
200
100
200
0
100
200
0
100
200
0
100
200
1.0
1.0
1.0
0.5
0.0
0.0
-0.5
-0.5
-0.5
-0.5
-0.5
-1.0
-1.0
-1.0
100
200
100
200
100
200
100
200
100
200
0.5
0.5
0.5
0.0
0.0
-0.5
-0.5
-0.5
00.5
-1.0
-1.0
0.5
0.0
-0.5
-0.5
00.5
-1.0
-1.0
100
200
100
200
100
200
100
200
100
200
1.0
0.5
0.5
0.0
0.0
-0.5
-1.0
-1.0
-0.5
-0.5
-0.5
100
200
100
200
100
200
100
200
100
200
1.0
1.0
1.0
1.0
0.5
0.5
0.5
0.0
0.0
0.0
0.5
-0.5
1.0
100
200
100
200
0
100
200
0
100
200
0
100
200
1.0
1.0
0.5
0.5
0.0
0.0
0.0
-0.5
-0.5
-0.5
-0.5
-0.5
-1.0
-1.0
100
200
100
200
100
200
100
200
100
200
Figure 13: Samples for the IceCube-Gen2 dataset. as obtained from the different. constrained models.
26


Reviews:
[{"review_id": "GFVnOEbieh", "reviewer": "Reviewer_1V6h", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall I appreciate what the method is pushing for, and it seems like an elegant way of incorporating prior knowledge. The paper is clear and easy to read, I think I could almost reimplement this just from the text.\n\nWhile the empirical evaluation is limited, there's still some good work done there where the synthetic example is a good illustration, and some design choices are tested. Unfortunately, there are still some choices that seem not very well justified (see below). It also feels like there's a missed opportunity here to test on way more domains with different priors. This could be a much more general paper considering how many domains this method could apply to, and this lack of scope is perhaps the main weakness of the paper.\n\nOn the novelty side, this is novel to me but I'm not up-to-date enough on this corner of the GAN world to be an authority here.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a novel regularization mechanism for GANs based on divergences between some differentiable statistics of the true and generated distributions. In particular in this work, those statistics are chosen based on domain knowledge and estimated with KDEs. Some additional work is done in terms of \"tricks\" that are part of them method, such as how to chose kernel parameters, or the weighing of divergences.\nEmpirically, the method is tested on time series data where it can be observed that the proposed method better recovers the desired statistics. A number of design choices are also tested.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The main change I'd like to see in this paper is better justification for some design choices, that appear to be tricks set by trial and error rather than methodically. For example, why set $\\bar\\sigma_s$ the way it is set? This is just a hyperparameter that should be empirically tuned. Similarly, the method to determine $f_\\sigma^s$ appears very heuristic, with some constants sprinkled in Algorithm 3. The method to determine $\\lambda_s$ is just as suspicious; where does this $0.1$ value (or even really the whole $\\eta$ formula) come from?\n\nSome text comments:\n- \"A major advantage of using f-divergence...\" this is fairly constrained for situations where one can easily integrate over x. This is rarely the case for a complex enough distribution (e.g. one parameterized by a deep model).\n- Equation 5 is wrong and shows the Jeffreys divergence rather than the JS divergence. The two are quite close but the difference matters numerically and empirically (what are the experiments using?)\n- \"We chose to employ KDE with Gaussian kernels\", this was a bit scary to read, but I was reassured later on by appendix B.1. I'd recommend presenting the method in a manner that's a bit more generic (but using Gaussian kernels as a running example may be helpful for readers).\n- Equation 10 suggests some recursive computation, is this really the case? If so is it truncated?\n- Writing `linspace(0, 20, 200)` belongs more in a technical report than a scientific paper. Consider revising.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "OXZlT5YaTr", "reviewer": "Reviewer_uLxz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. This paper introduces a new way to constrain GANs by matching distributions of chosen statistics.\n\n2. The use of f-divergences and kernel density estimation is well-justified.\n\n3. The approach can be applied to various GAN architectures and allows for arbitrary differentiable statistics to be constrained.\n\n4. The experiments cover both synthetic and real-world datasets, with a wide range of evaluation metrics and comparisons to existing methods. This paper also includes the detailed ablation studies.\n\nWeaknesses:\n\n1. The use of kernel density estimation may become computationally expensive for high-dimensional statistics. A more detailed discussion on the method's scalability would be beneficial. Additionally, the paper could be strengthened by demonstrating improvements on existing state-of-the-art GAN models for more complex tasks like image generation, rather than focusing primarily on toy datasets.\n\n2. While the paper makes good use of KDE to approximate statistics, it relies on hand-crafted features. An interesting extension would be to explore the use of learned statistics from existing models, which could potentially capture more complex or nuanced aspects of the data distribution. This approach could draw connections to the rich history of analysis-by-synthesis methods in statistical modeling.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes the probabilistically constrained GAN to improve the statistical fidelity of generated samples in GANs. The main contribution is a technique to match the distributions of certain statistics between real and generated data by adding a new loss term to the generator loss function. This approach aims to ensure that generated samples not only look realistic but also accurately reflect the true data distribution.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "An intriguing aspect to consider is the potential adaptability of the proposed method to other probabilistic generative models, such as energy-based models (EBMs). It would be valuable to explore how the core principles of the pcGAN approach might be applied beyond the GAN framework. Could the authors comment on the feasibility and potential benefits of extending their proposed model to these broader classes of generative models?", "broader_impact_concerns": "I do not see any significant broader impact concerns that would necessitate adding a Broader Impact Statement or expanding an existing one.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YWSqwdcJPp", "reviewer": "Reviewer_PNEX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n- The paper introduces a further form of supervision in training GANs for scientific applications to enforce a better match between generated and real data statistics.\n- The proposed idea is presented with clarity, and the author's contribution is well-stated.\n- The authors provide the source code to reproduce their results.\n- The paper is well-written and well-organized.\n\n## Weaknesses\n\n\n- The empirical evaluation is not convincing (see Requested Changes - 1):\n\n  - The number of runs (3 per model) is not enough to produce a convincing empirical evaluation, as the differences in performance cannot be proven to be statistically relevant with only 3 evaluations per model.\n  \n  - The experiments conducted on the real dataset are not exhaustive since they do not include the evaluation of adding the probabilistic constraints to different GAN variants, as done for the synthetic dataset. \n  \n  - The numerical results reported in Figure 7 do not show a clear improvement in pcGAN compared to the other models.\n\n  - The new loss term in pcGAN introduces an overhead both in hyper-parameters fine-tuning and training procedure. However, this aspect is not covered in the paper. As pointed out by the authors in the introduction, \"GANs may serve as surrogate models for expensive but highly accurate numerical simulations\", thus the tradeoff between performance improvement and computational overhead is crucial for their method, but it is missing in the analysis. \n\n- The results are not well-presented (see Requested Changes - 2).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In the context of data generation for scientific applications, the authors introduce a method to align the distributions of statistics of generated data with those of real data. Their approach is suited for Generative Adversarial Networks (GANs) and enhances the generator's loss function by incorporating a new term that measures distributional differences using f-divergences. By applying kernel density estimation, they represent the real distributions and estimate the corresponding generated distributions at each iteration. This method referred to as probabilistically constrained GAN (pcGAN), focuses on matching the overall shapes of distributions rather than individual sample properties. The approach is particularly beneficial for scientific applications, as domain-specific differentiable statistics can be selected. The authors perform the empirical evaluation on one synthetic and one real-world dataset.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "### 1. Empirical Evaluation (Critical)\n\n- Increase the number of runs per model to strengthen the results.\n- Evaluate the addition of the probabilistic constraints to different GAN variants on real data\n- In Section 5.2 (pag. 9) authors claim \"In therm of execution time, the pcGAN takes about twice as long to train as the unconstrained WGAN\". This claim needs to be supported with numerical results.  \n- Adding the new loss requires adjusting several parameters ($f_\\sigma$, $\\lambda$, $\\epsilon$, batch size). How expensive is the fine-tuning process? \n\n### 2. Results Presentation (Critical)\n\nThe graphical representation chosen by the authors to report the numerical results is 1) not easy to read and 2) not compact, forcing the authors to move important results in the Appendix. For example, the authors describe four metrics used in the empirical evaluation, but in Figures 4 and 5 they report only two of them, without providing any explanation for the missing results.\nThus, I strongly suggest the authors to: \n  - replace Figures 3, 5, and 7 with tables, and include the results for all the metrics;\n  - replace Figure 4 with a line chart with a confidence interval, and include the results for all the  metrics.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper does not qualify for a certification.

Overall, the paper meets the acceptance criteria of TMLR. The claims are supported by convincing experiments, and the proposed solution and analysis could be of interest to those working in the area of generative modelling and their applications to time series data.
Decision: Accept (poster)