Practical Synthesis of Mixed-Tailed Data with Normalizing Flows
Authors: Saba Amiri
Eric Nalisnick
Adam Belloum
Sander Klous
Leon Gommans
Venue: Accepted by TMLR ()

Abstract: Capturing the correct tail behavior is difficult, yet essential for a faithful generative model. In this work, we provide an improved framework for training flows-based models with robust capabilities to capture the tail behavior of mixed-tail data. We propose a combination of a tail-flexible base distribution and a robust training algorithm to enable the flow to model heterogeneous tail behavior in the target distribution. We support our claim with extensive experiments on synthetic and real world data.

Published in Transactions on Machine Learning Research (10/2024)
Practical Modelling of Mixed-Tailes Data with Normalizing Flows
Saba Amiri
s.mmrrsuuo..f ml
Informatics Institute
University of Amsterdam
Eric Nalisnick
malismick Baahiickkkjjhu.... edu
Department of Computer Science Johns Hopkins University
Adam Belloum
a. w.w.belllom@@uv....
Informatics Institute
University of Amsterdam
Sander Klous
Informalics Institute University of Amsterdam
Leon Gommans Air France- KLM
leon.gormans@kkm..comm
Reviewed on OpenReview: https: // openreview. net/ forum? id= uphskDjoOs
Abstract
Capturing the correct tail behavior is difficult, yet essential for a faithful generative model. In this work, we provide improved framework for training flows-based models with robust capabilities to capture the tail bchavior of mixed-tail data. We propose a combination of lail-flexible base distribution and a robust training algorithm to enable the flow to model heterogeneous tail behavior in the target distribution. We support our claim with extensive experiments on synthetic and real world data.
1 INTRODUCTION
Real-world data often show mixed tails - both heavy (representing 'black swan' events, seen in communication networks traffic and actuarial risk (Wang et al.) 2006 Afif et al.] 2020 ) and light (indicating events within narrow outcome ranges, seen certain pricing models and extreme engineering events (Singh Gor Jamissen ct al. )022)). Capturing tail behavior accurately is vital in data synthesis, especially in sectors such healthcare, where mis-estimation can cause significant inaccuracies in analyses based the synthesized data, leading to misinferred risks (Ibragimov et al. 2015).
Normalizing flows (Papamakarios et al. 2021 present an attractive model for mixed-tailed data synthesis. This due to the fact that the tail behavior of the base distribution directly and transparently allects the tail of the generated data (Jaini et al.) 2020). Other classes of deep generative models do not provide such an explicit mechanism parameterization of base density on the capabilities of normalizing flows on capturing the tail behavior of targets with different tail behaviors.
control their tails. Figure demonstrates the impact of mis-spocification of

---

Published in Transactions on Machine Learning Research (10/2024)
Base Gaussian
Target
Heavy tailed GGD
Light tailed GGD
Heavy talled GGD
Target
Light tailed GGD
Figure 1: The impact missspecification of base density on the capabilitics of normalizing flows to capture the tail behavior the target density. We train an RNVP on three short, normal and heavy-tailed Generalized Gaussian (GGD) targets (columns 2-4) with short, normal and heavy-tailed GGD base densities (rows 1-3). We plot the probability density of the plot. Mis-specified parameterization of base leads to flow being only able to estimate the target with the same tail behavior its base density accurately
The task of synthesizing mixed-tail data using normalizing flows is an intriguing research domain within generative modeling. Previous methods (e.g. (Jaini et al. 2020 Laszkiewicz et al. 2022)) select the base distribution to match the target tails. One potential concern with applying these methods to mixed-tail targets is that the base distribution might be overly conditioned on the estimated or perceived tail of the target. This makes the flow model brittle, as will be discussed in Section 3 We also posit that mixed-tail data can negatively impact model gradients during training regardless of the choice the family and tail behavior of the base density.
In this work, we focus on improving the flow model for mixod-tail targets. In contrast with previous methods which select. the base densily, we make the base density of the flow adapt to the target lail characteristies across broad family by making it flexible enough to capture the tailedness of the target through flexible mixture of Generalized Gaussians without binding the base to the tail propertics of the target distribution. To make the flow model itself capable of capturing the tail of the target distribution we make the training process robust to heavy-tailed gradients. By making the Monte Carlo estimator robust we stabilize the training process and enable model to capture the tail behavior of the target distribution and admit maddient-based adaptation of the tail behavior. Following prior works in this area (Jaini et al.] (2020): Laszkiewicz al. (2022),, validate our method on range of data synthesis (asks, benchmarking against other state of the art tail-adaptive flows and empirically show the advantage of our method for mixed-tal targets. We show general improvements in capturing the tail behavior of mixed-tail targets while maintaining high general utility.
Our contributions include:
Highlighting the problem with pre-hoe selection and estimation of base density tail and proposing a flexible tail-adaptive mixture for base density to fix these issues.
2

---

Published in Transactions on Machine Learning Research (10/2024)
Discussing the problem with maximum likelihood training of normalizing flows on mixed-tailed largets and propose robust method to counter the impact of heavy-tailed gradients.
Empirical results showing generally favorable performance compared to other tail-adaptive flow methods.
2 BACKGROUND
Notation We denote univariate and multivariate random variables with bold letters and observations with non-bold letters. We denote data by x and its observation by æ. The kth component of x is denoted by Ty.
2.1 Characterizing Tail Behavior
A random variable X is said to follow a heavy-tailed distribution if its tail is not exponentially bounded. This can be formally defined using the moment generating function (MGF): Mx(() = E[e!*]. distribution is heavy-tailed if there exists some t > such that Mx (t) is infinite or does not exist. An example of this category the Pareto distribution with scale parameter 7 > 0, for which the MGF is Mx(t) --6 which is undefined for t 7.7
The tails of light-tailed random variable X decay sub-exponentially. The Gumbel distribution, with location parameter and scale parameter 8 > 0, is example of light-tailed distribution as its survival function e (a-w1// indicates sub-exponential decay for large z.
In many real-world scenarios, encounter data with mixed tail properties. We define such data having mixed lails if some of its marginal distributions exhibit heavy lails and others light tails. This underscores the multifaceted nature of real-world data, where different variables or features may be governed by distinct distributional properties.
Tail Index Estimation The tail behavior of a probability distribution primarily characterized by their tail index. The tail index, which we denote as a, is defined the exponent the power-law tail of the distribution:
F(x) = P(X S 1-
where P(X 2 x) is the probability of the random variable X being greater than or equal to a certain value x, and F(x) is the cumulative distribution function of As a approaches zero, the distribution follows a power law a greater degree. Hill's estimator index. It is defined as follows:
Hill 1975 is widely-used method for estimating the tail
âн = k
(log( Xn-it1) - log( (og(X,, xx)())+))))
Here, represents the sample size, k is the number of upper order statistics, and Xn-i+1 denotes the ith largest observation in the sample. The parameter & is of crucial importance: large k inflates the variance of the estimator, while small & increases the bias (Fedotenkov 2020)
2.2 Normalizing Flows
Normalizing flows (NFs) (Tabak & Turner 2013 Rezende & Mohamed 2013 Papamakarios et al. 2021 are
models represented by sequence of invertible transformations that warp simple base distribution (such as standard normal) into a richer target distribution. The transformations are chosen such that their (log) Jacobian determinant is casy to compute, allowing for efficient computation of the likelihood. Let u ~ (u(u) denote the base density, and the sequence of invertible transformations be denoted To = TN-1 o... To, where o denotes the neural network parameters. We can evaluate the density of observation x by applying the change of variables formula:
(x(x)= = Pu (T> (x)) dot Jy. (x))
3

---

Published in Transactions on Machine Learning Research (10/2024)
where Jrz 1 is the Jacobian matrix of the inverse transformation. Due to the availability of the exact density function, maximum likelihood training can be carried out as usual:
(()) = KKL (xx() II (x - -Bp log (T (x: t)i + log |det Jr- ( Φ)) +
(1)
where C' is a constant, p" (x) denotes the true distribution of the data and sb and 0 represent the parameters of the base distribution Pa and the collective parameters of the model respectively. In practice, the expectation is calculated via Monte Carlo approximation using the training set.
3 CHALLENGES OF TRAINING TAIL-ADAPTIVE NORMALIZING FLOWS FOR MIXED-TAIL TARGETS
Reviewing literature indicates that to adapt a flow model's base density to the target tail, existing methods either incorporate assumptions about tail behavior into the model through a specific base density selection, as in Jaini et al. (2020). directly estimate the target's tail behavior, as in the marginal adaptive base method by Laszkiewicz el al. (2022) Furthermore, previous research (Behrmann et al. 2021 shows flow-based models suffering from the sxploding/vanishing vanishing gradients problem, which will be exacerbated in presence of tailed mini-batches during training. We explore the limitations of these approaches in modeling mixed-tailed data.
3.1 Challenges in Adjusting the Tail Behavior of the Flow Model's Base Density
Pre-Hoc Selection of Base Density Tail Behavior for Mixed-Tail Targets Real-world data may have tail behavior that deviate from any specific choice of base distribution family such as having a multi-modal mixed-tail distribution. Therefore, choosing a specific heavy-tailed parametric distribution family as the flow basc as suggested by Jaini et al.] (2020) a priori will inherently restrict the range of possible tail shapes and asymplotic decay rates that can be represented. The other issue with this method for mixed-tail targets is based the fact that normalizing flow transforms top of the base distribution can either alter its tail behavior significantly or not converge due to exploding/vanishing /vanishiing gradients. Thus to make the flow robust, the tail modeling capabilities must come more from the flows rather than the base distribution alone.
Pre-Hoc Estimation of Base Tail Index for Mixed-Tail Targets Adapting the tail behavior of the flow's base density directly on the target distribution is another way to make the base distribution tail adaptive, with prominent example being Marginal Tail Adaptive Flows (mIAF) by (Laszkiewicz al., 2022). TAFS calculate the tail index of each marginal of x. Since tail estimation is notoriously difficult, Laszkiewicz et al.] combine the results from three different tail index estimators. The base density for light-tail marginals is set to Gaussian and a univariate Student's t with degree of freedom equal to the estimated tail index for heavy tail marginals.
The limitation of these methods for modelling complicated mixed-tail targets is their reliance on estimated tail index of the target distribution. Tail index estimators generally suffer from high variance and sensitivity tuning parameters, especially for small sample sizes or multidimensional settings. For example, a major drawback of Hill's estimator is its sensitivity to the choice of k. Mason (Mason 1982 demonstrated that áæ is inconsistent. if l stays fixed as approaches infinity.
3.2 Challenges in Maximum Likelihood Training of Flow Models for Mixed- Tailed Targets
Regarcless of the choice for the base distribution and its tail properties, argue that it is simply not enough to enable the flow to be trained properly and capture the tail behavior of a mixed-tail target distribution. The reason is unstable training due to increased variance of the gradients when the target is heavy-tailed.
Consider the likelihood term in the maximum likelihood training objective (Equation (1)). Let u = T x) be the inverse transformation of the observation x. The term log (og),()) in the likelihood involves the logarithm of the base density evaluated at u. In the presence of heavy tails, the density can have thwer--han-expooentii decay, resulting in higher likelihood contributions from extreme values of u (and consequently, x). Similarly,
4

---

Published in Transactions on Machine Learning Research (10/2024)
log Jy-1 Φ can become large in magnitude when falls in the tails of the distribution since the Jacobian determinant accounts for the local expansion or contraction of the transformation and heavy tails can amplify this effect, leading to large values of the determinant.
4 METHOD
Our approach to tail adaptiveness in any normalizing flow hinges on: a) ensuring the base distribution 's flexibility to align with the target's tail behavior, and b) enhancing the flow's training process robustness to adequately capture the tail dynamics of the target dist.ribution.
4.1 Tail-Adaptive Base Density for Mixed-Tail Targets
We propose using a mixture of Generalized Gaussian Distributions (GGDs) as the base density of the flow. The Generalized Gaussian represents a family of continuous probability distributions that extend the concept of the Gaussian distribution by incorporating a shape parameter, denoted as > 0. Formally, the probability density function (PDF) of GGD with mean μ4, scale parameter 0, and shape parameter B > 0 is defined
B 0,, (0T(1/B)
exp -
(-(1== 1
where T()) is the garmaa function. This distribution encompasses wide range of density shapes including
but not limited to the Gaussian distribution 22) and Laplacian distribution (3 11) Higher values of
3 give rise to light-tailed distributions, whereas lower 3 values lead to heavy-tailed distributions. Figure shows the flexibility of the tail behavior of GGD with different shape parameters (for more thorough discussion on GGDs, see (Dytso et al., 2018)).
A Irainable mixture model as the base distribution is defined as:
M
Pu(u; Ti.... TM, ii...., WM) Tm p(u vm)
m=1
where m is the component index, am E [0, 1] is the weight of the mihh component such that Σm Tm 1, and ti are the parameters of the mth GGD component paramcterizod by (p, (,,0,
Our choice for using mixture for better modelling mixed tail behavior is substantiated by previous work that successfully used mixtures to model tailed distributions (Feldmann & Whitt 1998 Okada et al. 2020 Venturini et al. 22088). Hagemann & Neumayer (2021) show that normalizing flow training can be stabilized in cases (e.g. disjoint support of base and larget) by using a mixture for (u.)). We posít that training a flow model on mixed-tail targets will similarly introduce optimization difficulties that may only be exacerbated by having a ficxible basc density with widc-ranging tail behavior.
Satisfying the Tail Condition of the Base Density Jaini et al. (020) show that for Lipschitz triangular flows to be able to capture the tail of the target distribution, their base density should be at least heavy tailed. When the tails of the base density Pa are fixed, and the determinant of the Jacobian det Jy 1 (x) is bounded due to the Tipschitz property, it follows that Ps cannot exhibit heavier tails than Pas as the transformation TĄ cannot increase the rate of decay of Pu its tails. This implics that for u ~ pa(u) with known tail behavior, and the expressiveness of Té, ultimately constraining the model's capacity to represent target densities ps with tails heavier than those of the base density Pu.
(x),, the tail properties of pe are essentially inherited from pa. and limited by
To comply with this requirement, our choice of a flexible GGD mixture base is motivaled by the the ability of GGD to control its tail behavior through its shape parameter. We argue that the mixture base density requires only one component to be at least as heavy-tailed as the target density for the flow model adhere to the requirement of Jaini et al. (2020), while the flexible nature of the mixture will let the flow capture complex mixed-tail tail behavior of the target density.
5

---

Published in Transactions on Machine Learning Research (10/2024)
Q-Q Plot
Probability Density Function
g ==1
60
8 2
B 10
1
X
Theoretical quantiles
Figure 2: Probability density functions (left)) and quantile functions (right) of generalized Gaussian distribu- tions with shape parameter 8 = 1.0, 2.0, and 10.0. B = 2.0 yields a Gaussian. As 3 drops below 2.0, the PDF becomes heavy-tailed and the quantile function transitions from linear to concave down and vice versa.
Formally, consider mixture model Pa where component densities p(u: dm) decay super- epper-exponentially i.e., p(u; 4i )emeelll" a E (0,2), for some c > O. The slowest decaying component ensures the mixture's heavy tail, as Pu (u; : ......, (har) ~ Tm p(u; v/m) as x + oc.
Considering mixture of generalized Gaussians, p(u; 4mm) with & (Jm o. Bm), a heavy tail possible if there exists component j which satisfies (i) 83 < 3r vi # and (ii) B, S 2. This indicates the j-th component's s tail is heaviest and decays super-exponentially By making By sufficiently small, we ensure heavy lail, leading to (u) ~ 7yp(u; v))) for large u. A single heavy-tailed component is enough to induce heavy tails in the mixture, while the remaining components can vary to adjust the overall density shape. This logic extends to ereating arbitrarily light tails. It should be noted that this does not proclude the flow from capturing complex and heterogeneous lail behaviors as would be the case for non-mixture bases. The controllable tail of GGD lets the flow adapt shorter tails as well. The key advantage of using a mixture of GGDs as the base density is that it provides more flexibility to match multimodal marginals with complex tail behaviors, compared to using single parametric family. Our approach does not make any assumptions about isotropy impose identical marginal tail decay rates and each mixture marginal can have distinct tail properties.
4.2 Training Flows On Mixed- Tail Targets
In the previous section we proposed using a mixture of generalized Gaussians base density for modelling heavy and mixed-tail targets. We also argued that the flexibility to model the tail behavior should come from the flow itself and not be put entirely upon the base density. In this section, based the motivation provided in the Section 3.2 we propose our method for stabilizing the training process of the flow models by making it tail adaptive.
4.2.1 Our Proposed Robust Monte Carlo Maximum Likelihood -aasedd Training
To mitigate the problem described in previous section, we propose using robust estimation met during the maximum likelihood estimation. Robust gradient estimators have been employed similarly in the literature (c.g. Hsu Sabato (2016)). However, to the best of our knowledge they have not been applied to the problem of stabilizing MLE in presence of mixed-tail behavior. Specifically, we propose employing the Geometric Median (GM):
g = arg min
lg VoL(O: x1)22
g gERd
(2)
g
6=

---

Published in Transactions on Machine Learning Research (10/2024)
during the Monte Carlo estimation of the likelihood while training the flow to mitigate the problem of high variance gradients when the target is heavy mixed-tailed.
To motivate this, we pay attention to mean estimation in presence of heavy-tailed data. Let X1, X2, Xn be i.id. random variables with a heary-tailed distribution. Let X, = 1 Σi= X, be the sample mean and Med be the sample geometric median. Then Var(Xn) > Var(Xmed). For heavy-tailed distributions, the variance can be arbitrarily large. In fact, by definition of heavy tails, the tails of the distribution decay slower than an exponential distribution. Therefore, for any finite V, there exists some Zo such that:
f(z)dr > V
|0/
Where fxx is the probability density function. This means there is non-negligible probability mass in the heavy tails that can lead to extremely large values of X,, These extreme values disproportionately increase the variance of the sample mean X,, (more in depth discussion can be found in literature, e.g. (Sun et al., 2015)). On the other hand, the geometric median only depends on the relative ordering of the X; values, not their magnitudes. Therefore, has robust variance in the presence of heavy tails.
Next we posit that using the geometric median estimator for gradients instead of the sample mean when training normalizing flows on mixed-tailed data reduces the variance and leads to more stable optimization. From the training objective, the gradient is:
VeC(o) = Ep, Ve log loggx(f((z))
This expectation typically approximated with Monte Carlo sampling:
Vol(O) Vologps(fo(z))
n
We already established that when Px is heavy-tailed, the gradients Vo log (x(((z))) have very high variance. Therefore, using the geometric median instead of the sample mean to aggregate the gradients reduces the variance while still being a consistent estimator. This results in more stable optimization. Robust estimators like the geometric median can mitigate the training instability induced by mixed-tailed target distributions for normalizing flows. We demonstrate this phenomenon in Figure B which depicts the instability of the standard maximum likelihood estimation in presence of heavy tails. As can be seen, regardless of whether the model mis-specified or not or the tail behavior of the base and target densities, the loglikelihood estimate is impacted by the heavy-tailed minibatches, resulting in biased estimation of expectation. This bias is more prominent when the base density is heavy-tailed, which explains why the proposed method of Jainn et al. (2020) yields mixed results in practice. We employ the Weiszfeld method (Eftelioglu 2017 to estimate the geometric median.
5 EXPERIMENTS
We experiment using a range of simulated and tabular data. We train Real NVP (RNVP Dinh et al. 2017) and Masked Autoregressive (MAF (Papamakarios et al. 2017)) flows with GGD mixture base density (100 components, learnable parameters) and robust MLE. The choice of RNVP motivated by being a Lipschite-contintous function, which results in the density function having the same tail properties as its base, making the dynamics of the model simpler and easier to interpret. The shift and scale operations of RNVPs are modelled by an feed forward network with one hidden layer the width of 1024. The MAF is potentially more expressive than RNVP, letting us study the performance of our method for different flow models. Each flow model is 12 steps deep, with the MAFs having 12 autoregressive layers which use a stacked transformation with 1024 hidden units and blocks to model the conditional dependencies of the variables. Minibatch size is 1024 and we perform Adam optimization (Ir 10-5) All trainings are over 1000 iterations. We perform experiments in this section 5 times unless explicitly mentioned and report the mean and standard error.
7

---

Published in Transactions on Machine Learning Research (10/2024)
malia
Negative Loglikelihood
120 350
200 250 300
Iteration
Figure 3: The impact of heavy-tailed target and base on the maximum likelihood estimation. We plot the NLL of a Real NVP flow trained with Student's base density on a Student's target using mean (dashed lines) and geometric median (solid lines), with and without tail mis-specification of the base with respect to the target. The legend denotes (degrees of freedom of the base-> degrees of freedom of the target).
5.1 Data Quality metrics
To measure the performance of our proposed method we perform experiments on synthetic and real data and report the negalive log likelihood. Also lollowing Laszkiewizz et al. (2022) we report the tail-specific performance metries of Tail Value at Risk (tVaR) and Area Under Log-Log Plot (AULLP). The details of the metrics used can be found in the Appendix C
Average Negative Log Likelihood In the context of normalizing flows, the negative log-likelihodd (NLL) loss is often used a performance metric train the model. The NLL loss measures the model's ability to approximate the true data distribution. The negative log-likelihood of the model given the dataset. D is defined as:
N
C(0:D) = Σ log (x ())
Tail Value at Risk Tail Value at Risk (tVaR) is measure used to estimate expected losses beyond a certain threshold, known as the Value at Risk (VaR). Tt calculates the average loss above the VaR level based on a given quantile. The tVaR is obtained by taking the conditional expectation of the loss variable, given that the loss exceeds the VaR level. The VaR itself is determined as the minimum value of the loss variable for which the cumulative distribution function exceeds the specified quantile level. We report the tVaR difference between test dataset and synthetic dataset.
Area Under Log-Log Plot The Area Under Log-Log Plot (AULLP) involves integrating the logarithm of the survival function, which represents the probability of random variable excceding given value, log-log space beyond a high quantile threshold. We report the The AULLP difference between test dataset and hetic dataset.

---

Published in Transactions on Machine Learning Research (10/2024)
5.2 2D Dataset
Data We use synthetic datasets to demonstrate the performance of our proposed method. For our mixed-tail dataset, following Jaini et al. (2020) we choose our target distribution as a bi-variate Neal's funnel:
F4,1 N(7, 1),
51,2 N(O, exp(x.,1/2+ A3).
To better control the tail behavior of the distribution, add additive offset the variance of the second variable via parameter E I220.
Table 1: General performance metrics for the Neal's funnel for the average NLL well AULLP difference and tVaR difference for each variable separately.
Base Density
NLL 4
AULLP o14
AULLP v24
tVaR.24
Gaussian
3.93 .22
8.15 1.7
18.09 1 1.14
2.4c4 1 2n2
1.1c5 1 7.6c3
Mx. Gaussian
3.86 +
4.58 .19
11.15 .56
54.5 = 2.6
36.45 1.72
Student's t
3.86 + .39
89.9 10.9
93.6 11.6
2c30 + 4e27
2c31 1 9e28
Tix.Stmdent's
3.88
5.73 + .53
+4.15
8.8e3
8.8e5 + 9.2c3
GGD
3.83 331
7.09 = .45
13.93
8.61
161.17 8.32
Our Method
3.77 + .31
6.1 = 35
5.81 + .49
73.72 4
33.0 + 2.62
Target
Gaussian
Mx.Gaussian
Student's
Mx.Student's
GGD
Mx.GGD +Robust MLE
Ti Tiaie RNVP
Figure 4: Performance of RNVP flow with different base distributions for estimation of a mixed-tail target. Our modified Neal's funnel with X 6.0 has heavier-tailed and lighter-tailed variable.
Results Figure Canure/commme compares the performance of our method for an RNVP flow with baselines with multivariate Gaussian base (as is commonly used in practice), Student's and generalized Gaussian base densities. Our method performs well for both variables while the other methods' performance is sub-optimal for one or both variables.
Table 1 shows performance of an RNVP with difference base densities. Results reported here show the advantage of our proposed method. We observe meaningful advantage for our method, especially for the hcavier-tailed variable (v2). Of note is the high variance of the flow with Student's t base (following Jaini
et al. (2020)) ) which as discussed in section is due to altenuated tail misspecification in the presence of mixed- tailed targets. The mixture of Gaussian base reports better results for the lighter tailed variable (4.58 vs 6.1 for AULLP, 54.5 vs 73.72 for tVaR) which is to be expected since is standard Gaussian and has the same normal tail behavior as the flow base. As can observed in the next section, for targets with more complicated tail behavior, the mixture Gaussian base loses its adnatage.
5.3 Ablation Study
Data To measure the impact of each aspect of our method, we perform an ablation study. We use two base distributions, GGD and mixture of GGDs. We train an RNVP model on both bascs with and without robust MLE. For each base, we train flows on a two-dimensional distribution P(X,,X) such that X, follows generalized Gaussian distribution, and X₂ adheres to Student's t-distribution. X is parameterized by 7-

---

Published in Transactions on Machine Learning Research (10/2024)
The shape parameter of X1 is transformed by via skew-adjusted Gamma cumulative distribution function, while X2's degrees of freedom will be retrieved by 1 7- The parameter is explored within the interval [0, 1000] using a non-exhaustive, selective sampling strategy. Our setup enables us to test out method for a range of distributions with extreme tail behaviors, on onc end (2=0) having a heavy-tailed Cauchy and a close to uniform short-tailed GGD and at the other end spectrum ending up with a shorter-tailed Student'ss 's and a Laplace distribution. We train our ablation models for cach chosen , and subsequently report the AULLP-diff of two samples from the target and the trained model, each with size 50000.
Figure 5 shows the B and parameters of the random variable use for our ablation study as function of (as shown in the two plots on the right). Furthermore, we provide visual representation of the probability
density functions (pdf) for X and X2 at the extremities of the y spectrum, namely when 7 = 0 and = 1000 (illustrated in the two plots on the left).
random vartables for y
ablation random variabies for y=1000
Student's as function
function at
060. 4.3
668.6 5
500
05
OK
AXC
01
JK
34
1040
430 08C
Figure 5: Ablation study framework. The two graphs on the left depict the probability density functions (pdf) of our dual random variables at the extreme points of y. The pair of graphs on the right illustrate how the shape parameters y for the Student's t-distribution and / for the GGD vary as functions of 7.
Results The results of our ablation study depicted in Figure c show that the flow with mixture of GGD base outperform single GGD in terms of lower AULLP difference across diverse 7 settings, underscoring the versatility of the mixture in capturing tail behavior of our mixed tail target. Further, robust gradient estimation generally improves model fidelity, although its efficacy varies with the tail behavior of the distributions, which can be attributed to stochasticity of the models since we do not optimize hyperparameters for different. cases individually. These insights reveal that espocially for heavier tailed target densities (variable 2) both the GGD mixture base and the robust gradient. estimation are needed for the flow to be able to model the tail behavior of the target density accurately.
Table 2: Comparison with [Lazziewicz et al. 2022 We run the same experiment as Laszkiewizz al.
(2022)ss 's Table experiment for dn 4. We repeat our experiment 25 times and report the average results and standard error.
Mechod
NLL
AULLP-diff.
AULLP-diff,
tVaR-diffi
+VaR-diff.-
TAF
-8.09
0.42
4.05
0.80
4.36
mTAF
-8.55
0.25
2.6
0.57
STAF
-8.57
0.5
3.38
0.98
5.55
Ours
-9.03 + .08
0.26 :: .03
1.53 .21
0.61 4: .07
3.8 + .49
5.4 Performance Comparison with Pre-Hoc Tail Estimation Methods
In this section we report the performance of our method on synthetic experiment from Laszkiewicz et al. (2022) to compare the performance of our method to the state of the art in pre-hoc tail rstimatiion-deeeddent models.
Data The target dataset is generated by a 8-dimensional Gaussian copula. The marginals of the copula consist of two Gaussians, followed by a 2-mixture and 3-mixture of Gaussians. The last four marginals are a mixture two t-distributions with v 2. Mixtures have randomized means and variances and equal weights and the correlation matrix is randomized.
10

---

Published in Transactions on Machine Learning Research (10/2024)
AULLP for variable 1, shorter tailed
AULLP for variable 2, heavier tailed
.GG base
G50 F011-11
Mixture of GGD base
Figure 6: Ablation study results. The AULLP for models trained with two different base distributions are plotted on different rows. Variables X1 and X2 are plotted on different columns. The gray lines show the results of models without robust gradient estimation and the black lines show the results with robust gradient. estimation. The results show that Mixixee of GGD provides an improved and more stable performance in capturing the tail behavior of the target, b) robust MLE improves the tail-related performance of the model for both and c) a combination of GGD mixture base density and robust MLE provide the best performance across the board.
More details can be found in Appendix C.2 of Laszkiewiez al. (2022)
Results Table compares the performance of our method with that of Laszkiewicz et al. (2022) We report the results of training our model on an RNVP with the same architecture used in previous experiments with the same setup as per Section 4.1 Laszkiewczz et al. (2022). Results of Laszkiewicz et al. (0022) are copied from the da = section of their Table 1. The metrics are reported for heavy tailed and light tailed components separately (denoted by à and subscripts). We observe that our method performs better in case of hcavy-tailed components while Laszkicwicz components. We attribute our method's slightly lower performance in case of light tails to mis-specification of tails since Laszkiewicz et al. (2022) use Student's base marginals with the same degree of freedom as the synthetic example (v 2) for heavy-tailed target components. This pattern is consistent with results from Section
al. (2022] 's mTAF reports better results for light-tailed
5.5 Tabular Datasets
Data To test our method a more realistic setting, following and expanding on the empirical evaluations done by other methods this area we train our model on four tabular datasets from the UCI repository, namely Power, Gas, Miniboone and Hepmass. We focus tabular data. for the ease with which measure and evaluate the tailedness of the training and synthetized data. For other data types such images and language, due to their inherent digh-dimensionality and the rich spatial (for images) and contextual (for language) dependencies it is difficult to define, measure and evaluate the tailedness of the data and the tail-adaptiveness of the model. We follow the same preprocessing steps as Papamakarios et al. (2017) (According to Appendix Section of their paper). We train RNVP and MAF flows with different base densities as well our own method. We report negative log likelihood and mean AULLP difference and tVaR difference over all columns.
Results In Table 3 we present a comprehensive evaluation our proposed approach for an RNVP normalizing flow. The results are contrasted against various base distributions with trainable parameters. We report average tVaR and AULLP over all variables. Our method's superiority is evidenced by the lowest NLL and (Var for all datasets. We report lowest AULLPs for all datasets except MINIBOONE, (1.19 1.01). Even
11

---

Published in Transactions on Machine Learning Research (10/2024)
Table 3: Comparison of NLL, AULLP difference and tVaR difference for our proposed method applied to a
RNVP in comparison with the same model with different base distributions. R.MLE refers to the robust gradient estimation of Section 4.2.1.
*The tVR metric tends to grow exponentially when the tail is heavily undercstimated
HEPMASS
POWER
MINIBOONE AULAP
UNVP
NET
[Valt]
1.79
Seadran'es
2
3,38
0.21
GGD
9.02...22
5.58
0.64
2.1...15
4:27
Ms.CGD R.MLE 7.22
1..4
0.44=
9.18
1.12
8.04
1.09::.15
3.74
1.19
05.39 0.77
in this case the difference is relatively small. Notably, the stark tVaR discrepancy, especially in HEPMASS with Student'ss s base 1e9 vs 1.12), underscores the challenge of tail estimation which our method effectively addresses. Our method reports superior results across the board except for taar in HEPMASS dataset (1.08 vs 1.12), again with very small margin. This comparison showcases the robustness of our methodology for capturing the tail behavior of the mixed-tail targets. Also of note is the inferior performance of the Student's and GGD bases. Considering these results alongside results from section 5.2 we conclude that while the Gaussian base maintains& a fixed tail behavior and produces average results dependant on the tail behavior of the target distribution and its pathologics, since both Student's t and GGD's tail behavior is parameterized and trainable, the flow either pushes them to underestimate the tail violating the requirements set by [Jaini ct al. (2020) or overostimate the tail, leading to generation of samples with extreme values which will results in subpar performance in tail-specific metrics.
Table 4: Comparison of NLI, AULLP difference and tVaR difference for our proposed method applied to a MAF in comparison with the same model with different base distributions. R.MLE refers to the robust gradient estimation of Section 4.2.1.
GAS
HRPMASS AULLES
POWER AUTLPP
MINIROONR AULII
NLL.
Walt
NLL.
193011
NEA
Gerrecan hooe
-3.11
2.5.1.0
210
11.2
1.42
3.3
-M.121 6.97
0
2:21
3,87
7,61
1:37
13:06
-95.87
1.36
27.71:22:04
2..
$1.18
18.08
11,87
17
=2.42=2.40
Mc.GGD I.MLE
-5.05 558
15.18:1...00 1.41
5,36 1.8
1002.8 1: 7.20
0.94 1.08
8.05
.65
1.5
6,96 .63
1.17.1.30
-96.13 2
6.77
1.445
1.66
We also investigate the efficacy of our method on Masked Autoregrossive Flow. Our findings, summarized in Table g strongly suggest that integrating the GGD mixture base with robust MLE significantly improves the model's general performance - in terms of Negative Log Likelihood (NLL) as well as its capabilities to capture tail behavior- - as evident from metrics AULLP and tVaR, for which we report the difference.
Our proposed method achieves the lowest NLL for all datasets except POWER and in this exception the margin of difference is quite small. This improvement is most pronounced in the HEPMASS dataset, where the NLL reached a remarkable 102.44 Additionally, our approach also yields the smallest AULLP and tVaR values in most datascts, highlighting its capability to more accurately capture tail distributions and overall density estimation.
6 DISCUSSION AND FUTURE WORK
In this work we demonstrated the practical utility of using generalized Gaussians for base density coupled with a robust maximum likelihood training algorithm when training normalizing flows to model mixed-tailed data. We proposed new method of making normalizing flows capable of modelling mixed-tail targets without explicitly binding the tail behavior of the base density to that of the target and instead letting the gradient-based optimization shape the tail of the base. We showcased the general capabilities of our proposed method in estimating mixed-tail targets.
Our method could potentially benefit from being able to set the number of modes in the base mixture adaptively. We varied the number of mixture modes for different tasks and observed an inflection point in performance enhancement with the increase of mixture components, beyond which the model's effectiveness
12

---

Published in Transactions on Machine Learning Research (10/2024)
declines. An extension that uses an infinite mixture model could be a promising future direction. To model more complicated multimodal and asymmetric tail behaviors, mixture of mixtures structure could be a promising direction for future improvements. Our proposed method also enables, with slight modifications introduced in the Appendix B, selective generation from specific regions of the learned distribution by targeting components of the latent space mixture model, offering finer-grained control over generated samples. Future work should explore this selective gencration capability further, including its application in downstream tasks such aaire machine learning and anomaly detection.
Acknowledgments
This research has been performed as part of the Enabling Personalized Intervention (EPI) project. The EPI project funded by the Dutch Science Foundation in the Commit2Data program, grant number 028.011.028.
References
Ahmed Z Afify, Ahmed M Gemeay, and Noor Akma Ibrahim. The hcavy-tailed exponential distribution: risk measures, estimation, and application to actuarial data. Mathemalics, 8(8):1276, 2020.
Pierre Baldi, Kyle Cranmer, Taylor Faucett, Peter Sadowski, and Daniel Whiteson. Parameterized machine learning for high-energy physics. arxiv preprint urXiv:1601.0790 2016.
Jens Behrmann, Paul Vicol, Kuan-Chieh Wang, Roger Grosse, and Jörn-Henrik Jacobsen. Understanding and mitigating exploding inverses in invertible neural networks. In International Conference on Artificial Intelligence and Statistics, pp. 1792-1800. PMLR, 2021.
Laurent. Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. International Conference on Learning Representations, 2017.
Alex Dytso, Ronit Bustin, H Vincent Poor, and Shlomo Shamai. Analytical properties of generalized gaussian distributions. Journal of Statistical Distributions and Applications, 5(1):1-40, 2018.
Eare Eftelioglu. Geometric median., 2017.
Igor Fedotenkov. A review of more than one hundred pareto-tall tail index estimators. Statistica, 80(3):245-29) 2020.
Anja Feldmann and Ward Whitt. Fitting mixtures of exponentials to long-tail distributions to analyze network performance models. Performance evaluation, 31(3-4):245 279, 1998.
Jordi Fonollosa, Sadique Sheik, Ramón Huerta, and Santiago Marco. Reservoir computing compensates
slow response chemosensor arrays exposed fast varying gas concentrations in continuous monitoring. Sensors and Actuators B: Chemicol, 215:618 629, 2015.
Paul Hagemann and Sebastian Neumayer. Stabilizing invertible neural networks using mixture models. Inverse Problems, 3788000550 2021.
Georges Hebrail and Alice Berard. Individual household electric power consumption. UCI Machine Learning Repository, 2012. DOI: https://doi.orr/110 10. 10..004442//5110 C58K54
Bruce M Hill. A simple general approach to inference about the tail of distribution. The annals of statistics, pp. 1163 1174, 1975.
Daniel Hsu and Sivan Sabato. Loss minimization and parameter estimation with heavy tails. The Journal of Machine Learning Research, 17(1):543-556 582, 2016.
Marat Ibragimov, Rustam Ibragimov, and Johan Walden. Heavy-tailed distributions and robustness in economics and finance, volume 214. Springer, 2015.
Priyank Jaini, Ivan Kobyzev, Yaoliang Yu, and Mareus Brubaker. Tails of lipschitz triangular flows. In International Conference on Machine Learning, pp. 4673 4681. PMLR, 2020.
13

---

Published in Transactions on Machine Learning Research (10/2024)
Philip Lundberg Jamissen, Yihan Xing, and Yucong Ma. Probabilistic design of thin-walled cylindrical structures for application in large cargo submarines. In International Conference on Offshore Mechanics and Arctic Engineering, volume 85895, pp. V05AT06A010, American Society of Mechanical Engineers, 2022.
Mike Laszkiewiez, Johannes Lederer, and Asja Fischer. Marginal tail-adaptive normalizing flows. In International Conference on Machine Learning, pp. 12020 12048. PMLR, 2022.
David M Mason. Laws of large nunbers for sums of extreme values. The Annals of Probabilily, pp- 754 764, 1982.
Makoto Okada, Kenji Yamanishi, and Naoki Masuda. Long-tailed distributions of inter-event times mixtures of exponential distributions. Royal Society open science, 7(2):191643, 2020.
George Papamakarios, Theo Pavlakou, and lain Murray. Masked autoregressive flow for density estimation. Advances in neural information processing systems, 30, 2017.
George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshmi- narayanan. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 22(57):1 64, 2021. URL http://jmlr.orr
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International conference on machine learning, pp. 1530-1538. PMLR, 2015.
Byron P Roe, Hai-Jun Yang, Ji Zhu, Yong Liu, Ton Stancu, and Gordon McGregor. Boosted decision trees alternative to artificial neural networks for particle identification. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, 543(2-3): 577-584, 2005.
Akash Singh and Ravi Gor. Relevancy of pricing european put option based on truncated gumbel distribution in actual market.
Ying Sun, Prabhu Babu, and Daniel P Palomar. Regularized robust estimation of mean and covariance matrix under heavy-tailed distributions. IEEE Transactions on Signal Processing, 63(12):3096-3109, 2015.
Esteban G Tabak and Cristina V Turner. A family of nonparametric density estimation algorithms. Commu- nicalions Pure and Applied Mathemalics, 66(2):145 164, 2013.
Sergio Venturini, Francesca Dominici, and Giovanni Parmigiani. Gamma shape mixtures for heavy-tailed distributions. The Annols of Applied Statistics, 2(2):756-770) 2008.
Junfeng Wang, Hongxia Zhou, Minglian Zhou, and Lei Li. A general model for long-tailed network traffic approximation. The Journal of Supercomputing, 88:155-172, 2006.
A Points of Failure in Tail Estimators
Figure 7 shows the accuracy and consistency of the bootstrapped Hill's estimator ân for samples with varying sizes from a Pareto distribution with different tail indices. We report the difference between true and estimated tail indices. We observe that åH overestimates for heavy-tailed samples and underestimates for short-tailed ones. We also observe increased variance for smaller batch sizes closer the usual minibatch sizes from extremely heavy-tailed targets.
14

---

Published in Transactions on Machine Learning Research (10/2024)
Tail index
Target tail index 0.5
Target tail index 2.5
Target tail index 5.5
-5
Target tail index 7.5
Target tail index 10.5
200
400
600
800
1000
Number of Samples
Figure 7: Mis-ostimation of tail indices by bootstrapped Hill method for datasets sampled from Pareto distribution. Tail indices vary from heavy (a=0.5) to short (a=10.5) and sample sizes from 64 to 2048. Each experiment conducted 9 times and the mean solid lines and standard error shades of the difference between the true tail index and the estimated one are reported.
B Selective Generation by Exploring the Latent Space
tail-adaptive normalizing flows method for controlled data generation, enabling precise sampling from specific regions of complex probability distributions. By regularizing the training process to encourage the mixture components to spread out in the latent space, different components specialize in generating different regions the target distribution, from the tails to the high-density areas around the mean. This allows for targeted sampling from specific components to generate samples from dcsired regions, offering benefits in applications like fairncss-aware machine learning where oversampling minority groups in the tail regions is essential.
In this section, we introduce extension
We posit that, given considering that each component of the mixture can exhibit spectrum of tail behaviors, different individual components the mixture- or subsets thereof will, upon model convergence, be responsible for generating different parts of the target marginals relative to their tail behavior. In other words, some components will be responsible for gencrating the tail of the target distribution while alternate components are assigned represent the regions of high probability density, typically centered around the distribution's mean. To capture extreme tails, find that it is helpful for the components of the mixture base distribution to be
of tail-adaptive mixture to model the latent space of our flow model, and
15

---

Published in Transactions on Machine Learning Research (10/2024)
spread-out. We can encourage this behavior by adding extra term to the maximum likelihood objective:
K Σ Mlg - 1xll2
R(HI..... pK) = K
(3)
where My denotes the geometric median of the component locations (ft).. Since the means could, due initialization the stochastic nature of the optimization, have long tails, we calculate the geometric median of the means, Mg and then calculate the sum of the l2 norms of the means and the calculated medians as the total spread of the means. We add the R term to the model's loss function. The model is thereby rewarded during optimization by increasing the distance across the locations.
Our training algorithum including the regularization term will be:
Input: flow, (1, E, B, J: base for minibateh xi do
A(pl..... μK) = 2 Σ k = K lug pk|2
C = E (x, [x x;; 0)] + R(ul,...... , ,K)
g/4 = arg min g Ek= lgg VoL(O:
AO VAL
A & - v
end Algorithm 1: Training Algorithm for tail-adaptive normalizing flows with enhanced selective generation
The regularization term in our loss function incentivizes the model to push the means of the mixture modes apart, covering a larger part of the latest space including the tail area - instead of being centered around certain areas. This by design leads to each sample mapped to the latent space having a small subset of mixture modes as its nearest neighbors. We leverage this aspoct of our model for Selective Generation.
To generate samples that are on the tail area of the distribution real world dataset in specific area of the joint density of subset of its variables few samples from that area to the latent space, find k ncarest mixture modes to these samples - & being a predetermined number or the radius of a hypersphere around the geometric median of our subsample - and then generate our desired number of samples from these modes by sampling from them indiscriminately or by normalizing their weights to form a discrete mixing distribution and forming a separate mixture model.
its mean, in case of complex can map
The advantage of this method is especially evident for areas such as fairness where minority groups are defined in the joint tail section of several features of the dataset. We can upsample any subgroup within our population which has been shown to help mitigate some of the problems with fairness and more generally with performance of ML models. The advantage here is that we are not blindly exploring the latent space. Rather, we are systematically choosing the components in our base distribution with highest probability of being responsible for generating samples similar to our target sample or subsample and sample directly from those components to be pushed forward through the flow model and be transformed into our larget.
B.1 Selective Generation in Practice
Figure iillsstrate an example of our method's capabilities for selective generation. We train an RNVP using Algorithm D with a 20-component GGD mixture base and robust MLE on target dataset sampled from a two dimensional Student's t with v=2. We then sample a dataset with the size of 100000 from the Student's target and evaluate the probabilities of each sample on target distribution. We bin the probabilities into 15 equal sized bins, with samples belonging to bins with lower probabilities assumed to be further down either tails of the distribution and vice versa. We then evaluate the samples of cach bin on every single of the 20 training base mixture components. The component that exhibits the highest average probability value for each bin is assigned as the primary generator for that bin. Since the samples have the highest average probability value when evaluated on that specific component. Therefore, if we sample from that component in the latent space and push the samples through the flow, we will predominantly yield instances that are belong to that specific bin. Tn Figure g for each bin the component with the highest mean probability values
16

---

Published in Transactions on Machine Learning Research (10/2024)
19
Base Component w/ Highest Mean Prob
17
16
15
13
12
10
7
6
3
2
1
13
Bins
Figure Selective generation from an RNVP model with component GGD mixture base for 2d Student's larget.
is marked. We note that component 0 is primarily responsible for generating instances for the bin with the lowest average probabilities (i.e., instances the tail evaluated the target distribution itself), while the majority of the distribution mass is generated by components 3 and 18. Component 15 generates the remaining instances, notably those that are not extreme on the distribution's tail but are situated further from the distribution's mean. It should be noled that our larget is a two dimensional larget. However, to refrain from over- complicalin this analysis we do not discriminate between the dimensions when binning the samples. If each marginal is considered separately, it will offer a more nuanced control over the sample generation process.
In scenario with real world target dataset, will not bin the samples, but rather select representative sample of the type want to generate from the target dataset. We then map these representative samples to the latent space. Once we have the mappings in the latent space, we could, for example, choose the top k components with the highest probability values for our selective generation and use a simple weighting scherne (e.g., using the average likelihood values or distance to the component mean), creating sub-mixture for that particular subset of samples. This sub-mixture will then enable us to upsample our dataset through generation of samples similar our representative samples.
While not primary focus of the main paper, the concept of selective generation through targeted sampling within the latent space presents a promising avenue for future research. This approach could be particularly valuable in applications like fairness-aware machine learning, where oversampling minority groups represented in the tail regions of the data distribution is crucial.
C Extended Definition of Tail-Related Metrics
Tail Value at Risk The Tail Value at Risk (tVaR) at quantile level is an expected loss measure for losses exceeding the Value at Risk (VaR) at the same level, defined as:
tVRR, (X) EE[X X > VaR. (X))
with X being the loss variable and VaR (X):
17

---

Published in Transactions on Machine Learning Research (10/2024)
(X) = in E R F(x) > a}
where F(x) is X's cumulative distribution function.
The WVaR-difference metric, the absolute difference between empirical ((VaRa and model-generated (tVaR, (oodel) iaRRs:
VaR-difff tVaR - tVa.R
evaluates generative model's tail capture capability. Smaller VVaR-difference implies better model perfor- mance in mimicking the distribution's extreme quantiles.
Area Under Log-Log Plot The Area Under Log-Log Plot (AULLP) involves integrating the logarithm of the survival function, which represents the probability of random variable excceding given value, in log-log space beyond high quantile threshold. We report the The AULLP difference between test dataset and synthetic dataset.
The Area under the log-log plot (AULLP) measures a model's ability to capture the far distribution tail, emphasizing extreme values more than the Tail Value at Risk (tVaR). It involves integrating the survival function (S(x) = 1 F(x)), which represents the probability of a random variable X exceeding value æ, log-log space beyond high quantile threshold za (e.g., 95th or 99th percentile):
AULLP(X) = /
00
log ((x) d(log
Jog do
To gauge generative models' tail capture accuracy, the AULLP difference (AULLP-diff) between the empirical training data distribution and the model-generatedd distribution is calculated:
AULLP-diff AUT.Lemmp (X) AUT.I.Pmooi (x)
AULLP"MP(X) (X) the empirical distribution's AULLP, and AULLpmodel (X) is for the model-generated distribution.
D
Experiments
Our training was done two types of nodes depending availability, with either one NVIDIA A40 A100 GPU.
D.1 Data preparation
Following the steps from Papamakarios et al. 2017)) we prepared the data. The next parts of this section will give a short explanation of the preprocessing done for each dataset.
The POWER dataset (Hebrail & Berard 2012) includes records of household electricity use over 47 months. Although originally a time scries, records were treated as soparate, identical samples. The time was changed into an integer representing minutes in the day, and random noise was added. The date and the global reactive power parameter, which often shows a zero value, were removed to prevent unexpected changes in the distribution. Uniform noise was added cach feature. The magnitude of the added noise was large enough avoid duplication but small enough keep the data values largely the same. The training data has 1,659,917 examples featuring variables.
POWER
18

---

Published in Transactions on Machine Learning Research (10/2024)
The GAS dataset (Fonollosa et al. 2015) contains measurements from 16 chemical sensors exposed to gas mixtures over duration of 12 hours. Analogous to the POWER dataset, it is time series that has been handled as if each instance were an i.i.d sample from the marginal distribution. Data was exclusively used from the file ethylene Connne_OOtttt CO.tzt, which pertains to an ethylene and carbon monoxide mixture. The removal of highly correlated attributes resulted in an eight-dimensional dataset. The training data has 852,174 examples featuring variables.
GAS
HEPMASS The HEPMASS dataset (Baldi et al. 2016) contains particle collision measurements in the field of high-energy physics. Half of the instances denote particle-enneratnng collisions (positive), while the remainder originate from background source (negative). For this study, positive examples from the "1000' dataset were selected. Five features were excluded due to their high frequency of recurring values, as such repetition can induce spikes in the density and potentially yield misleading outcomes. The training data has 323,123 examples featuring variables.
MINIBOONE The MINIBOONE dataset (Roe et al. 2005): derived from the MiniBooNE, experiment conducted at Fermilab. Like HEPMASS, contains positive (electron neutrinos) and negative examples (muon neutrinos). For this study, only the positive examples were used. Notable outliers (11 instances) exhibiting a constant value of -1000 across all columns were removed, well as seven features displaying excessively high counts for specific values, such as 0.0. The training data has 29,556 examples featuring 43 variables.
19


Reviews:
[{"review_id": "GVW3gZmGup", "reviewer": "Reviewer_59rd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The proposed method is novel, simple and interesting.\n* The proposed solution is really well motivated. The challenges are clearly laid out and then the proposed method \"flows\" naturally from the challenges.\n* The results on both the 2D synthetic data and tabular datasets are very encouraging. \n* The paper includes adequate ablation experiments to highlight the utility of the proposed method.\n\nWeakness:\n* The paper uses only two types of flows: Real NVP and Masked Autoregressive flows. It would be interesting to add additional evaluations using nonlinear normalizing flows, e.g., (Ziegler & Rush, 2019). These experiments would further highlight the applicability of the proposed method across flows.\n\n* Lack of results on real-world datasets: The paper only includes results on the tubular datasets. While the results are encouraging it is unclear how mix-tailed these datasets are. It would be helpful to include experiments with additional real-world datasets with mix-tailed data.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a  improved framework for training flows-based models with robust capabilities to capture the tail behavior of mixed-tail data. To this end the paper proposes to use a mixture of Generalized Gaussian Distributions (GGDs) as the base density of the flow. Additionally, the paper proposes using employing the Geometric Median during the Monte Carlo estimation of the likelihood at training time. The proposed method is evaluated on 2D synthetic data and tabular datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The paper can be accepted largely as is.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ouXUxKx5l6", "reviewer": "Reviewer_kPeN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is sound and in appears technically correct, but some technical details are missing and hence it is somewhat difficult to evaluate the exact correctness. Even though the problem setup is not new and the previous works have provided more rigorous treatment of the topic, the paper  is likely to be somewhat interesting for practitioners looking for better methods to fit flows in cases of difficult tail behaviour. The solution is not particularly novel and combines two known techniques used previously in the literature (learnable mixtures as base distributions, and geometric median for robust learning for heavy-tailed distributions), but the specific solution has not been presented before. It is adequately evaluated against alternatives and shown to work sufficiently well in practice, but it is worth noting that all experiments are simplified and fairly artificial. Even the supposedly more realistic experiment only considers generic UCI data sets.\n\nFormally, the paper satisfies the evaluation criteria but this is partly because the claims are so weak and indirect ('we highlight and discuss problems' and 'generally favorable performance'). As said above, I do think the paper will be interesting for some readers, but it would benefit from sharper and more scientific writing to contribute better for the general literature. Making the claims more explicit would help in achieving that.\n\n**Strengths:**\n - The paper makes the importance of accounting for tail behaviour very clear, especially in Figure 1\n - The solution is natural and easy to understand, and if open source implementation was provided I presume it would be easy to use in practice\n\n**Weaknesses:**\n - The paper lacks theoretical rigor, especially compared to the most related work. Section 4.1 makes an attempt of theoretical analysis when discussing how the requirements of Jaini et al. (2020) are satisfied for the chosen prior, but this part is not particularly deep and the presentation is too verbal.\n - The empirical performance is not particularly convincing and there is no clear evidence the proposed method is truly useful. It is only compared against the previous method of Laszkiewicz et al. (2022) in one experiment that shows a small numerical improvement, but it is hard for the readers to grasp how significant the improvement is.\n - The actual method description is somewhat lacking. Appendix B mentions a specific regularizer and introduces a new concept of \"Selective Generation\" that appear to be essential components of the method, yet they are not described in the main paper at all (and there is no reference to Appendix B).\n - No open source implementation. As the paper is in general quite scarce on technical details, I believe we should be able to see the details at least in the code.\n - The writing style is somewhat informal and in general the paper falls short of the quality requirements of top-end venues. There are no clear specific weaknesses and e.g. the most critical citations are included, but the general impression is not that of a highly polished scientific paper and the discussion is not particularly deep.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies learning of normalizing flows in context of mixed-tail data, introducing a new solution that combines a mixture of generalized Gaussian distributions as a base distribution with a robust training algorithm that uses geometric median of subsample gradients. The method is compared against previous methods proposed for addressing tail behaviour of flows on a few simple data sets with somewhat mixed results. The proposed method is in general best in terms of NLL, but in some experiments is worse in terms of the metric focusing specifically on tails (tVaR and AULLP).", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The following list includes both concrete requests or recommendations and a few questions.\n\n 1) Clarify the role of Appendix B. If these additional elements were used in the experiments, then you need to explain them already in the main paper. If they were not, you should make it much more clear that Appendix B introduces some additional ideas that could be used but were not. Also, please write the regularization math as proper equations, not in-line within a paragraph.\n 2) Clarify the method itself. While you give the mathematical details for both the base distribution and geometric median, it would be near impossible to reproduce the work based on the material provided. I recommend also releasing the code.\n 3) Why do you use the word 'synthesis' in the title? For me that term refers to generation of artificial data (e.g. for privacy reasons), but I do not see a particular reason for this paper to emphasise synthesis. You are, after all, proposing a general method for generative modelling applications of normalising flows, right? Your experiments are evaluated in terms of NLL, not in terms of synthetized data.\n 4) Section 4.2.1 should cite previous works that considered geometric median at the beginning of the subsection, not at the end. This is important to avoid giving the impression the general idea of using GM for improving robustness of the gradient estimates was a new contribution. I would also strongly recommend mentioning GM explicitly already in the introduction, with proper citations.\n 5) The notation is inconsistent. In Section 2 you first claim to use bold letters for random variables and non-bold for observations, yet immediately after that say $\\mathbf{x}$ refers to observations and $X$ is used to refer to a random variable. \n 6) Section 4.1 talks about platykurtic and leptokurtic distributions, as specific formulation relating to tails. Please introduce this perspective where kurtosis is used for characterising the heaviness of the tails already in Section 2.1.\n 7) Check use of \\citet{} and \\citep{}; there are a few occasions where the wrong command is used.\n 8) Figure 1 is illustrative, but I would not *start* the whole Introduction with that. A reader wants to start the reading by reading some actual context, and hence the figure should be placed only on the 2nd page. Also, I would suggest adding one more column here with the proposed mixture as the base distribution. Now the figure only shows the importance of using a suitable base distribution, but with that additional column it would also show that your method works in all cases. Does it? Knowing whether it solves these problem instances is important and showing it in this figure is the easiest way of communicating it for the reader.\n 9) I don't see the point of Figure 2. It shows well that the likelihood is the best for the base distribution with the correct tail behaviour, but I do not understand what is the supposed message of the gradient magnitude plot or the visualisation of the progress over iterations. All cases appear quite similar throughout the optimization, and hence I do not see why the plot is as a function of iteration. Maybe think a bit more about what exactly you want to show and then create a more informative figure?\n 10) The NLL arrow is in the wrong way in Table 2, right?\n 11) Mention in Section 5.1 that the details of the metrics are provided in Appendix C.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lcwzGHrkjb", "reviewer": "Reviewer_frCD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The paper is well written.\n- The intuitive comparison against cited papers is well made.\n\n**Weaknesses**\n- As mentioned by the authors, it is not obvious that the mixture base wouldn't introduce significant training challenges in general cases\n- Some parameters of the model (namely the weights of the base mixture) lie in a simplex but it doesn't seem like the optimization procedure (simple Adam) takes care of this peculiar geometry.\n- Limited questioning on the architecture of the transport map itself.\n- How would the proposed geometric median method compare to classic exponential moving average or gradient clipping (which are very popular in the generative modeling community) ? It seems like the training stabilization method hasn't been benchmarked against very natural competitors.\n- The experiments show that the method is not very well suited for **mixed** tails data as it underperforms on light tail data compared the heavy tail one.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors address the complex problem of generative modeling on heavy tail data using normalizing flows, with a particular focus on mixed-tail distributions. Recognizing the inherent challenges in selecting or estimating the base distribution for such data, they introduce a novel method for base estimation grounded in generalized Gaussian distributions. This approach aims to more accurately capture the characteristics of mixed-tail data. In their exposition, the authors thoroughly discuss the difficulties and limitations encountered during the training process. To mitigate these issues, they propose the use of a more robust gradient estimator, which enhances the stability and effectiveness of the training phase. The efficacy of their proposed modifications is demonstrated through experiments on both synthetic (toy) models and real-world datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Clarify why \"the bias of MLE estimator is quite apparent\" in Fig. 2 \n- As mentioned by the authors, (Jaini et al., 2020) mentions that the Lipschitz behavior of triangular maps is a significant limitation. Could we consider non-triangular flows ? Similarly, (Cornish et al., 2020) were also limited by this Lipschitz behavior when working on multimodal data and decided to switch to continuous flows.\n- Explain how this method compares to changes made to the transport map itself as in (Hickling et al. 2024)\n\n\n[Cornish et al., 2020] Cornish, R., Caterini, A., Deligiannidis, G., and Doucet, A. Relaxing bijectivity constraints with continuously indexed normalising flows\n\n[Hickling et al. 2024] Tennessee Hickling, & Dennis Prangle. (2024). Flexible Tails for Normalizing Flows.", "broader_impact_concerns": "They are no ethical implications of this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is an addition to the normalizing flows literature, and during the rebuttal process, the authors addressed most of the reviewers' concerns. Therefore, both the reviewers and I recommend its acceptance.
Decision: Accept (poster)