Flexible Tails for Normalizing Flows
Authors: Tennessee Hickling
Dennis Prangle
Venue: ICML 2025 poster (2025)
Keywords: normalizing flows
extreme values
heavy tails
variational inference
Primary area: probabilistic_methods->everything_else

Abstract: Normalizing flows are a flexible class of probability distributions, expressed as transformations of a simple base distribution. A limitation of standard normalizing flows is representing distributions with heavy tails, which arise in applications to both density estimation and variational inference. A popular current solution to this problem is to use a heavy tailed base distribution. We argue this can lead to poor performance due to the difficulty of optimising neural networks, such as normalizing flows, under heavy tailed input. We propose an alternative, "tail transform flow'' (TTF), which uses a Gaussian base distribution and a final transformation layer which can produce heavy tails. Experimental results show this approach outperforms current methods, especially when the target distribution has large dimension or tail weight.

Flexible Tails for Normalizing Flows
Tennessee Hickling" Dennis Prangle"
Abstract
CURRENT PROPOSALS
Normalizing flows are flexible class of probabil- ity distributions, expressed as transformations of a simple base distribution. A limitation of standard normalizing flows is representing distributions with heavy tails, which arise in applications to both density estimation and variational inference. A popular current solution to this problem is use heavy tailed base distribution. We argue this can lead to poor performance due to the difficulty of optimising neural networks, such as normaliz- ing flows, under heavy tailed input. We propose an alternative, "tail transform flow" (TTF), which Gaussian base distribution and final trans- formation layer which can produce heavy tails. Experimental results show this approach outper- forms current methods, especially when the target distribution has large dimension or tail weight.
Distribution
Target Distribution
Extremes
Dependencies
PROPOSAL
Distribution
Target Distribution
Model Dependencies
Model Extramas
Figure 1: not the base distribution.
Our method models extremes in the final layer,
Modelling distributions with heavy tails is critical in many applications such as climate (Zsceisschlrr et al. 2018). con- tagious diseases (Cirillo & Taleb 2020) and finance Gilli & Kěllezi 2006). Indeed, in the context of density estima- tion, results from extreme value theory (EVT) (Coles 2001 Embrechts et al. 2013) show distributional tails often be modelled using particular heavy tailed distribution, the generalized Pareto distribution (GPD). Heavy tailed targets can also arise naturally in Bayesian inference, requiring heavy tailed approximate distributions in variational infer- ence (Liang et al. 2022) and providing challenges for MCMC methods (see e.g. Yang al. 2024).
1. Introduction
A normalizing flow (NF) expresses a complex probability distribution as parameterised transformation of a simpler base distribution. A NF sample
x = 0);
(1)
where: is a sample from the base distribution, typically (00, 7). A number of transformations have been proposed which produce flexible and tractable distributions. Typically multiple transformations are composed to form T with the desired level of flexibility. Applications include density estimation (fitting a transformation to observed data points) and variational inference (fitting a transformation to target distribution). In either case 0, the parameters controlling T', can be optimised using stochastic gradient methods for suitable objective function. For reviews of NFs see Kobyzev et al. 2020) and Papamakarios et al. (202T).
However standard NFs do not model heavy tails well. In particular.Daini et al. (020) prove that Gaussian tails cannot be mapped to GPD tails under Lipschitz transformations. (This is a special case of their main result, which we include as Theorem 1.2 below.) Many NFs use Lipschitz trans- formations of Gaussian base distributions, implying they produce distributions poorly approximating heavy tailed distributions.
Equal contribution 'School of Mathematics, University of Bristol, Bristol, UK. Correspondence to: Tennessce Hickling <ten- nessee.hickling bristol.ac.uk>.
Based on this result, several authors have proposed using heavy tailed base distributions. This includes the tail adap- tive flow (TAF) methods of Laszkiewicz et al. (2022) for density estimation, and similar methods for variational infer- ence (Liang et al. 2022). We argue that using heavy tailed
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Flexible Tails for Normalizing Flows
base distributions has an important drawback. Typical NF transformations involve passing the input 2 through neural network. However neural network optimisation can perform poorly under heavy tailed input-as as this can induce heavy tailed gradients, which are known to be problematic (Zhang 2020). We verify this problem empirically in Section 4.1 for NFs (and in Appendix pope a simpler neural network
1.1. Related Work
1.1.1. THEORY
A common definition of heavy tails (e.g. Foss et al. 2011 is as follows.
ett al.
Definition 1.1. A real valued random variable 2 is (right-) heavy-tailed when E/exp(AZ)] = oo for all X > 0. Other- wise Z is light-tailed.
regression example.)
To improve performance we propose an alternative: use Gaussian base distribution with final non-Lipschite trans- formation in T. We refer to this as the tail transform flow (TTF) approach. The difference is illustrated in Figure [ In Section I.I we provide a suitable final transformation 3 motivated by extreme value theory. This is easy to im- plement in automatic differentiation libraries it's based on standard special function: the complementary enror function. We also prove that this transformation converts Gaussian tails to heavy tails with tunable tail weights, and hence provides the capability of producing heavy tails when used with standard flows.
Examples of heavy tailed distributions include Student's T and Pareto distributions. An example of a light tailed distribution is Gaussian distributions.
Under Definition 1.1 the following result holds.
Theorem 1.2 (Jaini et al.) (020). Let Z be a light tailed real valued random variable and ' be a Lipschitz transfor- mation. Then TZZ) is also light tailed.
Jaini et al. (2020) also generalise this result to the multivari- ate case, and prove that several popular classes of NFs use Lipschitz transformations (see also| Liang et al. 2022).
We perform experiments showing our TTF approach out- performs current methods, especially when the target distri- bution has large dimension or tail weight. We concentrate on density estimation, investigating both synthetic and real data, but also include a small-scale variational inference experiment.
Other mathematical definitions of tail behaviour exist, in- cluding alternative usages of the term "heavy tails". Liang et al. (2022) apply such definitions to prove more detailed results how Lipschitz transformations affect tail weights (e.g. Theorem
our appendices).
To model heavy tailed distributions, Jaini et al. note that there is a choice "of either using source densities with the same heaviness as the target, or deploying more expressive transformations than Lipschitz functions". They pursue the former approach and this paper investigates the latter.
To summarise, our contributions are:
We illustrate the problem of using extreme inputs in NFs.
We introduce the TTF transformation equation ( and prove it converts Gaussian tails to heavy tails with tunable tail weights. This provides the capability of producing heavy tails when used with standard NFs.
1.1.2. HEAVY TAILED BASE DISTRIBUTIONS
Several NF papers propose using a heavy tailed base distri- bution. To discuss these, let Z be a random vector following the base distribution of the normalizing flow, and let Z; be its ith component.
We provide practical methodology using TTF in a nor- malizing flow architecture.
Firstly, for density estimation, Jaini et al. (2020) set Z, ~ ty: standard (location zero, scale one) Student's T distribution with v degrees of freedom. The Z; components are indepen- dent and identically distributed. The / parameter is learned jointly with the NF parameters. Optimising their objective function requires evaluating the Student's 7 log density and its derivative with respect to v, which is straightforward.
Jaina
We demonstrate improved empirical results for density estimation (synthetic and real data examples) and varia- tional inference (an artificial target example) compared to standard NFs, and other NF methods for heavy tails.
In the remainder of this section we review related prior work. Then Section [ outlines background material and Section describes our proposed transformation, as well as summarising our theoretical results. Section Sociionoooonn contains our examples, and Section S concludes. The appendices contain further technical details, including proofs. Code for all our examples can be found at httpt:::////ccc //gtthbb comm Com/Tennessee. Wa www.WWawaaaaahceea www.hhhtaiaaiaa wtw.h//tannnfllloo.o
An extension to allow tail anisotropy taiil behaviour varying across dimensions-byy setting (independently) Z; ty.. so the degrees of freedom can differ with i. In Laszkiewicz et al. (2022), two such approaches are pro- posed for density estimation. Marginal Tail-Adaptive Flows (mTAF) first learn v/ values using a standard estimator (a version of the Hill [975 estimator). Then the distribution of Z is fixed while learning T. Another feature of mTAF
2

---

Flexible Tails for Normalizing Flows
is it allows Gaussian marginals by using Gaussian base dis- tributions for appropriate Zi components, and then keeping Gaussian and non-Gaussian components separate through- out the flow transformation. Generalised Tail-Adaptive Flows (g train the vis and 0 jointly. Liang et al. (2022) propose a similar method to gTAF for the setting of varia- tional inference: Anisotropic Tail-Adaptive Flows (ATAF).
on the carrier function provide useful theoretical properties, and it would be interesting to explore analogous results for our method in future.
2. Background
2.1. Normalizing Flows
Consider vectors = E E and x = T(z) E R Suppose 2 is sample from a base density 9: (z). Then the transformation T defines normalizing flow density (r)).. Suppose T is diffeomorphism (a bijection where T and 2 are dif- ferentiable), then the standard change of variables formula gives
1.1.3. OTHER METHODS
Here discuss work on alternatives to Student's T base distribution. In the setting of density estimation, Amiri et al.) (2022) consider two alternative base distributions: Gaussian mixtures, and generalized Gaussian. Although the former are light tailed (see Appendix A). they argue that mixture distributions can in theory model any smooth density given enough components, and also that they are more stable in optimisation than heavy tailed base distributions. The latter has tails which can be heavier than Gaussian, but are lighter than Student's ' T.
qu (x(x) -
det Jy-:
Here Jr 1 (z) denotes the Jacobian of the inverse transfor- mation and det denotes determinant.
Typically parametric transformation T(2; 0) is used, and q= (z) is a fixed density, such as that of (O, Z) distribution. However some previous work uses parametric base density (s( 0). (So 0 denotes parameters defining both I and q:.) For instance some methods from Section 1.1 Juse a Student's T base distribution with variable degrees of freedom.
McDonald et al. 2022) propose COMET (copula multivari- ate extreme) flows for the setting of density estimation. This involves a preliminary stage of estimating marginal cumu- lative distribution functions (cdfs) for each component of the data. The body of each marginal is approximated using kernel density estimation, and the tails are taken as GPDs, with shape parameters chosen using maximum likelihood. The COMET flow then uses a Gaussian base distribution, followed by a copula transformation. This is an arbitrary normalizing flow followed by an elementwise logistic trans- formation so that output is in [0, 1js Finally the inverse marginal edfs are applied to each component. In Appendix C.2 we analyse the resulting tail behaviour and prove that the output cannot capture the full range of heavy tailed be- haviour. More formally, Theorem C.8 proves that the output is not in the Fréchet domain of attraction- see Definition below. This is in contrast to our proposed method, TTF, which we show in Appendix C.3 does produce in outputs in the Fréchet domain. Despite this our experiments in Section 4 find COMET flows often have good empirical performance, which we discuss in Section 5
Usually we have T = Tx o .... T2 Ti, a composition of several simpler transformations. Many such transformations have been proposed, with several desirable properties. These include producing flexible transformations and allowing evaluation (and differentiation) of T' 7-, and the Jacobian determinant. Such properties permit tractable sampling via [ and density evaluation via 2
Throughout the paper we describe NFs in the generative direction: by defining the transformations T, applied to sample from the base distribution. NFs can equivalently be described in the normalizing direction by defining the T:- transformations. We pick the generative direction simply to be concrete.
2.2. Density Estimation
Density estimation aims to approximate a target density p(z) from which we have independent samples { We assume x, E Rd.
The extreme value theory literature also proposes methods to jointly model heavy tails and the body of a distribution. See Huser et al. (2255) (Section for a review of several approaches. We comment on one particularly relevant uni- variate method: iPapastathopoulos & Tawn (2013); Naveau let al. (2016); de Carvalho et al. (2022) use edf G (²) which composes the GPD edf H with a carrier function G: [0, 1] + [0, 1] which is taken to be simple parametric function. Similarly, Stein (202)) composes several carrier functions. Our approach is similar, effectively using NF' in place of carrier functions, and covering the multivariate case. Naveau et al. (2016) proves that certain conditions
We can fit normalizing flow by minimising the objective
N
To = log Ou (2re5 0)-
This is equivalent to minimising Monte Carlo approxima- tion of the Kullback-Leible divergence K L[p(z)llq<(() 0)]. The objective gradient can be numerically evaluated using automatic differentiation. Thus optimisation is possible by stochastic gradient methods. This approach remains feasible
3

---

Flexible Tails for Normalizing Flows
under Student's T base distribution since its log density and required derivatives are tractable.
x) proportional to x 1/2-1 while for À < the upper tail has bounded support. In terms of Definition 1.1 X > 0 guarantees heavy tails and À < 0 guarantees light tails. Given X, the shape parameter of the GPD resulting from Pickands theorem is a measure of how heavy the tail of X is. A Gaussian distribution results in À = O (and requires a non-constant scaling function h())), and À > 0 represents heavier tails. Finally, we will use the following definition in our theoretical results later.
2.3. Variational Inference
Variational inference (VI) aims to approximate target den- sity p(æ), often Bayesian posterior for parameters I E RR Typically is used where only unnormalised target p(xz) can be evaluated. Then p(z) p(x)/ 2 but the normalizing constant 2 = fg. p(r)dr cannot easily be calculated.
The Fréchet domain of attraction with shape parameter X, Ox is the set of distributions resulting in À > 0 under Pickands theorem.
Definition 2.1.
VI aims to minimise K [L]g,(2; 0) [pp[))]] over a parame- terised set of densities 2z 8). In this paper qu (2; 8) is a
normalizing flow. An equivalent optimisation task is max- imising the ELBO objective
3. Methods and Theory
(+) = Ez~q. log ((x) log qa (2:
We propose producing normalizing flow samples R Thoyy (z), where 2 is a (N 1) sample, Thody is standard normalizing flow transformation and It is final transforma- tion. If Tody is Lipschitz transformation, then the input to R has light tails by Theorem T.2 Thus the resulting archi- tecture avoids the problem outlined in Section Tof of passing extreme values as inputs any neural network layers. The final transformation R should be able to output heavy tails of any desired tail weight. This section presents our proposal for a suitable transformation R.
This has tractable unbiased gradient estimate
M
M
Ve Vollogp(az - log gz (z 0)),
where 24 = (22;0) 0) and {=) are independent samples from the base distribution. Again, the gradient estimate can be numerically evaluated using automatic differentiation, allowing optimisation by stochastic gradient methods.
A generalisation of the above is use base distribution with parameters. In particular, ATAF (Liang et al. 2022) needs to learn degrees of freedom for Student's 7 distri- butions. The above approach remains feasible by using a Student's T sampling scheme which allows application of the reparameterization trick, as detailed in Appendix G.4
Section 3.1 describes our proposed transformation for the univariate case. Section 3.2 summarises our theoretical results on its tail behaviour. Sections 3.3 3.4 give further details produce practical general purposs method, and Section 3.5 comments universality. Proofs and technical details are given in the appendices. This also includes a discussion of alternative transformations and related results in the literature, in Appendix D
For more background on this form of VI see Rezende & Mohamed (2015); Blei et al. (2017): Murphy (023))
2.4. Extreme Value Theory
3.1. TTF Transformation
Extreme value theory (EVT) is the branch of statistics study-
We propose the tail transform flow (TTF) transformation R R
ing extreme events (Coles. 2001 Embrechts et al. 2013).
A classic result is Pickands theorem (Pickands III 1975); see Papastathopoulos & Tawn (2013) for review. Given a scalar vealev random variable X, consider the scaled excess random variable X-u |X > u, where u. > is large h(()
$ R(2; (-) = Xg
[erfe(|=]/V2]
1]. (3)
We use the notation sign(2), with X X+ for s = and A A_ for 11 The transformation is based on erfe, the complementary error function. This is a special function, reviewed in Appendix which is tractable for use in automatic differentiation using standard libraries. The parameters X > O, > 0 control tail weights for the upper and lower tails (also often referred as right and left tails respectively). This allows us model asymmetry in tail behaviour. The parameters μ E R and a > 0 are location and scale parameters. We found these helped performance, although similar effects could be achieved in principle by adjusting Toody-
threshold and h(u)= > 0 is an appropriate scaling function. The theorem states that if the scaled excess converges in dis- tribution to a non-degenerate distribution, then it converges to Generalized Pareto distribution (GPD).
A common EVT approach is to fix some u. treat h(u) as constant, and model tails of distributions as having GPD densities. The motivation is that this should be a good tail approximation near to u, while for & >> there is not enough data to estimate the behaviour of h(w).
The GPD distribution involves a shape parameter, À E R. For X= O, the GPD density is asymptotically (for large
To perform density evaluation via
? we need to evaluate
4

---

Flexible Tails for Normalizing Flows
the inverse and derivative of These, and some other properties, are provided in Appendix B.3
lowest 5% of data, to produce tail parameters for the positive and negative tails respectively.
We do not consider similar two stage procedure for VI, as preliminary estimation of tail weights from the unnor- malised target distribution not straightforward. However recent work on static analysis of probabilistic programs (Liag et al. 2024 provides progress in this direction.
3.2. Theory
Appendix proves results on the asymptotic properties of (3). Informally, if X has Gaussian tails then R(X) is in the Fréchet domain of attraction (sce Definition 2.1 with \_ controlling the tail shape parameters. A special case is that (0, 1) tails produce GPD output with shape parameters \+.7. A_
3.5. Universality
It's been proved that some NFs have a universality property: "the flow can learn any target density to any required pre- cision given sufficient capacity and data" (Kobyzev et al.) 2020). In Appendix [ we show that many NF universality results are preserved when the TTF transformation is added as a final layer.
So composing R. with existing NFs permits the output to have heavy tails with parameterised weights. This is shown by the following argument. Most NFs use a Gaussian base distribution and a Lipschitz transformation. By Theorem 1.2 the output has light tails. Since the NF transformation can be the identity, it is capable producing Gaussian tails.
As we' 've already seen, the situation under bounded capacity is different. Standard NFs cannot produce heavy tailed distributions (Jaini et al. [020]] reviewed as Theorem [.2 above). However adding our transformation does permit these (see Section
3.3. Multivariate Transformation
To extend our univariate transformation R R + R to the multivariate case, we simply transform each marginal with its own μ. o, A- parameters. In some cases we know particular marginals are light tailed. Then we could simply perform an identity transformation instead. However, we find fixing A+ x_ to low values (we use 1/1000) suffices.
So theoretically our method improves the set of distributions which can be modelled under bounded capacity without sac- rificing expressiveness in the limit of infinite capacity. The next section shows this is reflected by improved empirical performance modelling heavy tailed data.
A more flexible approach would be to allow dependence, for instance by using an autoregressive structure (Papamakarios et al. 1 to generate the Ā- parameters for each marginal. This could capture tail behaviour that varies in different parts of the distribution. However, exploratory work found that this approach is harder optimise, so we leave it for future research.
4. Experiments
This section contains our experiments. Firstly, recall that one motivation for our work in Section is the claim that neural network optimisation can perform poorly under heavy tailed inputs. We verify this empirically in Section [4. for NFs. This is also verified in Appendix [ for a simple neural network regression example. Secondly, Sections 4.1 4.3 contain normalizing flow examples, comparing our method to existing approaches for density estimation and a proof-of-concept variational inference example. Additional implementation details are provided in Appendix G
3.4. Two Stage Procedure
Joint optimisation of R and Thody can require careful initial- isation of the À parameters. (Details of how we do are provided in Appendix G.3) As an alternative, we propose two stage procedure for density estimation: TTFfix. Here the tail weight A_, A parameters of R are estimated in an initial step and then fixed while optimising Thody (and the H, a parameters of R.) This can be viewed as first trans- forming the data using R. to remove heavy tails, and then fitting standard normalizing flow to the transformed data. Similar approaches appear in McDonald et al. (2022) and
4.1. Density Estimation with Synthetic Data
This experiment looks at density estimation for data gener- ated from the following model, with d >
( ~ lu, Xa|Xa-1 ~ N(.a-1,
(4)
Laszkiewicz et al. 2022).
In this model the only non-trivial dependency to learn is between Xa-1 and X2. but there are also several heavy tailed nuisance variables.
Shape parameter estimators exist in the EVT literature, which we can apply to each marginal tail. We follow Laszkiewicz et al. (2022) in using the Hill double-bootstrap estimator (Danielsson al. 2001 Qi 2008). Note that this enforces x_ = X4. Alternatively McDonald et al. (2022) perform maximum likelihood estimation on the highest and
We use this example to investigate whether modelling tails in the final layer of a NF is superior to modelling the tails in the base distribution. Of particular interest is how performance varies with dimensionality and tail weight v. Appendix
5

---

Flexible Tails for Normalizing Flows
Fiswws that neural network regression can perform poorly under heavy tailed inputs. Here we investigate whether similar finding holds in the setting of NFs.
Table I: Density estimation results on synthetic example for d = 50. Each entry is a mean value of negative test log likelihood per dimension across 10 repeated experiments, with the standard error in brackets. Bold indicates methods whose mean log likelihood differs from the best mean by less than 2 standard errors (of the best mean). A dash indi- cates potential unstable optimisation (at least one repeat had a final loss above le5).
Some NF methods we test involve fixed tail weight parame- ters (either v for a Student's T base distribution or X+, A- for our transformation), and for this experiment we fix these to their known true values. This means our analysis is not confounded by the difficulty of estimating these parameters. The tail weights are known since the marginal density for each X; can be shown to be asymptoticllly proportional to x; e-
Flow normal m. nororall
== 0.5
== 1
V - 2
2.02 (0.01)
Note that only X+, A_ in 3 are fixed, not jt and a.
2.02 (0.00)
g_normal
2.01 (0.00)
Flow Architectures We investigate a selection of NF methods, including several which aim to address extremes. To conduct a fair comparison, we maintain as much consis- tency between the flow architectures as possible.
gTAF
7.49 (0.38)
2.65 (0.01)
1.99 (0.00)
TTF mTAF
3.68 (0.00)
2.54 (0.00)
1.98 (0.00)
5.22 (0.04)
2.62 (0.01)
1.98 (0.00)
TTFfix
3.68 (0.00)
2.54 (0.00)
1.98 (0.00)
COMET
3.74 (0.00)
2.55 (0.00)
1.97 (0.00)
A baseline method, normal, uses d-dimensional isotropic Gaussian base distribution. This is followed by an autore- gressive rational quadratic spline (RQS) layer then an au- toregressive affine layer. The latter should be capable of capturing linear dependency, while the former can adjust the shape of the body, but not the tails of the distribution.
for brevity. Its results appear in Appendix J.I
Experimental Details We run 10 repeats for each flow/target combination. Each repeat samples new set of data, with 5000 observations. which is split in proportion 40/20/40, to give training, validation and test sets respec- tively. We train using the Adam optimiser with a learning rate of 5e-3. We use an early stopping procedure, stopping once there has been improvement in validation loss in 100 epochs, and returning the model from the epoch with best validation loss. Optimisation loss plots were also visu- ally inspected to confirm convergence. The selected model was then evaluated on the test set to give a negative test log likelihood per dimension.
Our proposed approach, tail transform flow, modifies the architecture just described by adding an additional layer for dealing with the tails, as described in Section 3 TTF trains the tail parameters alongside the other parameters. TTFfix is a 2-stage approach, which fixes the tail parameters to the known true values.
Marginal tail adaptive flows (mTAF) have the same architec- ture as normal, but use Student's T base distribution, as detailed in Section 1.1 The degrees of freedom are fixed to the correct tail parameters. Generalised tail adaptive flows (gTAF) differ in that the degrees of freedom are optimised alongside all other parameters during the training procedure.
Results Table D shows a selection of results with d = 50. See Appendix J.I for other d and v values. The meth- ods not specifically designed to permit GPD tails (normal, m. m.normal, g normall are the worst performing, often not converging at all for / S 1. However, the difference be- tween methods is small for 2, and all methods are simi- lar for 30 (near-Gaussian tails-results in appendix).
As further variations on normal, we consider two alter- native base distributions suggested by Amiri et al. (2022) Gaussian mixture (m normall) and Generalised Normal (g. (g.normal). More details are provided in Appendix G
We also consider COMET flows (COMET), as detailed in Section 1.1 For the normalizing flow part of this method we use the same architecture as normal. Our analysis is based on the code McDnald et al. (2022) with some im- provements to implementation details (needed to run more complicated examples later). COMET is a 2-stage approach, which involves estimating tail parameters in the first stage, so again we fix these to the known correct values.
For particularly heavy tails (v < 1), the best performing methods are TTF, TTFfix and COMET: the approaches which model tails in the final transformation. We did not detect a significant difference between the two TTF meth- ods, but they both outperform COMET. A similar pattern present for other choices of d (results in appendix).
It is interesting that COMET performs competitively here, despite not inducing Fréchet tails. A possible reason is that COMET can produce log-normat tails (see Appendix C.2 and these have similar sub-as aub-asymptotic properties to GPD tails (Nair et al. 2022).
We also tested variant suggested by a reviewer, TTF tBase. This combines TTF with Student's 's T base distribution with trainable degrees of freedom. It performed worse than both TTF and TTFfix, so is omitted from the main paper results
6

---

Flexible Tails for Normalizing Flows
Another question is whether it's better to fix the tail parame- ters or optimise them. Shows that mTTAF outperforms gTAF, so fixing tail parameters is better for heavy tailed base distribution methods. However there is no sigmificant dif- ference between TTF methods. A further investigation of the results suggested by a reviewer does suggest possible advantages of TTF which might become more significant in other settings. See Appendix J.2 for details.
cases, especially the most complicated dataset, CLIMDEX.
Also, estimating tail parameters for each marginal in- volves extra compute costs, especially for high dimensional datasets such as CLIMDEX, where the cost was comparable to fitting the normalizing flow (although easy to parallelise).
4.3. Variational Inference for Artificial Target
As proof-of-concepy investigation of variational inference, we perform experiments using 44) as a target distribution. Recall that in Section 44. we used this to generate synthetic data for density estimation. This model allows us to easily vary tail weights (v) and number of nuisance variables (d).
4.2. Density Estimation with Real Data
This section investigates density estimation for several real datasets with extreme values, covering insurance, financial and weather applications. Three taken from Liang al. (2022); Laszkiewicz et al. (2022) and one is novel to this pa- per. Appendix has more information about the datasets and standard preprocessing applied before density estimation.
Flow Architectures We compare four flows which exhib- ited good performance in Section TTF, TTFlix, mTAF and gTAF. While COMET flows also performed well in the density estimation setting, they involve making kemnel density estimates of marginal distributions, which has no obvious equivalent for the VI setting. So we do not include them in this comparison. As in Section 4.1 for TTFfix and mTAF we fix the tail parameters to the known correct values.
In this experiment we compare NF architectures which performed reasonably well in Section TTF, TTFlix, mTAF, gTAF and COMET.
Flow Architectures
4.1
For most examples we reuse the NF architectures described in Section 4.1 with a slight alteration. We add trainable linear layers based the LU factorisation (Oliva et al.] 2018). For TTF methods, this immediately precedes the TTF layer. Otherwise, this is the final layer. We found adding LU layers greatly improved empirical performance.
Although mTAF and gTAF were proposed as density esti- mation methods, it is straightforward to use these heavy tail base distribution methods for this VI task. As noted ear- lier, gTAF [Lazzkiewizz et al. 2022) is equivalent to ATAF (Liang et al. 2022) in the context of variational inference, and trains the tail parameters alongside the other parameters. It is difficult to apply mTAF to VI in general, due to the difficulty of estimating the tail parameters, but in this case we have theoretical correct values.
For the most complex dataset (CLIMDEX) we use a deeper architecture, as used in Laszkiewicz al. (2022), to permit fair comparison with their results. This has 5 RQS layers, al- ternated with LU layers. We use this architecture for nTAFF and g gTAF, and modify it for TTF, TTFli and COMET in the same way as described in Section 4.1
Experimental Details We take d E {5, 10, 50) and v e (0.5, 1, 2,30 (heavier than Cauchy; Cauchy; lighter than Cauchy: close to Gaussian). We run 5 repeats for each flow/target combination. Each repeat uses 10,000 iterations of the Adam optimiser with learning rate 1e-3. Optimisation loss plots were visually inspected to confirm convergence.
Experimental Details For all examples we run 10 repeats for each flow/target combination. For most examples we train for 400 epochs using the Adam optimiser with a learn- ing rate of 5e-4. The exception is CLIMDEX, where we follow the more complex training setup of Laszkiewicz et al. (2022) (e.g. cosine annealing, more optimisation steps) to permit fair comparison with their results.
Diagnosties We measure the accuracy of our fitted vari- ational density using two diagnostics based on importance sampling and described in Appendix H ESS, and k.
The results are presented in Table 2. TTF outper- forms previous NF methods for density estimation: it is the best performing method for 3 examples, and second best to TTFfix for the other. The performance of TTFfix is more variable: it the best performing in one example, second best to TTF in two examples, but the worst performing in the remaining example.
Results
Results Table reports our results. TTFfix has best ESS. for heavier tails (v < 2). TTF produces slightly worse values, with mTAF and gTAF worse than either TTF method, especially for the I 1, d 50 cases (ESS, < 0.1). For v = 30 the best results are for mTAF, but all methods do well (ESS, > 0.7). All methods aside from gTAF achieve useful approximations (f below 0.7), in every setting where v N 1. In the most challenging setting- - = 0.5, d = 50- both TTF method achieve good le values.
Overall these results show TTF methods provide an improve- ment in density estimation performance, and that the addi- tional freedom of TTF to learn tail behaviour compared to TTFáx (e.g. it can learn tail asymmetry) is beneficial in some
7

---

Flexible Tails for Normalizing Flows
Table 2: Real data density estimation results. Each entry is a mean test negative log likelihood over 10 trials, with standard deviation reported in brackets. Values marked with are those reported by Laszkiewicz et al. (022). Results within one standard deviation of the best mean are highlighted in bold.
Model
Insurance
Fama 5
S&P 500
CLIMDEX
normal
1.41 (0.03)
4.65 (0.01)
334.01 (1.02)
-2101.91 (9.44)*
gTAF
1.41 (0.03)
4.68 (0.01)
321.81 (0.55)
-2113.48 (7.93)*
TTF
1.37 (0.02)
4.61 (0.01)
317.56 (0.56)
-2214.28 (13.25)
mTAF
1.52 (0.03)
4.90 (0.03)
322.98 (0.33)
-2121.38 (10.91)*
TTFfi
1.38 (0.01)
4.63 (0.01)
314.84 (0.46)
-2090.91 (11.90)
COMET
1.41 (0.03)
4.63 (0.01)
324.38 (0.58)
-2118.60 (8.61)
Overall the results show that, as before, is advantageous to model tails in the final transformation (TTF, TTFfxx.. Fixing tail parameters improves performance of TTFfix over TTF, perhaps because optimisation is more stable in this case.
more extreme value theory properties. For instance, our method doesn't 't explicitly allow tail dependence (Coles et al. 1999). This would be an interesting future direction e.g. by adapting the manifold copula method of (McDonald et al. 2022). Also, it would be interesting to design multivariate transformations with the max-stable property (Coles) 2001).
5. Conclusion
We found that fixing the tail parameters to known values performed well in our VI example on an artificial target. However in real VI applications, the tail parameters aren't t known, motivating methods to estimate them e.g. using static analysis of probabilistic programs (Liang et al.) 2024). Also, in exploratory work on real VI applications, we found it difficult improve VI results using our method. A possi- ble reason is that our experiments only show major improve- ments for especially heavy tails (v S 1 in Table which we found to be uncommon for real posteriors.
Most current methods for modelling extremes with NFs use heavy tailed base distributions. We demonstrate this can perform poorly, due to extreme inputs to neural networks causing slow convergence. We present an alternative: use a Gaussian base distribution and a final transformation which can induce heavy tails. We propose the TTF transformation, equation 3,, and prove it can indeed produce heavy tails. For density estimation, our TTF methods outperform exist- ing methods on real and synthetic data. Jointly learning the tail parameters and the rest of the flow is usually the best approach. However for variational inference, fixing the tail parameters is beneficial, perhaps by improving optimisation stability. Our approaches do not damage performance in examples without heavy tails here all types of NF we considered perform well.
A potential future application is simulation based inference. Here neural likelihood estimation methods (Papamakarios et al. 2019) use NF density estimation to estimate like- lihood function from simulated data. Our approach could improve results when the simulated data is heavy tailed.
Finally. it would be interesting to use our tail transforma- tion with other generative models. Here we briefly discuss differential equation based methods c.g. continuous normal- izing flows, diffusion models, flow matching (Lipman et al., 2023 These sample z0) from a base distribution such as N (0, T). apply differential equation
5.1. Limitations and Future Work
An arguable limitation of our transformation is that it al- ways converts Gaussian tails to heavy tails. Hence it cannot produce exactly Gaussian tails. It is not clear whether this is detrimental in practice, and it does not prevent good per- formance in all our experiments.
uf2 t) (or SDE also including diffusion term), and output x().. The vector field u is neural network, trained to produce a desired output distribution. Our work suggests it would be difficult to train a vector field for heavy tailed targets. To see this, suppose we desire output x x(1) = x². By continuity, we need z(1 - 6) x² for small 8. So we must evaluate u for input similar x², which can be extreme for a heavy tailed target. We' 'vee argued neural networks are hard to train with extreme inputs. This motivates using our tail transform as a final transformation to x1)), effectively lightening the tails of the target, avoiding extreme x* values. Either joint training or a sta-s approach could be tried, similar to TTF and TFFfx.
Another potential limitation is that the TTF transformation affects both the body and tail of the output distribution. The preceding layers of the flow must learn to model the body of the distribution, and also adapt to the final layer. It would be appealing to decouple the transformations, so that they can concentrate separately on the body and tail, which could make optimisation easier. This could be achieved by somehow ensuring the final transformation is approximately the identity in the body region.
Possible future work is to design methods which incorporate
8

---

Flexible Tails for Normalizing Flows
Table 3: Variational inference results for artificial target. Each entry is a mean value across repeated experiments. For ESS, columns, bold indicates the method with best value for each For & columns, bold indicates any value below 0.7.
d Flow
V/ = 0.5
1/
1
= 2
V/ 30 k
E
1
ESS.
k
ESSc
k
E.SS.
gTAF
0.47
0.89
0.53
0.93
0.90
0.79
0.97
0.53
TTF
0.59
0.67
0.83
0.40
0.89
0.43
0.98
0.18
5
MTAF
0.00
2.25
0.10
0.18
0.52
-0.18
0.99
0.31
TTF6X
0.62
0.82
0.97
0.37
0.98
0.35
0.98
0.19
gTAF
0.19
0.64
0.28
1.05
0.74
0.89
0.96
0.28
TTF
0.40
0.74
0.90
0.36
0.92
0.36
0,96
0.20
10
mTAF
0.00
3.13
0.09
0.31
0.52
-0.37
0.98
0.25
TTFNiX
0.45
0.88
0.93
0.33
0.95
0.32
0.97
0.07
gTAF
0.02
0.90
0.08
0.82
0.39
0.83
0.84
0.13
TTF
0.39
0.51
0.57
0.25
0.61
0.24
0.79
0.14
50
TTAF
0.00
7.92
0.02
0.57
0.41
0.03
0.90
0.17
TTFli
0.43
0.53
0.75
0.18
0.81
0.18
0.85
0.08
Impact Statement
Values. Springer London, London, 2001. ISBN 978-1- 84996-874-4.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Coles, S., Heffernan, J., and Tawn, J. Dependence measures for extreme value analyses. Extremes, 2:339-365, 1999.
Danielsson, J., Haan, L., Peng, L., and Vries, C. G. Using bootstrap method choose the sample fraction in tail index estimation. Journal of Multivariate analysis, 76(2):226-248, 2001.
Acknowledgements Thanks to Miguel de Carvalho, Seth Flaxman, lain Murray, Ioannis Papastathopoulos, Scott Sis- son, Jenny Wadsworth, Peng Zhong and anonymous review- for helpful discussions. Tennessee Hickling is supported by PhD studentship from the EPSRC Centre for Doc- toral Training in Computational Statistics and Data Science (COMPASS). A preliminary version this work appeared as Hickling & Prangle (023).
de Carvalho, M., Pereira, S., Pereira, P., and Zea Bermudez, P. An extreme value Bayesian lasso the conditional left and right tails. Journal of Agriculural, Biological and Environmental Statistics, pp. 1-18, 2022.
References
Draxler, F., Wahl, S., Schnoerr, C., and Koethe, U. On the universality of volume-preserving preserveeeergggge and coupling- coupling-based based normalizing flows. In International Conference on Ma- chine Learning, volume 235, pp. 11613-11641. PMLR, 2024.
Abiri, N. and Ohlsson, M. Variational auto-encoders with Student's t-prior. In ESANN 2019 Proceedings, pp. 837-848. ESANN, 2019.
Amiri, S., Nalisnick, E. T., Belloum, A., Klous, S., and Gommans, L. Generating heavy-tailed synthetic data with normalizing flows. In The 5th Workshop Tractable Probabilistic Modeling, 2022.
Durkan, C., Bekasov, A., Murray, I., and Papamakarios, G. Neural spline flows. In Advances in Neural Information Processing Systems, volume 32, 2019.
Blei, D. M., Kucukelbir, A., and McAuliffe, J. D. Varia- tional inference: A review for statisticians. Journal of the American Statistical Association, 11((518)859-8877 2017.
Durkan, C., Bekasov, A., Murray, I., and Papamakarios, G. nflows: normalizing flows in PyTorch, November 2020. URLhttps://gg.thu... wtt.:///.....mm..aaiiiaia.aaa nflows
Cirillo, and Taleb, N. N. Tail risk contagious diseases. Nature Physics, 16:606 613, 2020.
Embrechts, P., Klüppelberg, C., and Mikosch, T. Modelling extremal events: for insurance and finance. Springer Science & Business Media, 2013.
Coles, An Introduction Statistical Modeling of Extreme

---

Flexible Tails for Normalizing Flows
Foss, S., Korshunov, D., and Zachary, S. An introduction to heavy-tailed and subexponential distributions. Springer, 2011.
Lipman, Y., Chen, R. T. Q.. Ben-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling. In The Eleventh International Conference Learning Repre- sentations, 2023.
Fung, T. and Seneta, E. Quantile function expansion using regularly varying functions. Methodology and Computing in Applied Probability, 20: 10:1091-1103, 2018.
McDonald, A., Tan, P.-N,, and Luo, COMET flows:
Towards generative modeling of multivariate extremes and tail dependence. In International Joint Conference on Artificial Intelligence, 2022.
Gilli, M. and Këllezi, E. An application of extreme value theory for measuring financial risk. Computational Eco- nomics, 27:207-228, 2006.
Murphy, K. P. Probabilistic Machine Learning: Advanced Topics. MIT Press, 2023.
Hickling, T. and Prangle, D. Flexible tails for normalising flows, with application to the modelling of financial return data. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 374-386. Springer, 2023.
Nair, J., Wierman, A., and Zwart, B. The fundamentals of heavy tails: Properties, emergence, and estimation. Cambridge University Press, 2022.
Naveau, P., Huser, R., Ribereau, P., and Hannart, A. Model- ing jointly low, moderate, and heavy rainfall intensities without a threshold selection. Water Resources Research, 52(4):2753-2769, apr 2016.
Hill, B. M. A simple general approach inference about the tail of a distribution. The Annuls of Statistics, 3(5), September 1975.
Huang, C.-W., Krueger, D., Lacoste, A., and Courville, A. Neural autoregressive flows. In International Conference Machine Learning, pp.
Oliva, J., Dubey, A., Zaheer, M., Poczos, B., Salakhutdinov, R., Xing, E., and Schneider, J. Transformation autoregres- sive networks. In International Conference on Machine Learning, volume 80, pp. 3898-3907. PMLR, 2018.
PMLR, 2018.
Huser, R., Opitz, T., and Wadsworth, J. L. Modeling of spatial extremes in environmental data science: time to move away from max- max-stable processes. Environmental Data Science, 4:e3, 2025.
Papamakarios, G., Pavlakou, T., and Murray, I. Masked autoregressive flow for density estimation. In Advances in Neural Information Processing Systems, volume 30, 2017.
Jaini, P., Selby, K. A., and Yu, Y. Sum-of-squares poly- nomial flow. In International Conference on Machine Learning, pp. 3009-3018. PMLR, 2019.
Papamakarios, G., Sterratt, D., and Murray, I. Sequen- tial neural likelihood: Fast likelihood-fres inference with autoregressive flows. In The 22nd International Confer- ence on Artificial Intelligence and Statistics, pp. 837-848. PMLR, 2019.
Jaini, P., Kobyzev, I., Yu, Y., and Brubaker, M. Tails of Lipschitz triangular flows. In Proceedings of the 37th International Conference on Machine Learning, volume 119, pp. 4673-4688 PMLR, 2020.
Papamakarios, G., Nalisnick, E., Rezende, J., Mohamed, S., and Lakshminarayanan, B. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 22(57):1 1-4, 2021.
Kobyzev, I, Prince, S. J. D., and Brubaker, M. A. Nor- malizing flows: An introduction and review of current methods. IEEE transactions puttern analysis and machine intelligence, 43((1))3394
2020.
Papastathopoulos, I. and Tawn, J. A. Extended generalised Pareto models for tail estimation. Journal of Statistical Planning and Inference, 143(1):131-143, 2013.
Laszkiewicz, M., Lederer, J., and Fischer, A. Marginal tail- adaptive normalizing flows. In International Conference Machine Learning, volume 162, pp. 12020-12048 PMLR, 2022.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, 7.,, Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, Pytorch: An imperative style, nigh-performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8024- 8035. Curran Associates, 2019.
Liang F. T., Hodgkinson, L., and Mahoney, M. W. Fat- tailed variational inference with anisotropic tail adaptive flows. In International Conference on Machine Learning, volume 162, pp. 13257-13270 PMLR, 2022.
Liang, T., Hodgkinson, L., and Mahoney, M. W. A heavy- tailed algebra for probabilistic programming. Advances in Neural Information Processing Systems, 36, 2024.
Pickands III, J. Statistical inference using extreme order statistics. The Annals of Statistics, pp. 119-131, 1975.
10

---

Flexible Tails for Normalizing Flows
Qi, Y. Bootstrap and empirical likelihood methods in tremes. Extremes, 11:81-97, 2008.
Rezende, D. and Mohamed, S. Variational inference with normalizing flows. In International conference on ma- chine learning, volume 37, pp. 1530-1538. PMLR, 2015.
Robert, C. and Casella, G. Monte Carlo statistical methods. Springer Verlag, 2004.
Shaw, W. T., Luu, T., and Brickman, N. Quantile mechanies ii: changes of variables in Monte Carlo methods and GPU-optimised normal quantiles. Eu- ropean Journal of Applied Mathematics, 25(2):177-
212, January 2014. ISSN 1469 4425. doi: 10.1017/ $0956792513000441,, URL http:// /dxx org 10. 00.1017/8095511251300111
Stein, M. L. Parametric models for distributions when inter- is in extremes with an application to daily temperature. Extremes, 24(2):293-323, 2021.
Temme, N. M. Error functions, Dawson's and Fresnel in- tegrals. In NIST handbook of mathematical functions.
Cambridge university press, 2010.
Teshima, T., Ishikawa, I., Tojo, K., Oono, K., Ikeda, M., and Sugiyama, M. Coupling-basee invertible neural networks are universal diffcomorphism approximators. Advances in Neural Information Processing Systems, 33:3362-3373, 2020.
Troshin, V. V. On the maximum domain of attraction for transformations of normal random variable. Journal of Mathematical Sciences, 262(4):537-543) 2022.
Van der Vaart, A. W. Asymptotic statistics. Cambridge university press, 2000.
Yang, J., Latuszyíski. K., and Roberts, G. O. Stereographic Markov chain Monte Carlo. The Annals of Statistics, 52 (6):2692-2713. 2024.
Yao, Y., Vehtari, A., Simpson, D., and Gelman, A. Yes, but did it work?: Evaluating variational inference. In Proceedings of the 35th International Conference on Ma- chine Learning, volume 80, pp. 5581-5590, PMLR, 2018.
Zhang, J., Karimireddy, S. P., Veit, A., Kim, S., Reddi, S., Kumar, S., and Sra, S. Why are adaptive methods good for attention models? In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp- 15383-15393, Curran Associates, Inc., 2020.
Zscheischler, J., Westra, S., van den Hurk, B., Seneviratne, S. I., Ward, P. J., Pitman, A. J., Aghakouchak, A., Bresch, D. N., Leonard, M., Wahl, T., and Zhang, X. Future climate risk from compound events. Nature Climate Change, 1469-477, 2018.

---

Flexible Tails for Normalizing Flows
A. Gaussian mixtures are light tailed
As suggested by a reviewer, here we prove that Gaussian mixtures are light tailed. Then, by Theorem 1.2 using a Gaussian mixture base distribution with Lipschitz NF produces light tailed output. Amiri et al. (2022) suggest this combination (reviewed in Section I.1.3), and we consider in our experiments under the name m_normal.
Suppose the real valued random variable Z has density
((2) Σ (;)((2)
where E Wi 1, we (0, 1) and di (=(=) is a Zi N (us of) density. The moment generating function of Za is
Mz, (A) = Elexp(AZ:)) exp/Axs + o22
Hence the moment generating function of is
M² (A) = Elexp(AZ)) =
Σ
w. M2 (2) =
Σ exp Apt 2222/22]
So (A do for all À > and Z light tailed under Definition
In this paper we concentrate univariate properties. However, note that for Z which is mixture of multivariate Gaussians, consider any univariate random variable of the form Z' = a Z (with aTa 0). Since Z' is clearly mixture of univariate Gaussians, is light tailed by the argument above.
B. TTF details
This appendix gives more details of our TTF transformation:
R(2; = μ + [erfo[I=//V2] d -1].
X
Recall that sign(z) and
A
for s = 1,
A
(5)
A_ for s = 1.
First, Section B.1 reviews some background on the complementary emror function. Then Section B.2 presents a motivation for the transformation, and Section B.3 discusses some its properties.
B.1. Complementary error function
For E IR, the error function and complementary error function are defined as
erf(2) 2 V / exp(-t))dt,
erfe(=) = erf(z).
For large 2, erfe(2) No 0, so it can be represented accurately in floating point arithmetic.
Note that
Fx(2) = (1 erf(=/v2)),
(6)
where (x) the A/ 1) cumulative distribution function. This implies
erfe (x) - (x/2)/V2.
(7)
Efficient numerical evaluation of erfe and its gradient is possible, as it is standard special function (Temme 2010 implemented in many computer packages. For instance PyTorch provides the torch. speciall also require erfe -1 which is less commonly implemented directly. However using ( we can compute erfe -1 using the more common standard normal quantile function Fy' For instance PyTorch implements this as torch. special nnt (although this can have problems for small inputs see Appendix G.5).
function. Later we
12

---

Flexible Tails for Normalizing Flows
B.2. Motivation
This section motivates the TTF transformation by sketching how can be derived from some simpler transformations. This gives informal derivation that R transforms (0, 1) tails to GPD tails. The formal version, Theorem C.9) is presented later and also appropriately generalises the result to [(,, o²) inputs.
GPD transform
Section 2.4 motivates the transformation P: :0 + R given by the GPD quantile function
P(u; A) - 111 1( -)
(8)
where À > is the shape parameter. Using transforms a distribution with support [0, 1] one on R+ with tunable tail weight.
We note that (8) is also valid for À 0 and maps [0, 1] to [0, Ł Here the image of the transformation depends on the parameter values, which is not a useful property for normalizing flow. Hence we do not consider negative À in our work.
Two tailed transform We can extend to a transformation Q [-1, R,
un|) ->x 1].
A
Here A > 0, A_ > 0 are shape parameters for the positive and negative tails. We use the notation s = (u), with A defined as (5)
Using Q transforms distribution with support [-1,1] to one on with tunable weights for both tails.
Real domain transform We would like a transformation R: R. - IK which can transform Gaussian tails to GPD tails. Consider E IR, and let u = (=) 1, where F'v is the (o,,, 1) cumulative distribution function. Then w E [-1, 1] and we can output Q(u). A drawback is that large |>| can give u values which are rounded to =1 numerically.
Using 66 shows that
1 - |u| erfc(1=//v2),
(10)
so there a standard special function to compute |u| directly. Large |=| gives 1 1- |u| 0, which can be represented to high accuracy in floating point arithmetic, avoiding catastrophic rounding which could result from working directly with Substituting
110 into
(9) and adding location and scale parameters results in our TTF transformation R.
B.3. Properties
Here we derive several properties of the TTF transformation, R. These include expressions which are useful in an automatic differentiation implementation of R for NFs: R-1 and derivatives of R and R
Forward transformation
The derivative of with respect to 2 given by
aR
(=: =o /2) orfcll//// -
(11)
dz
All of these factors are positive, which implies that for all parameter settings monotonically increasing.
an (z) > 0. so the transformation is dz
The transformation is continuously differentiable at 2 0 as
aR (0: (0: = o
2
(12)
dz
This has no dependence on the X4, A_ and the limit - 0 from above or below.
13

---

Flexible Tails for Normalizing Flows
Inverse transformation
Define y Xe|(2 p1/al +. Then the inverse transformation is
R R- (2: x,, )_) - 2v -1 "yy
(13)
Note here we can take sign(æ - 11). The gradient is
0R d
(,
1/2.-1 exp (eyfc-(y-11/,))) 2
(14)
C. Asymptotic results
This appendix proves asymptotic results on the tails produced by COMET flows and our TIF transformation. Our main aim to prove whether or the distributions are in the Fréchet domain of attraction e) (see Definition 2.1). We will also find the tail behaviour in detail for some special cases.
Throughout this appendix use the following asymptotic notation. In particular, is not used to describe probability distributions, it is elsewhere in the paper.
Definition C.1. For functions f : R + R+ and IR + RR+, we write
1. f(z) O(g(z)) lim sups f(x)/ < 00,
2. f(z) ~ g(x) 4o0 f=))/(()=11.
C.1. Background
The material in this section will be used in the proofs later in this appendix. The first results are on regularly varying functions, based on the presentation in Nair et al. (2022).
Definition C.2. A function f IR+ > IR is regularly varying of index p E IR if
lim [(ko)/{(t)= k2
$0000
for all & > 0. When p 0, fx) is slowly varying.
Theorem C.3 Nair et al. Lemma 2.7). For slowly varying fxx)
if < 0, 00
lim x f(z) =
if 0.
Theorem C.4
Nair et al.
Theorem 2.8). A function
:R+ + IR+ is regularly varying of index p if and only if
f(x) x 1²((x)
for some slowly varying L.
The following result is an immediate corollary of Theorem 7.5(i) of Nair et al.
Theorem C.5 (Nair et al., corrolary), Consider a real-valued random variable X with density g(x). Its distribution is in O if and only g regularly varying of index -1-
Finally, we present result of Liang et al. (022).
Definition C.6. Let E be the set of random variables X such that for some > 0
Pr([X] x) O(exp[=ax=))
Note that Eh includes Gaussian distributions, and does intersect with O for any >> 0.
The following result is special case of Theorem 3.2 of Liang et al. and is more precise extension of Theorem 1.2 Theorem C.7 LLangg et al. special case). E is closed under Lipschitz transformations.
Of interest is a Gaussian base distribution transformed by a Lipschitz transformation, such as most standard NFs, The result shows that the output in E2, and not the Fréchet domain of attraction.
14

---

Flexible Tails for Normalizing Flows
C.2. COMET flow tails
Recall that COMET flows (MDonall al. 2022). reviewed in Section T.I use a Gaussian base distribution followed by
generic normalizing flow. Then each component of the output is transformed by h 9. where is the logistic transformation and h is GPD quantile function. The following result describes the resulting tails.
Theorem C.8. Let Z be real random variable with density p(z). Define X g( where
g(z) 1 exp(-z)
h(z) = [[1 -1)-))= 2)- 1].
Let PN (2: o²) denote a (N(,, density. Then:
1. 7 E E (see Definition C.6 implies X & Ox (the Fréchet domain of attraction).
2. For p(z) ~ ppN(z;; (, X has density g(x)) (I, O, >²o²), a log-normal density with location zero and scale x²02.
Statement 1 implies that the tails produced by COMET flows are not suitably heavy tailed under most standard NFs. The argument is as follows. Recall that (Jaini et al., 2020 Liang et al. 2022) give details showing that most standard NFs are Lipschitz. Theorem C.7 shows that applying a Lipschitz transformation to a Gaussian base distribution produces a distribution in E² Then statement shows that COMET flows do not map this to the Fréchet domain of attraction, which the desired property by Definition 2.1
Statement 2 shows that in one particular case, the resulting tails log-normal. While the log-norma distribution is heavy tailed under Definition 1.] it is not in the Fréchet domain of attraction (see e.g. Embrechts et al. 2013 Example 3.3.31). Hence it cannot capture the full range of asymptotic tail behaviours. For instance, all moments exist for a log normal distribution but not for a GPD.
Proof. Let x = h(v), u = g(z). It's straightforward to derive that
2 x log
(15)
exp(A=)/A g((=) ~ (Ax) 11/2
(16)
(17)
h'(v) (Az)) 111A
(18)
The change of variables theorem gives
p(=)
p(2)
((x) 1 (()))
(19)
Ax
Statement 1
We'll prove the contrapositive:
X E O implies Z E
By Theorems
C.4 and C.5
X Ox implies g(x) = x ->-1(((x) with X and slowly varying &. Thus
p(z)
qxx))x
from 19
=x-> ---))
expression for g(x))
exp( (xxp(~A[++ 11))1~+22(((1)
from (16
Let 8(2) = A(A 1) exp(- + 1]z). Then
p(2) Drl(x)),
s(2)
with D - X /(A+1) 1). By Theorem C.3 this ratio converges to
follows that there exists zn such that zn implies p(z) > s(z). Thus for = > zp:
Pr(Z 2) p(t)dt s(t)dt exp(-A[A 1]z).
This contradicts Pr([Z) > 2) = O(expl 0z 2l2 for some
0, so Z & E
15

---

Flexible Tails for Normalizing Flows
Statement 2
From
(9))
19
pN //x.
Ax. Substinuting in the Gaussian density and 15) gives
g(z) ~ r r(z) Ax expl- x;)].
where A required.
1e 1/ expl- 20
B
and C' 2x 0. For μ 0, r(z:) is log log-normal LN (2o2) density, as
Remark The proof of statement suggests that Z with exponential distribution would produce output in the Fréchet domain of attraction. This motivates using Laplace distribution a base distribution for a COMET flow. A drawback is the lack of smoothness of the Laplace density at the origin.
C.3. TTF tails
Our transformation R is defined in 3 Here consider simplified version S R +R+
S(x; X) = [erfc(2/V2)-
(20)
This modifies R by setting the location parameter to zero and scale parameter to one, and only considering the upper tail. To simplify notation uses in place of X+. We provide the following result on tails produced by this transformation.
Theorem C.9. Let be a rundom variable with density p(z) ~ (N(=;, N(u, o²) density. Define X -S( A). Then:
1. The distribution of X is in the Fréchet domain of attraction with shape parameter Xo
2. For === 00, 1, X has density q(x) (,rrp(;;A, GPD density with shape parameter X and scale 23/(2+1/A) (see Definition C.10 below).
The result for the lower tail is a simple corollary. Also, the result is unaffected by adding final location and scale transformation-as in our full TTF transformation R-except that the latter will modify the scale parameter in statement 2.
As argued in Section 3.1 most NFs standard normal base distribution, and capable of producing the identity transformation. So statement 2 of Theorem C.9 shows that composing them with R can produce output which has heavy tails with parameterised weights. Statement provides robustness: the result remains true when the input to R is Gaussian with arbitrary location and scale parameters.
C. LEMMAS
Here present lemmas which are used in the proof of Theorem C.9 Definition C.10. GPD with shape 0, location O, and scale o > has density
(oPD(a; = (1 Xx/o)-
Lemma C.11. For l: >> O,
kgGPD(I; ~ A
The left hand side GPD density with scale 1, multiplied by constant The lemma shows this is asymptotically equivalent GPD density with unchanged shape but modified scale.
Proof. Observe that
(G 2-11/ --) -Xx1 = 1))
2CGPD(E; A. k 1-11211/111) 1/(++))))
16

---

Flexible Tails for Normalizing Flows
Lemma C.12. Suppose x S(z; A). Then for large 2;
1 log Ar + 1)) + o(1),
(21)
2² log(Av
log log(Ax 1) log N/4 + (1).
(22)
Proof. Let (-)() be the (O(, 1) edf. The result is corollary of the following asymptotic expansion from (Fung & Seneta 2018 for p +
log log po
)
(x) (p) log2 log logp log 2x 2x/1//2 1/2
1 + 0
(log p)²
By definition, x [lerl(=z/V2) -11 So = - (2rrcc-(()) where - (1 + A 1/2, Using gives 2 -F (p/2). (w/2)
11.
Hence we get
2 = 1 log(Ar (Ax + +)]1 + log log Au + 1) + log x + o11) 1/2
( (1+0)
log log(Ar + 1) [log(Az Ar + 1)]2
1
$ log(Ar
Expanding the middle factor using Taylor series and checking the order of the remaining terms gives 21 Squaring the asymptotic expansion for and checking the order of terms gives (22)
Remark A consequence of (22 which we will use later is:
2² = 7(()) - log v1(x) + o(); where 7)x)) = 3 log(Ao - 1) + log 3-
(23)
C.3.2. PROOF OF THEOREM C.9
Let (2; A). The change of variables theorem gives:
((x) ====
Recall that
1
plz) pN(2;
exp 2 (2 1,)
1/20
and from
2 S'(2;A) = -v exp(-22 22 22/2) crfc(2/V2)
Hence
1
((x) ~
2 (r)
where
re;; exp -2 2z 11/2 - =
Rearranging
20 gives (1 Ar) Ar) 1/2 erfc(2/ 22) So
((x) (=; + Xx) -1-11A)
(24)
Note that 02 = Then the proof of statement 2 concludes by applying Lemma C.11 to 224
The remainder of the proof is for statement 1, and uses capital letters to represent constants with respect to r. We have
logr(zip, 1 o 22 + Az + B.
Using Lemma C.12 gives
gives
logr(2;11,0) = 111 -2) log(Ax +) + Cllog(Ar + 1))1/2 D log(o +1)+ + E + o(1).
17

---

Flexible Tails for Normalizing Flows
So
r(2;14)) F(Ax 1)(1-1 31/2 exp {CologAdr + 1))42) [oo(Adr + 1)) + o(1))
Substituting into 24 gives
1-> q(z) ~ 2 21 (1)) -1-2 to 2 F Fexp (CllogAAr + +))1/² [lo(AA + +1)
((2) -11 0- ((x),
where
(x) G exp {Cllog(Ax: 1))1/} [log(Az + 11) + o(1)
can easily be checked be slowly varying function. Applying Theorems C.4 and C.5 domain of attraction with shape Ao² required.
shows that Y is the Fréchet
D. TTF Transform: Variations and Related Work
This appendix discusses variations on our TTF transform R, 33 and related work in the literature. For simplicity we compare to R with location zero and scale one.
As well as those discussed below, we expect many other variations are possible with equivalent asymptotic properties, providing much scope for potential future study.
Student's T' CDF Transform A straightforward alternative transformation which converts Gaussian tails to heavy tails
Raar F.T FN
where Fr and FN are edfs of Student's T and Gaussian distributions (both with location and scale onc). Rather than use the Student's 7 distribution for the base distribution, in the prior work reviewed in Section 1.1.2 this moves the use of the 7' distribution to the final transformation. Unfortunately, F-- does not have closed form allowing automatic differentiation, so it lacks the flexibilty of TTF. In particular we cannot learn the tail weight parameters. However, Rear could be applied a two step procedure for density estimation similar to TTFfix (see Section 3.4.. Exploratory work suggests this has comparable performance TTFfx.
Unit Scale Transformation Another alternative transformation
{ erfo((=//V2)) } - Rate )+, Ag
This only differs from our TTF transform in that factor of 1/2 has been included. Equation 3.15 Shaw et al. 2014 shows that for oc, Rat is asymptotically equivalent Rear- Hence Rah converts (O, 1) tails to GPD tails. In this case it has the advantage of producing output with scale one, which the TTF transformation does not do (see Theorem C.9 statement 2). However, the reason we don't 't use Rai is due to disadvantage -theee can be a discontinuity at 2 = 0, since
lim Ran X )_) = (2 -1//A+>
lim Ran --) (22- 11//2 -0
Another related result from (Troshin 2022). This paper investigates necessary and sufficient conditions for transformations to map (0, 1) inputs to the Fréchet domain of attraction. This produces family of functions which includes R and Rai see Troshin's s Remark 2. However, unlike our Appendix C this work does not provide results for N(,, o²) inputs.
E. Universality
Here we prove some universality propertics relating to the TTF transformation, as referred to in Section 3.5 which also includes broader discussion on universality.
18

---

Flexible Tails for Normalizing Flows
E.1. Background
We define universality for our purposes in Sections
E.2 as follows.
Definition E.1. Let S be set of transformations s R - R Let Z ~ (O, I) be the usual base distribution. We call S universal if for every random variable X on I2 there is a sequence Sn E S such that sn (Z) + XX in the sense of weak convergence as n - 00.
Universality has been proved for some NFs including SoS polynomial flows and Neural Autoregressive Flows (Huang et al. 2018 Jaini et al. 2019; Kobyzev et al. 2020).
For all these results the set of transformations S comprises NFs of particular type with unbounded capacity. The sequence 8n typically requires increasing capacity as n + oo. So Definition E. means that an idealised sequence of NFs can approximate any target distribution. In practice, capacity is limited and only an imperfect approximation is usually achievable.
For many other common NFs it is unknown whether universality holds: see Jaini al. 2019 Table but note progress has been made on coupling flows, discussed in Section BK
E.2. TTF preserves universality
Let S be the set of transformations corresponding universal NF. Let R. be the set of TTF transformations under all valid parameter choices. Let S {r o R, E S}. We now prove that universality of S implies universality of S. Note that R does not contain the identity, so the result is not immediate.
Fix any F E R. and some target X. Since S is universal, it contains a sequence {8p} such that 8n, (Z) (X). By the continuous mapping theorem (sec e.g. Van der Vaart 2000 Theorem 2.3), the continuity of implics Fo s2) + X, as required.
E.3. Coupling flows
Draxler et al. (2024) investigate universality of coupling flows. They replace Definition E.I by an alternative convergence metric specialised to the problem. Given a target random variable X and a normalising flow s acting on base random variable ~ (0, I), it considers the closeness of -"(X to (0, I) (in terms of the improvement possible by using t where i is single coupling flow layer). See Definition 5.1 of Draxler et al. (2024) for full details.
Their main result- - Theorem 5.4 of Draxler et al. 2024 applies to any X with full support, continuous density and finite first and second moments. They show there exists a sequence of coupling flows {s,} achieving a convergence metric tending to zero
The assumption that X has finite first and second moments excludes sufficiently heavy tailed distributions from this result. This improves on earlier work such as Teshima al. (020)) which looks weak convergence bounded subset of IR and does not consider tails at all.
It is plausible that our TTF transformation can preserve the result of Draxler et al.) (022)) and extend it to heavier tailed targets. Given some target X, the idea is to select i e R such that X' = 7- --((X) ) has finite first and second moments. That is, F-1 acts to lighten tails of X which are too heavy. Take the sequence } from applying the result of Draxler et al. (2024) to X'. Then for target X the sequence {f s,) produces a convergence metric tending to zero
However further results are needed to prove when a suitable exists. We leave this for future work.
F. Neural Network Regression with Extreme Inputs
In Section we claimed that neural network optimisation can perform poorly under heavy tailed input. This forms part of the motivation for proposing our methods. In Section 4.1 we illustrated this claim empirically for an example involving NFs. Here we provide further support in a simpler supervised learning example.
Our experiment considers the following simple regression problem
(x,) ~ tw, Y Y~ N~~aX 1).
19

---

Flexible Tails for Normalizing Flows
Table 4: Neural network regression example results. Values are the median test MSE over 5 trials, with the maximum provided in brackets. (We use median as it's more robust than mean to extreme outliers. Maximum illustrates the presence and magnitude of such outliers.)
d
V
Sigmoid activation
ReLU activation
30.0
1.01 (1.02)
1.01 (1.04)
5
2.0
1.74 (6.19)
1.06 (1.08)
1.0
1.57e+03 (8.57e+06)
3.45 (1.93e+03)
30.0
1.02 (1.02)
1.02 (1.05)
10
2.0
2.04 (7.28)
1.03 (1.05)
1.0
4.13e+03 (1.09e+06)
8.41 (49.6)
30.0
1.01 (1.04)
1.08 (1.09)
50
2.0
3.09 (5.26)
1.27 (12.3)
1.0
1.03e+04 (1.73e+06)
55.8 (6.59c+03)
30.0
1.02 (1.07)
1.15 (1.2)
100
2.0
3.41 (4.33)
1.60 (1.74)
1.0
1.02c+05 (1.94c+05)
336 (1.2c+03)
The d-dimensional input is heavy tailed. The output equals one of the inputs plus Gaussian noise. The other inputs act as nuisance variables.
Experimental Details We consider a number of tail weight (v) and dimension (d) combinations and generate 5000 train, validation and test samples for each. Our models are simple layer multi-layer perceptrons with 50 nodes in each hidden layer. The models are optimised with Adam to minimise mean square error, selecting the model with smallest validation loss found during training. We consider both sigmoid and ReLU activations. This is to investigate whether the main issue is saturation of sigmoid activation functions. The experiment was repeated 5 times, each trial sampling new set of data.
Results Table shows the results of the experiment. Both models perform well for v = 30 (light tailed inputs), worse for 2, and very badly for 1. Performance also decays as increases, but the effect is weaker. The ReLU activation function performs little better overall, but can still result in very large MSE values.
Overall, the results demonstrate the failure of neural network methods to capture a simple relationship in the presence of heavy tailed inputs. The problem persists under ReLU activation function, showing that it is not caused solely by saturation of the sigmoid activation function.
G. Implementation Details
This appendix contains more details of how implement our experiments in Section of the main paper.
G.1. Normalizing Flows
We use the nflows package (Durkan et al., 2020) implement the NF models. This depends on Py PoTch (Paskke et al. 2019) for automatic differentiation.
With the exception of the CLIMDEX example, our spline and autoregressive affine NF layers output the required transfor- mation parameters from masked neural networks. These use 2 hidden layers, each with a width equal to the input dimension plus 10. Spline layers are configured to use bounding box of [-2.5, 2.5] with 5 bins.
For the more complicated CLIMDEX dataset, which requires more capacity, we reuse the architecture and tuning choices used in Laszkiewicz al. 2002). as discussed in Section 4.2
G.2. Alternative Base Distributions
Standard NFs use a Gaussian base distribution, and a Student's 's T base distribution is also popular for heavy tailed targets. In addition to these, we also tried two alternative base distributions suggested by Amiri al. (2022) which we describe here.
20

---

Flexible Tails for Normalizing Flows
The first is Gaussian mixture model base distribution (m (m_normal) with 5 components (preliminary work found 10 15 components had no significant difference in performance). For each component optimise the d dimensional mean and diagonal covariance terms along with other parameters.
The second alternative base distribution is independent generalised normal distributions sg-normal). The generalised normal family does not have Pareto tails, so we cannot fix them to have the true tail weight. Instead, we optimise the shape parameter each marginal.
G.3. Tail Parameter Initialisation
For density estimation of heavy tailed data, the tail parameters must be initialised sufficiently high that large observations aren't it mapped to vanishingly low probability regions of the base distribution. If this does occur, it is possible to get numerical overflow during optimisation. For variational inference related problem can occur: if some tail parameters are too high, then we can sample points of very low target density and get numerical overflow. In practice, we find that initialising À uniformly from [0.05, provides sufficient stability.
For two-stage methods, Section 3.4 describes our initial stage of tail parameter estimation. Equivalent tail parameters are used for mTAF, TTFfix and COMET. In some cases, this estimation procedure finds that marginal distribution is light tailed. In these cases, as mentioned in Section 3.3 we set the tail parameter to a small value, corresponding to a degree of freedom 1000.
G.4. Sampling Student's T
Consider sampling from a Student's 7 distribution with v degrees of freedom. Algorithm implements this, while allowing the reparamcterization trick i.e differentiation with respect to p.
Algorithm 1 Sampling Student's T
Require: Input:
degrees of freedom v, threshold 2 0.
Sample g ~ Gamma(v/2,, 1).
2: Let g max(g,,))
3: Sample (NO 1).
4: Return 2v 2g
Abiri & Ohlsson (019)) suggest this algorithm with O, and comment on how to differentiate through Gamma distribution. This method is implemented in PyTorch. However we found that taking the reciprocal of 0 can result in an overflow error. Hence we make slight adjustment for numerical stability: we clamp to e = le-24.
G.S. Inverse Transformation for Small Inputs
Equation 13) gives an expression for R Rz'(z), the inverse of our TTF transformation R, in terms of erfe (yy (yy-1/// where (()) = A (x p)/ol + 1. However we experience numerical issues when implementing (13 for small r. Therefore for y 1/a, < 10 our code uses the approximation
R- [π(x) - log n(z)),,
where nTx)= - log (xx) log
and s = =sign(z p). We have (x) xe (x) using 23
G.6. Optimisation
All of the experiments use the Adam optimiser. Table 5 gives learning rate and batch size values.
Most experiments use pytorch defaults for other tuning choices. One exception is density estimation for CLIMDEX, where we follow the optimisation setup from Laszkiewicz et al. (2022). Another is the / = 0.5 variational inference example, where we clip the gradient norm to 5, which was beneficial the stability of mTAF and TTFfix
21

---

Flexible Tails for Normalizing Flows
Table 5: Optimisation Hyperparameters.
Experiment Group
Learning Rate
Batch Size
Synthetic Density Estimation (Section 4.1)
5e-3
None (full pass)
Real Data (Section 4.2
5c-4
512
Variational Inference (Section 4.3
le-3
100
H. Variational Inference Diagnostics
We measure the quality of a density (x) produced by VI using two diagnosties based on importance sampling. This
involves calculating w' = p(z;)// / (x()) where p(z) is the target density and 24 ~ (z) (independently) for i = 1, 2....... In our examples we use n = 10, 000.
Our first diagnostic is based on effective sample size (Robert Casella 2004)
2
n
n
ESS(n)=
Wi
W
We report ESS efficiency, ESS,(n) = (n)(n/ A value of (n),(n) 2 indicates (x(x)) s p(æ).
Our second diagnostic is from Yao et al. (2018). who fit a GPD to the wys and return the estimated shape parameter À Lower values indicate a better approximation of p(x)) by ga (zr). Yao et al. (2018) argue that < 0.7 indicates a useful variational approximation, so we highlight this threshold in our tables.
Data
Table 6 summarises the real data sets used in our experiments.
Table 6: Real data sets information.
Name
Dimension
Average
Topic
Source
Insurance
2
2.17
Medical claims
Liang al. (2022)
Fama 5
5
2.36
Daily returns of 5 major indices
Liang al. (2022)
S&P 500
300
4.78
Daily returns of the 300 most traded US stocks High dimensional meteorological data
Novel to our paper
CLIMDEX
412
4.24
Laszkiewicz et al. (022
All datasets pre-processed before density estimation is performed. To do so, data is normalized variance using the estimated mean and variance from the training and validation sets, as is standard practice in NF density estimation PPapamakarios et al. 2017: Durkan et al.) 2019).
and unit
I.1. S&P 500
Here we describe the S&P 500 dataset which we introduce. These financial returns are example of moderately high dimensional multivariate data with extreme values. We take the closing prices of the top 300 most traded S&P 500 stocks, and convert them in standard fashion to log returns i.c. the log returns are xj = log( # where s, is stock closing price on day j. We use data covering the time period 2010-01-04 to 2022-10-27, corresponding to 3227 days in total. In this example we concentrate on the tails of the data, rather than time series structure. As such, we treat each day of log returns as an independent observation in idd.
The test set is comprised of observations after 2017-09-14, with train and validation sampled uniformly from the period up to and including this date. This corresponds to 1292 training, 645 validation and 1290 test observations respectively.
22

---

Flexible Tails for Normalizing Flows
J. Additional Results for Density Estimation with Synthetie Data
J.1. Other Experimental Settings
Table 7
expands on the results from Table from the main paper, by including more values of and v.
Dashes in the table indicate potential unstable optimisation: least one repeat had very large final loss (above le5). In these cases, even excluding losses this large resulted in significantly worse mean losses than other methods. These results confirm that all the methods not specifically designed to permit GPD tails become increasingly unstable the data becomes more heavy tailed.
The table includes results for the TTF_tBase Btaee architecture. As described Section 4.1 this is similar to TTF, but uses a Student's T base distribution with trainable degrees of freedom. TTF ttase always does worse than TTF and TTFfix. For largest values - and 30 it worst all methods. also has the largest standard error values, suggesting that optimisation may be more difficult once these two methods are combined and there are sets of tail parameters to tune.
Table 7: Density estimation results synthetic example. Each entry is mean value of negative test log likelihood per dimension across 10 repeated experiments, with the standard error in brackets. (Dividing by dimension acts normalises the values, allowing for easier comparison across dimensions.) Bold indicates methods whose mean log likelihood differs from the best mean by less than standard errors (of the best mean). A dash indicates potential unstable optimisation (at least one repeat had final loss above le5). No entry indicates that the experiment was not ran.
d
Flow normal m normal
== 0.5
V == 1
V == 2
V = 30
2.01 (0.07)
1,46 (0.00)
403.24 (239.15)
1.94 (0.02)
1,46 (0.00)
g noormal
1.93 (0.01)
1.46 (0.00)
gTAF
6.42 (0.27)
2.49 (0.02)
1.90 (0.01)
1.46 (0.00)
TTF
3.33 (0.01)
2.34 (0.01)
1.89 (0.01)
1,47 (0.00)
mTAF
4.08 (0.03)
2.49 (0.02)
1.92 (0.01)
1.46 (0.00)
TTFlix
3.33 (0.01)
2.35 (0.01)
1.89 (0.01)
1.47 (0.00)
COMET
3.42 (0.01)
2.35 (0.01)
1.89 (0.00)
1.46
3.39 (0.02)
2.51 (0.04)
2.04 (0.02)
1.65 (0.03)
normal
2.00 (0.02)
1.46 (0.00)
m. monormal
2.04 (0.07)
1.46
g normal
1.98 (0.02)
1.46 (0.00)
gTAF
7.13
2.63 (0.02)
1.95 (0.00)
1.47 (0.00)
10
TTF
3.55 (0.01)
2.47 (0.00)
1.93 (0.00)
1.47 (0.00)
mTAA
4.48 (0.04)
2.63 (0.01)
1.95 (0.00)
1,46 (0.00)
TTFfx
3.54 (0.01)
2.46 (0.00)
1.93 (0.00)
1.47 (0.00)
COMET
3.63 (0.01)
2.46 (0.00)
1.93 (0.00)
1.47 (0.00)
TTF_Basee
3.67 (0.01)
2.63 (0.01)
2.08 (0.01)
1.62 (0.01)
normal
2.02 (0.01)
1.47 (0.00)
m normal
2.02 (0.00)
1.47 (0.00)
g gonormal
2.01 (0.00)
1.47 (0.00)
gTAF
7.49 (0.38)
2.65 (0.01)
1.99 (0.00)
1.47 (0.00)
50
TTF
3.68 (0.00)
2.54 (0.00)
1.98 (0.00)
1.47 (0.00)
TTAF
5.22 (0.04)
2.62 (0.01)
1.98 (0.00)
1.47 (0.00)
TTFlx
3.68 (0.00)
2.54 (0.00)
1.98 (0.00)
1.47 (0.00)
COMET
3.74 (0.00)
2.55 (0.00)
1.97 (0.00)
1.47 (0.00)
TiF_Base
4.17 (0.01)
2.84 (0.04)
2.35 (0.04)
1.82 (0.06)
J.2. Learned Tail Parameters
Figure fillustrates tail parameters learned by TTF corresponding to the d = 5 experiment Table These results show that the learned parameters don't exactly match the TTFfix values chosen to match the known tail shapes. Our theory Theorem C.9 statement 2 shows that the two should equal for good tail fit if our transformation were applied to
23

---

Flexible Tails for Normalizing Flows
Final tail parameters for synthetic density estimation d=5
6.0
30
0.9
2.0
5.5
25
1.8
5.0
0.8
арprox TTF df
approx TTF df
approx TTF df
approx TTF df
4.5
20
1.6
4.0
15
1.4
3.5
0.6
10
1.2
2.5
5
0.5
1.0
2.0
0.5
1.0
2.0
30.0 target
target df
target df
target df
Figure 2: Box plots of TTF final tail shape parameters from repeated optimiser runs. The dashed horizontal lines show the TTFfix values. The y-axis shows 1/A, as then the TTFfix values directly match the target Student's ' T degrees of freedom.
a aO 1) random variable. However Theorem C.9 statement shows that changing the input random variable e.g. to [u,, o²) can produce resulting distribution with a different tail shape. So the lack of match in practice suggests the learned TTF parameters adjust slightly from the TTFfix values to compensate for the effect of the other normalising flow layers.
Adjusting in this way could be an advantage in learning the tail parameters (as in our TTF method) rather than fixing them in advance (as in our TTFfix method). However in this case any such advantage small, as the overall results are very similar Table Also, as we have seen other experiments, TTFfxx is often competitive practice, and can be easier train.
24


Reviews:
[{"review_id": "EvARTiFL4D", "reviewer": "Reviewer_a6XW", "summary": "Normalizing Flows have been shown to be difficult to make work when data/density to be represented has heavy tails, something common to see to atleast some degree in tasks such as density estimation and Variational inference. Neural networks are known to have difficulty in converging during training where heavy tails are involved. Also some recent work has shown that existing standard NF algorithms with light tailed Gaussians as base distribution does not work well which serves as a motivation for this work. Some recent work has suggested to using heavy tailed distributions as base distributions, and then passing it through a Lipschitz transformation, this work suggests doing it the other way, using a standard light tailed ddensity as the base distribution while making the transformation non-Lipschitz so that the final transformation becomes heavy tailed. The transformation function to induce heavy tails has to be both invertible and differentiable for this to work, for which the authors emply the ecdf function  \nThe experimental results show some improvements over the other algorithms based on the other approach on real world datasets but marginal improvements on synthetic datasets.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the positive review of our paper. We appreciate you taking the time to read it!\n\n## Left/right tails\n\nThis is a great suggestion. We will update the paper to include terminology around left/right tails.\n\n## Additional experiments\n\nUnfortunately each experiment on real world data took several weeks to set up and run, including the time to find suitable datasets and perform preliminary data cleaning. So we aren't able to add any more datasets at this stage in the review process. We argue that our method does show reasonable improvement for real datasets. We summarise the argument below.\n\nTable 2 shows our method provides improvements for all experimental datasets. In line with our synthetic experiments these gains are modest for lower dimensional datasets (Insurance, Fama 5), but larger for higher dimensional datasets (CLIMDEX and S&P returns).\n\nIn particular for the highest dimensional dataset, CLIMDEX, we reach a NLL value of -2214. This is a significant improvement on competing methods, which provided values of -2113 (gTAF), -2121 (mTAF), -2118 (COMET), and had only a modest advantage on the value of -2102 from using no tail method.", "claims_and_evidence": "I think the paper has done a good job at supporting their claims with experiments. I particularly liked how sections in Appendix B3 and D are provided by authors to elaborate and explain the claims which they used as motivation in the paper. Section D supports previous claim made on universality by the work of Kobyzev et al. The appendix is so well done and covers the practical and finer optimization details quite adequately, which are especially needed to practitioners especially when it is common to encounter numerical overflow and underflow issues when working with heavy tailed and datasets containing outliers.", "methods_and_evaluation_criteria": "The authors used both real and simulated datasets in the experiments and had a proper cross validation set up to compare with the baselines.", "theoretical_claims": "I found the theoretical  claims well done and supoorted by arguments using theory from previous recent work esp. that of Jaini 20.", "experimental_designs_or_analyses": "I found the analysis adequate and supporting the claims made by the paper. I have also felt in my first hand experience that working with heavy tailed base distributions can be more tricky in VI annd NF especially in higher dimensions and challenging posterior, I have also felt that fixing the tail parameters results in more stable optimization than optimizing it along with other parameters, and this work also makes similar conclusions based on their experimental results.", "relation_to_broader_scientific_literature": "The work uses many recently proposed algorithms as baselines which means that there is still interest in the community for this nature and direction of work.", "essential_references_not_discussed": "I am moderately familiar with the literature on flow methods and extreme value theory, I think the authors cover all the most noteworthy papers that I know of.", "other_strengths_and_weaknesses": "1. I think the paper is a really nice read, it introduces the reader with a lot of interesting concepts and methods in EVT, Normalizing flow and tail-index estimation like modelling the tails and estimating shape parameter of the GPD to researchers who work with standard Normalzing flows or standard VI using light tailed Gaussians. \n2. The appendix is well written and the paper makes an effort to elaborate whenever it touches a claim in the appendix.\n3. The empirical results are mostly supportive, it is good to see that in some cases where the target is quite difficult, standard methods with heavy tailed base distributions can fail to converge ash shown in Table 1. \n4. The paper lists its limitations in quite transparent and matter of fact way, something which is missing in a lot of papers.\n5. For a user who is really concerned about accuracy, and maybe dealing with smaller datasets, the results are encouraging and Table 3 clearly shows that. \nWeaknesses\n1. I would have hoped for a bit better results, meaning seeing higher degree of improvement in Table 2.", "other_comments_or_suggestions": "1. I have personally seen left and right tails in literature in place of lower and upper, maybe you can mention that somewhere to make it clearer.\n2. Can you find some real world dataset where you can show a bigger improvement in results in Table 2.\nI would still like to accept this work as it is so well written and explained.", "questions_for_authors": "Listed above already.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NDtQOks5hV", "reviewer": "Reviewer_mY4v", "summary": "The paper proposes a way to enhance normalising flows models by incorporating ideas from Extreme Value Theory literature for representing distributions with heavy tails. The authors propose adding a new invertible layer after the traditional flow layers that does not have a Lipschitz transformation and hence provide more flexibility.\n\n## update after rebuttal\nI am still not convinced of the added value of the universality result and if the method can be applied to more classes of flow-based models, hence I will keep my score.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the helpful review. We respond to each of your questions below, and invite you to increase your score if you think we have addressed the points fully.\n\n## Flow-Matching\n\nThanks for raising this interesting question: does it remain challenging to fit heavy tailed observations in Flow-Matching? We expect similar problems would occur here and in other related methods e.g. diffusion models and continuous normalizing flows. The reason is as follows.\n\nAll these methods involve learning a vector field $u(x,t)$ controlling a differential equation. The vector field is taken to be a neural network. In a successful generative model we will sometimes have $x \\approx x_i$ for any given training data item $x_i$. So if there are extreme values in the training data, the neural network will need to be able to cope with such input. A key argument of our paper is that neural networks are poor at this task.\n\nA plausible fix is to use our tail transform as a final discrete layer following a Flow-Matching generative model. Either joint training or a two-stage approach could be tried, similarly to the methods in our paper for discrete normalizing flows. Empirical investigation is beyond the scope of our current paper, but is a very interesting avenue for future work.\n\nWe will add a paragraph to our discussion section commenting on this topic.\n\n## Universality\n\nYou ask a natural question about the theoretical value of the TTF transformation given that it does not improve existing universality properties. We do comment on this in the paper, but it may not be easy to find as it's in the appendices (Appendix D.3). Here's an extended version of our argument.\n\nA normalising flow defines a set of probability densities depending on the capacity of the flow (e.g. number of knots in a spline flow, number of neural network parameters etc). Roughly speaking, universality means that under the limit of large capacity the set contains all probability densities. We expect improved tail behaviour modelling means that it's easy to get good approximations from lower capacity. This is because only a small number of parameters are required to produce heavy tails. In contrast a spline flow with a Gaussian base distribution (for example) would need a large or infinite number of knots to modify a Gaussian tail to become heavier.\n\nIn summary, universality properties are about the limit of infinite capacity. We expect the benefits of our approach occur for any given finite capacity. Further theoretical developments would be needed to formalise this argument mathematically, but it is consistent with our empirical findings.\n\nWe'll add a brief summary of the above to the main paper's section on universality.\n\n## Metrics\n\nThanks for the suggestion of MMD and TVD as metrics. We have concentrated on test log likelihood as it's the main metric in the past literature on this topic. For instance Table 2 contains a direct comparison to log likelihood values from a previous paper. We've not been able to repeat our experiments to report additional metrics within the review period, but we will explore this in future work. We note care may be needed applying MMD in higher dimensional examples: https://arxiv.org/abs/1406.2083.\n\n## Combining new layer and T-distribution for base \n\nThanks for the intriguing suggestion of combining these approaches i.e. (1) a Students T base (2) our final TTF layer. In the paper we present these as two competing methods, and didn't consider making a combination.\n\nWe have run some experiments on combining the methods. For the setting of Table 1 with $\\nu=0.5$, we get a scaled NLL of 4.17 with standard error of 0.01. This is better than the heavy tailed base distribution methods, but not as good as the TTF methods.\n\nIn another setting - $d=5$, $\\nu=0.5$ - we find similar results. The combined method gets a scaled NLL of 3.40 with standard error of 0.02. Again this is better than the heavy tailed base distribution methods, but not as good as the TTF methods (see Table 7 in the appendices). However the differences between the methods are smaller now.\n\nFurther experiments would be needed to explore why the combined method has worse performance.", "claims_and_evidence": "Yes, the evidence is convincing.", "methods_and_evaluation_criteria": "The evaluation follows commonly used  negative log-likelihood metric.", "theoretical_claims": "Yes, the theoretical claims appear valid.", "experimental_designs_or_analyses": "Yes!", "relation_to_broader_scientific_literature": "The prior works often consider using heavy tail distributions such as T-distributions (or Gaussian mixtures, generalised Gaussian), whereas the authors propose a more flexible approach and substantiate their claims empirically.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Taking motivation from Extreme Value Theory Literature was an ingenius idea", "other_comments_or_suggestions": "NA", "questions_for_authors": "I add most of my questions and concerns here.\n\n1. Normalising flows are great models with attractive properties such as density estimation. Their real power appears to be in the form of Flow-Matching, though, where they have rivalled Diffusion models. The implicit modelling approach only requires learning the associated vector field instead of explicit mapping. Now my question is, if the problem of poor modelling of heavy tailed distribution also appears in flow matching. Since flow matching does not constrain the architecture, I doubt it. If it is still an issue, how could the proposed methods be applied to non-discrete versions of Normalising Flows?\n2. If I understood correctly, the authors show that the additional layer does not change the existing universality proofs. In that case, I question the theoretical value of the TTF transformation, given that it doesn't relax any assumption on modelled class in the universality proof.\n3.Whilst the density estimation results are better for the proposed method, I believe log-likelihood alone is not enough to show the power of the proposed additional layer, I believe metrics like MMD or TVD could shed more light on the efficacy of the method.\n4. Could the authors show how the proposed layer help if the base distribution is chosen to be the T-distibution.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bakjIxvjyO", "reviewer": "Reviewer_ocLN", "summary": "The paper introduces an invertible transformation to overcome difficulties that exist when fitting normalising flow models to heavy-tailed data distributions. The transformation is to be used at the tip of a normalising flow \"chain\", handling the heavy tails of the data and enabling normalising flows to be fit to heavy-tailed data without restricting the flow to a specific architecture.\n\n## Update after rebuttal\nNo changes after the rebuttal, as no further comments were made by the authors and the interactions with other reviewers did not change my evaluation of the paper.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the positive review of our paper. We appreciate you taking the time to read it!", "claims_and_evidence": "The claims made are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The evaluation criteria is sensible and uses reasonable baselines and data sets. There experimental settings cover both controlled scenarios and real data, allowing a clearer picture to be made.", "theoretical_claims": "The theoretical claims were verified insofar as covered by Appendix A.1, where I found no issues.", "experimental_designs_or_analyses": "The experimental settings and analyses are valid, though I have not verified the contents in the Appendix.", "relation_to_broader_scientific_literature": "The paper introduces a transformation to be applied to normalising flows to handle heavy-tailed data, thus relating closely to existing methods that address the same problem. Because it applies to most normalising flows that could be applied in tandem with the proposed method, it naturally relates to most of them.", "essential_references_not_discussed": "I have not identified any lacking references.", "other_strengths_and_weaknesses": "**Strengths**\n\nThe paper is very well-written, with an easy-to-follow structure. The method itself is quite simple to implement and understand, consisting of a simple transformation parameterised also in a straightforward manner, and is thoroughly studied by the authors. I also appreciate the authors discussed important limitations of their work.\n\n**Weaknesses**\n\nGiven how much the authors rely on the theory and some of the notable aspects (such as universality) being so fundamental, I do wonder why the paper is organised with so much didactic introduction to already existing work. I do believe this point is rather subjective, so I consider this only a very minor weakness.", "other_comments_or_suggestions": "No other suggestions or comments at this point.", "questions_for_authors": "No questions at this point.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Wun10noTi0", "reviewer": "Reviewer_h4DD", "summary": "This paper addresses learning heavy tailed distributions with generative models and normalizing flows in particular. Similar to the previous approach COMET, it first transforms the tails of the input data to be light tailed and then applies a classical normalizing flow.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the detailed review which has provided some helpful new insights into our results.\n\n## Success rate\n\nFailure rates for normal, m_normal, g_normal in Table 1 were 100% for $\\nu=0.5$, and in the range 10-20% for $\\nu=1$.\n\nFor $\\nu=1$, even in runs which did converge for these 3 methods, the best loss (NLL of 49.62) was significantly worse than results for other methods.\n\nThese results confirm that all the methods not specifically designed to permit GPD tails become increasingly unstable as the data becomes more heavy tailed.\n\n(On checking our code, we note \"did not converge\" is not as we defined it in the paper. In fact it means at least one run had a final loss above 1e5, corresponding to extremely unstable optimisation. We'll update this in the paper.)\n\n## NLL metric\n\nNLL has some sensitivity to both body and tail behaviour. For instance under a $N(0,1)$ density estimate, a single observed value of 10 would contribute a very large amount to NLL.\n\nWe have concentrated on NLL as it's the main metric in the past literature on this topic. For instance Table 2 contains a direct comparison to log likelihood values from a previous paper. \n\nWe agree it's desirable to explore metrics more specialised to tail behaviour in future e.g. see our reply to reviewer mY4v.\n\n## Learned tails\n\nA rough plot of learned tail parameters for the experiment of Table 7 with $d=5$ is at [this link](https://pasteboard.co/oKRznj9WA3UL.png). The y-axis shows $1/\\lambda$, with the target $\\nu$ value shown as a horizontal line. The two would need to be equal for a good tail fit if our transformation were applied to a $N(0,I)$ random variable. As you suggest there is an offset due to the Lipschitz transformation.\n\nThis is an interesting finding - thanks for suggesting this plot! It suggests a potential advantage to jointly learning the Lipschitz transform and the tail parameter. We'll write this up as an appendix.\n\n## $\\hat{k}$ threshold\n\nTable 3 highlights the threshold $\\hat{k} < 0.7$ since Yao et al recommend this as a cut-off for a variational approximation to be useful.\n\nThanks for pointing out that this isn't clear from our current discussion of Yao et al. We'll extend this.\n\n## How well do VI metrics capture tails?\n\nESS and $\\hat{k}$ are general VI metrics. Informally we expect both are sensitive to tail behaviour. This is because both are based on importance weights $w(x) = p_{\\text{target}}(x)/q_{\\text{approx}}(x)$. When $p_\\text{target}(x)$ is heavier tailed than $q_{\\text{approx}}(x)$, this typically results in a few very large $w$ values leading to poor ESS and $\\hat{k}$. We agree that specialised VI metrics for tail behaviour would be useful, but we are not aware of any in the related literature.\n\n## Fréchet domain of attraction\n\nThe Fréchet domain of attraction captures power-law (Pareto) tails and closely related tail behaviour. Our theory shows that COMET flows cannot represent such tails, although they can produce milder heavy tails similar to log-normal.\n\nThis is an argument in support of TTF: it allows a much wider range of heavy tails than COMET flows.\n\nWe'll add a sentence to summarise this in the main paper.\n\n## VI benchmark\n\nWe've chosen our VI example as it allows the tail weight to be varied. We found that established benchmarks didn't easily allow exploration of heavier tails, such as those corresponding to $\\nu = 0.5$. We agree investigating established benchmarks is a useful future step. \n\n## Mixture of Gaussians\n\nThe suggested result should indeed strengthen the paper and be straightforward to prove, so we'll add it to an appendix.\n\n## Draxler et al\n\nThanks for pointing out that the theory of Draxler et al doesn't use KL convergence. Instead their metric is based on the effect of extending a sequence of coupling layers by one element. It's not obvious how to extend this to our setting where the sequence instead terminates in our TTF transform. So we'll remove our comments in D.4 on the possibility of extending the work of Draxler et al, and instead just note that it exists. This doesn't affect our main argument in Appendix D on extending universality under weak convergence.\n\n## Line 56\n\nThanks for highlighting this. We do mean z not x. Our presentation describes the \"generative direction\" of normalizing flows i.e. a function converting z to x. We do this simply to be concrete and consistent, and it does not affect our results. Note z can be extreme when a heavy tailed base distribution is used. \n\nWe'll add a clarifying comment about \"generative direction\" around line 56 in the paper, as the current version doesn't mention this until line 139.\n\n## Sub-Gaussian tails\n\nFor heavy tails, we motivate making a transformation at the data end to avoid providing extreme input to a neural network. For sub-Gaussian tails this argument does not apply, so we expect that adapting the tail at either end will work well. Which is most beneficial likely depends on which is easier to implement in practice.", "claims_and_evidence": "Let's span this by claimed contributions:\n\n- \"Illustrate the problem of using extreme inputs\": I somewhat agree, but it is unclear how bad it is -- for the normal/m_normal/g_normal approaches normal, m_normal and g_normal, what is the success rate of training a model? Table 1 only says that for heavy tails, at least one out of ten runs fails.\n- \"We introduce ... TTF ... and prove it converts Gaussian tail to heavy tails with tunable tail weights\" -- yes.\n- \"We demonstrate improved empirical results for density estimation ... and VI ... compared to standard NFs, and other NF methods for heavy tails.\" -- I think there are some important comparisons missing, see below.\n\n# Update: Addressed\n\nI am convinced that the method improves the quality of learning heavy tails with normalizing flows.", "methods_and_evaluation_criteria": "Datasets: Artificial targets make sense, cannot judge for the real-world data.\n\nMetrics:\n\n- KL: Correct me if I'm wrong, but the KL divergence = NLL is more sensitive to body than tail behavior due to the vanishing weight of these regions. Please also report the learned tails (which presumably even makes sense for the cases where the tails are learned and not fixed using the known values, given that there could still be some offset produced by the Lipschitz transformation).\n- hat k: In the appendix, the authors write \" Yao et al. (2018) argue that k>0.7 corresponds to a very poor approximation, and k<0.5 indicates a good fit.\", but they then proceed to mark values already below 0.7 in Table 3 as bold, which is inconsistent. Also, more details on this metric would be crucial instead of the three lines of text in l. 349-351: How well do these metrics capture tails?\n\n# Update: Addressed\n\nMain argument: KL is indeed sensitive to extreme values, which I agree to.", "theoretical_claims": "I did not check proofs, but the statements seem plausible.\n\nThe authors argue in the Related Work + Appendix B.2 that COMET is not in the Fréchet domain of attraction -- why is this important? COMET performs pretty well experimentally, and also I would assume the tail behavior can be rich, so why pick this particular instantiation? To my understanding, COMET could just well fulfill some other class of heavy-tailedness.\n\n# Update: Largely addressed\n\nThe authors argue that COMET learns a class of less heavy-tailed distributions. I think this argument and the fact that they outperform COMET is good enough to motivate their method.", "experimental_designs_or_analyses": "In terms of density estimation, Table 2 merges several benchmarks from previous papers.\n\nIn terms of variational inference, it would be good to have a comparison to an established benchmark instead of a new artificial one.\n\n\n# No update", "relation_to_broader_scientific_literature": "Learning heavy tails correctly is an important consideration for designing generative models, which can help them be successful in a broader set of applications. Like COMET, the method indicates that feeding heavy-tailed data into a neural network may be detrimental to performance; instead, the heavy tails should be converted first.", "essential_references_not_discussed": "Seems like nothing is missed.", "other_strengths_and_weaknesses": "No additional comments.", "other_comments_or_suggestions": "One idea to strengthen the paper is by showing that a mixture of Gaussians with a finite number of components as a latent distribution does not have an influence on the tail behavior of the learned distribution. It seems like this is an easy proof -- the asymptotic behavior in any direction is determined by the largest standard deviation component only, and it is light tailed.\n\nI read [Draxler et al. 2024] differently: From their Limitations section: \"Secondly, it is unclear how the convergence metric [in] Section 5.2 is related to convergence in the loss used in practice, the KL divergence given in Equation (3).\" So while I agree that they consider the full space (which contains the tails), their theory is not about KL convergence (they only have experimental evidence for KL). Given that their statements apply to any continuous distribution with finite second moment, it would be interesting to characterize what class of r is needed to convert any heavy into light tails.\n\nl. 56: z should be x.\n\n\n# Update: Found common ground\n\nThe authors will clarify the statements regarding universality, they might add the proof I suggested regarding the tails of GMMs being always light.", "questions_for_authors": "What is the success rate of normal/g_normal/m_normal in Table 1?\n\nHow relevant is the Fréchet basin of attraction? What tails can/cannot be explained by it? Is this an argument for why TTF should be preferred over COMET?\n\nGiven that a heavy-to-light transformation should be applied on the data end: For sub-Gaussian tails, do the authors think that adapting the tails on the latent end is beneficial?\n\nHow well do NLL, ESS and k capture tails?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Understanding how generative models handle extreme values and haevy tailed distribution is an important open task. This paper proposes a novel approach for handling haevy tails in normalizing and provide supporting theory.  The reviwers agreed that the paper is well written and that the prposed method and theory is intersting and novel (3 oout of 4 reviewers voted for accpetance apfter the rebuttal). Remaining questions where clearly answered during the discussion period.
Decision: Accept (poster)