Marginal Tail-Adaptive Normalizing Flows
Authors: Mike Laszkiewicz
Johannes Lederer
Asja Fischer
Venue: ICLR 2022 Submitted (2022)
Keywords: Normalizing Flows
Heavy-Tailed Data
Generative Models

Abstract: Learning the tail behavior of a distribution is a notoriously difficult problem. The number of samples from the tail is small, and deep generative models, such as normalizing flows, tend to concentrate on learning the body of the distribution. In this paper, we focus on improving the ability of normalizing flows to correctly capture the tail behavior and, thus, form more accurate models. We prove that the marginal tailedness of a triangular flow can be controlled via the tailedness of the marginals of the base distribution of the normalizing flow. This theoretical insight leads us to a novel type of triangular flows based on learnable base distributions and data-driven permutations. Since the proposed flows preserve marginal tailedness, we call them marginal tail-adaptive flows (mTAFs). An empirical analysis on synthetic data shows that mTAF improves on the robustness and efficiency of vanilla flows and—motivated by our theory—allows to successfully generate tail samples from the distributions. More generally, our experiments affirm that a careful choice of the base distribution is an effective way to introducing inductive biases to normalizing flows.

Under review as a conference paper at ICLR 2022
MARGINAL TAIL-ADAPTIVE NORMALIZING FLOWS
Anonymous authors
Paper under double-blind review
ABSTRACT
Learning the tail behavior of distribution a notoriously difficult problem. The number of samples from the tail is small, and deep generative models, such normalizing flows, tend to concentrate on learning the body of the distribution. In this paper, focus improving the ability of normalizing flows to correctly capture the tail behavior and, thus, form more accurate models. We prove that the marginal tailedness of a triangular flow be controlled via the tailedness of the marginals of the base distribution of the normalizing flow. This theoretical insight leads us novel type of triangular flows based on learnable base distributions and data-driven permutations. Since the proposed flows preserve marginal tailed- ness, we call them marginal tail tail-adaptive flows (mTAFs). An empirical analysis on synthetic data shows that m n'TAF improves the robustness and efficiency of vanilla flows and motivvaedd by our theory-allows to successfully generate tail samples from the distributions. More generally, experiments affirm that a careful choice of the base distribution is an effective way to introducing inductive biases to normalizing flows.
INTRODUCTION
Heavy-tailed distributions are known to occur in various applications in biology, finance, social sciences, and more. Examples for such observations include the length of protein sequences in genomes (Koonin et af. 2006), returns of stocks (Gabaix et al., 2003). or the size of cities (Gabaix) 1999]. Applications that are tightly connected to typical deep learning applications include the frequency of class examples in image classification (Horn & Perona 2017) and the frequency of words ZZipi 1949) in natural language processing. From a theoretical point of view, this is not surprising since heavy-tailed distributions emerge from several circumstances, including the limiting distribution in the generalized central limit theorem, of multiplicative process, the limit of an extremal process (Nair et al. 2013). Given the frequency of occurrence, developing generative models that allow learn heavy-tailed distributions is essential.
Normalizing Flows (NFs (Rippel & Adams 2013 Tabak & Turner 2013 Dinh et al. 2015 Rezende & Mohamed 2013) a popular class of deep generative models. Despite their success in learning tractable distributions where both sampling and density evaluation can be efficient and exact, their ability to model heavy tailed distributions is known to be limited. [Jaini al. (2020) identified the problem that a range of NFs (c.g. vanilla triangular flows with a Gaussian base distribution) are unable to map a light-tailed distribution to a heavy-tailed distribution. They propose solve this issue by replacing the Gaussian base distribution by a multivariate t-distribution with one learnable degree of freedom. While this allows to model distributions with a heavy-tailes euclidean norm, we show that modeling multivariate distributions, where some of the marginals are heavy- and some are light-tailed, still poses problem.
Contributions Our contributions in this work, that extend the results Jaini et al. (002), are the following. First, we prove that triangular affine NF using a base distribution with solely heavy- tailed marginals is only able to provide a target distribution with just heavy-tailed marginals as well. Consequently, such a NF is not capable of learning distributions with mixed marginal tail behavior. Second, we derive result that states conditions under which the marginal tailedness of the base distribution be preserved. Third, based these theoretical findings, of triangular NF that allows to learn distributions with heavy- and light-tailed marginals. The new model is called marginally Tail-Adaptive Flows (mTAFss) and illustrated in Figure combines
propose novel kind

---

Under review as a conference paper at ICLR 2022
Step 1: Estimate the Marginal Tail Indices
Step 2: Defining the Base Distribution
Step 3: Flow-Layer with Data- Driven Permutation Scheme
Employ Tail Index Estimators
Flexible Base Distribution
Rearranging Marginals
Triangular Transformation
Tell-Dependent
Groupwise Permutation
Light- Thiled
Target Distribution
Tall Index Ptz
A/0.11
Light-
indloc
(0
An Overview of m.AF. In first step, apply estimators from extreme value theory to classify the marginals as heavy- or light-tailed. This classification defines flexible base distribution consisting of marginal Gaussians and marginal d-distributions with trainable degree of freedom, as illustrated by Step 2 of this figure. Further, we rearrange the marginals such that the first d marginals light-tailed, whereas the remaining marginals heavy-tailed, mTAF is then constructed using several flow-layers as visualized in Step 3: we employ a triangular mapping, followed by 2- 2-group permutation scheme, i.c. we just permute within the of light-tailed marginals and within the set of heavy-tailed marginals. At the end, we restore the original ordering using the inverse of the permutation employed in Step 2. Using Theorem we prove that m'TAFs are marginally tail- adaptive (Corollary 1).
Figure 1:
estimators from extreme value theory (Hill 1975 Dekkers et al. 1989 Csorgo et al. initially assess Seavy-tailednnss of the target's s marginals;
1985
flexible and trainable base distribution based on the estimated tail behavior of the target distribution;
new permutation-ssheme between flow-layers that ensures the correct tail behavior of the estimated target distribution.
Finally, we conduct an experimental analysis demonstrating the superior performance of the pro- posed mTAFs comparison to other flow models. Furthermore, we present a new sample genera- tion scheme, motivated by theory, which successfully generates joint samples that are from the tails of a specified marginal.
Notational Conventions In the following, we will denote random variables by bold letters, such as and its realisations by non-bold letters, We use this notation for multivariate and for univariate random variables. Further, we denote the jth component of x by X,, and Xı, or X<j are the first j components of respectively. We denote the random variable representing the base distribution by z and the random variable representing the target distribution by x. Further, for notational convenience, whereas marginal PDFs are denoted by and 4,, respectively. Finally, we assume that both random variables x and z have continuous and positive density on R ie p(x), ((2) > 0 for all x, 2 E R.", where D is the dimensionality of and z.
denote the probability density functions (PDFs) of x and z by p and q,
2

---

Under review as a conference paper at ICLR 2022
2 BACKGROUND
In this section, we give brief introduction to heavy-tailed distributions and present needed back- ground knowledge about normalizing flows.
2.1 HEAVY-TAILED DISTRIBUTIONS
Heavy-tailed distributions are distributions that have heavier tails (i.e. decay slower) than the ex- ponential distribution. Loosely speaking. slowly decaying tails allow to model distributions that generate samples, which differ by a large magnitude from the rest of the samples, For univariate random variable x we define heavy-tailedness via its nomenntgeneraatng function
Definition (Heavy-Tailed Random Variables). Consider a random variable E R with PDF p.
Then, we say that x is heavy-tailed if and only if
AYA 0 [x[
=
The function (X) :- [x(exp(Xx)] is known the moment-generating fanction of Random variables that are not heavy-tailed are said to be light-tailed.
Note that this definition is, strictly speaking, merely definition for heavy right tails. We say random variable x E R has heavy left tails if xx has heavy tails according to Definition I For simplicity of derivations and w.l.o.g. we proceed with this definition but the derived results can analogously be applied to left tails.
We can assess the degree of tailedness of distribution. While there are many equivalent notions of the called tail index, the most straight-forward definition is via the existence of moments:
Definition 2 (Tail Index). A random variable x E R with PDF p is said to have tail inde holds that
F a if
{< 00. ifB<a. Ex [x[
00 if a.
Since the tail index is tightly related to the decay rate of the PDF, it enables us to assess the degree of heavy-tailedness of a random variable. Therefore, estimation the tail index became an important objective in extreme value theory and statistical risk assessment (see e.g. |Emberchts et al.) (013). Since the existence (see Proposition
the moment does not depend on the "body" of but only on the tails of 3 in Section A. in the Appendix), estimating the tail index by fitting a full parametric model to all data e.g. via likelihood maximization leads to a biased estimator. Instead, semi-paranie cemi-parametric estimators have been developed, which aim to fit distribution only the tails. Popular methods for tail estimation include the Hill estimator (Hill) 1975). the moment estimator (Dekkers al. 1989), and kernel-based estimators (Csorgo et al. 1985). In Section B. of the Appendix, we discuss these tail estimators and review some practical issues with these.
An example of heavy tailed distribution the standardized l-distribution, which has parameter v > referred to as the degree of freedom and a density function given by
T( V
p
(1
IER,
where is the Gamma function. is known that the I-distribution has tail index (see e.g. Kirkby et al. (2019) for a detailed reference).
In the multivariate setting, there exist various definitions of heavy For instance Resnick (004) make use of a definition based on multivariate regular variation. JJini et al. (2020) define multivariate random variable x to be heavy-tailed if the f--norm heavy-tailed, a property which we refer to fe-heavy tailed, and which is formally defined as follows:
'One readily show that this definition is equivalent to the definition, which compares the tails of the tails exponential distribution. See Section 1 in Nair (0013).
"Notice that the notion of a tail index is only valid for eguularlyvvarying random variables, which are subclass of heavy-tailed random variables. For the purpose of this work, it sufficient to consider regularly varying random variables. More details can he found in Nair al. (2013.
3

---

Under review as a conference paper at ICLR 2022
Definition 3 (ey-Heavy- Ha--HeeayTaieedde Tailed), Letx IR' be multivariate random variable. Then, we call lg-heavy-taile if it holds that ||ж||| is univariately heavy-tailer according to Definition | denotes the l--nomm Otherwise, we call x Ez-light-tailed
where ||
2.2 NORMALIZING FLOWS
The fundamental idea behind NFs is based on the sity functions (PDFs) given in the following theorem.
formula for probability den-
Theorem 1 (Change-of- -faaiabl--Variabless). Consider random variables x.2 E RD and diffeomorphic map T IR - R such that x T(z). Then, it holds that the PDF of x satisfies
p(x) = q (x)=(x)))))))))(((((()()>))()) det (x)) Vx E IRD
(1)
where JT (z) is the Jacobian of T evaluated at a R..
This formula allows us to evaluate the possibly intractable PDF of x if we can evaluate both, the PDF of and (x). and efficiently calculate the Jacobian-determinant det Jr (x). As I maps z to x, we denote the distribution of z and x as the base and the target distribution, respectively.
To model the PDF of using NFs, common to set the base distribution standard normal distribution (i.e., z ~ I)) and to employ likelihood maximization to learn parameterized transformation
To T= T (b) 7
which, yet, remains tractable and diffeomorphic. Masked autoregressive flows (MAFs (Papamakar ios et al. 2017)) are one popular architecture, which employ transformations T (Ti...., Tn) of the form
T²(2) == (zجح) exp(a,(z<)))z for (I...., D3
where μuy and oj are neural networks, which obtain the first j- components of 2 input and output a scalar. Composing several transformations of the form (2). we obtain the MAF, The autoregressive efficiently evaluate the facobian-Determinant due to the diagonal form of (xr) A crucial issue such autoregressive models that a component X, only depends on the previous outputs xzj and, therefore, they cannot model causal relationship in which x; causes x; if < j. This issue can be solved by applying permutation before each transformation T(() , TT( This is usually random permutation the one that reverses the ordering of the components. Therefore, in summary a MAF consists of multiple consecutive layers T p(t), where p!) DD is a permutation. MAFs belong to the class of triangular flows, which are defined as flows that consist of diffeomorphisms whose jth output only depends on Z-z-- Other examples for triangular flows include RealNVP (Dinh et al. 2017). NAF (Huang et al. 2018), and SOS (Jaini et al. 2019). the triangular maps are affine linear (such as in (2). we call the resulting flow triangular affine flow. Further types of NFs include invertible ResNets (Jacobsen et al. 2018 Behrmann et al. 2019 Chen et al.] 2019). continuous flows (Chen et al.) 2018 Grathwohl et al, 2019), and many more (Kobyzev et al. 2020
form in 2 allows
Tail-Adaptive Flows. Jaini et al. (2020) investigated the ability of triangular flows to learn heavy- tailed distributions. The authors have shown that if a triangular affine flow transforms a fz-light- tailed distribution, such as the multivariate Gaussian distribution, to a Eg fz-heavy-tailee target distri- bution, then To cannot be Lipschitz continuous. And more explicitly, it holds the following.
Theorem 2. Jaini al. 2020 Let z be Ez-light-tailed random variable and T be affine triangular flow such that (3(3s) = (5ام)))) + az (Zzz) )2 for all j. a is bounded above and p. Lipschitz, for all, j, then the transformed variable x is also Ez-light-tailed.
Furthermore, the authors prove that any triangular mapping from an elliptical distribution to heavier-tailed elliptical distribution must have an unbounded facobian-determinan
Clearly, these results illuminate that learning a heavy-tailed distribution using NEs leads to non-Lipschitz trans- formations and unbounded Jacobians, which inevitably affects training robustness (Behrmann et al.) 2021). Motivated by these result, Jaini et al. (2020) propose Ada--Ade Flows (TAF), which place the Gaussian base distribution by a multivariate t-distribution with one learnalee degree of freedom.
4

---

Under review as a conference paper at ICLR 2022
3 LEARNING THE CORRECT MARGINAL TAIL BEHAVIOR WITH MTAF
In this section, we present a simple extension to triangular affine flows that allows to model distri- butions with flexible tail behavior. We start by presenting our theoretical results in Section 3.1 Motivated by these results, we propose marginally Tail-Adaptive Flow (mTAF) in Section 3.2
3.1 THE NECESSITY OF A FLEXIBLE BASE DISTRIBUTION
In this work, we investigate the tailedness of NFs more thoroughly through the lense of marginal tailedness, i.e. we consider the univariate tailedness of the marginal distributions of xy. Therefore,, we introduce the following definitions:
Definition 4 Hi-Hevvy--ailess Mixed-Tailed, Fully Heavy-Tailed, Equal Tail Behavior). We call random variable x E ID j-heavy-tailed if its jth marginal xj is heavy-tailed according to Defini- tion [ Otherwise, we call x j-light-tailed. is said to be mixed-tailed there exists 11, J2 such that x is jı taile and se-light-tailed, Further, we say that x is fully heavy-tailed if x is j-heavy- tailed for all j E DD....... We define two random variables x and z to have equal tail behavior if it holds for all j that
x is j-heavy tailed <
2. is j-heavy tailed
We found the following relation to Definition 3
Proposition 1 (j-Heavy-Tailednes. induces (g-Heavy-Taileddess Assume that x is j-heavy-tailed
for any Then, x is also fz-heavy-tailed.
The proof can be found in Section A. in the Appendix. The proposition shows that j-heavy- tailedness is more general notion of mu nultivari heavy-tailedness than tg-heavy-talledness, which allows narrow inspection of the tail behavior. More precisely. the new notion allows us to differ- entiate between fully heavy-tailed random variables and mixed-tailed random variables, which are both Ez- fz-heavy-tailed. One can now wonder how the tails, described by the novel notation, behave for NFs, which we answer by the following two results. The first result states that, under mild technical conditions, fully heavy-tailedness of the base distribution is preserved by triangular affine maps.
Proposition 2 (Triangular Affine Maps preserve Fully Heavy- Tailedness). Let z be a fully heavy- and let T be a triangular affine map, that is, p, + +az(zes) )2j with 0j > 0. Then, it holds that T'zz) is also fully heavy-tailed.
tailed random variable that satisfies Assumption [
formal proof can be found in Section A.2 the Appendix. Assumption D is a mild condition on the decay rate of the copula density of z. We explain this condition in more detail and give various examples in Section A.3 of the Appendix.
It is clear that permuting the marginals does not change the heavy-tailedness. Hence, by iterative application of Proposition [ we deduce that affine triangular flows with a fully heavy-tailed base distribution are unable to model mixed tailed distributions. Implicitly, Proposition 2 states that Lipschitz normalizing flow as proposed by Jaini et al.] (2020) is not able to model mixed-tailed distributions. However, the following Theorem that consider as our main result guides towards a flow architecture that able to model target distributions that are mixed-taile adaptive way.
a marginally tail-
Theorem 3 (Learning the correct Tail Behavior). Consider cundom-variably z that is j-light- tailed for j E ([,,...,d) .d) for some < D and j-heavy-tailed for j E {d + I....,D). Then, under the same conditions in Theorem 2 and Proposition 2 it holds that and T(z) have the same tail behavior.
Proof. Since the result combines Theorem Proposition i an evident fashion, we just quickly present a sketch of the proof. First, let us consider j I d. Then it holds for the moment-generating function of x, that
/
e (T)(ess) = / [ (T,(oss) (es ps)(Es))
mxy (A)
J20
TThis Assumption can he found in Section A.2 in the Appendix.
5

---

Under review as a conference paper at ICLR 2022
which has been shown to be bounded for some À > 0 (see the proof of Theorem Jaini et al. 20200). Therefore, x is j- j-light-tailer for all j I d. In the case j > d, we notice that the proof for haaly-tailednnes of T3 (Z,z) involves just the heavy-tailedness of z, and not of any other component of zej. Hence, if 2 is heavy-tailed, then x, = ; (z,ي) is also heavy-tailed, regardless of z<j- Therefore, x is j-heavy-tailed for all > d, which completes the proof. Note that in general we cannot deduce the latter conclusion for light-tailed marginals, i.c. if z; is light-tailed, this does not mean that xj is also light-tailed. This only the case, if all žzj are light-tailed as well.
3.2 MARGINALLY TAIL-AAAPTIVE FLOW (MTAF)
Our main result, Theorem B prompts that if we maintain an ordering of the marginals such that the first marginals are light-tailed and the following are heavy-tailed in each flow step, we retain the marginal tail behavior of the base distribution in the estimated target distribution. This finding motivates the novel NF proposed in this paper. The proposed approach combines research findings from extreme value theory (Embrechts et al.] 2013 Nair al. 2013), recent findings about nor- malizing flows (Jaini et al. 2020] Alexanderson & Henter 20200] Laszkiewicz et al. [202T). and the results presented herein. The proposed m'TAFs consists out of three steps depicted in Figure | and
described in the following:
Step 1: Estimating the marginal tail indices and defining the marginal distributions. For each marginal, i.e. for the marginal distribution zj of each x,. I....,,. D, we use the moments double- bootstrap estimator (Draisma et al., 1999) and the kernel-type bootssrp estimator (Groene- boom et al.) (203)) to assess heavy- tailedees of the data distribution. If both estimators predict a light-tailed distribution, we set the corresponding marginal base distribution q; to be standard nor- mal distributed, i.e. 2j -N(0, 1). Otherwise we set the marginal to the standardized (-distribution with the estimated degree of freedom, i.e. 2, ~ lp..> where ū is the Hill double- poott-bpotttrappp estimator (Danielsson et al., 2001 Qi| 2008). In Section B.1 the Appendix, we present more details about the tail sall-assessment scheme.
Step 2: Defining the base distribution. We construct the base distribution as the mean-field ap- proximation of the marginals, i.e. has the density q(=) = IT Uz (z(2,) with marginal densities 4 defined in step I. Further, to satisfy the assumptions of Theorem 3 we need to permute the marginals such that it holds Zj ~ (N, 1) for j S and 2j ~ tp, for j > d. We apply the same permutation to restructure our data according the base components. To account for tail index esti- mation errors and for more flexible learning, we make the tail indices (i.e. the degrees of freedom of each t-distribution) learnable. That is, we initialize the degree of freedom of the jth marginal with by but adapt the parameter together with the network parameters throughout training.
Step 3: A data-driven permutation scheme. Recall, that vanilla triangular flows employ a permu- tation step after each transformation to enhance the mixing of variables. However, purely random permutations might lead to violation on the ordering of marginals, which is necessary to ensure Theorem 3 Therefore, we permute only within the set of heavy-tailed marginals and within the set of light-tailed marginals, to ensure the validity of Theorem 3 Within these groups one can choose any permutation scheme.
Without loss of generality, we assume that the first d components of z are light-tailed and the re- maining D - d components are heavy-tailed [5) Then, the training objective is to optimize for flow parameters â and degrees of freedom & [Dat1...,, PD) to maximize the log-like likelihood
N
D
L(0,0;X)=
Σ log
II क T- (o))))
II v=d+1
to (1)1 log det Jr, (2001)
j=1
log (2(())
Σ log to (2 log det J'y (zt00)
+
i=d+1
where X := (z) .... (N)) is the data, and π and to the PDF of the standard normal distribution and the standard t-distribution with á degrees of freedom, respectively.
*For details, we refer to the proof of Proposition 2 in the Appendix. 'Oteewiise we permute the marginals as deserihed in Step 2.

---

Under review as a conference paper at ICLR 2022
When applying theoretical results presented in the previous section to the proposed mTAF, can show that it fulfills the desired tail-preserving property, as formalized by the following corollary:
Corollary (Marginal Tail- Addapivee) Under the same assumptions as in Theorem Theoremaand in Propo- sition 2 mTAFs are marginally tail-adaptive, that is, and x (z) have the same tail behavion.
EXPERIMENTAL ANALYSIS ON SYNTHETIC DATA
To investigate the benefits of mTAF we perform an empirical analysis in which we conduct ex- periments synthetic heavy-tailed data. We construct synthetic 16-dimensional data sets and in- vestigate eight different settings: The marginals are chosen to contain h E {1,2, 4, 8) mixtures of two I-distributions, four Gaussian distributions, two mixtures of three Gaussian distributions, and 10 10-/ mixtures of two Gaussian distributions. The degree of freedom of all (-distributions is either 3. In all mixtures all mixture components are weighted equally, means are uniformly sampled from [4, 4]. and
are sampled from [1, 2]. Using Gaussian Copula, we construct a complex joint distribution with mixed-taile marginals. A detailed description of the data set generation can be found in Section B B.2jin the Appendix. Details about hyperparameters can be found in Section [B.4] in the Appendix. We provide PyTorch implementation and the code for all experiments along the submission.
4.1 MODEL ACCURACY
In first experiment, we compare the performance of 5 different MAFs: vanilla flow with z ~ (O, I). the TAF, a variation of TAF (TAF(E),... .D)) in which all marginals have their own independent degree of freedom (i.e. 2j l2, (0, 1) with trainable degrees of freedom é and mTAF as described in Section 3.2 with learnable and fixed tail indices, where the letter is denoted as m mTAFF
(fixed P))
Table summarizes the test losses archived by the five models in the eight different settings. The results demonstrate that mTAF can efficiently learn multivariate heavy-tailed distribution, whereas the vanilla approach, i.e. fixing z ~ (0,0 I), is prone to optimization instabilities (as can be seen by high standard deviations) and leads to higher values of the negative log-likelihood. Moreover, TAF performs worse than mTAAF (and is also less stable) which could be contributed to having only one joint degree of freedom to model the tailedness of all marginals. Using independent degrees of freedom however only leads to slight improvements, illustrating the necessity of the proposed permutation scheme of mTA. The performance increase is especially severe for marginals with small tails index, i.e., when the degree of freedom of the marginals of the data aatta-ennrratin distribution V = 2. Fixing the tail indices to the estimated values already leads to great results, which can further be increased by adapting the parameters during training.
4.2 GENERATING TAIL EVENTS
An often-used property of NFs is the diffeomorphic relation between base and target distribution. which can be used, for instance, for image interpolation or for constructing counterfactualg] In this work, we present another practical advantage, which we motivate by the derived theory, namely the generation of tail samples. One generic way to generate tail samples in the target space is to generate tail samples in the base space, which are then pushed towards the target space. However, in this work we propose to solve more specific problem, which is the generation of samples that are from the j- j-tails, i.c. samples from the joint distribution o x with the jth component x, being from the tail of the jth marginal. Recall that in the proof of Proposition we show that if z, is heavy- tailed, then T, (2<,) resulting from applying one affine triangular flow layer is also heavy-tailed, independently of Zzj. Inspired by this result, we hypothesize that we generate multivariate samples x with heavy j-tails by sampling tail events of z,. complementing them with remaining components žyj sampled from Zyjz and pushing the resulting = through the flow, which retains the j
However, since we apply permutation after each layer, to sample from the
by interpolating the base distrihution and pushing them forward using the NF Kingma & Dhariwal
2018).
by applying gradient ascent in the hase space Dombrowski
2021

---

Under review as a conference paper at ICLR 2022
Table 1: Average loss (lower is better) on the synthetic data in various settings over 10 trials. The error bars represent one standard deviation and the numbers in brackets denote the number of crashed runs. Models that are significantly better than the other models (according to one-sided (-test) are highlighted.
number of heavy-tailed components h
8
Vanilla
35.68 + 1.92
36.68 = 1.77
38.19 1.71
45.78 + 13.91
TAF
36.29 0.61 (1)
35.54 + 1.08 (2)
36.04 0.36
38.33 + 0.40
TAF(O ip)
35.23 + 0.76
35.69 +: 0.44 (1)
36.44 + 0.58
38.63 + 0.36
mTAF (fixed 0)
33.48 + 0.04
34.17 0.07
35,46
0.09
38.52 + 0.36
mTAF
33.48 = 0.04
34.11 = 0.06
35.39 + 0.13
38.44 = 0.28
3
Vanilla
33,47 + 0.20
33.80 0.19
34.43 0.22
36.09 = 0.13
TAF
33.81 + 0.37
34.16 0.45
34.48 + 0.20
35.75 + 0.09
TAF(E1..... DD)
33.54 = 0.17
33.83 0.19
34.63 0.58
35.80 = 0.15
TTAF (fixed D)
33.34 0.04
33.66 0.04
34.28 0.07
35.80 = 0.09
mTAF
33.31 = 0.05
33.63 = 0.04
34.21 0.04
35.63 = 0.08
0.14
0.12
0.10
0.08
0.06
0.04
0.02
10
10
20
20
20
Figure 2: Visualizing the 16th marginal of samples resulting from the tail generation procedure from Section 4.2 in orange in the settings h E {1,2,4,88 and v = (from left to right). In blue, we visualize samples from the true distribution.
jth marginal of the target distribution, we need to employ the above procedure with tail samples of Ax+())> where k {1...., D} + {I...., is the permutation mapping resulting from applying all permutation layers, and - is its inverse.
For all models trained on the data sets with v = 2, we generate 000 samples of the heavy-tailed variable Zg-1(16) and consider the 50 with largest absolute value as tail samples. In Figure visualize the results for mTAF: We depict the 16th component X16 of the samples generated by the described procedure for the distributions with different amounts of heavy-tailed components. becomes clear that the sampling procedure indeed results only in samples from the joint distribution where the 16th component stems from the tails of the corresponding marginal. Surprisingly, even in the case in which 8 heavy-tailed components are permuted (i.e. h = 8) the sampling procedure succeeds by ignoring the body of the distribution and generating exclusively tail samples in the 16th component. As further proof of concept, we demonstrate in Figure 5 in Section B.3 in the Appendix that sampling tail events from other marginals than Zz-1(16) does not result in tail events in the 16th component of Hence, this result substantiates "responsible" factor for generating j-tail samples. Furthermore, we rerun the sampling procedure with vanilla flow and TAF, which visualize in Figure While TAF is able to generate
intuition that Zg-1G) is the
8

---

Under review as a conference paper at ICLR 2022
Vanilla
TAF
Vanilla
TAF
0.075
0.050
0.050
0.025
0.025
0.000
0.000
-250
250
-20
200
200
10
(a) and
(b) and 8.
Figure 3: Applying the tail sample generation procedure form Section 4.2 with vanilla flow and a TAF in two different settings. Samples from the true distribution are visualized in blue and flow samples are orange.
reasonable tail samples comparable to m TAF, the vanilla flow is generating extreme outliers, and is thus, not able to efficiently sample from the tails.
5 CONCLUSION AND FUTURE RESEARCH DIRECTIONS
In this work, deepen the mathematical understanding of the tail behavior of triangular flows. We note that the distribution we want model may have heavy- as well as light-tailed marginals, and show how the marginal tail behavior of the target distribution of the flow relates to the tail behavior of its base distribution. Based on these theoretical findings we propose a new algorithm, which we refer to mTAF. In partieular, estimates of the target, and employ data-driven permutation scheme that guarantees the correct tail behavior of the target distribution. An in-depth empirical analysis on synthetic data with heavy- and light-tailed marginals shows that mAA benefits in terms of estimation performance and robustness of the training. Moreover, we provide sampling strategy inspired by out theory that allows to generate joint samples from the target distribution with a customizable tail behavior. In summary, we introduce novel way impose an inductive bias normalizing flows that allows model the tails of the target distribution.
initialize trainable base distribution based statistical tail
Yet, we think that there exists much potential for interesting follow-up works. For instance, we are aware that most SOTA architectures are indeed non-affine, such as NSF (Durkan et al. (2019), which are based on monotone quuional-qaadratic splines, or SOS (Jaini al.] 2019), which are based on strictly increasing polynomials. However, similarly to (Jaini et al. (020). trade-off between a flexible base distributions and the complexity of the flow leads to simpler and more efficient models. This, and the investigation the impact of mTAF real world data remains subject of further future studies. Furthermore, the attentive might have noticed that the proposed mTAF employs restricted set of permutations, which limits its generality. One possible solution for this issue is to employ more expressive base distributions that introduce marginal dependencies, which extends the generality of mTAF, while preserving its ability be marginally tail-adaptive. For modeling these more expressive base distributions it is evident to study copula theory, which allows grounded treatment of various kinds of multivariate dependency concepts, even beyond the scope of pearson correlation, such as tail dependencies (Joe) (004)). In addition, we believe that mTAFss and further improvements of them, bear lots of potential for sampling methods, which benefit from the whole range of samples from the target distribution, including tail samples. Generation of such samples might improve methods for simulation basedd inference (Cranmer et al. (220)) such as VAEs and are essential for the analysis of extremes such in weather forecasting.
hypothesize that the
Last but not least, we want to highlight that our theory is not limited to maximum likelihood training and one could think of various other training paradigms, for instance based on f-divergences. Us- ing alternative optimization metrics, we can potentially put more emphasize on the learning of the correct tail behavior.

---

Under review as a conference paper at ICLR 2022
Reproducibility Statement This work concentrates the mathematical understanding of nor- malizing flows, and in doing so, includes technical proofs, which might on abstract assumptions. Nonetheless, we put our best effort in making all the proofs and the derived theory in Section AA.] and A.2 precise, accessible, and correct. To avoid misconceptions, we dedicate Section A.3 to the explanation and clarification of Assumption We provide documented source code including code execution instructions, which allows the reproducibility of our empirical results. Additionally, all algorithms and hyperparameters are presented in Section B
REFERENCES
Simon Alexanderson and Gustav Eje Henter. Robust model training and generalisation with studen- tising flows. In ICML Workshop Invertible Neural Networks, Normalizing Flows, and Explicit Likelihood Models, 2020.
Jens Behrmann, Will Grathwohl, Ricky TQ Chen, David Duvenaud, and Jörn-Henrie Jacobsen. Invertible residual networks. International Conference on Machine Learning, pp. 573-582. PMLR, 2019.
Jens Behrmann, Paul Vicol, Kuan-Chieh Wang, Roger Grosse, and Joern-Henrik Jacobsen. Under- standing and mitigating exploding inverses in invertible neural networks. In Proceedings of The 24th International Conference on Artificial Intellience and Statistics, volume 130, pp. 1792- 1800, Virtual Event, 2021.
Ricky T. Q. Chen, Jens Behrmann, David K Duvenaud, and Joern-Henrik Jacobsen. Residual flows for invertible generative modeling. In Advances in Neural Information Processing Systems, vol- ume 32, pp. 9913-9923, Vancouver, BC. Canada, 2019.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differ- ential equations. In Advances in Neural Information Processing Systems, volume 31, Montreal, Canada, 2018.
Kyle Cranmer, Johann Brehmer, and Gilles Louppe. The frontier of simulation-based inference. Proceedings of the National Academy of Sciences, 1 117549):30055-3006,, 2020.
Sandor Csorgo, Paul Deheuvels, and David Mason. Kernel Estimates of the Tail Index of a Distri- bution. The Annals of Statistics, 13(3):1050 1077, 1985.
Danielsson, L. de Haan, L. Peng, and C.G. de Vries. Using a bootstrap method to choose the sample fraction in tail index estimation. Journal of Multivariate Analysis, 76(2):226-248, 2001.
A. L. M. Dekkers, J. H. J. Einmahl, and L. De Haan. A moment estimator for the index of an extreme-value distribution. The Annals of Statistics, 17(4):183311855, 1989.
Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: non-linear independent components es- timation. In 3rd International Conference on Learning Representations, ICLR Workshop Track Proceedings, San Diego, CA, USA, 2015.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. In International Conference Learning Representations, 2017.
Ann-Kathrin Dombrowski, Jan E Gerken, and Pan Kessel. Diffcomorphic explanations with normal- izing flows. In ICML Workshop on Invertible Neural Networks, Normalizing Flows, and Explicit Likelihood Models, 2021.
Gerrit Draisma, Laurens de Haan, Liang Peng, and T Themido Pereira. A bootstrap-based method to achieve optimality in estimating the extreme-value index. Extremes, 2(4):367-404, 1999.
Conor Durkan, Artur Bekasov, Iain Murray, and George Papamakarios. Neural spline flows. Ad- vances Neural Information Processing Systems, 32:7511-7522, 2019.
Conor Durkan, Artur Bekasov, Iain Murray, and George Papamakarios. nflows: normalizing flows in PyTorch, 2020.
10

---

Under review as a conference paper at ICLR 2022
Paul Embrechts, Claudia Klüppelberg, and Thomas Mikosch. Modelling extremal events: for insur- ance and finance. volume 33. Springer Science Business Media, 2013.
Xavier Gabaix. Zipf's s law for cities: An explanation. The Quarterly Journal of Economics, 114(3): 739-767, 1999.
Xavier Gabaix, Parameswaran Gopikrishnan, Vasiliki Plerou, and H Eugene Stanley. A theory of power-law distributions in financial market fluctuations. Nature, 423(6937) 2003.
Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. Ffjord:
Free-form continuous dynamics for scalable reversible generative models. In International Con- ference on Learning Representations, New Orleans, LA, USA, 2019.
Groeneboom, H.P. Lopuhaä, and P.P. de Wolf. Kernel-type estimators for the extreme value index. The Annals of Statistics, 31(6):1956 1995, 2003.
Bruce M Hill. A simple general approach to inference about the tail of distribution. The annals of statistics, pp. 1163-1174, 1975.
Marius Hofert, Ivan Kojadinovie, Martin Maechler, and Jun Yan. Elements of Copula Modeling with R. Springer Use R! Series, 2018.
Grant Van Horn and Pictro Perona. The devil is in the tails: Fine-grained classification in the wild. arXiv preprint 17Xv:07709 2017.
Maxim
Asymptotics of inverse of normal
cdf. Mathematics Stack Exchange. U.L.thtpp//mmii...eccccccccooooo (version:
2018-10-22).
Chin-We Wei Huang, David Krueger, Alexandre Lacoste, and Aaron Courville. Neural autoregressive flows. In International Conference on Machine Learning, pp. 2078-2087. PMLR, 2018.
Jörn-Henrik Jacobsen, Amoll Smeulders, and Edouard Oyallon. i-revnet: Deep invertible networks. In International Conference on Learning Representations, Vancouver, Canada, 2018.
Priyank Jaini, Kira A Selby, and Yaoliang Yu. Sum-of-squares polynomial flow. In International Conference on Machine Learning, pp. 3009-3018, PMLR, 2019.
Priyank Jaini, Ivan Kobyzev, Yaoliang Yu, and Marcus Brubaker. Tails of Lipschitz triangular flows. In Proceedings of the 37th International Conference on Machine Learning, volume 119, pp. 4673-4681, Virtual Event, 2020.
Harry Joc. Dependence modeling with copulas. CRC press, 2014.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. In Advances in Neural Information Processing Systems, volume 31, 2018
J. Lars Kirkby, Dang Hai Nguyen, and Duy Nguyen. Moments of student's t-distribution: A unified approach. Computation Theory eJournal, 2019.
Ivan Kobyzev, Simon Prince, and Marcus Brubaker. Normalizing flows: An introduction and review of current methods. In IEEE Transactions on Pattern Analysis and Machine Intelligence. IEEE, 2020.
Eugene Koonin, Yuri Wolf, and Georgy Karev. Power Laws, Scale-Free Networks and Genome Biology. 01 2006.
Mike Laszkiewicz. Johannes Lederer, and Asja Fischer. Copula-based normalizing flows. In ICML Workshop on Invertible Neural Networks, Normalizing Flows, and Explicit Likelihood Models, 2021.
John Liu. Seymour Lipschutz, and Murray Spiegel. Schaum's Outline of Mathematical Handbook of Formulas and Tables, 4th Edition. McGraw-Hill, 2012.

---

Under review as a conference paper at ICLR 2022
Jayakrishnan Nair, Adam Wierman, and Bert Zwart. The fundamentals of heavy-tails: Properties, emergence, and identification. In Proceedings of the ACM SICMETRICS/mmeration Confer- Measurement and Modeling of Computer Systems, pp. 387-388, New York, NY, USA, 2013.
George Papamakarios, Theo Pavlakou, and lain Murray. Masked autoregressive flow for density estimation. Advances Neural Information Processing Systems, volume 30, pp. 2338-2347, Long Beach, CA, USA, 2017.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vitito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. H. Wallach, H. Larochelle, Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024 8035. Curran Associates, Inc., 2019.
Yongcheng Qi. Bootstrap and empirical likelihood methods in extremes. Extremes, 11:81-97, 03 2008.
Sidney Resnick. On the foundations of multivariate heavy-tail analysis. Journal of Applied Proba- bility, 41:191-212, 2004.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Interna- tional conference on machine learning, pp. 1530-1538. PMLR, 2015.
Oren Rippel and Ryan Prescott Adams. High-dimensional probability estimation with deep density models. CoRR,
2013.
E. G. Tabak and Cristina V. Turner. A family of nonparametric density estimation algorithms. Com- munications on Pure and Applied Mathematics, 66(2):145-164 2013.
Ivan Voitalov, Pim van der Hoorn, Remco van der Hofstad, and Dmitri Krioukov. Scale-free net- works well done. Phys. Rev. Research, 1:033034, Oct 2019.
George Kingsley Zipf. Human behavior and the principle of least effort: An introd. to human ecology. Addison- Aadison-Wesley Press, 1949.
A THEORY AND PROOFS
In this Section, we derive and proof all of our theoretical results. We start to present some prelimi- nary results in Section A. which help us providing the proof of our main result in Section In Section A.3 we illuminate the technical Assumption and provide examples and intuitions.
A.1 PRELIMINARY THEORETICAL RESULTS
Proposition 3. Let x E R be a random variable. Then, it holds that x has tail index less or equal than a iff for any B > and any C > 0 it
J 1e1 = 00
Proof. Let first assume that has tail index at most a. Then, according to Definition we know that for any B > and C > 0
x = 1 = f | p(x)d = I JMSSC p())d( I IMp (el%p(x)de
$ CC¹ C
Jusc p(a)dx + 1 ploydd
12

---

Under review as a conference paper at ICLR 2022
Since assume be continuous, bound p the compact interval [-C, CJ2 and hence, the first above integral must be bounded. Therefore, it
f M p(r)dr
To prove the back-direction, let us consider a 3 >> and C' > O. Very similarly to the forward-proof, can now that
S 111S p())dd
-08 S |a|sC
p(z)ddx + / >c
|xr p())dd
[1[]]]
where the second equality follows from the finiteness of the integral. Since this follows for any B a, x must have tail index a or less.
This simple result demonstrates that the tail index, as indicated by the name, depends on the tail of the distribution, i.c. on the PDFs behavior for large values [x] >> C. This fact motivates why maximum likelihood estimations of the tail index, which depend on the whole distribution are biased. For more elaborate details, we refer Section in Nair al. (2013).
In a similar fashion to the previous result, the next technical lemma states that unboundedness of the comment-generating function is due the unboundedness of the integrand for tail events, i.e. for > 2 This little lemma turns out to be useful in the proof of Proposition 2
Lemma 1. Let be heavy-tailed tailed. Then it holds for any z" E R and that
f e² Pa(edd =0
Proof. Since z E IR is heavy-tailed, we know that for all X >>0 O
oo = =mz(A)= = / xe pp(x2))
[ p==2a(2)dz I palpa(e)de
Jest
S Fa(=")e>e" + +1 pa(xd2)
where F2 is the CDF of z. The last inequality follows from the fact that exp(Az) monotonic increasing in Since the first summand is bounded, it follows that the second summand must be unbounded. This completes the proof.
Recall that in Proposition I we state that j j-heavy-tailednee induces talleeey-ss In the following, we provide a formal proof of this result.
Proof of Proposition For this proof, employ the equivalent definition of heavy-tailedness of Xj via the decay rate of its distribution function (see e.g. Lemma 1.1. in Nair et al. (013)) i.e.
1- F=F5(
limsup
for all À > 0
-
25 +00
where F, is the CDF of x,. Since 2j < || for all x E R.,, we can conclude that Ff 2 (a) for a E R. Therefore,
F |x||| (a)
1- (a)
+ 2o for + 00
Aa
Aa
According to the equivalent definition in tailed.
3). |x|| heavy- which proves that x is & -eaayy
13

---

Under review as a conference paper at ICLR 2022
The following is well-known implication of the change of variables formula and the integration rule by substitution, which we are going to apply in the subsequent proofs.
Lemma (Substitution in the Moment Moment-Generann Generating Function). Let be a diffeomorphism such that T(z) for some random variables x, R.. Then, can rewrite
J e² p(x)dx
/ g
For completeness, give a brief proof of this result.
Proof. Using change variables formula, see we can write
[ p(= f Av ( (x) JT 1(x) dr
Now, we can rewrite exp(Ar) exp(AT and substitute = = Integration by substi- tution completes the proof.
Next, we present how we can use copulae to reformulate a multivariate PDF.
Definition (Copula), A copula is a multivariate distribution with cumulative distribution function funiitttionfunctioon (CDF) C: [0. 1]" [0, 1] that has standard uniform marginals, i.e. the marginals Cj of C satisfy C ~ UO, 1].
Theorem (Sklar's Theorem). Taken from Hofert et al. (2018)
1. For any D-dimensional CDF F with marginal CDFs Fi...., FD, there exists copula C
such that
F(z) C(F())) FD(ED))
(4)
for all 2 E R.D. The copula uniquely defined on U = nT Tm(FF), where Tm(F;) is the image of Fj. For all U it is given by
C(u) F F(F (u1), FD F
where F are the right-inverser of Fj.
2. Conversely, given any D-dimensional copula C and marginal CDFs F1, .D. funcntiion as defined in 4 is a D-dimensional CDF with marginals F1,..., FD.
Therefore, if F absolutely continuous, we can differentiate 44 to obtain the PDF
D
((2) = c(F((z), FD((D))
II (T()))
f=1
where e denotes the PDF of the copula C.
Lastly, we present the following asymptotic behavior of the inverse CDF of standard Gaussian distribution, which we use Section A.3 to explain Assumption [
Lemma 3 (Asymptotic Behavior of + (1 y)). Denote by the CDF of standard Gaussian
distribution. Then, it holds for the inverse of db that
Φ Φ - y) ~ / 1 22ggg)) for y - +
Proof. First, note that
+ (x)
1
erf
x²
2
x/2
"The idea of the proof is due
14

---

Under review as a conference paper at ICLR 2022
which is well-known asymptotic (Liu et al. 2012). Here, erf denotes the iiror-function Rearrang- ing terms gives
x²
log(1 - (x)) - log a /v 2x
as x + 00
2
2
Finally, we can invert the above asymptotic equation to obtain
Φ- (y) = -2 - 8)
for + 1
or equivalently
Φ (1 y) v 6/2 2log(y)
for y
A.2 PROOF OF THE MAIN RESULT
The proof of Proposition 2 relies on lower-bounding the noment-generating function of each marginal Xy. In order to derive such a bound of multivariate integral, we rewrite the joint dis- tributions 4sy using their copula densities:
(s - (F (z,2,)
II (((2)) i<j
for any j E (1..... D} and for corresponding copula density oj. Our proof relies on the following technical condition on the decay rate of the copula densities.
Assumption (Bounding the Marginal Decay Rate of the Copula Densities). For all j E {I...., D} and O there exists a compact set S RR-1 with positive (Lebesgue- Jmass, constant 2j > 0, scaling constant O, and function f(zez) < <>o(ze;) for zej E & such that
(F((()))) F((z)) 2 se" ((
for 2j > 2 and zej E
(5)
where Cj is the copula density of 9:3
This assumption sets a bound on the decay rate of the copula density with respect to 2j. We clarify this assumption in Section A.3 with additional examples.
Now, we preliminaries to prove Proposition 2
Proof of Proposition 2 therefore
We start by considering the case j = 1. In this case it is μ + oz and
mx, (A)
Ar p
e Mlnntora)) qn (2(2)d2 (Lemma)
Ann / 201 (z(z))d11.
Defining X' = Ao of zı, see Definition Therefore, mx, (A) oo for all X > 0, which proves the heavy-tailedness of x1.
that the last integral is unbounded due to the heavy-tailedness
Next, we consider the case j > 1. Again, we examine the moment generating function of x,. Define the jth canonical basis vector 2y =
0, ,0.....0.1..00 O.....
Then
An "n]"d(xxdd (LOTUS)
mxx = m.J x
Rg
(T:(ssiss)) (())dz (Lemma
RD
Aulzzc) cullzc)) qe
euliess / co(les) q (25s dzzdzzz. (d JR
(6)
Note that for the sake of clarity, leave out the index in μı and o3-
15

---

Under review as a conference paper at ICLR 2022
Using Sklar's Theorem (Theorem 4). we can write any joint PDF as the product of marginals and copula density oj such that
4s ( (Fi(z)). 1z(2))) a=))
icj
We plug 7 into (6) to obtain
mxx (A) Aulzed gerles) /
...., Fy (5 (zzddze
2 [ (eulecc)gsslees))
Fz(2))) cdolecs) ( dzzdzes (
(8)
since all quantities within the integral are positive. Using Assumption we can bound the inner integral of the above equation, which we denote by A(zes); and get
A(zz)) 2
(Nr(Zz))) ((ses))= - ( (2z)
2,22]
4i (2j)
(define A' >=>=)((zs)) f(=es)
25223
for all 2c E
due to the heavy-tailedness of 2j and Lemma Since is compact, μr and q are both continuous, and q is positive, deduce that (=<Ap
is lower-bounded (by constant larger than 0) in S. Therefore, employing (8) and using that has positive mass, we can lower-bound the
function by co, which proves the heavy-tailedness of xj. In summary, is j j-heavy-tailed for all j E (1...., D).
A.3 NOTES ON ASSUMPTION I
Assumption I might look troublesome first sight, but we will illustrate in this section that the condition is indeed very reasonable. We will show how to verify it in simple examples, and we will introduce a simpler, more intuitive sufficient condition for it.
First of all, let us present a restricted but more intuitive version of Assumption 1
Assumption (Simplification of Assumption D.. For all j E (1...., D) and X 0 holds for S = [a, 6-:1 that there exist constants 2 and > 0 such that
F,), 2 se" (Aa-c)=; for 2j > 23 and zej E
(9)
where Cj is the copula density of 953 Da is a lower bound of (o(Ze)) > is small such that Aa - 0.
Let us summarize the simplifications that we make in Assumption First of all, restricted S to be a closed cube [a, b)š-1, which is obviously specific instant of a compact set with positive mass. Further, we assumed a to be continuous, and thus, (a(22)) must be lower-bounded in S. This allows us to replace the function by the constant X for arbitrary small E > 0.
After giving this simplified sufficient condition, we provide some intuition by presenting some ex- amples where Assumption[]hont true.
Example (Independent Variables). Consider a random variable z with independent components, Lee. q(z) = I1 (2()). Then, the associated copula is the independence copula (Figure 4 which is uniform random distribution on [0, 1J Therefore is c(Fri FFpp)),....FF(zp))
for all R and Assumption 2 follows immediately since sexp(((> - =))z) O for 2j + oc.
Example 2 (Bounded Copula Density). Consider lower-bounded copula density, i.e. there exists a lower bound a > such that
c(ui...., uD) 2 a for all E []
Again, the validity of Assumption
2 in this setting is clear.
16

---

Under review as a conference paper at ICLR 2022
1.0
1.0
0.8
0.8
0.4
0.4
0.0
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
(a) 2-dimensional Independence copula: (b) 2-dimensional Gaussian copula with cor- relation f122
c(21, 22) 1.
0.7
Furthermore, this assumption is obviously not limited to bounded copula densities but also holds for
copula densities that converge to 0 but whose decay rate 2j is lower-bounded by (9) To visualize
the intuition, consider the 2-dimensional copula density of a Gaussian copula in Figure 4b Imagine
fixing S such that Fi (21) [0.5, 0.75). which compact for continuous Fi. Then (
bounds the
decay rate within the "tube" [0.5, 0.75] if we consider 22 00, i.e. if F2(22) + 1. Next, we show how we can formally prove the assumption for Gaussian copulae.
Example 3 (Gaussian Copula). The Gaussian copula with correlation matrix R E RDD has density function
c(u)
exp
1 (
(ui).... (
( -1)(00-(
Vdlet R
(10)
where I E DDDD is the identity matrix, 1- is the inverse CDF the univariate standard Gaussian distribution, and u E [0, PP In the following, we consider Assumption for = =D.
Note that S is assumed to be compact therefore Fz(2z) are all upper and lower-bounded by some value for zcj E S. This makes all polynomials of them also bounded. Hence, we can find constants a' b,, e' such that
lower-bound the term within the exponential in (0
2 (FF(=))F(Ez))))
+ 1/ +V +d. (FF((z))
Plugging the above into 100 gives
((Fi(z)...... Fb(=D)) 2 ddet R
1
exp (a''0 -140-11--- (Fb + ccd )
-a (FF-((F)(z))) + 1x - (for some b)
exp
-aa|o-1 (Fb(zz)) 16/2 11 (Fb(z)))
exp
exp
-la|0-110 (FF(==)))))))
(11)
where the last line applies f 2 0, which is satisfied 2² is large enougl 10
Next, we use the asymptotic relation from Lemma
① --1 - log En(en))
for instance if 2* is larger than the median of zD
17

---

Under review as a conference paper at ICLR 2022
Hence, for each O there exists 2" large enough such that
① (Fb(zD))
<E
2oo FD(ED))
which can be rearranged to Plugging the above into 11 we obtain
(FD(zD)) <v 22oo FD(ED)) +
((F((z).. Fo((D)) 2 exp 2/0 (1+ log(1 Fo(2D))
= =(1-Fb(ZD) 2
(12)
where we define à == 2|a((1 + E)². Hence, we are left to lower-bound 12 range of heavy-tailed marginal distributions such as:
which we can do for
1. Pareto distribution: The Pareto distribution with shape parameter has CDF
F(z) =
Therefore,
(1- (D(=)) 1 2/a/a
2n
= exp( 2āa log(=D))
2 exp( -
for 2D 2 e.
2. Scale invariant distributions: Following the same argument as above, each distribution with CDF
F(2) = 1- ba for 2 >>> 2
(13)
for constants b, a. > 0 satisfies the bound
(1 FD(2D)) 2 2
Each distribution with CDF in 13 is scale- invariant distribution (see e.g. Theorem 2.1 in Nair et al. (2013)).
3. Exponentially decaying distributions: Every distribution that satisfies the bound
F(z) S - exp(2z-
for 2" and a > In this case, we can again show that Assumption 7 ii valid:
- FD(D))
Lastly, we want to emphasize that Corollary is derived by an iterative application of Theo-
rem Therefore, Assumption must hold for all "flow steps", i.c. T = ( o (
we need to ensure validity of Assumption for (0) 2, (1) = (2), (2) (2)
( ...,, (4-1) = ( .
In Example we show that this assumption holds true for (0) since we define our base distribution under the mean-field assumption. Furthermore, random variable with bounded copula density, then the transformed random variable must also have bounded copula den- sity. Hence, Assumption [ would be valid for all "flow steps" (see Example 2 when starting with mean-field base distribution (=)) I1 (zz).. However, this conjecture needs to be studied in further research.
we conjecture that if we apply Lipschitz-continuoud diffeomorphism
18

---

Under review as a conference paper at ICLR 2022
B ALGORITHMS AND COMPUTATIONAL DETAILS
B.1 TAIL ESTIMATION
Many heavy-tailed distributions can be characterized by their tail index, which include the set of regularly varying distributions [T such as the t-distribution, the Pareto distribution, and many more. However, as already shown in Section 2.1 the tail index does not depend on the body of the distri- bution, and hence, non-tail samples must typically be discarded for tail index estimation. Although a variety of estimators for the tail index exist, such as the Hill estimator (Hill, 1975). the moment estimator (Dekkers et al. 1989,, and kernel-based estimators (Csorgo et al. 1985). none of them is considered to as superior in all settings. A major issue of all mentioned estimators is that they based on threshold defining the tail, i.c. the needs to input statements of the form "the k largest samples are considered to be tail events". Even though there exist some strategies to find k, there is none working robustly in all settings. In fact, one construct simple counter examples for all estimators that lead to failures of tail estimation. This is due to undesired properties of the esti- mators, such as the lack of translation invariance of the Hills estimator (while the tail index clearly is location invariant). We refer to Section 9 in Nair et al. (2013) for a detailed text book treatment of tail index estimation. In summary, robust tail estimation still considered an unsolved problem, which forces practitioners to consider multiple estimators to make well-founded decision. Fur- thermore, we note that the Hills estimator can only be applied for regularly varying distributions, which excludes the application of the Hills estimator to classify light-tailed distributions. In con- trast, the moments and the kernel estimator can both be applied to identify heavy-tailed marginals and to assess tail index.
To implement the tail assessment scheme, see Step I of the proposed method Section 3.2 we
found that Algorithm| [ works fine in classifying the correct tail behavior and giving a decent ini- tialization for the tail indices. We reused the code by Voitalov et al. (2019). which implements all tail estimation procedures" from Algorithm. Notice that we clip the tail index by 20, i.c. the algorithm classifies marginals with a tail index larger than 20 as light-tailed, which prevents a too restrictive set of allowed permutations, see Step in Section 3.2 For illustration, consider the following simple example. Assume that we estimate all except of one marginal to be heavy-tailed. Then, the first component the flow never allowed permute with other components, since they are classified as heavy-tailed. Hence, the mixing of the first component would be severely restricted. Further, since large tail indices indicate less heavy-taile distribution, is reasonable clip the tail index at some threshold.
Algorithm Marginal Tail Estimation
Require: Data_val
tail.est
for j in {L..... D} do
marginal Data val[:.
moments <- moments ests marginal]]
D O if marginal| is estimated to be light-tailed
kernel f- [
> 0 if marginal) is estimated be light-tailed
if moments==kemell-0) then
tail tail.est.append())
> light-tailed if moments and kernel estimate light-tailed marginal
else
hill + hills_est( marginal) hills.es((marginal))
if hill> 20 then
tail est.append(O)
> light-tailed if hills estimator predicts high tail index
else
tail tail.est.append(hil)
end if
end if
end for
return tail est
see Section in Nair et al. (013) for further details inncluding the hyperparameter selection (Danielsson et al.
2001
Qi
2008
Draisma et al.
1999
Groene-
hoom al. 2003
19

---

Under review as a conference paper at ICLR 2022
0.175
0.125
0.100
0.050
0.025
0.000
-10
10
-10 10
-10
10
-10
10
Figure 5: Applying the sample generation procedure from Section 4.2 but with tail samples from other base marginals and visualizing the 16th target marginal. Samples from the true distribution visualized blue and flow samples are orange. We sample from the 3rd, 7th, 11th, and the 15th base marginal, respectively (from left to right). The visualized setting is and h 8.
B.2 SYNTHETIC DATA GENERATION
Our generation the synthetic distribution consists steps: Generating the marginal distri- butions, 2. Defining copula distribution, 3. Combining the marginal and the copula obtain multivariate joint distribution.
Generating the marginal distributions. The first four marginals are defined to be Gaussians. The following marginals are two 2-mixtures of two Gaussians and two mixtures of three Gaussians. The last 21 4,8) components are mixture of t-distributions and the remaining marginals are again mixtures of two Gaussians. All mixtures have equal weight for each mixture component and all means and -eviaadddeviations are randomized. Means are constructed by uniformly sampling from [-4 4). whereas standard-deviatioon are sampled from [1.
Defining a copula distribution. Recall, a Gaussian copula (T] is parameterized by a correlation
matrix R. To generate R, randomly sample 16 different pairs (i, {l,..., 16)2 with and set the corresponding entry of the correlation matrix
R = 0.25
Obtaining a joint distribution. Lastly, we combine the marginals with the Gaussian copula using Sklar's Theorem 4 This gives multivariate distribution with specified and complex marginals with a dependency structure given by the copula, see Joe (2014) for more details on the induced dependencies.
To construct the training, test, and validation sets 15.000, 500, and 5 000 samples from this distri- bution are sampled, respectively.
B.3 SAMPLING MARGINALLY TAIL EVENTS
In the main text, we have seen that the proposed tail generation procedure is successful in generating samples that are marginally the tail of some specified marginal (Figure(2). In Figure present the 16th marginal when repeating the same experiment but with marginal tail samples from other marginals z, than the proposed Z)16(
B.4 HYPERPARAMETERS AND EXPERIMENTAL DETAILS
In all our experiments, constructed model using L 5 MAF flow layers, where each emerging neural networks contains hidden layer and 200 hidden neurons. The tail index TAF is initially set to & 10. As permutation scheme, employ random permutations in the vanilla flow and
20

---

Under review as a conference paper at ICLR 2022
TAF, and random permutations within the sets of light- and heavy-tailed marginals, respectively, in mTAF. In addition, we applied nomhaloomaaliztiion after each triangular transformation. For optimization,
Adam optimizer with learning rate of 1e 5 and weight-decay of le 6. 6. The learning rates of the tail indices are set to 0.1 for TAF and for mTAA. We train until 200 epochs use early stopping if training gives no improvement after 30 epochs. We use batch size of 256.
Our code is built with PyTorch (Paszke et al. 2019) and we make extensive use of the package rflows (Durkan et al. 2020 implement TAA.. All code provided along the submission.
21


Reviews:
[{"review_id": "fBKF_mFZRN", "reviewer": "Reviewer_pWUG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\n1. Solid and rigorous mathematics foundation \n\nThe theoretical proofs are very helpful and provide a clear insight, which explains the motivation and intuition. \n\n2. Careful discussion about the related work and current limitations\n\nThe authors provide a good review and comparison of the existing works. Also, the limitations and future directions are helpful and insightful. \n\n3. Well-written and easy to follow\n\nThe paper is well-structured and easy to follow. The theoretical proof strongly connects with the experiments, which makes the papers much easier to understand. \n\nWeakness\n\n 1. The novel contribution is marginal \n\nThis work is mainly inspired by Jaini et al, 2020, who proposed to model long-tailed distribution via normalizing flows. Although the theoretical contribution is strong, the new proposed mTAFs did not show a significant improvement, as shown in Table 1, compared with vanilla and TAF.  \n\n2. Lack of baseline methods and comparison \n\nThe SOTA flow models and architectures are not included in the baselines.  Although it is argued by the authors in the potential work, I still believe the comparison is necessary. Many papers have shown that the affine coupling layers, such as RealNVP, have limited expressivity in handling complex distribution. Either light-tailed or heave-tailed distribution would be more challenging. So the worse performance might be due to the limited representation capability of the vanilla flows. \n\n3. Experiments need to be improved with large-scale and high-dimensional datasets\n\nCurrently, only synthetic toy examples are provided to demonstrate the performance. The dimensionality is also low. If the proposed algorithm is able to scale to high-dimensional problems, it would be very helpful to increase the impact. \n", "summary_of_the_paper": "This paper focuses on understanding the tail behavior of normalizing flows through a mathematical and statistical way. Motivated by Jaini et al 2020's work on learning long-tailed distribution via triangular flows, this work proves that the marginal tailedness can be controlled by the tailedness of the marginals of the base distribution in flow-based models. Based on this theoretical insight, the authors propose a new algorithm by leveraging a data-driven permutation scheme to enable a correct tail behavior of the target distribution. \n\n ", "summary_of_the_review": "The paper provides a strong theoretical insight but the experiments and baselines are weak.  \bThe contribution seems limited with a marginal improvement compared with the current work.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "AjVmIGXaECS", "reviewer": "Reviewer_dggh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": " Strengths:\n- The addressed problem is important. Vanilla normalizing flows (such as other deep density estimators) are poor at tail-estimation, which limits their applicability to many problems in science and engineering. \n- The paper is very well written and it serves as a good introduction to both normalizing flows and heavy-tailed distributions. \n- The offered solution is technically sound. \n- The theoretical analysis is sound and convincing.\n\nWeaknesses: \n- The main weakness of the paper is its very limited novelty. All theoretical analysis and methodological improvements are relatively minor modifications of the work in (Jaini, 2020). The present paper does not contain major new ideas.\n- The proposed method is somewhat inelegant as it proposes the use of a separate off-the-shelf tail estimator prior to the flow training. I do agree that it could be the right approach in many applications, however it is a rather obvious idea, not really worth of a top conference publication.", "summary_of_the_paper": "The paper is composed of two main parts: 1) A theoretical section where the authors prove that (Lipschitz) triangular normalizing flows cannot map either heavy- or light-tailed base distributions into target distributions with different tail indices for different marginals; 2) An algorithmic section where the authors introduce a new method for modeling distributions with different tail indices for different marginals.\n\nBoth the theoretical and the algorithmic parts are a straightforward extension of the analysis and methods introduced in the paper \"Tail-adaptive flows\" (Jaini, 2020). ", "summary_of_the_review": "I highly appreciate the clarity and technical soundness  of the paper. However, I cannot recommend acceptance given the very limited novelty.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "AEam6n-fgEp", "reviewer": "Reviewer_ZEHf", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is  The problem of estimating the tail behavior \n\n**Pros**: Modelling tail-phenomena (or rare events) in general is a challenging problem made even more difficult in higher dimensions due to the lack of any definition of heavy/light tails in higher dimension. The problem considered by the authors of capturing tail behavior by normalizing flows and limitations of the chosen architecture for flow layers imposes is an interesting and valid problem. The proposed solution that uses estimators to estimate the tail-coefficient for choice of base distribution is interesting and a nice addition to start the optimization of the $\\nu$ at a favorable location and to be closer to whats needed. Overall, the paper is very well written and easy to follow. The paper develops the idea in a natural and easy to understand manner.\n\n**Cons**: \n\n1)  **Motivation**: I found that the problem under consideration was not properly motivated and this issue lingers throughout the paper essentially making the paper come across as just an extension of Jaini et.al 2020. For example, it is not clear from the paper why capturing tails in variational inference paradigms is of importance. It can be shown that the error in modelling a probability density can be bounded arbitrarily well by learning the density properly on a bounded subset of the support of the density. Thus, what are the drawbacks in the model if it is unable to capture the tail phenomena present in the problem?\n\nThe authors do make statements regarding the limitations of the push-forward density given the choice of base density and the transformation map. However, I believe a more thorough discussion about the implications of these results (both in general and particularly for normalizing flows) and some recipe or ideas to alleviate these problems will help the paper tremendously.\n\n2) **Significance**: Another weakness I believe of the paper itself is the significance itself. The main result of the paper ie that of specifying and ensuring that marginal distributions have the correct tail coefficient is a direct extension of the work of Jaini et.al 2020. In some ways, the optimization problem presented in Jaini et.al already can encompasses the correct marginal tails by optimizing over the $\\nu$ vector. This weakness of significance is further amplified by the lack of strong empirical results. In the present form, the experimental results come across more as proofs-of concept rather than proving strong empirical support. Furthermore, I'd be interested to see the gain in starting the optimization with tail-coefficients estimated using the various estimators vs random initialization and letting the process figure these out. Will the first step of using estimators lead to any significant gains?\n\n**Other comments:**\n- Definition 4 it seems is a bit restrictive as well since it completely side-steps any issues with differences in tail-coefficients. For examplke different ,marginals can be heavy tailed but have different degrees of heaviness. In that case, is it correct to call the two distributions having the same tail behavior?\n\n- It seems in the experiments that if the tail estimator estimates that some marginals are light-tailed, a normal distribution is used. However, again there can be degrees of light-tailedness (see Jaini et.al 2020) eg. uniform vs normal. In these scenarios too, a lighter tailed distribution cannot be pushed-forward to another light-tailed distribution but with higher tail coefficient with lipschitz maps. Thus, the problem of mismatched tails may still persist.", "summary_of_the_paper": "The paper proposes an extension to Tail-adaptive flows for learning the tail behavior of target distributions using normalizing flows. The authors propose to learn the tail behavior by learning flows that match the tail properties of the marginal distributions. They achieve this by using a source distribution consisting of marginal distributions with tail properties matching the target distribution. The tail coefficient of the source distribution is set in a data-driven manner using estimators that can estimate this tail coefficient. ", "summary_of_the_review": "Overall, the paper studies a pertinent and difficult problem. However, in the current form, the present manusript provides only initial proofs-of-concept for potentially interesting ideas. These ideas need to be demonstrated and explored in more detail both in theory and empirically to make the manuscript stronger.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "-gObd5Huwo6", "reviewer": "Reviewer_EStc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths:** I thought the paper was interesting. NFs definitely do have their limitations despite their expressivity, and I don’t think the problem of generating distributions with a mixture of both heavy- and light-tailed marginals has been considered before. The paper provides a more general definition of heavy tailedness that extends existing work and uses it to construct their mTAF method.\n\n**Weaknesses:** That being said, I think the paper still requires a significant amount of work in order to demonstrate the efficacy of mTAF. \n- First, it’s not clear to me when you would run into situations where you want to generate distributions with mixed-tail behavior in the marginals. I understand that it would be desirable to generate distributions with heavy tails, but when do we encounter cases where we would like to do both? I think making this clear would definitely strengthen the paper, and could also guide some downstream evaluation tasks.\n- The experiments were probably the weakest aspect of the paper. mTAF was only evaluated on a synthetic dataset of 16 dimensions, which seems too small (even for tabular datasets commonly used for evaluating NFs). Also, the evaluations conducted in the experiments did not clearly demonstrate the advantage of mTAF over existing methods. For example in Table 1, does mTAF capture both the light-tailed and heavy-tailed components better than TAF/the base method? (this is hard to tell with just a simple average). Additionally, it’s hard to tell the difference between mTAF and TAF in Figures 2 and 3. I think the paper would be much stronger if the authors could find some compelling use cases of the method beyond synthetic Gaussians, and demonstrate that mTAF both captures all marginals more faithfully (via likelihoods) and can generate samples properly in the tails.\n- I also think a big limitation of the method is that mTAF essentially requires separating out the light-tailed marginals from the heavy-tailed marginals (the permutation step where such marginals are grouped into 2 categories). This seems particularly problematic as the real advantage of using NFs is to learn complicated dependencies between all dimensions of the data to best capture the overall density. This is also why I was asking whether there are real-world examples where such mixtures occur, and whether this kind of ``independence assumption’’ makes sense in these scenarios. It seems like mTAF is very restrictive, and I am wondering if maybe that is why it doesn’t significantly outperform TAF and the vanilla baseline.\n\n**Questions:**\n- I’m also curious if the method performs worse relative to conventional flows (e.g. MAF) when the distribution in question is only light-tailed or heavy-tailed. It seems like if the tail index estimator is correct, mTAF should return the correct “tail behavior” of each marginal and generate either a light-tailed or heavy-tailed distribution only. Is this the case? Or does mTAF do a worse job at modeling, say, the light-tailed components, etc?\n- Additionally, the vanilla baseline has pretty high variance and sometimes seems to perform on par with TAF -- would the authors elaborate upon this point?\n\n**Miscellaneous/minor typos:**\n- “Allow [us] to” in Section 2.1\n- “By out theory” in Section 5\n\n", "summary_of_the_paper": "This paper introduces Marginally Tail-Adaptive Flows (mTAFs), which extend existing work on TAFs to better learn a generative model of heavy-tailed distributions. In particular, they propose a new type of normalizing flow (NF) that can learn marginals with mixed-tail behavior.", "summary_of_the_review": "Although the paper extends an existing approach to learn generative models of distributions exhibiting mixed-tail behavior, the paper has a number of weaknesses: (1) it’s not clear when such mixed-tail behavior arises in the real-world; (2) the class of flows considered are quite restrictive (affine, coupled with a permutation that requires light-tailed and heavy-tailed marginals to be split into two consecutive blocks); and (3) the empirical results are lacking: they only provide experiments on datasets of dim=16. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses the performance of normalizing flows in the tail of the distribution. It does this by controlling tail properties in the marginals of the high-dimensional distribution. The paper is well-motivated, and the key theoretical insight has merit. However, the general perspective and methodology appears to be incremental relative to past results. Furthermore, some concerns over correctness remain after discussion with authors. Also, clear baselines and more realistic settings are lacking in the experimental results. Thus, while the paper generally has promising ideas on a pertinent topic, it appears to be not developed enough to merit dissemination.
Decision: Reject