Year: 2021

Abstract:
Learning the tail behavior of a distribution is a notoriously difficult problem. The number of samples from the tail is small, and deep generative models, such as normalizing flows, tend to concentrate on learning the body of the distribution. In this paper, we focus on improving the ability of normalizing flows to correctly capture the tail behavior and, thus, form more accurate models. We prove that the marginal tailedness of a triangular flow can be controlled via the tailedness of the marginals of the base distribution of the normalizing flow. This theoretical insight leads us to a novel type of triangular flows based on learnable base distributions and data-driven permutations. Since the proposed flows preserve marginal tailedness, we call them marginal tail-adaptive flows (mTAFs). An empirical analysis on synthetic data shows that mTAF improves on the robustness and efficiency of vanilla flows and—motivated by our theory—allows to successfully generate tail samples from the distributions. More generally, our experiments affirm that a careful choice of the base distribution is an effective way to introducing inductive biases to normalizing flows.

heading: 1 INTRODUCTION
text: Heavy-tailed distributions are known to occur in various applications in biology, finance, social sciences, and more. Examples for such observations include the length of protein sequences in genomes (Koonin et al., 2006), returns of stocks (Gabaix et al., 2003), or the size of cities (Gabaix, 1999). Applications that are tightly connected to typical deep learning applications include the frequency of class examples in image classification (Horn & Perona, 2017) and the frequency of words (Zipf, 1949) in natural language processing. From a theoretical point of view, this is not surprising since heavy-tailed distributions emerge from several circumstances, including the limiting distribution in the generalized central limit theorem, of a multiplicative process, or as the limit of an extremal process (Nair et al., 2013). Given the frequency of occurrence, developing generative models that allow to learn heavy-tailed distributions is essential.
Normalizing Flows (NFs (Rippel & Adams, 2013; Tabak & Turner, 2013; Dinh et al., 2015; Rezende & Mohamed, 2015)) are a popular class of deep generative models. Despite their success in learning tractable distributions where both sampling and density evaluation can be efficient and exact, their ability to model heavy tailed distributions is known to be limited. Jaini et al. (2020) identified the problem that a range of NFs (e.g. vanilla triangular flows with a Gaussian base distribution) are unable to map a light-tailed distribution to a heavy-tailed distribution. They propose to solve this issue by replacing the Gaussian base distribution by a multivariate t-distribution with one learnable degree of freedom. While this allows to model distributions with a heavy-tailed euclidean norm, we show that modeling multivariate distributions, where some of the marginals are heavy- and some are light-tailed, still poses a problem.
Contributions Our contributions in this work, that extend the results of Jaini et al. (2020), are the following. First, we prove that a triangular affine NF using a base distribution with solely heavytailed marginals is only able to provide a target distribution with just heavy-tailed marginals as well. Consequently, such a NF is not capable of learning distributions with mixed marginal tail behavior. Second, we derive a result that states conditions under which the marginal tailedness of the base distribution can be preserved. Third, based on these theoretical findings, we propose a novel kind of triangular NF that allows to learn distributions with heavy- and light-tailed marginals. The new model is called marginally Tail-Adaptive Flows (mTAFs), and as illustrated in Figure 1, combines
• estimators from extreme value theory (Hill, 1975; Dekkers et al., 1989; Csorgo et al., 1985) to initially assess heavy-tailedness of the target’s marginals;
• a flexible and trainable base distribution based on the estimated tail behavior of the target distribution;
• a new permutation-scheme between flow-layers that ensures the correct tail behavior of the estimated target distribution.
Finally, we conduct an experimental analysis demonstrating the superior performance of the proposed mTAFs in comparison to other flow models. Furthermore, we present a new sample generation scheme, motivated by our theory, which successfully generates joint samples that are from the tails of a specified marginal.
Notational Conventions In the following, we will denote random variables by bold letters, such as x, and its realisations by non-bold letters, x. We use this notation for multivariate and for univariate random variables. Further, we denote the jth component of x by xj , and x≤j or x<j are the first j or j − 1 components of x, respectively. We denote the random variable representing the base distribution by z and the random variable representing the target distribution by x. Further, for notational convenience, we denote the probability density functions (PDFs) of x and z by p and q, whereas marginal PDFs are denoted by pj and qj , respectively. Finally, we assume that both random variables x and z have continuous and positive density on RD, i.e p(x), q(z) > 0 for all x, z ∈ RD, where D is the dimensionality of x and z.

heading: 2 BACKGROUND
text: In this section, we give a brief introduction to heavy-tailed distributions and present needed background knowledge about normalizing flows.

heading: 2.1 HEAVY-TAILED DISTRIBUTIONS
text: Heavy-tailed distributions are distributions that have heavier tails (i.e. decay slower) than the exponential distribution. Loosely speaking, slowly decaying tails allow to model distributions that generate samples, which differ by a large magnitude from the rest of the samples. For a univariate random variable x we define heavy-tailedness via its moment-generating function1: Definition 1 (Heavy-Tailed Random Variables). Consider a random variable x ∈ R with PDF p. Then, we say that x is heavy-tailed if and only if
∀λ > 0 : Ex [ eλx ] =∞ .
The function mp(λ) := Ex[exp(λx)] is known as the moment-generating function of x. Random variables that are not heavy-tailed are said to be light-tailed.
Note that this definition is, strictly speaking, merely a definition for heavy right tails. We say a random variable x ∈ R has heavy left tails if −x has heavy tails according to Definition 1. For simplicity of derivations and w.l.o.g., we proceed with this definition but the derived results can analogously be applied to left tails.
We can assess the degree of tailedness of a distribution. While there are many equivalent notions of the so called tail index, the most straight-forward definition is via the existence of moments: Definition 2 (Tail Index). A random variable x ∈ R with PDF p is said to have tail index2 α if it holds that
Ex[|x|β ] { <∞ , if β < α , =∞ , if β > α .
Since the tail index is tightly related to the decay rate of the PDF, it enables us to assess the degree of heavy-tailedness of a random variable. Therefore, estimation of the tail index became an important objective in extreme value theory and statistical risk assessment (see e.g. Embrechts et al., 2013). Since the existence of the moment does not depend on the “body” of x but only on the tails of x (see Proposition 3 in Section A.1 in the Appendix), estimating the tail index by fitting a full parametric model to all data e.g. via likelihood maximization leads to a biased estimator. Instead, semi-parametric estimators have been developed, which aim to fit a distribution only on the tails. Popular methods for tail estimation include the Hill estimator (Hill, 1975), the moment estimator (Dekkers et al., 1989), and kernel-based estimators (Csorgo et al., 1985). In Section B.1 of the Appendix, we discuss these tail estimators and review some practical issues with these.
An example of a heavy tailed distribution is the standardized t-distribution, which has parameter ν > 0 referred to as the degree of freedom and a density function given by
p(x) := Γ ( ν+1 2 ) √ νπΓ ( ν 2 )(1 + x2 ν )− ν+12 , x ∈ R , where Γ is the Gamma function. It is known that the t-distribution has tail index ν (see e.g. Kirkby et al. (2019) for a detailed reference).
In the multivariate setting, there exist various definitions of heavy-tailedness. For instance Resnick (2004) make use of a definition based on multivariate regular variation. Jaini et al. (2020) define a multivariate random variable x to be heavy-tailed if the `2-norm is heavy-tailed, a property which we refer to as `2-heavy tailed, and which is formally defined as follows:
1One can readily show that this definition is equivalent to the definition, which compares the tails of x to the tails of an exponential distribution. See Section 1 in Nair et al. (2013).
2Notice that the notion of a tail index is only valid for regularly-varying random variables, which are a subclass of heavy-tailed random variables. For the purpose of this work, it is sufficient to consider regularly varying random variables. More details can be found in Nair et al. (2013).
Definition 3 (`2-Heavy-Tailed). Let x ∈ RD be a multivariate random variable. Then, we call x `2-heavy-tailed if it holds that ‖x‖ is univariately heavy-tailed according to Definition 1, where ‖ · ‖ denotes the `2-norm. Otherwise, we call x `2-light-tailed.

heading: 2.2 NORMALIZING FLOWS
text: The fundamental idea behind NFs is based on the change-of-variables formula for probability density functions (PDFs) given in the following theorem. Theorem 1 (Change-of-Variables). Consider random variables x, z ∈ RD and a diffeomorphic map T : RD → RD such that x = T (z). Then, it holds that the PDF of x satisfies
p(x) = q ( T−1(x) )∣∣det JT−1(x)∣∣ ∀x ∈ RD , (1) where JT−1(x) is the Jacobian of T−1 evaluated at x ∈ RD.
This formula allows us to evaluate the possibly intractable PDF of x if we can evaluate both, the PDF of z and T−1(x), and efficiently calculate the Jacobian-determinant det JT−1(x). As T maps z to x, we denote the distribution of z and x as the base and the target distribution, respectively.
To model the PDF of x using NFs, it is common to set the base distribution to a standard normal distribution (i.e., z ∼ N (0, I)) and to employ likelihood maximization to learn a parameterized transformation
Tθ := T (L) θ ◦ · · · ◦ T (1) θ ,
which, yet, remains tractable and diffeomorphic. Masked autoregressive flows (MAFs (Papamakarios et al., 2017)) are one popular architecture, which employ transformations T = (T1, . . . , TD)> of the form Tj(z) := µj(z<j) + exp(σj(z<j))zj for j ∈ {1, . . . , D} , (2) where µj and σj are neural networks, which obtain the first j−1 components of z as input and output a scalar. Composing several transformations of the form (2), we obtain the MAF. The autoregressive form in (2) allows us to efficiently evaluate the Jacobian-Determinant due to the diagonal form of JT (x). A crucial issue of such autoregressive models is that a component xj only depends on the previous outputs x<j and, therefore, they cannot model a causal relationship in which xj causes xi if i < j. This issue can be solved by applying a permutation before each transformation T
(1) θ , . . . , T (L) θ . This is usually a random permutation or the one that reverses the ordering of the components. Therefore, in summary a MAF consists of multiple consecutive layers T (l)θ ◦ P (l), where P (l) ∈ RD×D is a permutation. MAFs belong to the class of triangular flows, which are defined as flows that consist of diffeomorphisms whose jth output only depends on z≤j . Other examples for triangular flows include RealNVP (Dinh et al., 2017), NAF (Huang et al., 2018), and SOS (Jaini et al., 2019). If the triangular maps are affine linear (such as in (2)), we call the resulting flow a triangular affine flow. Further types of NFs include invertible ResNets (Jacobsen et al., 2018; Behrmann et al., 2019; Chen et al., 2019), continuous flows (Chen et al., 2018; Grathwohl et al., 2019), and many more (Kobyzev et al., 2020).
Tail-Adaptive Flows. Jaini et al. (2020) investigated the ability of triangular flows to learn heavytailed distributions. The authors have shown that if a triangular affine flow transforms a `2-lighttailed distribution, such as the multivariate Gaussian distribution, to a `2-heavy-tailed target distribution, then Tθ cannot be Lipschitz continuous. And more explicitly, it holds the following. Theorem 2. (Jaini et al., 2020) Let z be a `2-light-tailed random variable and T be an affine triangular flow such that Tj(z≤j) = µj(z<j) + σj(z<j)zj for all j. If σj is bounded above and µj is Lipschitz for all j, then the transformed variable x is also `2-light-tailed.
Furthermore, the authors prove that any triangular mapping from an elliptical distribution to a heavier-tailed elliptical distribution must have an unbounded Jacobian-determinant. Clearly, these results illuminate that learning a heavy-tailed distribution using NFs leads to non-Lipschitz transformations and unbounded Jacobians, which inevitably affects training robustness (Behrmann et al., 2021). Motivated by these result, Jaini et al. (2020) propose Tail-Adaptive Flows (TAF), which replace the Gaussian base distribution by a multivariate t-distribution with one learnable degree of freedom.

heading: 3 LEARNING THE CORRECT MARGINAL TAIL BEHAVIOR WITH MTAF
text: In this section, we present a simple extension to triangular affine flows that allows to model distributions with a flexible tail behavior. We start by presenting our theoretical results in Section 3.1. Motivated by these results, we propose marginally Tail-Adaptive Flow (mTAF) in Section 3.2.

heading: 3.1 THE NECESSITY OF A FLEXIBLE BASE DISTRIBUTION
text: In this work, we investigate the tailedness of NFs more thoroughly through the lense of marginal tailedness, i.e. we consider the univariate tailedness of the marginal distributions of xj .Therefore, we introduce the following definitions: Definition 4 (j-Heavy-Tailed, Mixed-Tailed, Fully Heavy-Tailed, Equal Tail Behavior). We call a random variable x ∈ RD j-heavy-tailed if its jth marginal xj is heavy-tailed according to Definition 1. Otherwise, we call x j-light-tailed. x is said to be mixed-tailed if there exists j1, j2 such that x is j1-heavy-tailed and j2-light-tailed. Further, we say that x is fully heavy-tailed if x is j-heavytailed for all j ∈ {1, . . . , D}. We define two random variables x and z to have equal tail behavior if it holds for all j that
x is j-heavy tailed ⇔ z is j-heavy tailed .
We found the following relation to Definition 3. Proposition 1 (j-Heavy-Tailedness induces `2-Heavy-Tailedness). Assume that x is j-heavy-tailed for any j. Then, x is also `2-heavy-tailed.
The proof can be found in Section A.1 in the Appendix. The proposition shows that j-heavytailedness is a more general notion of multivariate heavy-tailedness than `2-heavy-tailedness, which allows a narrow inspection of the tail behavior. More precisely, the new notion allows us to differentiate between fully heavy-tailed random variables and mixed-tailed random variables, which are both `2-heavy-tailed. One can now wonder how the tails, described by the novel notation, behave for NFs, which we answer by the following two results. The first result states that, under mild technical conditions, fully heavy-tailedness of the base distribution is preserved by triangular affine maps. Proposition 2 (Triangular Affine Maps preserve Fully Heavy-Tailedness). Let z be a fully heavytailed random variable that satisfies Assumption 13 and let T be a a triangular affine map, that is, Tj(zj , z<j) = µj(z<j) +σj(z<j)zj with σj > 0. Then, it holds that T (z) is also fully heavy-tailed.
A formal proof can be found in Section A.2 of the Appendix. Assumption 1 is a mild condition on the decay rate of the copula density of z. We explain this condition in more detail and give various examples in Section A.3 of the Appendix.
It is clear that permuting the marginals does not change the heavy-tailedness. Hence, by iterative application of Proposition 2, we deduce that affine triangular flows with a fully heavy-tailed base distribution are unable to model mixed tailed distributions. Implicitly, Proposition 2 states that a Lipschitz normalizing flow as proposed by Jaini et al. (2020) is not able to model mixed-tailed distributions. However, the following Theorem that we consider as our main result guides us towards a flow architecture that is able to model target distributions that are mixed-tailed in a marginally tailadaptive way. Theorem 3 (Learning the correct Tail Behavior). Consider a random-variable z that is j-lighttailed for j ∈ {1, . . . , d} for some d < D and j-heavy-tailed for j ∈ {d + 1, . . . , D}. Then, under the same conditions as in Theorem 2 and Proposition 2, it holds that z and T (z) have the same tail behavior.
Proof. Since the result combines Theorem 2 and Proposition 2 in an evident fashion, we just quickly present a sketch of the proof. First, let us consider j ≤ d. Then it holds for the moment-generating function of xj that
mxj (λ) = ∫ RD eλTj(z≤j)q(z)dz = ∫ Rj eλTj(z≤j)p≤j(z≤j)dz≤j ,
3This Assumption can be found in Section A.2 in the Appendix.
which has been shown to be bounded for some λ > 0 (see the proof of Theorem 2 in Jaini et al. (2020)). Therefore, x is j-light-tailed for all j ≤ d. In the case j > d, we notice4 that the proof for heavy-tailedness of Tj(z≤j) involves just the heavy-tailedness of zj and not of any other component of z<j . Hence, if zj is heavy-tailed, then xj = Tj(z≤j) is also heavy-tailed, regardless of z<j . Therefore, x is j-heavy-tailed for all j > d, which completes the proof. Note that in general we cannot deduce the latter conclusion for light-tailed marginals, i.e. if zj is light-tailed, this does not mean that xj is also light-tailed. This is only the case, if all z<j are light-tailed as well.

heading: 3.2 MARGINALLY TAIL-ADAPTIVE FLOW (MTAF)
text: Our main result, Theorem 3, prompts that if we maintain an ordering of the marginals such that the first marginals are light-tailed and the following are heavy-tailed in each flow step, we retain the marginal tail behavior of the base distribution in the estimated target distribution. This finding motivates the novel NF proposed in this paper. The proposed approach combines research findings from extreme value theory (Embrechts et al., 2013; Nair et al., 2013), recent findings about normalizing flows (Jaini et al., 2020; Alexanderson & Henter, 2020; Laszkiewicz et al., 2021), and the results presented herein. The proposed mTAFs consists out of three steps depicted in Figure 1 and described in the following:
Step 1: Estimating the marginal tail indices and defining the marginal distributions. For each marginal, i.e. for the marginal distribution qj of each xj , j = 1, . . . , D, we use the moments doublebootstrap estimator (Draisma et al., 1999) and the kernel-type double-bootstrap estimator (Groeneboom et al., 2003) to assess heavy-tailedness of the data distribution. If both estimators predict a light-tailed distribution, we set the corresponding marginal base distribution qj to be standard normal distributed, i.e. zj ∼ N (0, 1). Otherwise we set the marginal to the standardized t-distribution with the estimated degree of freedom, i.e. zj ∼ tν̂j , where ν̂j is the Hill double-bootstrap estimator (Danielsson et al., 2001; Qi, 2008). In Section B.1 of the Appendix, we present more details about the tail-assessment scheme.
Step 2: Defining the base distribution. We construct the base distribution as the mean-field approximation of the marginals, i.e. z has the density q(z) := ∏D j=1 qj(zj) with marginal densities qj defined in step 1. Further, to satisfy the assumptions of Theorem 3, we need to permute the marginals such that it holds zj ∼ N (0, 1) for j ≤ d and zj ∼ tν̂j for j > d. We apply the same permutation to restructure our data according to the base components. To account for tail index estimation errors and for more flexible learning, we make the tail indices (i.e. the degrees of freedom of each t-distribution) learnable. That is, we initialize the degree of freedom of the jth marginal with ν̂j but adapt the parameter together with the network parameters throughout training.
Step 3: A data-driven permutation scheme. Recall, that vanilla triangular flows employ a permutation step after each transformation to enhance the mixing of variables. However, purely random permutations might lead to a violation on the ordering of marginals, which is necessary to ensure Theorem 3. Therefore, we permute only within the set of heavy-tailed marginals and within the set of light-tailed marginals, to ensure the validity of Theorem 3. Within these groups one can choose any permutation scheme.
Without loss of generality, we assume that the first d components of z are light-tailed and the remaining D − d components are heavy-tailed5. Then, the training objective is to optimize for flow parameters θ̂ and degrees of freedom ν̂ = [ν̂d+1, . . . , ν̂D] to maximize the log-likelihood
L(θ̂, ν̂;X) = N∑ j=1
{ log ( d∏ i=1 π ( T−1 θ̂ ( x(j) ) i ) · D∏ i=d+1 tν̂i ( T−1 θ̂ ( x(j) ) i )) − log detJTθ̂ ( x(j) )}
= N∑ j=1 { d∑ i=1 log π ( T−1 θ̂ ( x(j) ) i ) + D∑ i=d+1 log tν̂i ( T−1 θ̂ ( x(j) ) i ) − log detJTθ̂ ( x(j) )}
whereX := (x(1), . . . x(N)) is the data, and π and tν̂ are the PDF of the standard normal distribution and the standard t-distribution with ν̂ degrees of freedom, respectively.
4For details, we refer to the proof of Proposition 2 in the Appendix. 5Otherwise we permute the marginals as described in Step 2.
When applying our theoretical results presented in the previous section to the proposed mTAF, we can show that it fulfills the desired tail-preserving property, as formalized by the following corollary:
Corollary 1 (Marginal Tail-Adaptive). Under the same assumptions as in Theorem 2 and in Proposition 2, mTAFs are marginally tail-adaptive, that is, z and x = T (z) have the same tail behavior.

heading: 4 EXPERIMENTAL ANALYSIS ON SYNTHETIC DATA
text: To investigate the benefits of mTAF we perform an empirical analysis in which we conduct experiments on synthetic heavy-tailed data. We construct synthetic 16-dimensional data sets and investigate eight different settings: The marginals are chosen to contain h ∈ {1, 2, 4, 8} mixtures of two t-distributions, four Gaussian distributions, two mixtures of three Gaussian distributions, and 10− h mixtures of two Gaussian distributions. The degree of freedom of all t-distributions is either ν = 2 or ν = 3. In all mixtures all mixture components are weighted equally, means are uniformly sampled from [4, 4], and standard-deviations are sampled from [1, 2]. Using a Gaussian Copula, we construct a complex joint distribution with mixed-tailed marginals. A detailed description of the data set generation can be found in Section B.2 in the Appendix. Details about hyperparameters can be found in Section B.4 in the Appendix. We provide a PyTorch implementation and the code for all experiments along the submission.

heading: 4.1 MODEL ACCURACY
text: In our first experiment, we compare the performance of 5 different MAFs: a vanilla flow with z ∼ N (0, I), the TAF, a variation of TAF (TAF(ν̂1, . . . , ν̂D) in which all marginals have their own independent degree of freedom (i.e. zj ∼ tν̂j (0, 1) with trainable degrees of freedom ν̂j), and mTAF as described in Section 3.2 with learnable and fixed tail indices, where the letter is denoted as mTAF (fixed ν̂).
Table 1 summarizes the test losses archived by the five models in the eight different settings. The results demonstrate that mTAF can efficiently learn a multivariate heavy-tailed distribution, whereas the vanilla approach, i.e. fixing z ∼ N (0, I), is prone to optimization instabilities (as can be seen by high standard deviations) and leads to higher values of the negative log-likelihood. Moreover, TAF performs worse than mTAF (and is also less stable) which could be contributed to having only one joint degree of freedom to model the tailedness of all marginals. Using independent degrees of freedom however only leads to slight improvements, illustrating the necessity of the proposed permutation scheme of mTAF. The performance increase is especially severe for marginals with small tails index, i.e., when the degree of freedom of the marginals of the data-generating distribution is ν = 2. Fixing the tail indices to the estimated values already leads to great results, which can further be increased by adapting the parameters during training.

heading: 4.2 GENERATING TAIL EVENTS
text: An often-used property of NFs is the diffeomorphic relation between base and target distribution, which can be used, for instance, for image interpolation6 or for constructing counterfactuals7. In this work, we present another practical advantage, which we motivate by the derived theory, namely the generation of tail samples. One generic way to generate tail samples in the target space is to generate tail samples in the base space, which are then pushed towards the target space. However, in this work we propose to solve a more specific problem, which is the generation of samples that are from the j-tails, i.e. samples from the joint distribution of x with the jth component xj being from the tail of the jth marginal. Recall that in the proof of Proposition 2 we show that if zj is heavytailed, then Tj(z≤j) resulting from applying one affine triangular flow layer is also heavy-tailed, independently of z<j . Inspired by this result, we hypothesize that we can generate multivariate samples x with heavy j-tails by sampling tail events of zj , complementing them with remaining components z6=j sampled from z 6=j , and pushing the resulting z through the flow, which retains the j-heavy-tailedness. However, since we apply a permutation after each layer, to sample from the
6by interpolating in the base distribution and pushing them forward using the NF (Kingma & Dhariwal, 2018).
7by applying gradient ascent in the base space (Dombrowski et al., 2021).
jth marginal of the target distribution, we need to employ the above procedure with tail samples of zk−1(j), where k : {1, . . . , D} → {1, . . . , D} is the permutation mapping resulting from applying all permutation layers, and k−1 is its inverse.
For all models trained on the data sets with ν = 2, we generate 1 000 samples of the heavy-tailed variable zk−1(16) and consider the 50 with largest absolute value as tail samples. In Figure 2 we visualize the results for mTAF: We depict the 16th component x16 of the samples generated by the described procedure for the distributions with different amounts of heavy-tailed components. It becomes clear that the sampling procedure indeed results only in samples from the joint distribution where the 16th component stems from the tails of the corresponding marginal. Surprisingly, even in the case in which 8 heavy-tailed components are permuted (i.e. h = 8) the sampling procedure succeeds by ignoring the body of the distribution and generating exclusively tail samples in the 16th component. As a further proof of concept, we demonstrate in Figure 5 in Section B.3 in the Appendix that sampling tail events from other marginals than zk−1(16) does not result in tail events in the 16th component of x. Hence, this result substantiates our intuition that zk−1(j) is the “responsible” factor for generating j-tail samples. Furthermore, we rerun the sampling procedure with a vanilla flow and a TAF, which we visualize in Figure 3. While TAF is able to generate
reasonable tail samples comparable to mTAF, the vanilla flow is generating extreme outliers, and is thus, not able to efficiently sample from the tails.

heading: 5 CONCLUSION AND FUTURE RESEARCH DIRECTIONS
text: In this work, we deepen the mathematical understanding of the tail behavior of triangular flows. We note that the distribution we want model may have heavy- as well as light-tailed marginals, and show how the marginal tail behavior of the target distribution of the flow relates to the tail behavior of its base distribution. Based on these theoretical findings we propose a new algorithm, which we refer to as mTAF. In particular, we initialize a trainable base distribution based on statistical tail estimates of the target, and employ a data-driven permutation scheme that guarantees the correct tail behavior of the target distribution. An in-depth empirical analysis on synthetic data with heavy- and light-tailed marginals shows that mTAF benefits in terms of estimation performance and robustness of the training. Moreover, we provide a sampling strategy inspired by out theory that allows to generate joint samples from the target distribution with a customizable tail behavior. In summary, we introduce a novel way to impose an inductive bias in normalizing flows that allows to model the tails of the target distribution.
Yet, we think that there exists much potential for interesting follow-up works. For instance, we are aware that most SOTA architectures are indeed non-affine, such as NSF (Durkan et al., 2019), which are based on monotone rational-quadratic splines, or SOS (Jaini et al., 2019), which are based on strictly increasing polynomials. However, similarly to (Jaini et al., 2020), we hypothesize that the trade-off between a flexible base distributions and the complexity of the flow leads to simpler and more efficient models. This, and the investigation on the impact of mTAF on real world data remains subject of further future studies. Furthermore, the attentive might have noticed that the proposed mTAF employs a restricted set of permutations, which limits its generality. One possible solution for this issue is to employ more expressive base distributions that introduce marginal dependencies, which extends the generality of mTAF, while preserving its ability to be marginally tail-adaptive. For modeling these more expressive base distributions it is evident to study copula theory, which allows a grounded treatment of various kinds of multivariate dependency concepts, even beyond the scope of pearson correlation, such as tail dependencies (Joe, 2014). In addition, we believe that mTAFs, and further improvements of them, bear lots of potential for sampling methods, which benefit from the whole range of samples from the target distribution, including tail samples. Generation of such samples might improve methods for simulation-based inference (Cranmer et al., 2020) such as VAEs and are essential for the analysis of extremes such as in weather forecasting.
Last but not least, we want to highlight that our theory is not limited to maximum likelihood training and one could think of various other training paradigms, for instance based on f -divergences. Using alternative optimization metrics, we can potentially put more emphasize on the learning of the correct tail behavior.
Reproducibility Statement This work concentrates on the mathematical understanding of normalizing flows, and in doing so, includes technical proofs, which might rely on abstract assumptions. Nonetheless, we put our best effort in making all the proofs and the derived theory in Section A.1 and A.2 precise, accessible, and correct. To avoid misconceptions, we dedicate Section A.3 to the explanation and clarification of Assumption 1. We provide documented source code including code execution instructions, which allows the reproducibility of our empirical results. Additionally, all algorithms and hyperparameters are presented in Section B.

heading: A THEORY AND PROOFS
text: In this Section, we derive and proof all of our theoretical results. We start to present some preliminary results in Section A.1, which help us providing the proof of our main result in Section A.2. In Section A.3, we illuminate the technical Assumption 1 and provide examples and intuitions.

heading: A.1 PRELIMINARY THEORETICAL RESULTS
text: Proposition 3. Let x ∈ R be a random variable. Then, it holds that x has tail index less or equal than α iff. for any β > α and any C > 0 it is∫
|x|>C |x|βp(x)dx =∞ .
Proof. Let us first assume that x has tail index at most α. Then, according to Definition 2, we know that for any β > α and C > 0
∞ = Ex[|x|β ] = ∫ R |x|βp(x)dx = ∫ |x|≤C |x|βp(x)dx+ ∫ |x|>C |x|βp(x)dx
≤ Cβ ∫ |x|≤C p(x)dx+ ∫ |x|>C |x|βp(x)dx .
Since we assume p to be continuous, we can bound p on the compact interval [−C,C], and hence, the first above integral must be bounded. Therefore, it is∫
|x|>C |x|βp(x)dx =∞ .
To prove the back-direction, let us consider a β > α andC > 0. Very similarly to the forward-proof, we can now see that
∞ = ∫ |x|>C |x|βp(x)dx
= Cβ ∫ |x|≤C p(x)dx+ ∫ |x|>C |x|βp(x)dx
= Ex [ |x|β ] ,
where the second equality follows from the finiteness of the integral. Since this follows for any β > α, x must have tail index α or less.
This simple result demonstrates that the tail index, as indicated by the name, depends on the tail of the distribution, i.e. on the PDFs behavior for large values |x| > C. This fact motivates why maximum likelihood estimations of the tail index, which depend on the whole distribution are biased. For more elaborate details, we refer to Section 9 in Nair et al. (2013).
In a similar fashion to the previous result, the next technical lemma states that unboundedness of the moment-generating function is due to the unboundedness of the integrand for tail events, i.e. for z > z∗. This little lemma turns out to be useful in the proof of Proposition 2. Lemma 1. Let z ∈ R be heavy-tailed. Then it holds for any z∗ ∈ R and λ > 0 that∫
z>z∗ eλzpz(z)dz =∞ .
Proof. Since z ∈ R is heavy-tailed, we know that for all λ > 0 ∞ = mz(λ) = ∫ R eλzpz(z)dz
= ∫ z≤z∗ eλzpz(z)dz + ∫ z>z∗ eλzpz(z)dz
≤ Fz(z∗)eλz ∗ + ∫ z>z∗ eλzpz(z)dz ,
where Fz is the CDF of z. The last inequality follows from the fact that exp(λz) is monotonic increasing in z. Since the first summand is bounded, it follows that the second summand must be unbounded. This completes the proof.
Recall that in Proposition 1 we state that j-heavy-tailedness induces `2-heavy-tailedness. In the following, we provide a formal proof of this result.
Proof of Proposition 1. For this proof, we employ the equivalent definition of heavy-tailedness of xj via the decay rate of its distribution function (see e.g. Lemma 1.1. in Nair et al. (2013)), i.e.
lim sup xj→∞ 1− Fj(xj) e−λxj =∞ for all λ > 0 , (3)
where Fj is the CDF of xj . Since xj ≤ ‖x‖ for all x ∈ RD, we can conclude that Fj(a) ≥ F‖x‖(a) for a ∈ R. Therefore,
1− F‖x‖(a) e−λa ≥ 1− Fxj (a) e−λa →∞ for a→∞ .
According to the equivalent definition in (3), ‖x‖ is heavy-tailed, which proves that x is `2-heavytailed.
The following is a well-known implication of the change of variables formula and the integration rule by substitution, which we are going to apply in the subsequent proofs.
Lemma 2 (Substitution in the Moment-Generating Function). Let T be a diffeomorphism such that T (z) = x for some random variables x, z ∈ RD. Then, we can rewrite∫
RD eλxp(x)dx = ∫ RD eλT (z)q(z)dz .
For completeness, we give a brief proof of this result.
Proof. Using the change of variables formula, see (1), we can write∫ RD eλxp(x)dx = ∫ RD eλxq ( T−1(x) )∣∣det JT−1(x)∣∣dx . Now, we can rewrite exp(λx) = exp(λT ( T−1(x) ) and substitute z = T−1. Integration by substitution completes the proof.
Next, we present how we can use copulae to reformulate a multivariate PDF.
Definition 5 (Copula). A copula is a multivariate distribution with cumulative distribution function (CDF) C : [0, 1]D → [0, 1] that has standard uniform marginals, i.e. the marginals Cj of C satisfy Cj ∼ U [0, 1]. Theorem 4 (Sklar’s Theorem). Taken from Hofert et al. (2018).
1. For any D-dimensional CDF F with marginal CDFs F1, . . . , FD, there exists a copula C such that F (z) = C ( F1(z1), . . . , FD(zD) ) (4)
for all z ∈ RD. The copula is uniquely defined on U := ∏D j=1 Im(Fj), where Im(Fj) is the image of Fj . For all u ∈ U it is given by
C(u) = F ( F←1 (u1), . . . , F ← D (uD) ) ,
where F←j are the right-inverses of Fj .
2. Conversely, given any D-dimensional copula C and marginal CDFs F1, . . . FD, a function F as defined in (4) is a D-dimensional CDF with marginals F1, . . . , FD.
Therefore, if F is absolutely continuous, we can differentiate (4) to obtain the PDF of z
q(z) = c ( F1(z1), . . . , FD(zD) ) D∏ j=1 qj(zj) ,
where c denotes the PDF of the copula C.
Lastly, we present the following asymptotic behavior of the inverse CDF of a standard Gaussian distribution, which we use in Section A.3 to explain Assumption 1.
Lemma 3 (Asymptotic Behavior8 of Φ−1(1 − y)). Denote by Φ the CDF of a standard Gaussian distribution. Then, it holds for the inverse of Φ that
Φ−1(1− y) ∼ √ −2 log(y) for y → 0 .
Proof. First, we note that
Φ(x) = 1
2 +
1 2 erf ( x√ 2 ) ∼ 1− 1 x √ 2 e−x 2/2 ,
8The idea of the proof is due to (https://math.stackexchange.com/users/491644/maxim)
which is a well-known asymptotic (Liu et al., 2012). Here, erf denotes the error-function. Rearranging terms gives
log ( 1− Φ(x) ) ∼ − log ( x √ 2π ) − x 2
2 ∼ −x
2
2 as x→∞ .
Finally, we can invert the above asymptotic equation to obtain Φ−1(y) = √ −2 log(1− y) for y → 1
or equivalently Φ−1(1− y) = √ −2 log(y) for y → 0 .

heading: A.2 PROOF OF THE MAIN RESULT
text: The proof of Proposition 2 relies on lower-bounding the moment-generating function of each marginal xj . In order to derive such a bound of a multivariate integral, we rewrite the joint distributions q≤j using their copula densities:
q≤j(z≤j) = cj ( F1(z1), . . . , Fj(zj) )∏ i<j qi(zi)
for any j ∈ {1, . . . , D} and for corresponding copula density cj . Our proof relies on the following technical condition on the decay rate of the copula densities. Assumption 1 (Bounding the Marginal Decay Rate of the Copula Densities). For all j ∈ {1, . . . , D} and λ > 0 there exists a compact set S ⊂ Rj−1 with positive (Lebesgue-)mass, a constant z∗j > 0, a scaling constant s > 0, and a function f(z<j) < λσ(z<j) for z<j ∈ S such that
cj ( F1(z1), . . . , Fj(zj) ) ≥ se−f(z<j)zj for zj > z∗j and z<j ∈ S , (5)
where cj is the copula density of q≤j
This assumption sets a bound on the decay rate of the copula density with respect to zj . We clarify this assumption in Section A.3 with additional examples.
Now, we set all preliminaries to prove Proposition 2.
Proof of Proposition 2. We start by considering the case j = 1. In this case it is x1 = µ+ σz1 and therefore
mx1(λ) = ∫ R eλx1p1(x1)dx1
= ∫ R eλ(µ1+σ1z1)q1(z1)dz1 (Lemma 2)
= eλµ1 ∫ R eλσ1z1q1(z1)dz1 .
Defining λ′ := λσ1 > 0, we can see that the last integral is unbounded due to the heavy-tailedness of z1, see Definition 1. Therefore, mx1(λ) = ∞ for all λ > 0, which proves the heavy-tailedness of x1. Next, we consider the case j > 1. Again, we examine the moment-generating function of xj . Define the jth canonical basis vector vj := (0, . . . , 0, 1, 0, . . . , 0)>. Then,9
mxj (λ) = mv>j x = ∫ RD eλv > j xp(x)dx (LOTUS)
= ∫ RD eλTj(zj ,z<j)q(z)dz (Lemma 2)
= ∫ Rj eλµ(z<j)+λσ(z<j)zjq≤j(z≤j)dz≤j
= ∫ Rj−1 eλµ(z<j) ∫ R eλσ(z<j)zjq≤j(z≤j)dzjdz<j . (6)
9Note that for the sake of clarity, we leave out the index j in µj and σj .
Using Sklar’s Theorem (Theorem 4), we can write any joint PDF as the product of marginals and a copula density cj such that
q≤j(z≤j) = cj ( F1(z1), . . . , Fj(zj) )∏ i<j qi(zi) . (7)
We plug (7) into (6) to obtain
mxj (λ) = ∫ Rj−1 eλµ(z<j)q<j(z<j) ∫ R eλσ(z<j)zjcj ( F1(z1), . . . , Fj(zj) ) qj(zj)dzjdz<j
≥ ∫ S eλµ(z<j)q<j(z<j) ∫ zj>z∗j eλσ(z<j)zjcj ( F1(z1), . . . , Fj(zj) ) qj(zj)dzjdz<j , (8)
since all quantities within the integral are positive. Using Assumption 1, we can bound the inner integral of the above equation, which we denote by A(z<j), and get
A(z<j) ≥ s ∫ zj>z∗j e(λσ(z<j)−f(z<j))zjqj(zj)dzj
= s ∫ zj>z∗j eλ ′zjqj(zj)dzj (define λ′ := λ− σ(z<j)− f(z<j) )
=∞ for all z<j ∈ S , due to the heavy-tailedness of zj and Lemma 1. Since S is compact, µ and q are both continuous, and q is positive, we deduce that exp(λµ(z<j))q<j(z<j) is lower-bounded (by a constant larger than 0) in S. Therefore, employing (8) and using that S has positive mass, we can lower-bound the moment-generating function by∞, which proves the heavy-tailedness of xj . In summary, x is j-heavy-tailed for all j ∈ {1, . . . , D}.

heading: A.3 NOTES ON ASSUMPTION 1
text: Assumption 1 might look troublesome at first sight, but we will illustrate in this section that the condition is indeed very reasonable. We will show how to verify it in simple examples, and we will introduce a simpler, more intuitive sufficient condition for it.
First of all, let us present a restricted but more intuitive version of Assumption 1. Assumption 2 (Simplification of Assumption 1). For all j ∈ {1, . . . , D} and λ > 0 it holds for S := [a, b]j−1 that there exist constants z∗j and s > 0 such that
c ( F1(z1), . . . , Fj(zj) ) ≥ se−(λσ−ε)zj for zj > z∗j and z<j ∈ S , (9)
where cj is the copula density of q≤j , λσ is a lower bound of λσ(z<j), ε > 0 is small such that λσ − ε > 0.
Let us summarize the simplifications that we make in Assumption 2. First of all, we restricted S to be a closed cube [a, b]j−1, which is obviously a specific instant of a compact set with positive mass. Further, we assumed σ to be continuous, and thus, λσ(z<j) must be lower-bounded in S. This allows us to replace the function f by the constant λσ − ε for arbitrary small ε > 0. After giving this simplified sufficient condition, we provide some intuition by presenting some examples where Assumption 2 holds true. Example 1 (Independent Variables). Consider a random variable z with independent components, i.e. q(z) = ∏D j=1 qj(zj). Then, the associated copula is the independence copula (Figure 4a),
which is a uniform random distribution on [0, 1]D. Therefore it is c ( F1(z1), . . . , FD(zD) ) = 1 for all z ∈ RD and Assumption 2 follows immediately since s exp(−(λσ − ε)zj)→ 0 for zj →∞. Example 2 (Bounded Copula Density). Consider a lower-bounded copula density, i.e. there exists a lower bound a > 0 such that
c(u1, . . . , uD) ≥ a for all u ∈ [0, 1]D . Again, the validity of Assumption 2 in this setting is clear.
0.00 0.25 0.50 0.75 1.00 0.0
0.2
0.4
0.6
0.8
1.0
(a) 2-dimensional Independence copula: c(z1, z2) = 1.
0.00 0.25 0.50 0.75 1.00 0.0
0.2
0.4
0.6
0.8
1.0
(b) 2-dimensional Gaussian copula with correlation ρ12 = 0.7 .
Furthermore, this assumption is obviously not limited to bounded copula densities but also holds for copula densities that converge to 0 but whose decay rate in zj is lower-bounded by (9). To visualize the intuition, consider the 2-dimensional copula density of a Gaussian copula in Figure 4b. Imagine fixing S such that F1(z1) ∈ [0.5, 0.75], which is compact for continuous F1. Then (9) bounds the decay rate within the “tube” [0.5, 0.75] if we consider z2 → ∞, i.e. if F2(z2) → 1. Next, we show how we can formally prove the assumption for Gaussian copulae. Example 3 (Gaussian Copula). The Gaussian copula with correlation matrix R ∈ RD×D has density function
c(u) = 1√
detR exp
( −1
2
( Φ−1(u1), . . .Φ −1(uD) )( R−1 − I )( Φ−1(u1), . . .Φ −1(uD) )>) ,
(10) where I ∈ RD×D is the identity matrix, Φ−1 is the inverse CDF of the univariate standard Gaussian distribution, and u ∈ [0, 1]D. In the following, we consider Assumption 2 for j = D. Note that S is assumed to be a compact set, therefore Fj(zj) are all upper and lower-bounded by some value for z<j ∈ S. This makes all polynomials of them also bounded. Hence, we can find constants a′, b′, c′ such that we can lower-bound the term within the exponential in (10) by
−1 2
( a′Φ−1 ( FD(zD) )2 + b′Φ−1 ( FD(zD) ) + c′ .
Plugging the above into (10) gives
c ( F1(z1), . . . , FD(zD) ) ≥ 1√
detR exp
( −1
2
( a′Φ−1 ( FD(zD) )2 + b′Φ−1 ( FD(zD) ) + c′
)
∝ exp ( −aΦ−1 ( FD(zD) )2 + bΦ−1 ( FD(zD) )) (for some a, b)
≥ exp ( −|a|Φ−1 ( FD(zD) )2 + |b|Φ−1 ( FD(zD) )) ≥ exp ( −|a|Φ−1 ( FD(zD) )2) , (11)
where the last line applies if Φ−1 ( FD(zD) ) ≥ 0, which is satisfied if z∗ is large enough10.
Next, we use the asymptotic relation from Lemma 3 Φ−1 ( FD(zD) ) ∼ √ −2 log ( 1− FD(zD) ) .
10for instance if z∗ is larger than the median of zD
Hence, for each ε > 0 there exists a z∗ large enough such that∣∣∣∣ Φ−1 ( FD(zD) )√ −2 log ( 1− FD(zD) ) − 1∣∣∣∣ < ε , which can be rearranged to
Φ−1 ( FD(zD) ) < √ −2 log ( 1− FD(zD) ) (1 + ε) .
Plugging the above into (11), we obtain
c ( F1(z1), . . . , FD(zD) ) ≥ exp ( 2|a|(1 + ε)2 log ( 1− FD(zD) )) = ( 1− FD(zD) )2ã , (12)
where we define ã := 2|a|(1 + ε)2. Hence, we are left to lower-bound (12), which we can do for a range of heavy-tailed marginal distributions such as:
1. Pareto distribution: The Pareto distribution with shape parameter α has CDF
F (z) = 1− 1 z
α
.
Therefore, ( 1− FD(zD) )2ã = 1
zD
2|a|α
= exp ( −2ãα log(zD) ) ≥ exp ( −2ãαzD
) for zD ≥ e.
2. Scale invariant distributions: Following the same argument as above, each distribution with CDF
F (z) = 1− bz−α for z > z∗ (13) for constants b, α, z∗ > 0 satisfies the bound(
1− FD(zD) )2ã ≥ b exp(−2ãαzD) .
Each distribution with CDF as in (13) is a scale-invariant distribution (see e.g. Theorem 2.1 in Nair et al. (2013)).
3. Exponentially decaying distributions: Every distribution that satisfies the bound
F (z) ≤ 1− exp(z)−α
for z > z∗ and α > 0. In this case, we can again show that Assumption 1 is valid:( 1− FD(zD) )2ã ≥ exp(−2ãαzD) . Lastly, we want to emphasize that Corollary 1 is derived by an iterative application of Theorem 3. Therefore, Assumption 1 must hold for all “flow steps”, i.e. if T = T (L) ◦ · · · ◦ T (1), we need to ensure validity of Assumption 1 for z(0) := z, z(1) := T (1)(z), z(2) := T (2) ◦ T (1)(z), . . . , z(L−1) := T (L−1) ◦ · · · ◦ T (1)(z). In Example 1, we show that this assumption holds true for z(0) since we define our base distribution under the mean-field assumption. Furthermore, we conjecture that if we apply a Lipschitz-continuous diffeomorphism on a random variable with bounded copula density, then the transformed random variable must also have a bounded copula density. Hence, Assumption 1 would be valid for all “flow steps” (see Example 2) when starting with a mean-field base distribution q(z) = ∏D j=1 qj(zj). However, this conjecture needs to be studied in further research.

heading: B ALGORITHMS AND COMPUTATIONAL DETAILS

heading: B.1 TAIL ESTIMATION
text: Many heavy-tailed distributions can be characterized by their tail index, which include the set of regularly varying distributions,11 such as the t-distribution, the Pareto distribution, and many more. However, as already shown in Section 2.1, the tail index does not depend on the body of the distribution, and hence, non-tail samples must typically be discarded for tail index estimation. Although a variety of estimators for the tail index exist, such as the Hill estimator (Hill, 1975), the moment estimator (Dekkers et al., 1989), and kernel-based estimators (Csorgo et al., 1985), none of them is considered to be as superior in all settings. A major issue of all mentioned estimators is that they are based on a threshold defining the tail, i.e. the user needs to input statements of the form “the k largest samples are considered to be tail events”. Even though there exist some strategies to find k, there is none working robustly in all settings. In fact, one can construct simple counter examples for all estimators that lead to failures of tail estimation. This is due to undesired properties of the estimators, such as the lack of translation invariance of the Hills estimator (while the tail index clearly is location invariant). We refer to Section 9 in Nair et al. (2013) for a detailed text book treatment of tail index estimation. In summary, robust tail estimation is still considered as an unsolved problem, which forces practitioners to consider multiple estimators to make a well-founded decision. Furthermore, we note that the Hills estimator can only be applied for regularly varying distributions, which excludes the application of the Hills estimator to classify light-tailed distributions. In contrast, the moments and the kernel estimator can both be applied to identify heavy-tailed marginals and to assess a tail index.
To implement the tail assessment scheme, see Step 1 of the proposed method in Section 3.2, we found that Algorithm 1 works fine in classifying the correct tail behavior and giving a decent initialization for the tail indices. We reused the code by Voitalov et al. (2019), which implements all tail estimation procedures12 from our Algorithm. Notice that we clip the tail index by 20, i.e. the algorithm classifies marginals with a tail index larger than 20 as light-tailed, which prevents a too restrictive set of allowed permutations, see Step 3 in Section 3.2. For illustration, consider the following simple example. Assume that we estimate all except of one marginal to be heavy-tailed. Then, the first component of the flow is never allowed to permute with other components, since they are classified as heavy-tailed. Hence, the mixing of the first component would be severely restricted. Further, since large tail indices indicate a less heavy-tailed distribution, it is reasonable to clip the tail index at some threshold.
Algorithm 1 Marginal Tail Estimation Require: Data val
tail.est← [ ] for j in {1, . . . , D} do
marginal← Data val[:, j] moments← moments est[|marginal|] . 0 if |marginal| is estimated to be light-tailed kernel← kernel est[|marginal|] . 0 if |marginal| is estimated to be light-tailed if moments==kernel==0 then
tail est.append(0) . light-tailed if moments and kernel estimate a light-tailed marginal else
hill← hills est(|marginal|) if hill> 20 then
tail est.append(0) . light-tailed if hills estimator predicts high tail index else
tail est.append(hill) end if
end if end for return tail est
11see Section 2 in Nair et al. (2013) for further details 12including the hyperparameter selection (Danielsson et al., 2001; Qi, 2008; Draisma et al., 1999; Groene-
boom et al., 2003)

heading: B.2 SYNTHETIC DATA GENERATION
text: Our generation of the synthetic distribution consists of 3 steps: 1. Generating the marginal distributions, 2. Defining a copula distribution, 3. Combining the marginal and the copula to obtain a multivariate joint distribution.
Generating the marginal distributions. The first four marginals are defined to be Gaussians. The following marginals are two 2-mixtures of two Gaussians and two mixtures of three Gaussians. The last h ∈ {1, 2, 4, 8} components are a mixture of two t-distributions and the remaining marginals are again mixtures of two Gaussians. All mixtures have equal weight for each mixture component and all means and standard-deviations are randomized. Means are constructed by uniformly sampling from [−4, 4], whereas standard-deviations are sampled from [1, 2].
Defining a copula distribution. Recall, a Gaussian copula (10) is parameterized by a correlation matrix R. To generate R, we randomly sample 16 different pairs (i, j) ∈ {1, . . . , 16}2 with i 6= j and set the corresponding entry of the correlation matrix
Ri,j := 0.25 .
Obtaining a joint distribution. Lastly, we combine the marginals with the Gaussian copula using Sklar’s Theorem 4. This gives us a multivariate distribution with specified and complex marginals with a dependency structure given by the copula, see Joe (2014) for more details on the induced dependencies.
To construct the training, test, and validation sets 15.000, 5 000, and 5 000 samples from this distribution are sampled, respectively.

heading: B.3 SAMPLING MARGINALLY TAIL EVENTS
text: In the main text, we have seen that the proposed tail generation procedure is successful in generating samples that are marginally in the tail of some specified marginal (Figure 2). In Figure 5, we present the 16th marginal when repeating the same experiment but with marginal tail samples from other marginals zj than the proposed zk−1(16).

heading: B.4 HYPERPARAMETERS AND EXPERIMENTAL DETAILS
text: In all our experiments, we constructed a model using L = 5 MAF flow layers, where each emerging neural networks contains 1 hidden layer and 200 hidden neurons. The tail index in TAF is initially set to ν̂ = 10. As a permutation scheme, we employ random permutations in the vanilla flow and
TAF, and random permutations within the sets of light- and heavy-tailed marginals, respectively, in mTAF. In addition, we applied batch-normalization after each triangular transformation. For optimization, we use an Adam optimizer with a learning rate of 1e−5 and a weight-decay of 1e−6. The learning rates of the tail indices are set to 0.1 for TAF and for mTAF. We train until 200 epochs or use early stopping if training gives no improvement after 30 epochs. We use a batch size of 256.
Our code is built with PyTorch (Paszke et al., 2019) and we make extensive use of the package nflows (Durkan et al., 2020) to implement mTAF. All code is provided along the submission.

Review summaries:
This paper focuses on understanding the tail behavior of normalizing flows through a mathematical and statistical way. Motivated by Jaini et al 2020's work on learning long-tailed distribution via triangular flows, this work proves that the marginal tailedness can be controlled by the tailedness of the marginals of the base distribution in flow-based models. Based on this theoretical insight, the authors propose a new algorithm by leveraging a data-driven permutation scheme to enable a correct tail behavior of the target distribution.    

The paper proposes an extension to Tail-adaptive flows for learning the tail behavior of target distributions using normalizing flows. The authors propose to learn the tail behavior by learning flows that match the tail properties of the marginal distributions. They achieve this by using a source distribution consisting of marginal distributions with tail properties matching the target distribution. The tail coefficient of the source distribution is set in a data-driven manner using estimators that can estimate this tail coefficient.