Marginal Tail-Adaptive Normalizing Flows

1 INTRODUCTION . Heavy-tailed distributions are known to occur in various applications in biology , finance , social sciences , and more . Examples for such observations include the length of protein sequences in genomes ( Koonin et al. , 2006 ) , returns of stocks ( Gabaix et al. , 2003 ) , or the size of cities ( Gabaix , 1999 ) . Applications that are tightly connected to typical deep learning applications include the frequency of class examples in image classification ( Horn & Perona , 2017 ) and the frequency of words ( Zipf , 1949 ) in natural language processing . From a theoretical point of view , this is not surprising since heavy-tailed distributions emerge from several circumstances , including the limiting distribution in the generalized central limit theorem , of a multiplicative process , or as the limit of an extremal process ( Nair et al. , 2013 ) . Given the frequency of occurrence , developing generative models that allow to learn heavy-tailed distributions is essential . Normalizing Flows ( NFs ( Rippel & Adams , 2013 ; Tabak & Turner , 2013 ; Dinh et al. , 2015 ; Rezende & Mohamed , 2015 ) ) are a popular class of deep generative models . Despite their success in learning tractable distributions where both sampling and density evaluation can be efficient and exact , their ability to model heavy tailed distributions is known to be limited . Jaini et al . ( 2020 ) identified the problem that a range of NFs ( e.g . vanilla triangular flows with a Gaussian base distribution ) are unable to map a light-tailed distribution to a heavy-tailed distribution . They propose to solve this issue by replacing the Gaussian base distribution by a multivariate t-distribution with one learnable degree of freedom . While this allows to model distributions with a heavy-tailed euclidean norm , we show that modeling multivariate distributions , where some of the marginals are heavy- and some are light-tailed , still poses a problem . Contributions Our contributions in this work , that extend the results of Jaini et al . ( 2020 ) , are the following . First , we prove that a triangular affine NF using a base distribution with solely heavytailed marginals is only able to provide a target distribution with just heavy-tailed marginals as well . Consequently , such a NF is not capable of learning distributions with mixed marginal tail behavior . Second , we derive a result that states conditions under which the marginal tailedness of the base distribution can be preserved . Third , based on these theoretical findings , we propose a novel kind of triangular NF that allows to learn distributions with heavy- and light-tailed marginals . The new model is called marginally Tail-Adaptive Flows ( mTAFs ) , and as illustrated in Figure 1 , combines • estimators from extreme value theory ( Hill , 1975 ; Dekkers et al. , 1989 ; Csorgo et al. , 1985 ) to initially assess heavy-tailedness of the target ’ s marginals ; • a flexible and trainable base distribution based on the estimated tail behavior of the target distribution ; • a new permutation-scheme between flow-layers that ensures the correct tail behavior of the estimated target distribution . Finally , we conduct an experimental analysis demonstrating the superior performance of the proposed mTAFs in comparison to other flow models . Furthermore , we present a new sample generation scheme , motivated by our theory , which successfully generates joint samples that are from the tails of a specified marginal . Notational Conventions In the following , we will denote random variables by bold letters , such as x , and its realisations by non-bold letters , x . We use this notation for multivariate and for univariate random variables . Further , we denote the jth component of x by xj , and x≤j or x < j are the first j or j − 1 components of x , respectively . We denote the random variable representing the base distribution by z and the random variable representing the target distribution by x . Further , for notational convenience , we denote the probability density functions ( PDFs ) of x and z by p and q , whereas marginal PDFs are denoted by pj and qj , respectively . Finally , we assume that both random variables x and z have continuous and positive density on RD , i.e p ( x ) , q ( z ) > 0 for all x , z ∈ RD , where D is the dimensionality of x and z . 2 BACKGROUND . In this section , we give a brief introduction to heavy-tailed distributions and present needed background knowledge about normalizing flows . 2.1 HEAVY-TAILED DISTRIBUTIONS . Heavy-tailed distributions are distributions that have heavier tails ( i.e . decay slower ) than the exponential distribution . Loosely speaking , slowly decaying tails allow to model distributions that generate samples , which differ by a large magnitude from the rest of the samples . For a univariate random variable x we define heavy-tailedness via its moment-generating function1 : Definition 1 ( Heavy-Tailed Random Variables ) . Consider a random variable x ∈ R with PDF p. Then , we say that x is heavy-tailed if and only if ∀λ > 0 : Ex [ eλx ] =∞ . The function mp ( λ ) : = Ex [ exp ( λx ) ] is known as the moment-generating function of x . Random variables that are not heavy-tailed are said to be light-tailed . Note that this definition is , strictly speaking , merely a definition for heavy right tails . We say a random variable x ∈ R has heavy left tails if −x has heavy tails according to Definition 1 . For simplicity of derivations and w.l.o.g. , we proceed with this definition but the derived results can analogously be applied to left tails . We can assess the degree of tailedness of a distribution . While there are many equivalent notions of the so called tail index , the most straight-forward definition is via the existence of moments : Definition 2 ( Tail Index ) . A random variable x ∈ R with PDF p is said to have tail index2 α if it holds that Ex [ |x|β ] { < ∞ , if β < α , =∞ , if β > α . Since the tail index is tightly related to the decay rate of the PDF , it enables us to assess the degree of heavy-tailedness of a random variable . Therefore , estimation of the tail index became an important objective in extreme value theory and statistical risk assessment ( see e.g . Embrechts et al. , 2013 ) . Since the existence of the moment does not depend on the “ body ” of x but only on the tails of x ( see Proposition 3 in Section A.1 in the Appendix ) , estimating the tail index by fitting a full parametric model to all data e.g . via likelihood maximization leads to a biased estimator . Instead , semi-parametric estimators have been developed , which aim to fit a distribution only on the tails . Popular methods for tail estimation include the Hill estimator ( Hill , 1975 ) , the moment estimator ( Dekkers et al. , 1989 ) , and kernel-based estimators ( Csorgo et al. , 1985 ) . In Section B.1 of the Appendix , we discuss these tail estimators and review some practical issues with these . An example of a heavy tailed distribution is the standardized t-distribution , which has parameter ν > 0 referred to as the degree of freedom and a density function given by p ( x ) : = Γ ( ν+1 2 ) √ νπΓ ( ν 2 ) ( 1 + x2 ν ) − ν+12 , x ∈ R , where Γ is the Gamma function . It is known that the t-distribution has tail index ν ( see e.g . Kirkby et al . ( 2019 ) for a detailed reference ) . In the multivariate setting , there exist various definitions of heavy-tailedness . For instance Resnick ( 2004 ) make use of a definition based on multivariate regular variation . Jaini et al . ( 2020 ) define a multivariate random variable x to be heavy-tailed if the ` 2-norm is heavy-tailed , a property which we refer to as ` 2-heavy tailed , and which is formally defined as follows : 1One can readily show that this definition is equivalent to the definition , which compares the tails of x to the tails of an exponential distribution . See Section 1 in Nair et al . ( 2013 ) . 2Notice that the notion of a tail index is only valid for regularly-varying random variables , which are a subclass of heavy-tailed random variables . For the purpose of this work , it is sufficient to consider regularly varying random variables . More details can be found in Nair et al . ( 2013 ) . Definition 3 ( ` 2-Heavy-Tailed ) . Let x ∈ RD be a multivariate random variable . Then , we call x ` 2-heavy-tailed if it holds that ‖x‖ is univariately heavy-tailed according to Definition 1 , where ‖ · ‖ denotes the ` 2-norm . Otherwise , we call x ` 2-light-tailed . 2.2 NORMALIZING FLOWS . The fundamental idea behind NFs is based on the change-of-variables formula for probability density functions ( PDFs ) given in the following theorem . Theorem 1 ( Change-of-Variables ) . Consider random variables x , z ∈ RD and a diffeomorphic map T : RD → RD such that x = T ( z ) . Then , it holds that the PDF of x satisfies p ( x ) = q ( T−1 ( x ) ) ∣∣det JT−1 ( x ) ∣∣ ∀x ∈ RD , ( 1 ) where JT−1 ( x ) is the Jacobian of T−1 evaluated at x ∈ RD . This formula allows us to evaluate the possibly intractable PDF of x if we can evaluate both , the PDF of z and T−1 ( x ) , and efficiently calculate the Jacobian-determinant det JT−1 ( x ) . As T maps z to x , we denote the distribution of z and x as the base and the target distribution , respectively . To model the PDF of x using NFs , it is common to set the base distribution to a standard normal distribution ( i.e. , z ∼ N ( 0 , I ) ) and to employ likelihood maximization to learn a parameterized transformation Tθ : = T ( L ) θ ◦ · · · ◦ T ( 1 ) θ , which , yet , remains tractable and diffeomorphic . Masked autoregressive flows ( MAFs ( Papamakarios et al. , 2017 ) ) are one popular architecture , which employ transformations T = ( T1 , . . . , TD ) > of the form Tj ( z ) : = µj ( z < j ) + exp ( σj ( z < j ) ) zj for j ∈ { 1 , . . . , D } , ( 2 ) where µj and σj are neural networks , which obtain the first j−1 components of z as input and output a scalar . Composing several transformations of the form ( 2 ) , we obtain the MAF . The autoregressive form in ( 2 ) allows us to efficiently evaluate the Jacobian-Determinant due to the diagonal form of JT ( x ) . A crucial issue of such autoregressive models is that a component xj only depends on the previous outputs x < j and , therefore , they can not model a causal relationship in which xj causes xi if i < j . This issue can be solved by applying a permutation before each transformation T ( 1 ) θ , . . . , T ( L ) θ . This is usually a random permutation or the one that reverses the ordering of the components . Therefore , in summary a MAF consists of multiple consecutive layers T ( l ) θ ◦ P ( l ) , where P ( l ) ∈ RD×D is a permutation . MAFs belong to the class of triangular flows , which are defined as flows that consist of diffeomorphisms whose jth output only depends on z≤j . Other examples for triangular flows include RealNVP ( Dinh et al. , 2017 ) , NAF ( Huang et al. , 2018 ) , and SOS ( Jaini et al. , 2019 ) . If the triangular maps are affine linear ( such as in ( 2 ) ) , we call the resulting flow a triangular affine flow . Further types of NFs include invertible ResNets ( Jacobsen et al. , 2018 ; Behrmann et al. , 2019 ; Chen et al. , 2019 ) , continuous flows ( Chen et al. , 2018 ; Grathwohl et al. , 2019 ) , and many more ( Kobyzev et al. , 2020 ) . Tail-Adaptive Flows . Jaini et al . ( 2020 ) investigated the ability of triangular flows to learn heavytailed distributions . The authors have shown that if a triangular affine flow transforms a ` 2-lighttailed distribution , such as the multivariate Gaussian distribution , to a ` 2-heavy-tailed target distribution , then Tθ can not be Lipschitz continuous . And more explicitly , it holds the following . Theorem 2 . ( Jaini et al. , 2020 ) Let z be a ` 2-light-tailed random variable and T be an affine triangular flow such that Tj ( z≤j ) = µj ( z < j ) + σj ( z < j ) zj for all j . If σj is bounded above and µj is Lipschitz for all j , then the transformed variable x is also ` 2-light-tailed . Furthermore , the authors prove that any triangular mapping from an elliptical distribution to a heavier-tailed elliptical distribution must have an unbounded Jacobian-determinant . Clearly , these results illuminate that learning a heavy-tailed distribution using NFs leads to non-Lipschitz transformations and unbounded Jacobians , which inevitably affects training robustness ( Behrmann et al. , 2021 ) . Motivated by these result , Jaini et al . ( 2020 ) propose Tail-Adaptive Flows ( TAF ) , which replace the Gaussian base distribution by a multivariate t-distribution with one learnable degree of freedom .

Summary: The paper proposes an extension to Tail-adaptive flows for learning the tail behavior of target distributions using normalizing flows. The authors propose to learn the tail behavior by learning flows that match the tail properties of the marginal distributions. They achieve this by using a source distribution consisting of marginal distributions with tail properties matching the target distribution. The tail coefficient of the source distribution is set in a data-driven manner using estimators that can estimate this tail coefficient.