Marginal Tail-Adaptive Normalizing Flows

1 INTRODUCTION . Heavy-tailed distributions are known to occur in various applications in biology , finance , social sciences , and more . Examples for such observations include the length of protein sequences in genomes ( Koonin et al. , 2006 ) , returns of stocks ( Gabaix et al. , 2003 ) , or the size of cities ( Gabaix , 1999 ) . Applications that are tightly connected to typical deep learning applications include the frequency of class examples in image classification ( Horn & Perona , 2017 ) and the frequency of words ( Zipf , 1949 ) in natural language processing . From a theoretical point of view , this is not surprising since heavy-tailed distributions emerge from several circumstances , including the limiting distribution in the generalized central limit theorem , of a multiplicative process , or as the limit of an extremal process ( Nair et al. , 2013 ) . Given the frequency of occurrence , developing generative models that allow to learn heavy-tailed distributions is essential . Normalizing Flows ( NFs ( Rippel & Adams , 2013 ; Tabak & Turner , 2013 ; Dinh et al. , 2015 ; Rezende & Mohamed , 2015 ) ) are a popular class of deep generative models . Despite their success in learning tractable distributions where both sampling and density evaluation can be efficient and exact , their ability to model heavy tailed distributions is known to be limited . Jaini et al . ( 2020 ) identified the problem that a range of NFs ( e.g . vanilla triangular flows with a Gaussian base distribution ) are unable to map a light-tailed distribution to a heavy-tailed distribution . They propose to solve this issue by replacing the Gaussian base distribution by a multivariate t-distribution with one learnable degree of freedom . While this allows to model distributions with a heavy-tailed euclidean norm , we show that modeling multivariate distributions , where some of the marginals are heavy- and some are light-tailed , still poses a problem . Contributions Our contributions in this work , that extend the results of Jaini et al . ( 2020 ) , are the following . First , we prove that a triangular affine NF using a base distribution with solely heavytailed marginals is only able to provide a target distribution with just heavy-tailed marginals as well . Consequently , such a NF is not capable of learning distributions with mixed marginal tail behavior . Second , we derive a result that states conditions under which the marginal tailedness of the base distribution can be preserved . Third , based on these theoretical findings , we propose a novel kind of triangular NF that allows to learn distributions with heavy- and light-tailed marginals . The new model is called marginally Tail-Adaptive Flows ( mTAFs ) , and as illustrated in Figure 1 , combines • estimators from extreme value theory ( Hill , 1975 ; Dekkers et al. , 1989 ; Csorgo et al. , 1985 ) to initially assess heavy-tailedness of the target ’ s marginals ; • a flexible and trainable base distribution based on the estimated tail behavior of the target distribution ; • a new permutation-scheme between flow-layers that ensures the correct tail behavior of the estimated target distribution . Finally , we conduct an experimental analysis demonstrating the superior performance of the proposed mTAFs in comparison to other flow models . Furthermore , we present a new sample generation scheme , motivated by our theory , which successfully generates joint samples that are from the tails of a specified marginal . Notational Conventions In the following , we will denote random variables by bold letters , such as x , and its realisations by non-bold letters , x . We use this notation for multivariate and for univariate random variables . Further , we denote the jth component of x by xj , and x≤j or x < j are the first j or j − 1 components of x , respectively . We denote the random variable representing the base distribution by z and the random variable representing the target distribution by x . Further , for notational convenience , we denote the probability density functions ( PDFs ) of x and z by p and q , whereas marginal PDFs are denoted by pj and qj , respectively . Finally , we assume that both random variables x and z have continuous and positive density on RD , i.e p ( x ) , q ( z ) > 0 for all x , z ∈ RD , where D is the dimensionality of x and z . 2 BACKGROUND . In this section , we give a brief introduction to heavy-tailed distributions and present needed background knowledge about normalizing flows . 2.1 HEAVY-TAILED DISTRIBUTIONS . Heavy-tailed distributions are distributions that have heavier tails ( i.e . decay slower ) than the exponential distribution . Loosely speaking , slowly decaying tails allow to model distributions that generate samples , which differ by a large magnitude from the rest of the samples . For a univariate random variable x we define heavy-tailedness via its moment-generating function1 : Definition 1 ( Heavy-Tailed Random Variables ) . Consider a random variable x ∈ R with PDF p. Then , we say that x is heavy-tailed if and only if ∀λ > 0 : Ex [ eλx ] =∞ . The function mp ( λ ) : = Ex [ exp ( λx ) ] is known as the moment-generating function of x . Random variables that are not heavy-tailed are said to be light-tailed . Note that this definition is , strictly speaking , merely a definition for heavy right tails . We say a random variable x ∈ R has heavy left tails if −x has heavy tails according to Definition 1 . For simplicity of derivations and w.l.o.g. , we proceed with this definition but the derived results can analogously be applied to left tails . We can assess the degree of tailedness of a distribution . While there are many equivalent notions of the so called tail index , the most straight-forward definition is via the existence of moments : Definition 2 ( Tail Index ) . A random variable x ∈ R with PDF p is said to have tail index2 α if it holds that Ex [ |x|β ] { < ∞ , if β < α , =∞ , if β > α . Since the tail index is tightly related to the decay rate of the PDF , it enables us to assess the degree of heavy-tailedness of a random variable . Therefore , estimation of the tail index became an important objective in extreme value theory and statistical risk assessment ( see e.g . Embrechts et al. , 2013 ) . Since the existence of the moment does not depend on the “ body ” of x but only on the tails of x ( see Proposition 3 in Section A.1 in the Appendix ) , estimating the tail index by fitting a full parametric model to all data e.g . via likelihood maximization leads to a biased estimator . Instead , semi-parametric estimators have been developed , which aim to fit a distribution only on the tails . Popular methods for tail estimation include the Hill estimator ( Hill , 1975 ) , the moment estimator ( Dekkers et al. , 1989 ) , and kernel-based estimators ( Csorgo et al. , 1985 ) . In Section B.1 of the Appendix , we discuss these tail estimators and review some practical issues with these . An example of a heavy tailed distribution is the standardized t-distribution , which has parameter ν > 0 referred to as the degree of freedom and a density function given by p ( x ) : = Γ ( ν+1 2 ) √ νπΓ ( ν 2 ) ( 1 + x2 ν ) − ν+12 , x ∈ R , where Γ is the Gamma function . It is known that the t-distribution has tail index ν ( see e.g . Kirkby et al . ( 2019 ) for a detailed reference ) . In the multivariate setting , there exist various definitions of heavy-tailedness . For instance Resnick ( 2004 ) make use of a definition based on multivariate regular variation . Jaini et al . ( 2020 ) define a multivariate random variable x to be heavy-tailed if the ` 2-norm is heavy-tailed , a property which we refer to as ` 2-heavy tailed , and which is formally defined as follows : 1One can readily show that this definition is equivalent to the definition , which compares the tails of x to the tails of an exponential distribution . See Section 1 in Nair et al . ( 2013 ) . 2Notice that the notion of a tail index is only valid for regularly-varying random variables , which are a subclass of heavy-tailed random variables . For the purpose of this work , it is sufficient to consider regularly varying random variables . More details can be found in Nair et al . ( 2013 ) . Definition 3 ( ` 2-Heavy-Tailed ) . Let x ∈ RD be a multivariate random variable . Then , we call x ` 2-heavy-tailed if it holds that ‖x‖ is univariately heavy-tailed according to Definition 1 , where ‖ · ‖ denotes the ` 2-norm . Otherwise , we call x ` 2-light-tailed . 2.2 NORMALIZING FLOWS . The fundamental idea behind NFs is based on the change-of-variables formula for probability density functions ( PDFs ) given in the following theorem . Theorem 1 ( Change-of-Variables ) . Consider random variables x , z ∈ RD and a diffeomorphic map T : RD → RD such that x = T ( z ) . Then , it holds that the PDF of x satisfies p ( x ) = q ( T−1 ( x ) ) ∣∣det JT−1 ( x ) ∣∣ ∀x ∈ RD , ( 1 ) where JT−1 ( x ) is the Jacobian of T−1 evaluated at x ∈ RD . This formula allows us to evaluate the possibly intractable PDF of x if we can evaluate both , the PDF of z and T−1 ( x ) , and efficiently calculate the Jacobian-determinant det JT−1 ( x ) . As T maps z to x , we denote the distribution of z and x as the base and the target distribution , respectively . To model the PDF of x using NFs , it is common to set the base distribution to a standard normal distribution ( i.e. , z ∼ N ( 0 , I ) ) and to employ likelihood maximization to learn a parameterized transformation Tθ : = T ( L ) θ ◦ · · · ◦ T ( 1 ) θ , which , yet , remains tractable and diffeomorphic . Masked autoregressive flows ( MAFs ( Papamakarios et al. , 2017 ) ) are one popular architecture , which employ transformations T = ( T1 , . . . , TD ) > of the form Tj ( z ) : = µj ( z < j ) + exp ( σj ( z < j ) ) zj for j ∈ { 1 , . . . , D } , ( 2 ) where µj and σj are neural networks , which obtain the first j−1 components of z as input and output a scalar . Composing several transformations of the form ( 2 ) , we obtain the MAF . The autoregressive form in ( 2 ) allows us to efficiently evaluate the Jacobian-Determinant due to the diagonal form of JT ( x ) . A crucial issue of such autoregressive models is that a component xj only depends on the previous outputs x < j and , therefore , they can not model a causal relationship in which xj causes xi if i < j . This issue can be solved by applying a permutation before each transformation T ( 1 ) θ , . . . , T ( L ) θ . This is usually a random permutation or the one that reverses the ordering of the components . Therefore , in summary a MAF consists of multiple consecutive layers T ( l ) θ ◦ P ( l ) , where P ( l ) ∈ RD×D is a permutation . MAFs belong to the class of triangular flows , which are defined as flows that consist of diffeomorphisms whose jth output only depends on z≤j . Other examples for triangular flows include RealNVP ( Dinh et al. , 2017 ) , NAF ( Huang et al. , 2018 ) , and SOS ( Jaini et al. , 2019 ) . If the triangular maps are affine linear ( such as in ( 2 ) ) , we call the resulting flow a triangular affine flow . Further types of NFs include invertible ResNets ( Jacobsen et al. , 2018 ; Behrmann et al. , 2019 ; Chen et al. , 2019 ) , continuous flows ( Chen et al. , 2018 ; Grathwohl et al. , 2019 ) , and many more ( Kobyzev et al. , 2020 ) . Tail-Adaptive Flows . Jaini et al . ( 2020 ) investigated the ability of triangular flows to learn heavytailed distributions . The authors have shown that if a triangular affine flow transforms a ` 2-lighttailed distribution , such as the multivariate Gaussian distribution , to a ` 2-heavy-tailed target distribution , then Tθ can not be Lipschitz continuous . And more explicitly , it holds the following . Theorem 2 . ( Jaini et al. , 2020 ) Let z be a ` 2-light-tailed random variable and T be an affine triangular flow such that Tj ( z≤j ) = µj ( z < j ) + σj ( z < j ) zj for all j . If σj is bounded above and µj is Lipschitz for all j , then the transformed variable x is also ` 2-light-tailed . Furthermore , the authors prove that any triangular mapping from an elliptical distribution to a heavier-tailed elliptical distribution must have an unbounded Jacobian-determinant . Clearly , these results illuminate that learning a heavy-tailed distribution using NFs leads to non-Lipschitz transformations and unbounded Jacobians , which inevitably affects training robustness ( Behrmann et al. , 2021 ) . Motivated by these result , Jaini et al . ( 2020 ) propose Tail-Adaptive Flows ( TAF ) , which replace the Gaussian base distribution by a multivariate t-distribution with one learnable degree of freedom .

Summary: This paper focuses on understanding the tail behavior of normalizing flows through a mathematical and statistical way. Motivated by Jaini et al 2020's work on learning long-tailed distribution via triangular flows, this work proves that the marginal tailedness can be controlled by the tailedness of the marginals of the base distribution in flow-based models. Based on this theoretical insight, the authors propose a new algorithm by leveraging a data-driven permutation scheme to enable a correct tail behavior of the target distribution.