Efficient Estimators for Heavy-Tailed Machine Learning

1 INTRODUCTION . Existing estimators in machine learning are largely designed for “ thin-tailed ” data , such as those coming from a Gaussian distribution . Past work in statistical estimation has given sufficient evidence that in the absence of these “ thin-tails ” , classical estimators based on minimizing the empirical error perform poorly ( Catoni , 2012 ; Lugosi et al. , 2019 ) . Theoretical guarantees for methods commonly used in machine learning usually place assumptions on the tails of the underlying distributions that are analyzed . For instance , rates of convergences proven for a variety of stochastic optimization procedures assume that the distribution of gradients have bounded variance ( for e.g. , Zou et al . ( 2018 ) ) or in some cases are sub-Gaussian ( for e.g. , Li & Orabona ( 2019 ) ) . Thus , these guarantees are no longer applicable for heavy-tailed gradient distributions . From a practical point of view however , this is a less than desirable state of affairs : heavy-tailed distributions are ubiquitous in a variety of fields including large scale biological datasets and financial datasets among others ( Fan et al. , 2016 ; Zhou et al. , 2017 ; Fan et al. , 2017 ) . While this may be argued as just artifacts of the domain , recent work has found interesting evidence of heavy-tailed distributions in the intermediate outputs of machine learning algorithms . Specifically , recent work by Simsekli et al . ( 2019 ) and Zhang et al . ( 2019 ) have provided empirical evidence about the existence of such heavy-tailed distributions , especially during neural network training for supervised learning tasks . Following these empirical analyses of Simsekli et al . ( 2019 ) and Zhang et al . ( 2019 ) , we look for sources of heavy-tailed gradients arising during the training of modern generative model based unsupervised learning tasks as well . In our preliminary investigation , we noticed that the distribution of gradient norms i.e. , �gt�2 are indeed heavy-tailed . These are showcased in Figure 1 ; Figures 1a and 1b show the distribution of gradient norms obtained while training the generator of a DCGAN ( Radford et al. , 2015 ) and Real-NVP ( Dinh et al. , 2016 ) on the CIFAR-10 dataset , respectively . These distributions are noticeably heavy-tailed , especially when juxtaposed with those obtained from a Gaussian distribution ( Figure 1c ) . We discuss more about the empirical setup in Section 5.2 . Interestingly , in all the supervised and unsupervised machine learning problems discussed above , we merely need to compute expectations of these varied random heavy-tailed quantities . For instance , mini-batch gradient descent involves aggregating a batch of gradients pertaining to each sample in the mini-batch . Typically , this aggregation is performed by considering the sample mean , and this is a reasonable choice due to its simplicity as an estimate of the expectation of the random gradient . For computing the mean of such heavy-tailed gradient distributions , the sample mean however is highly sub-optimal . This is because sample mean estimates are greatly skewed by samples on the tail . Thus gradient estimates using these sub-optimal sample means of gradients do not necessarily point in the right direction leading to bad solutions , prolonged training time , or a mixture of both . Thus , a critical requirement for training of modern machine learning models is a scalable estimation of the mean of a heavy-tailed random vector . Note that such computations of mean of sample gradients are done in each iteration of ( stochastic ) gradient descent , so that we require that the heavy-tailed mean estimation be extremely scalable , yet with strong guarantees . Note that once we have such a scalable heavy-tailed mean estimator , we could simply use it to compute robust gradient estimates Prasad et al . ( 2020 ) , and learn generic statistical models . We summarize our contributions as follows : • We extend recent analyses of heavy-tailed behavior in machine learning , and provide novel em- pirical evidence of heavy-tailed gradients while training modern generative models such as generative adversarial networks ( GANs ) and invertible flow models . • To combat the issue of aggregating gradient samples from a heavy-tailed distribution , we propose a practical and easy-to-implement algorithm for heavy-tailed mean estimation with provable guarantees on the error of the estimate . • We use the proposed mean estimator to compute robust gradient estimates , which allows us to learn generalized linear models in the heavy-tailed setting , with strong guarantees on the estimation errors . • Finally , we propose a heuristic approximation of the mean estimation algorithm , which scales to random vectors with millions of variables . Accordingly , we use this heuristic to compute robust gradients of large-scale deep learning models with millions of parameters . We show that training with this heuristic outperforms a variety of practical baselines . Notation and other definitions . Let x be a random vector with mean µ . We say that the x has bounded 2k-moments if for all v ∈ Sp−1 ( unit ball ) , E [ ( vT ( x− µ ) ) 2k ] ≤ C2k � E [ ( vT ( x− µ ) ) 2 ] �k . Throughout the paper , we use c , c1 , c2 , . . . , C , C1 , C2 , . . . to denote positive universal constants . 2 EFFICIENT AND PRACTICAL MEAN ESTIMATION . We begin by formalizing the notion of heavy-tailed distributions . Definition 1 ( Heavy-Tailed Distribution ( Resnick , 2007 ) ) . A non-negative random variable X is called heavy-tailed if the tail probability P ( X > t ) is asymptotically proportional to t−α ∗ , where α∗ is a positive constant called the tail index of X . Intuitively , this definition states that if the tail of the distribution P ( X > t ) decreases at a rate slower that e−t , then the distribution is heavy-tailed . An interesting consequence of this definition is the non-existence of higher order moments . Specifically , one can show that the quantity E [ Xα ] is finite for any α if and only if α < α∗ and X is a heavy-tailed random variable with tail index α∗ . In recent statistical estimation literature ( for e.g. , Minsker ( 2015 ) ; Hopkins ( 2018 ) ; Lugosi & Mendelson ( 2019 ) ) , heavy-tailed distributions are defined by the absence of finite higher order moments . In the heavy-tailed mean estimation task , we observe n samples x1 , . . . , xn drawn independently from a distribution P where xi ∈ Rp , which is only assumed to have finite low-order moments , therefore heavy-tailed . The goal of past work ( Catoni , 2012 ; Minsker , 2015 ; Lugosi et al. , 2019 ; Catoni & Giulini , 2017 ) has been to design an estimator �θn of the true mean µ of P which has a small �2-error with high-probability . As a benchmark for estimators in the heavy-tailed model , we observe that when P is the multivariate normal ( or equivalently a sub-Gaussian ) distribution with mean µ and covariance Σ , the sample mean �µn = 1/n � i xi satisfies , with probability at least 1− δ 1 : ��µn − µ�2 � � trace ( Σ ) n + � �Σ�2 log ( 1/δ ) n def = OPTn , Σ , δ ( 1 ) Seminal work by Catoni ( 2012 ) showed that the sample mean is extremely sub-optimal , while more recent work by Lugosi et al . ( 2019 ) showed that the sub-Gaussian error bound is achievable while only assuming that P has finite variance i.e. , 2nd moment . In the multivariate setting , Minsker ( 2015 ) showed that the extremely practical geometric-median-of-means estimator ( GMOM ) achieves a sub-optimal error bound by showing that with probability at least 1− δ : ��θMOM , δ − µ�2 � � trace ( Σ ) log ( 1/δ ) n . ( 2 ) Computationally intractable estimators that truly achieve the sub-Gaussian error bound were proposed by Lugosi et al . ( 2019 ) ; Catoni & Giulini ( 2017 ) . Hopkins ( 2018 ) and later Cherapanamjeri et al . ( 2019 ) developed a sum-of-squares based relaxation of the estimator in Lugosi et al . ( 2019 ) , thereby giving a polynomial time algorithm which achieves optimal rates . More recent work has studied the problem of mean estimation , focusing on constructing theoretically fast polynomial time estimators ( Dong et al. , 2019 ; Lugosi & Mendelson , 2019 ; Diakonikolas & Kane , 2019 ; Lei et al. , 2020 ; Lecué & Depersin , 2019 ) . However , these estimators have several hyperparameters , which require to be tuned for practice , making them impractical . Now , we present our algorithm Filterpd for heavy-tailed mean estimation , and is formally stated as Algorithm 1 . It proceeds in an iterative fashion , by ( 1 ) computing the leading eigenvector of the empirical covariance matrix ( Step ( 3 ) ) , ( 2 ) projecting points along this leading eigenvector ( Step ( 4 ) ) , and ( 3 ) randomly sampling points based on their projection scores ( Step ( 5 ) and ( 6 ) ) . This procedure is repeated for a fixed number of steps . Algorithm 1 Filterpd - Heavy Tailed Mean Estimator Require : Samples S = { zi } ni=1 , Iterations T ∗ 1 : for t = 1 to T ∗ do 2 : Compute �θS = 1|S| |S|� i=1 zi and �ΣS = 1|S| |S|� i=1 ( zi − �θS ) ⊗2 3 : Let v be the leading eigenvector of �ΣS 4 : For each zi , let τi def = � vT ( zi − �θS ) �2 be its score . 5 : Sample a point z ∼ S according to Pr ( zi ) ∝ τi 6 : Remove sample z from S i.e . S = S \ { z } 7 : end for 8 : return 1|S| |S|� i=1 zi Our proposed algorithm is primarily based on the SVD-based filtering algorithm , which has appeared in different forms ( Klivans et al. , 2009 ; Awasthi et al. , 2014 ) and was recently reused in Diakonikolas et al . ( 2016 ; 2017 ) for adversarial mean estimation . For instance , the algorithm in Diakonikolas et al . ( 2017 ) follows a similar procedure , but remove a subset of points at a step depending on magnitude of the projection score . Our first main result is presented as follows : Theorem 1 . Suppose { zi } ni=1 ∼ P with zi ∈ Rp for all i , where P has bounded 4th moment and n satisfies n ≥ Cr2 ( Σ ) log 2 ( p/δ ) log ( 1/δ ) , r ( Σ ) def = trace ( Σ ) ||Σ||2 ( 3 ) 1Here and throughout our paper we use the notation � to denote an inequality with universal constants dropped for conciseness . Then , Filterpd when instantiated for T ∗ = �C log ( 1/δ ) � steps returns an estimate �θδ which satisfies with probability at least 1− 4δ , δ ∈ ( 0 , 0.25 ) : ��θδ − µ�2 � OPTn , Σ , δ Remarks : Theorem 1 shows that when n is sufficiently large , Filterpd returns a mean estimate that achieves the optimal sub-Gaussian deviation bound . This algorithm is also extremely practical as compared to existing algorithms ( for e.g. , Lei et al . ( 2020 ) ) and engenders development of scalable variants , which we later describe in Section 4 . We defer the proofs of Theorem 1 to the appendix .

Summary: The paper studies the problem of high-probability mean estimation for heavy-tailed distributions, i.e., constructing a high-probability confidence intervals for the mean, when the underlying distribution has only finite low-degree moments.  The paper motivates this problem from the view-point of machine learning algorithms, where the gradients are heavy-tailed. 	Especially in deep generative networks, the paper  highlights the heavy-tailed nature of gradients via experiments. On a theoretical side, the paper derives bounds for mean estimation when the distribution has bounded fourth-moments.