ROBUST ESTIMATION VIA GENERATIVE ADVERSARIAL NETWORKS

1 INTRODUCTION . In the setting of Huber ’ s -contamination model ( Huber , 1964 ; 1965 ) , one has i.i.d observations X1 , ... , Xn ∼ ( 1− ) Pθ + Q , ( 1 ) and the goal is to estimate the model parameter θ . Under the data generating process ( 1 ) , each observation has a 1 − probability to be drawn from Pθ and the other probability to be drawn from the contamination distributionQ . The presence of an unknown contamination distribution poses both statistical and computational challenges to the problem . For example , consider a normal mean estimation problem with Pθ = N ( θ , Ip ) . Due to the contamination of data , the sample average , which is optimal when = 0 , can be arbitrarily far away from the true mean if Q charges a positive probability at infinity . Moreover , even robust estimators such as coordinatewise median and geometric median are proved to be suboptimal under the setting of ( 1 ) ( Chen et al. , 2018 ; Diakonikolas et al. , 2016a ; Lai et al. , 2016 ) . The search for both statistically optimal and computationally feasible procedures has become a fundamental problem in areas including statistics and computer science . For the normal mean estimation problem , it has been shown in Chen et al . ( 2018 ) that the minimax rate with respect to the squared ` 2 loss is pn ∨ 2 , and is achieved by Tukey ’ s median ( Tukey , 1975 ) . Despite the statistical optimality of Tukey ’ s median , its computation is not tractable . In fact , even an approximate algorithm takes O ( eCp ) in time ( Amenta et al. , 2000 ; Chan , 2004 ; Rousseeuw & Struyf , 1998 ) . Recent developments in theoretical computer science are focused on the search of computationally tractable algorithms for estimating θ under Huber ’ s -contamination model ( 1 ) . The success of the efforts started from two fundamental papers Diakonikolas et al . ( 2016a ) ; Lai et al . ( 2016 ) , where two different but related computational strategies “ iterative filtering ” and “ dimension halving ” were proposed to robustly estimate the normal mean . These algorithms can provably achieve the minimax rate pn ∨ 2 up to a poly-logarithmic factor in polynomial time . The main idea behind the two methods is a critical fact that a good robust moment estimator can be certified efficiently by higher moments . This idea was later further extended ( Diakonikolas et al. , 2017 ; Du et al. , 2017 ; Diakonikolas et al. , 2016b ; 2018a ; c ; b ; Kothari et al. , 2018 ) to develop robust and computable procedures for various other problems . However , many of the computationally feasible procedures for robust mean estimation in the literature rely on the knowledge of covariance matrix and sometimes the knowledge of contamination proportion . Even though these assumptions can be relaxed , nontrivial modifications of the algorithms are required for such extensions and statistical error rates may also be affected . Compared with these computationally feasible procedures proposed in the recent literature for robust estimation , Tukey ’ s median ( 9 ) and other depth-based estimators ( Rousseeuw & Hubert , 1999 ; Mizera , 2002 ; Zhang , 2002 ; Mizera & Müller , 2004 ; Paindaveine & Van Bever , 2017 ) have some indispensable advantages in terms of their statistical properties . First , the depth-based estimators have clear objective functions that can be interpreted from the perspective of projection pursuit ( Mizera , 2002 ) . Second , the depth-based procedures are adaptive to unknown nuisance parameters in the models such as covariance structures , contamination proportion , and error distributions ( Chen et al. , 2018 ; Gao , 2017 ) . Last but not least , Tukey ’ s depth and other depth functions are mostly designed for robust quantile estimation , while the recent advancements in the theoretical computer science literature are all focused on robust moments estimation . Although this is not an issue when it comes to normal mean estimation , the difference is fundamental for robust estimation under general settings such as elliptical distributions where moments do not necessarily exist . Given the desirable statistical properties discussed above , this paper is focused on the development of computational strategies of depth-like procedures . Our key observation is that robust estimators that are maximizers of depth functions , including halfspace depth , regression depth and covariance matrix depth , can all be derived under the framework of f -GAN ( Nowozin et al. , 2016 ) . As a result , these depth-based estimators can be viewed as minimizers of variational lower bounds of the total variation distance between the empirical measure and the model distribution ( Proposition 2.1 ) . This observation allows us to leverage the recent developments in the deep learning literature to compute these variational lower bounds through neural network approximations . Our theoretical results give insights on how to choose appropriate neural network classes that lead to minimax optimal robust estimation under Huber ’ s -contamination model . In particular , Theorem 3.1 and 3.2 characterize the networks which can robustly estimate the Gaussian mean by TV-GAN and JS-GAN , respectively ; Theorem 4.1 is an extension to robust location estimation under the class of elliptical distributions which includes Cauchy distribution whose mean does not exist . Numerical experiments in Section 5 are provided to show the success of these GANs . 2 ROBUST ESTIMATION AND f -GAN We start with the definition of f -divergence ( Csiszár , 1964 ; Ali & Silvey , 1966 ) . Given a strictly convex function f that satisfies f ( 1 ) = 0 , the f -GAN between two probability distributions P and Q is defined by Df ( P‖Q ) = ∫ f ( p q ) dQ . ( 2 ) Here , we use p ( · ) and q ( · ) to stand for the density functions of P and Q with respect to some common dominating measure . For a fully rigorous definition , see Polyanskiy & Wu ( 2017 ) . Let f∗ be the convex conjugate of f . That is , f∗ ( t ) = supu∈domf ( ut− f ( u ) ) . A variational lower bound of ( 2 ) is Df ( P‖Q ) ≥ sup T∈T [ EPT ( X ) − EQf∗ ( T ( X ) ) ] . ( 3 ) Note that the inequality ( 3 ) holds for any class T , and it becomes an equality whenever the class T contains the function f ′ ( p/q ) ( Nguyen et al. , 2010 ) . For notational simplicity , we also use f ′ for an arbitrary element of the subdifferential when the derivative does not exist . With i.i.d . observations X1 , ... , Xn ∼ P , the variational lower bound ( 3 ) naturally leads to the following learning method P̂ = argmin Q∈Q sup T∈T [ 1 n n∑ i=1 T ( Xi ) − EQf∗ ( T ( X ) ) ] . ( 4 ) The formula ( 4 ) is a powerful and general way to learn the distribution P from its i.i.d . observations . It is known as f -GAN ( Nowozin et al. , 2016 ) , an extension of GAN ( Goodfellow et al. , 2014 ) , which stands for generative adversarial networks . The idea is to find a P̂ so that the best discriminator T in the class T can not tell the difference between P̂ and the empirical distribution 1n ∑n i=1 δXi . 2.1 f -LEARNING : A UNIFIED FRAMEWORK Our f -Learning framework is based on a special case of the variational lower bound ( 3 ) . That is , Df ( P‖Q ) ≥ sup Q̃∈Q̃Q [ EP f ′ ( q̃ ( X ) q ( X ) ) − EQf∗ ( f ′ ( q̃ ( X ) q ( X ) ) ) ] , ( 5 ) where q̃ ( · ) stands for the density function of Q̃ . Note that here we allow the class Q̃Q to depend on the distribution Q in the second argument of Df ( P‖Q ) . Compare ( 5 ) with ( 3 ) , and it is easy to realize that ( 5 ) is a special case of ( 3 ) with T = TQ = { f ′ ( q̃ q ) : q̃ ∈ Q̃Q } . ( 6 ) Moreover , the inequality ( 5 ) becomes an equality as long as P ∈ Q̃Q . The sample version of ( 5 ) leads to the following learning method P̂ = argmin Q∈Q sup Q̃∈Q̃Q [ 1 n n∑ i=1 f ′ ( q̃ ( Xi ) q ( Xi ) ) − EQf∗ ( f ′ ( q̃ ( X ) q ( X ) ) ) ] . ( 7 ) The learning method ( 7 ) will be referred to as f -Learning in the sequel . It is a very general framework that covers many important learning procedures as special cases . For example , consider the special case where Q̃Q = Q̃ independent of Q , Q = Q̃ , and f ( x ) = x log x . Direct calculations give f ′ ( x ) = log x + 1 and f∗ ( t ) = et−1 . Therefore , ( 7 ) becomes P̂ = argmin Q∈Q sup Q̃∈Q 1 n n∑ i=1 log q̃ ( Xi ) q ( Xi ) = argmax q∈Q 1 n n∑ i=1 log q ( Xi ) , which is the maximum likelihood estimator ( MLE ) . 2.2 TV-LEARNING AND DEPTH-BASED ESTIMATORS . An important generator f that we will discuss here is f ( x ) = ( x−1 ) + . This leads to the total variation distance Df ( P‖Q ) = 12 ∫ |p− q| . With f ′ ( x ) = I { x ≥ 1 } and f∗ ( t ) = tI { 0 ≤ t ≤ 1 } , the TV-Learning is given by P̂ = argmin Q∈Q sup Q̃∈QQ [ 1 n n∑ i=1 I { q̃ ( Xi ) q ( Xi ) ≥ 1 } −Q ( q̃ q ≥ 1 ) ] . ( 8 ) A closely related idea was previously explored by Yatracos ( 1985 ) ; Devroye & Lugosi ( 2012 ) . The following proposition shows that when Q̃Q approaches toQ in some neighborhood , TV-Learning leads to robust estimators that are defined as the maximizers of various depth functions including Tukey ’ s depth , regression depth , and covariance depth . Proposition 2.1 . The TV-Learning ( 8 ) includes the following special cases : 1 . Tukey ’ s halfspace depth : Take Q = { N ( η , Ip ) : η ∈ Rp } and Q̃η = { N ( η̃ , Ip ) : ‖η̃ − η‖ ≤ r } . As r → 0 , ( 8 ) becomes θ̂ = argmax η∈Rp inf ‖u‖=1 1 n n∑ i=1 I { uT ( Xi − η ) ≥ 0 } . ( 9 ) 2 . Regression depth : Take Q = { Py , X = Py|XPX : Py|X = N ( X T η , 1 ) , η ∈ Rp } , and Q̃η = { Py , X = Py|XPX : Py|X = N ( X T η̃ , 1 ) , ‖η̃ − η‖ ≤ r } . As r → 0 , ( 8 ) becomes θ̂ = argmax η∈Rp inf ‖u‖=1 1 n n∑ i=1 I { uTXi ( yi −XTi η ) ≥ 0 } . ( 10 ) 3 . Covariance matrix depth : Take Q = { N ( 0 , Γ ) : Γ ∈ Ep } , where Ep stands for the class of p × p covariance matrices , and Q̃Γ = { N ( 0 , Γ̃ ) : Γ̃−1 = Γ−1 + r̃uuT ∈ Ep , |r̃| ≤ r , ‖u‖ = 1 } . As r → 0 , ( 8 ) becomes Σ̂ = argmin Γ∈Ep sup ‖u‖=1 [ ( 1 n n∑ i=1 I { |uTXi|2 ≤ uTΓu } − P ( χ21 ≤ 1 ) ) ( 11 ) ∨ ( 1 n n∑ i=1 I { |uTXi|2 > uTΓu } − P ( χ21 > 1 ) ) ] . The formula ( 9 ) is recognized as Tukey ’ s median , the maximizer of Tukey ’ s halfspace depth . A traditional understanding of Tukey ’ s median is that ( 9 ) maximizes the halfspace depth ( Donoho & Gasko , 1992 ) so that θ̂ is close to the centers of all one-dimensional projections of the data . In the f -Learning framework , N ( θ̂ , Ip ) is understood to be the minimizer of a variational lower bound of the total variation distance . The formula ( 10 ) gives the estimator that maximizes the regression depth proposed by Rousseeuw & Hubert ( 1999 ) . It is worth noting that the derivation of ( 10 ) does not depend on the marginal distribution PX in the linear regression model . Finally , ( 11 ) is related to the covariance matrix depth ( Zhang , 2002 ; Chen et al. , 2018 ; Paindaveine & Van Bever , 2017 ) . All of the estimators ( 9 ) , ( 10 ) and ( 11 ) are proved to achieve the minimax rate for the corresponding problems under Huber ’ s -contamination model ( Chen et al. , 2018 ; Gao , 2017 ) . 2.3 FROM f -LEARNING TO f -GAN The connection to various depth functions shows the importance of TV-Learning in robust estimation . However , it is well-known that depth-based estimators are very hard to compute ( Amenta et al. , 2000 ; van Kreveld et al. , 1999 ; Rousseeuw & Struyf , 1998 ) , which limits their applications only for very low-dimensional problems . On the other hand , the general f -GAN framework ( 4 ) has been successfully applied to learn complex distributions and images in practice ( Goodfellow et al. , 2014 ; Radford et al. , 2015 ; Salimans et al. , 2016 ) . The major difference that gives the computational advantage to f -GAN is its flexibility in terms of designing the discriminator class T using neural networks compared with the pre-specified choice ( 6 ) in f -Learning . While f -Learning provides a unified perspective in understanding various depth-based procedures in robust estimation , we can step back into the more general f -GAN for its computational advantages , and to design efficient computational strategies .

Summary: The paper considers the problem of robust high dimensional estimation in Huber’s contamination model. The algorithm is given samples from a distribution (1 - eps) * P + eps * Q, where P is a “nice” distribution (e.g. a Gaussian), eps is the fraction of contaminated points, and Q is some unconstrained noise distribution. The goal is then to estimate parameters of P as well as possible, given this noise. The settings they primarily consider in this paper are when P is a Gaussian with unknown mean and identity covariance, or when it is a Gaussian with unknown covariance. Classical estimators such as Tukey depth or matrix depth for these problems achieve optimal minimax rates, but are computationally expensive to compute. However, recent work of [1,2] propose efficient estimators for this problem that (nearly) achieve these rates.