ROBUST ESTIMATION VIA GENERATIVE ADVERSARIAL NETWORKS

1 INTRODUCTION . In the setting of Huber ’ s -contamination model ( Huber , 1964 ; 1965 ) , one has i.i.d observations X1 , ... , Xn ∼ ( 1− ) Pθ + Q , ( 1 ) and the goal is to estimate the model parameter θ . Under the data generating process ( 1 ) , each observation has a 1 − probability to be drawn from Pθ and the other probability to be drawn from the contamination distributionQ . The presence of an unknown contamination distribution poses both statistical and computational challenges to the problem . For example , consider a normal mean estimation problem with Pθ = N ( θ , Ip ) . Due to the contamination of data , the sample average , which is optimal when = 0 , can be arbitrarily far away from the true mean if Q charges a positive probability at infinity . Moreover , even robust estimators such as coordinatewise median and geometric median are proved to be suboptimal under the setting of ( 1 ) ( Chen et al. , 2018 ; Diakonikolas et al. , 2016a ; Lai et al. , 2016 ) . The search for both statistically optimal and computationally feasible procedures has become a fundamental problem in areas including statistics and computer science . For the normal mean estimation problem , it has been shown in Chen et al . ( 2018 ) that the minimax rate with respect to the squared ` 2 loss is pn ∨ 2 , and is achieved by Tukey ’ s median ( Tukey , 1975 ) . Despite the statistical optimality of Tukey ’ s median , its computation is not tractable . In fact , even an approximate algorithm takes O ( eCp ) in time ( Amenta et al. , 2000 ; Chan , 2004 ; Rousseeuw & Struyf , 1998 ) . Recent developments in theoretical computer science are focused on the search of computationally tractable algorithms for estimating θ under Huber ’ s -contamination model ( 1 ) . The success of the efforts started from two fundamental papers Diakonikolas et al . ( 2016a ) ; Lai et al . ( 2016 ) , where two different but related computational strategies “ iterative filtering ” and “ dimension halving ” were proposed to robustly estimate the normal mean . These algorithms can provably achieve the minimax rate pn ∨ 2 up to a poly-logarithmic factor in polynomial time . The main idea behind the two methods is a critical fact that a good robust moment estimator can be certified efficiently by higher moments . This idea was later further extended ( Diakonikolas et al. , 2017 ; Du et al. , 2017 ; Diakonikolas et al. , 2016b ; 2018a ; c ; b ; Kothari et al. , 2018 ) to develop robust and computable procedures for various other problems . However , many of the computationally feasible procedures for robust mean estimation in the literature rely on the knowledge of covariance matrix and sometimes the knowledge of contamination proportion . Even though these assumptions can be relaxed , nontrivial modifications of the algorithms are required for such extensions and statistical error rates may also be affected . Compared with these computationally feasible procedures proposed in the recent literature for robust estimation , Tukey ’ s median ( 9 ) and other depth-based estimators ( Rousseeuw & Hubert , 1999 ; Mizera , 2002 ; Zhang , 2002 ; Mizera & Müller , 2004 ; Paindaveine & Van Bever , 2017 ) have some indispensable advantages in terms of their statistical properties . First , the depth-based estimators have clear objective functions that can be interpreted from the perspective of projection pursuit ( Mizera , 2002 ) . Second , the depth-based procedures are adaptive to unknown nuisance parameters in the models such as covariance structures , contamination proportion , and error distributions ( Chen et al. , 2018 ; Gao , 2017 ) . Last but not least , Tukey ’ s depth and other depth functions are mostly designed for robust quantile estimation , while the recent advancements in the theoretical computer science literature are all focused on robust moments estimation . Although this is not an issue when it comes to normal mean estimation , the difference is fundamental for robust estimation under general settings such as elliptical distributions where moments do not necessarily exist . Given the desirable statistical properties discussed above , this paper is focused on the development of computational strategies of depth-like procedures . Our key observation is that robust estimators that are maximizers of depth functions , including halfspace depth , regression depth and covariance matrix depth , can all be derived under the framework of f -GAN ( Nowozin et al. , 2016 ) . As a result , these depth-based estimators can be viewed as minimizers of variational lower bounds of the total variation distance between the empirical measure and the model distribution ( Proposition 2.1 ) . This observation allows us to leverage the recent developments in the deep learning literature to compute these variational lower bounds through neural network approximations . Our theoretical results give insights on how to choose appropriate neural network classes that lead to minimax optimal robust estimation under Huber ’ s -contamination model . In particular , Theorem 3.1 and 3.2 characterize the networks which can robustly estimate the Gaussian mean by TV-GAN and JS-GAN , respectively ; Theorem 4.1 is an extension to robust location estimation under the class of elliptical distributions which includes Cauchy distribution whose mean does not exist . Numerical experiments in Section 5 are provided to show the success of these GANs . 2 ROBUST ESTIMATION AND f -GAN We start with the definition of f -divergence ( Csiszár , 1964 ; Ali & Silvey , 1966 ) . Given a strictly convex function f that satisfies f ( 1 ) = 0 , the f -GAN between two probability distributions P and Q is defined by Df ( P‖Q ) = ∫ f ( p q ) dQ . ( 2 ) Here , we use p ( · ) and q ( · ) to stand for the density functions of P and Q with respect to some common dominating measure . For a fully rigorous definition , see Polyanskiy & Wu ( 2017 ) . Let f∗ be the convex conjugate of f . That is , f∗ ( t ) = supu∈domf ( ut− f ( u ) ) . A variational lower bound of ( 2 ) is Df ( P‖Q ) ≥ sup T∈T [ EPT ( X ) − EQf∗ ( T ( X ) ) ] . ( 3 ) Note that the inequality ( 3 ) holds for any class T , and it becomes an equality whenever the class T contains the function f ′ ( p/q ) ( Nguyen et al. , 2010 ) . For notational simplicity , we also use f ′ for an arbitrary element of the subdifferential when the derivative does not exist . With i.i.d . observations X1 , ... , Xn ∼ P , the variational lower bound ( 3 ) naturally leads to the following learning method P̂ = argmin Q∈Q sup T∈T [ 1 n n∑ i=1 T ( Xi ) − EQf∗ ( T ( X ) ) ] . ( 4 ) The formula ( 4 ) is a powerful and general way to learn the distribution P from its i.i.d . observations . It is known as f -GAN ( Nowozin et al. , 2016 ) , an extension of GAN ( Goodfellow et al. , 2014 ) , which stands for generative adversarial networks . The idea is to find a P̂ so that the best discriminator T in the class T can not tell the difference between P̂ and the empirical distribution 1n ∑n i=1 δXi . 2.1 f -LEARNING : A UNIFIED FRAMEWORK Our f -Learning framework is based on a special case of the variational lower bound ( 3 ) . That is , Df ( P‖Q ) ≥ sup Q̃∈Q̃Q [ EP f ′ ( q̃ ( X ) q ( X ) ) − EQf∗ ( f ′ ( q̃ ( X ) q ( X ) ) ) ] , ( 5 ) where q̃ ( · ) stands for the density function of Q̃ . Note that here we allow the class Q̃Q to depend on the distribution Q in the second argument of Df ( P‖Q ) . Compare ( 5 ) with ( 3 ) , and it is easy to realize that ( 5 ) is a special case of ( 3 ) with T = TQ = { f ′ ( q̃ q ) : q̃ ∈ Q̃Q } . ( 6 ) Moreover , the inequality ( 5 ) becomes an equality as long as P ∈ Q̃Q . The sample version of ( 5 ) leads to the following learning method P̂ = argmin Q∈Q sup Q̃∈Q̃Q [ 1 n n∑ i=1 f ′ ( q̃ ( Xi ) q ( Xi ) ) − EQf∗ ( f ′ ( q̃ ( X ) q ( X ) ) ) ] . ( 7 ) The learning method ( 7 ) will be referred to as f -Learning in the sequel . It is a very general framework that covers many important learning procedures as special cases . For example , consider the special case where Q̃Q = Q̃ independent of Q , Q = Q̃ , and f ( x ) = x log x . Direct calculations give f ′ ( x ) = log x + 1 and f∗ ( t ) = et−1 . Therefore , ( 7 ) becomes P̂ = argmin Q∈Q sup Q̃∈Q 1 n n∑ i=1 log q̃ ( Xi ) q ( Xi ) = argmax q∈Q 1 n n∑ i=1 log q ( Xi ) , which is the maximum likelihood estimator ( MLE ) . 2.2 TV-LEARNING AND DEPTH-BASED ESTIMATORS . An important generator f that we will discuss here is f ( x ) = ( x−1 ) + . This leads to the total variation distance Df ( P‖Q ) = 12 ∫ |p− q| . With f ′ ( x ) = I { x ≥ 1 } and f∗ ( t ) = tI { 0 ≤ t ≤ 1 } , the TV-Learning is given by P̂ = argmin Q∈Q sup Q̃∈QQ [ 1 n n∑ i=1 I { q̃ ( Xi ) q ( Xi ) ≥ 1 } −Q ( q̃ q ≥ 1 ) ] . ( 8 ) A closely related idea was previously explored by Yatracos ( 1985 ) ; Devroye & Lugosi ( 2012 ) . The following proposition shows that when Q̃Q approaches toQ in some neighborhood , TV-Learning leads to robust estimators that are defined as the maximizers of various depth functions including Tukey ’ s depth , regression depth , and covariance depth . Proposition 2.1 . The TV-Learning ( 8 ) includes the following special cases : 1 . Tukey ’ s halfspace depth : Take Q = { N ( η , Ip ) : η ∈ Rp } and Q̃η = { N ( η̃ , Ip ) : ‖η̃ − η‖ ≤ r } . As r → 0 , ( 8 ) becomes θ̂ = argmax η∈Rp inf ‖u‖=1 1 n n∑ i=1 I { uT ( Xi − η ) ≥ 0 } . ( 9 ) 2 . Regression depth : Take Q = { Py , X = Py|XPX : Py|X = N ( X T η , 1 ) , η ∈ Rp } , and Q̃η = { Py , X = Py|XPX : Py|X = N ( X T η̃ , 1 ) , ‖η̃ − η‖ ≤ r } . As r → 0 , ( 8 ) becomes θ̂ = argmax η∈Rp inf ‖u‖=1 1 n n∑ i=1 I { uTXi ( yi −XTi η ) ≥ 0 } . ( 10 ) 3 . Covariance matrix depth : Take Q = { N ( 0 , Γ ) : Γ ∈ Ep } , where Ep stands for the class of p × p covariance matrices , and Q̃Γ = { N ( 0 , Γ̃ ) : Γ̃−1 = Γ−1 + r̃uuT ∈ Ep , |r̃| ≤ r , ‖u‖ = 1 } . As r → 0 , ( 8 ) becomes Σ̂ = argmin Γ∈Ep sup ‖u‖=1 [ ( 1 n n∑ i=1 I { |uTXi|2 ≤ uTΓu } − P ( χ21 ≤ 1 ) ) ( 11 ) ∨ ( 1 n n∑ i=1 I { |uTXi|2 > uTΓu } − P ( χ21 > 1 ) ) ] . The formula ( 9 ) is recognized as Tukey ’ s median , the maximizer of Tukey ’ s halfspace depth . A traditional understanding of Tukey ’ s median is that ( 9 ) maximizes the halfspace depth ( Donoho & Gasko , 1992 ) so that θ̂ is close to the centers of all one-dimensional projections of the data . In the f -Learning framework , N ( θ̂ , Ip ) is understood to be the minimizer of a variational lower bound of the total variation distance . The formula ( 10 ) gives the estimator that maximizes the regression depth proposed by Rousseeuw & Hubert ( 1999 ) . It is worth noting that the derivation of ( 10 ) does not depend on the marginal distribution PX in the linear regression model . Finally , ( 11 ) is related to the covariance matrix depth ( Zhang , 2002 ; Chen et al. , 2018 ; Paindaveine & Van Bever , 2017 ) . All of the estimators ( 9 ) , ( 10 ) and ( 11 ) are proved to achieve the minimax rate for the corresponding problems under Huber ’ s -contamination model ( Chen et al. , 2018 ; Gao , 2017 ) . 2.3 FROM f -LEARNING TO f -GAN The connection to various depth functions shows the importance of TV-Learning in robust estimation . However , it is well-known that depth-based estimators are very hard to compute ( Amenta et al. , 2000 ; van Kreveld et al. , 1999 ; Rousseeuw & Struyf , 1998 ) , which limits their applications only for very low-dimensional problems . On the other hand , the general f -GAN framework ( 4 ) has been successfully applied to learn complex distributions and images in practice ( Goodfellow et al. , 2014 ; Radford et al. , 2015 ; Salimans et al. , 2016 ) . The major difference that gives the computational advantage to f -GAN is its flexibility in terms of designing the discriminator class T using neural networks compared with the pre-specified choice ( 6 ) in f -Learning . While f -Learning provides a unified perspective in understanding various depth-based procedures in robust estimation , we can step back into the more general f -GAN for its computational advantages , and to design efficient computational strategies .

Summary: This paper considers the robust estimation problem under Huber’s \epsilon-contamination model. This problem is a hot topic in theoretical statistics and theoretical computer science community in recent 3 years.  From theoretical statistics community, the main approach is through depth functions. Solving the robust estimation problem can be reduced to solving a min-max problem. While the formulation is clean and can achieve the optimal statistical rate, solving the min-max problem is computationally intractable in general. On the other hand, approaches from TCS community are more involved and sometimes cannot achieve the optimal statistical rate (especially for the general distribution).