Robust Learning of Fixed-Structure Bayesian Networks in Nearly-Linear Time

1 INTRODUCTION . Probabilistic graphical models ( Koller & Friedman , 2009 ) offer an elegant and succinct way to represent structured high-dimensional distributions . The problem of inference and learning in probabilistic graphical models is an important problem that arises in many disciplines ( see Wainwright & Jordan ( 2008 ) and the references therein ) , which has been studied extensively during the past decades ( see , e.g. , Chow & Liu ( 1968 ) ; Dasgupta ( 1997 ) ; Abbeel et al . ( 2006 ) ; Wainwright et al . ( 2006 ) ; Anandkumar et al . ( 2012 ) ; Santhanam & Wainwright ( 2012 ) ; Loh & Wainwright ( 2012 ) ; Bresler et al . ( 2013 ; 2014 ) ; Bresler ( 2015 ) ) . Bayesian networks ( Jensen & Nielsen , 2007 ) are an important family of probabilistic graphical models that represent conditional dependence by a directed graph ( see Section 2 for a formal definition ) . In this paper , we study the problem of learning Bayesian networks where an -fraction of the samples are adversarially corrupted . We focus on the simplest setting : all variables are binary and observable , and the structure of the Bayesian network is given to the algorithm . Formally , we work with the following corruption model : Definition 1.1 ( -Corrupted Set of Samples ) . Given 0 < < 1/2 and a distribution family P on Rd , the algorithm first specifies the number of samples N , and N samples X1 , X2 , . . . , XN are drawn from some unknown P ∈ P . The adversary inspects the samples , the ground-truth distribution P , and the algorithm , and then replaces N samples with arbitrary points . The set of N points is given to the algorithm as input . We say that a set of samples is -corrupted if it is generated by this process . This is a strong corruption model which generalizes many existing models . In particular , it is stronger than Huber ’ s contamination model ( Huber , 1964 ) , because we allow the adversary to add bad samples and remove good samples , and he can do so adaptively . Our goal is to design robust algorithms for learning Bayesian networks with dimension-independent error . More specifically , given as input an -corrupted set of samples drawn from some groundtruth Bayesian network P and the graph structure of P , we want the algorithm to output a Bayesian network Q , such that the total variation distance between P and Q is upper bounded by a function that depends only on ( the fraction of corruption ) but not d ( the number of variables in P ) . In the fully-observable fixed-structure setting , the problem is straightforward when there is no corruption . We know that the empirical estimator ( which computes the empirical conditional probabilities ) is sample efficient and runs in linear time ( Dasgupta , 1997 ) . It turns out that the problem becomes much more challenging when there is corruption . Even for robust learning of binary product distributions ( i.e. , a Bayesian network with an empty dependency graph ) , the first computational efficient algorithms with dimension-independent error was only discovered in ( Diakonikolas et al. , 2019a ) . Subsequently , ( Cheng et al. , 2018 ) gave the first polynomialtime algorithms for robust learning of fixed-structured Bayesian networks . The main drawback of the algorithm in ( Cheng et al. , 2018 ) is that it runs in time Ω ( Nd2/ ) , which is slower by at least a factor of ( d/ ) compared to the fastest non-robust estimator . Motivated by this gap in the running time , in this work we want to resolve the following question : Can we design a robust algorithm for learning Bayesian networks in the fixedstructure fully-observable setting that runs in nearly-linear time ? 1.1 OUR RESULTS AND CONTRIBUTIONS . We resolve this question affirmatively by proving Theorem 1.2 . We say a Bayesian network is cbalanced if all its conditional probabilities are between c and 1 − c. For the ground-truth Bayesian network P , let m be the size of its conditional probability table and α be its minimum parental configuration probability ( see Section 2 for formal definitions ) . Theorem 1.2 ( informal statement ) . Consider an -corrupted set of N = Ω̃ ( m/ 2 ) samples drawn from a d-dimensional Bayesian network P . Suppose P is c-balanced and has minimum parental configuration probability α , where both c and α are universal constants . We can compute a Bayesian network Q in time Õ ( Nd ) such that dTV ( P , Q ) ≤ √ ln ( 1/ ) . 1 For simplicity , we stated our result in the very special case where both c and α are Ω ( 1 ) . Our approach works for general values of α and c , where our error guarantee degrades gracefully as α and c gets smaller . A formal version of Theorem 1.2 is given as Theorem 4.1 in Section 4 . Our algorithm has optimal error guarantee , sample complexity , and running time ( up to logarithmic factors ) . There is an information-theoretic lower bound of Ω ( ) on the error guarantee , which holds even for Bayesian networks with only one variable . A sample complexity lower bound of Ω ( m/ 2 ) holds even without corruption ( see , e.g. , ( Canonne et al. , 2017 ) ) . Our Contributions . We establish a novel connection between robust learning of Bayesian networks and robust mean estimation . At a high level , we show that one can essentially reduce the former to the latter . This allows us to take advantage of the recent ( and future ) advances in robust mean estimation and apply the algorithms almost directly to obtain new algorithms for learning Bayesian networks . Our algorithm and analysis are considerably simpler than those in previous work . For simplicity , consider learning binary product distributions as an example . Cheng et al . ( 2018 ) tried to remove samples to make the empirical covariance matrix closer to a diagonal matrix ( since the true covariance matrix is diagonal because each coordinate is independent ) . They used a “ filtering ” approach which requires proving specific tail bounds on the samples . In contrast , we show that it suffices to use any robust mean estimation algorithms which minimize the spectral norm of the empirical covariance matrix ( regardless of whether it is close to being diagonal or not ) . As a subroutine in our approach , we develop the first robust mean estimation algorithm that runs in nearly input-sparsity time ( i.e. , in time nearly linear in the total number of nonzero entries in the input ) , which may be of independent interest . The main computation bottleneck of current nearlylinear time robust mean estimation algorithms ( Cheng et al. , 2019a ; Depersin & Lecué , 2019 ; Dong et al. , 2019 ) is running matrix multiplication weight update with the Johnson-Lindenstrauss lemma , which we show can be done in nearly input-sparsity time . 1Throughout the paper , we use Õ ( f ) to denote O ( f polylog ( f ) ) . 1.2 RELATED WORK . Bayesian Networks . Probabilistic graphical models ( Koller & Friedman , 2009 ) provide an appealing and unifying formalism to succinctly represent structured high-dimensional distributions . The general problem of inference in graphical models is of fundamental importance and arises in many applications across several scientific disciplines ( see Wainwright & Jordan ( 2008 ) and references therein ) . The problem of learning graphical models from data ( Neapolitan , 2003 ; Daly et al. , 2011 ) has many variants : ( i ) the family of graphical models ( e.g. , directed , undirected ) , ( ii ) whether the data is fully or partially observable , and ( iii ) whether the graph structure is known or not . This learning problem has been studied extensively ( see , e.g. , Chow & Liu ( 1968 ) ; Dasgupta ( 1997 ) ; Abbeel et al . ( 2006 ) ; Wainwright et al . ( 2006 ) ; Anandkumar et al . ( 2012 ) ; Santhanam & Wainwright ( 2012 ) ; Loh & Wainwright ( 2012 ) ; Bresler et al . ( 2013 ; 2014 ) ; Bresler ( 2015 ) ) , resulting in a beautiful theory and a collection of algorithms in various settings . Robust Statistics . Learning in the presence of outliers has been studied since the 1960s ( Huber , 1964 ) . For the most basic problem of robust mean estimation , it is well-known that the empirical median works in one dimension . However , most natural generalizations of the median to high dimensions ( e.g. , coordinate-wise median , geometric median ) would incur an error of Ω ( √ d ) , even in the infinite sample regime ( see , e.g. , Diakonikolas et al . ( 2019a ) ; Lai et al . ( 2016 ) ) . After decades of work , sample-efficient robust estimators have been discovered ( e.g. , the Tukey median ( Tukey , 1975 ; Devroye & Györfi , 1985 ; Chen et al. , 2018 ) ) . However , the Tukey median is NP-Hard to compute in the worse case ( Johnson & Preparata , 1978 ; Amaldi & Kann , 1995 ) and many heuristics for approximating it perform poorly as the dimension scales ( Clarkson et al. , 1993 ; Chan , 2004 ; Miller & Sheehy , 2010 ) . Computational Efficient Robust Estimators . Recent work ( Diakonikolas et al. , 2019a ; Lai et al. , 2016 ) gave the first polynomial-time algorithms several high-dimensional unsupervised learning tasks ( e.g. , mean and covariance estimation ) with dimension-independent error guarantees . After the dissemination of ( Diakonikolas et al. , 2019a ; Lai et al. , 2016 ) , algorithmic high-dimensional robust statistics has attracted a lot of recent attention and there has been a flurry of research that obtained polynomial-time robust algorithms for a wide range of machine learning and statistical tasks ( see , e.g. , Balakrishnan et al . ( 2017 ) ; Charikar et al . ( 2017 ) ; Diakonikolas et al . ( 2017a ; b ) ; Steinhardt et al . ( 2018 ) ; Diakonikolas et al . ( 2018 ) ; Hopkins & Li ( 2018 ) ; Kothari et al . ( 2018 ) ; Prasad et al . ( 2020 ) ; Diakonikolas et al . ( 2019b ) ; Klivans et al . ( 2018 ) ; Diakonikolas et al . ( 2019c ) ; Liu et al . ( 2020 ) ; Cheng et al . ( 2020 ) ; Zhu et al . ( 2020 ) ) . In particular , the most relevant prior work is ( Cheng et al. , 2018 ) , which gave the first polynomial-time algorithms for robust learning of fixed-structure Bayesian networks . Faster Robust Estimators . While recent work gave polynomial-time robust algorithms for many tasks , these algorithms are often significantly slower than the fastest non-robust ones ( e.g. , sample average for mean estimation ) . Cheng et al . ( 2019a ) gave the first nearly-linear time algorithm for robust mean estimation and initiated the research direction of designing robust estimators that are as efficient as their non-robust counterparts . Since then , there have been several works that develop faster robust algorithms for various learning and statistical tasks , including robust mean estimation for heavy-tailed distributions Dong et al . ( 2019 ) ; Depersin & Lecué ( 2019 ) , robust covariance estimation Cheng et al . ( 2019b ) ; Li & Ye ( 2020 ) , robust linear regression Cherapanamjeri et al . ( 2020a ) , and list-decodable mean estimation Cherapanamjeri et al . ( 2020b ) ; Diakonikolas et al . ( 2020 ) . Organization . In Section 2 , we define our notations and provide some background on robust learning of Bayesian networks and robust mean estimation . In Section 3 , we give an overview of our approach and highlight some of our key technical results . In Section 4 , we present our algorithm for robust learning of Bayesian networks and prove our main result .

Summary: The paper considers the problem of robustly learning fixed structure Bayesian networks in nearly-linear time. Previous work by Cheng et al. gives a runtime of O(Nd^2/eps). The paper improves this to O(Nd). The algorithm works by directly relating the problem to robust mean estimation, and then leveraging the algorithm of Dong et al. for robust mean estimation which works in nearly-linear time. The authors have to modify the runtime analysis of the algorithm of Dong et al. to work in time linear in the sparsity, rather than dimension.