Robustifying Algorithms of Learning Latent Trees with Vector Variables

1 Introduction . Latent graphical models provide a succinct representation of the dependencies among observed and latent variables . Each node in the graphical model represents a random variable or a random vector , and the dependencies among these variables are captured by the edges among nodes . Graphical models are widely used in domains from biology [ 1 ] , computer vision [ 2 ] and social networks [ 3 ] . This paper focuses on the structure learning problem of latent tree-structured Gaussian graphical models ( GGM ) in which the node observations are random vectors and a subset of the observations can be arbitrarily corrupted . This classical problem , in which the variables are clean scalar random variables , has been studied extensively in the past decades . The first information distance-based method , NJ , was proposed in [ 1 ] to learn the structure of phylogenetic trees . This method makes use of additive information distances to deduce the existence of hidden nodes and introduce edges between hidden and observed nodes . RG , proposed in [ 4 ] , generalizes the information distance-based methods to make it applicable for the latent graphical models with general structures . Different from these information distance-based methods , quartet-based methods [ 5 ] utilize the relative geometry of every four nodes to estimate the structure of the whole graph . Although experimental comparisons of these algorithms were conducted in some works [ 4 , 6 , 7 ] , since there is no instance-dependent 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . impossibility result of the sample complexity of structure learning problem of latent tree graphical models , no thorough theoretical comparisons have been made , and the optimal dependencies on the diameter of graphs and the maximal distance between nodes ρmax have not been found . The success of the previously-mentioned algorithms relies on the assumption that the observations are i.i.d . samples from the generating distribution . The structure learning of latent graphical models in presence of ( random or adversarial ) noise remains a relatively unexplored problem . The presence of the noise in the samples violates the i.i.d . assumption . Consequently , classical algorithms may suffer from severe performance degradation in the noisy setting . There are some works studying the problem of structure learning of graphical models with noisy samples , where all the nodes in the graphical models are observed and not hidden . Several assumptions on the additive noise are made in these works , which limit the use of these proposed algorithms . For example , the covariance matrix of the noise is specified in [ 8 ] , and the independence and/or distribution of the noise is assumed in [ 9–11 , 7 ] . In contrast , we consider the structure learning of latent tree graphical models with arbitrary corruptions , where assumptions on the distribution and independence of the noise across nodes are not required [ 12 ] . Furthermore , the corruptions are allowed to be presented at any position in the data matrix ; they do not appear solely as outliers . In this work , we derive bounds on the maximum number of corruptions that can be tolerated for a variety of algorithms , and yet structure learning can succeed with high probability . Firstly , we derive the sample complexities of RG and CLRG where each node represents a random vector ; this differs from previous works where each node is scalar random variable ( e.g. , [ 4 , 13 ] ) . We explore the dependence of the sample complexities on the parameters of the graph . Compared with [ 4 , Theorem 12 ] , the derived sample complexities are applicable to a wider class of latent trees and capture the dependencies on more parameters of the underlying graphical models , such as ρmax , the maximum distance between any two nodes , and δmin , the minimum over all determinants of the covariance matrices of the vector variables . Our sample complexity analysis clearly demonstrates and precisely quantifies the effectiveness of the Chow-Liu [ 14 ] initialization step in CLRG ; this has been only verified experimentally [ 4 ] . For the particular case of the HMM , we show that the Chow-Liu initialization step reduces the sample complexity of RG which isO ( ( 92 ) Diam ( T ) ) toO ( log Diam ( T ) ) , where Diam ( T ) is the tree diameter . Secondly , we robustify RG , CLRG , NJ and SNJ by using the truncated inner product [ 15 ] to estimate the information distances in the presence of arbitrary corruptions . We derive their sample complexities and show that they can tolerate n1 = O ( √n2 logn2 ) corruptions , where n2 is the number of clean samples . Finally , we derive the first known instance-dependent impossibility result for learning latent trees . The dependencies on the number of observed nodes |Vobs| and the maximum distance ρmax are delineated . The comparison of the sample complexities of the structure learning algorithms and the impossibility result demonstrates the optimality of Robust Chow-Liu Recursive Grouping ( RCLRG ) and Robust Neighbor Joining ( RNJ ) in Diam ( T ) for some archetypal latent tree structures . Notation We use san-serif letters x , boldface letters x , and bold uppercase letters X to denote variables , vectors and matrices , respectively . The notations [ x ] i , [ X ] ij , [ X ] : ,j and diag ( X ) are respectively the ith entry of vector x , the ( i , j ) th entry of X , the jth column of X , and the diagonal entries of matrix X . The notation x ( k ) represents the kth sample of x . ‖x‖0 is the l0 norm of the vector x , i.e. , the number of non-zero terms in x . The set { 1 , . . . , n } is denoted as [ n ] . For a tree T = ( V , E ) , the internal ( non-leaf ) nodes , the maximal degree and the diameter of T are denoted as Int ( T ) , Deg ( T ) , and Diam ( T ) , respectively . We denote the closed neighborhood and the degree of xi as nbd [ xi ; T ] and deg ( i ) , respectively . The length of the ( unique ) path connecting xi and xj is denoted as dT ( xi , xj ) . 2 Preliminaries and problem statement . A GGM [ 16 , 17 ] is a multivariate Gaussian distribution that factorizes according to an undirected graph G = ( V , E ) . More precisely , a lsum-dimensional random vector x = [ x > 1 , . . . , x > p ] > , where xi ∈ Rli and lsum = ∑p i=1 li , follows a Gaussian distribution N ( 0 , Σ ) , and it is said to be Markov on a graph G = ( V , E ) with vertex set V = { x1 , . . . , xp } and edge set E ⊆ ( V 2 ) and ( xi , xj ) ∈ E if and only if the ( i , j ) th block Θij of the precision Θ = Σ−1 is not the zero matrix 0 . We focus on tree-structured graphical models , which factorize according to acyclic and connected ( tree ) graphs . A special class of graphical models is the set of latent graphical models G = ( V , E ) . The vertex set V is decomposed as V = Vhid ∪ Vobs . We only have access to n i.i.d . samples drawn from the observed set of nodes Vobs . The two goals of any structure learning algorithm are to learn the identities of the hidden nodes Vhid and how they are connected to the observed nodes . 2.1 System model for arbitrary corruptions . We consider tree-structured GGMs T = ( V , E ) with observed nodes Vobs = { x1 , · · · , xo } and hidden nodes Vhid = { xo+1 , · · · , xo+h } , where V = Vhid ∪ Vobs and E ⊆ ( V 2 ) . Each node xi represents a random vector xi ∈ Rli . The concatenation of these random vectors is a multivariate Gaussian random vector with zero mean and covariance matrix Σ with size lsum × lsum . We have n i.i.d . samples X̃j = [ x̃ ( j ) > 1 , · · · , x̃ ( j ) > o ] > ∈ Rlsum , j = 1 , . . . , n drawn from the observed nodes Vobs = { x1 , · · · , xo } . However , the observed data matrix X̃n1 = [ X̃1 , · · · , X̃n ] > ∈ Rn×lsum may contain some corrupted elements . We allow an level- ( n1/2 ) arbitrary corruption in the data matrix . This is made precise in the following definition . Definition 1 ( Level-m arbitrary corruption ) . For the data matrix X̃n1 ∈ Rn×k formed by n clean samples of k random variables ( or a random vector of dimension k ) , an level-m arbitrary corruption transforms X̃n1 into X n 1 ∈ Rn×k such that ‖ [ X̃n1 ] ; , i − [ Xn1 ] ; , i‖0 ≤ m for all i = 1 , . . . , k. ( 1 ) Definition 1 implies that there are at most n1/2 corrupted terms in each column of Xn1 ; the remaining n− n1/2 samples in this column are clean . In particular , the corrupted samples in different columns need not to be in the same rows . If the corruptions in different columns lie in the same rows , as shown in ( the left of ) Fig . 3 , all the samples in the corresponding rows are corrupted ; these are called outliers . Obviously , outliers form a special case of our corruption model . Since each variable has at most n1/2 corrupted samples , the sample-wise inner product between two variables has at least n2 = n− n1 clean samples . There is no constraint on the statistical dependence or patterns of the corruptions . Unlike fixing the covariance matrix of the noise [ 8 ] or keeping the noise independent [ 9 ] , we allow arbitrary corruptions on the samples , which means that the noise can have unbounded amplitude , can be dependent , and even can be generated from another graphical model ( as we will see in the experimental results in Section 3.6 ) . 2.2 Structural and distributional assumptions . To construct the correct latent tree from samples of observed nodes , it is imperative to constrain the class of latent trees to guarantee that the information from the distribution of observed nodes p ( x1 , . . . , xo ) is sufficient to construct the tree . The distribution p ( x1 , . . . , xo+h ) of the observed and hidden nodes is said to have a redundant hidden node xj if the distribution of the observed nodes p ( x1 , . . . , xo ) remains the same after we marginalize over xj . To ensure that a latent tree can be constructed with no ambiguity , we need to guarantee that the true distribution does not have any redundant hidden node ( s ) , which is achieved by following two conditions [ 18 ] : ( C1 ) Each hidden node has at least three neighbors ; the set of such latent trees is denoted as T≥3 ; ( C2 ) Any two variables connected by an edge are neither perfectly dependent nor independent . Assumption 1 . The dimensions of all the random vectors are all equal to lmax . In fact , we only require the random vectors of the internal ( non-leaf ) nodes to have the same length . However , for ease of notation , we assume that the dimensions of all the random vectors are lmax . Assumption 2 . For every xi , xj ∈ V , the covariance matrix Σij = E [ xix > j ] has full rank , and the smallest singular value of Σij is lower bounded by γmin , i.e. , σlmax ( Σij ) ≥ γmin for all xi , xj ∈ V , ( 2 ) where σi ( Σ ) is the ith largest singular value of Σ . This assumption is a strengthening of Condition ( C2 ) when each node represents a random vector . Assumption 3 . The determinant of the covariance matrix of any node Σii = E [ xix > i ] is lower bounded by δmin , and the diagonal terms of the covariance matrix are upper bounded by σ2max , i.e. , min xi∈V det ( Σii ) ≥ δmin and max xi∈V diag ( Σii ) ≤ σ2max . ( 3 ) Assumption 3 is natural ; otherwise , Σii may be arbitrarily close to a singular matrix . Assumption 4 . The degree of each node is upper bounded by dmax , i.e. , Deg ( T ) ≤ dmax .

Summary: Summary:  The paper studies the structural learning problems on latent tree models, in which a subset of vector observations have been arbitrarily corrupted. The paper proposes a series of "robustified" algorithms, including the robustified recursive grouping (RRG), robustified Chow-Liu recursive grouping (CLRG), robustified neighbor joining (RNJ), and robustified spectral neighbor joining (RSNJ). The authors analyze the sample complexity of the algorithms and show that a number of corruptions up to the square root of the number of clean samples can be tolerated. The paper also provides an information theoretic lower bound for this class of graphical models.