Scalable Inference of Sparsely-changing Gaussian Markov Random Fields 

1 Introduction . Contemporary systems are comprised of massive numbers of interconnected components that interact according to a hierarchy of complex , unknown , and time-varying topologies . For example , with billions of neurons and hundreds of thousands of voxels , the human brain is considered as one of the most complex physiological networks [ 18 , 22 , 28 , 30 , 37 ] . The temporal behavior of today ’ s interconnected systems can be captured via time-varying Markov random fields ( MRF ) . Time-varying MRFs are associated with a temporal sequence of undirected Markov graphs Gt ( V , Et ) , where V and Et are the set of nodes and edges in the graph at time t. The node set V represents the random variables in the model , while the edge set Et captures the conditional dependency between these variables at time t. A popular approach for the inference of MRFs is based on the maximum-likelihood estimation ( MLE ) : to obtain a model based on which the observed data is most probable to occur [ 42 ] . Despite being known as theoretically powerful tools [ 20 , 39 ] , MLE-based methods suffer from several fundamental drawbacks which render them impractical in realistic settings . First , they often suffer from notoriously high computational cost in massive problems , where the number of variables to be inferred is in the order of millions , or more . Second , they struggle to incorporate sparsity amongst 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . their components , which is pervasive in large-scale systems . In particular , while sparsely-changing MRFs can in theory be accurately estimated using sparsity-promoting regularizers ( such as ` 0 penalty ) , most of the existing methods resort to relaxed or weaker variants of such regularization ( such as ` 1 penalty ) , thereby suffering from inferior statistical guarantees . To address the aforementioned challenges , we propose a class of constrained optimization problems that achieve superior statistical and computational guarantees , compared to the regularized MLE , for the inference of time-varying Gaussian MRFs ( GMRFs ) . Our approach departs from the usual wisdom in statistics and machine learning that inference problems with nonconvex ` 0 terms are intractable , and convex proxies should be used instead . In particular , we show that the inference of sparsely-changing Gaussian MRFs can be solved efficiently via nonconvex ` 0 penalties . Notations . The ith element of a time-series vector vt is denoted as vt ; i ; the ( i , j ) th element of a time-series matrix Vt is denoted as Vt ; ij . For a vector v , the notation vi : j is used to denote the subvector of v from index i to j . For a vector v , the notations kvk1 , kvk2 , kvk0 denote the ` 1 norm , ` 2 norm , and the number of nonzero elements , respectively . Moreover , for a matrix M , the notations kMk2 , kMk1 , kMk1/1 , kMk1/1 refer to the induced 2-norm , induced1-norm , ` 1/ ` 1 norm , and ` 1/ ` 1 norm , respectively . Moreover , we define kMko↵ = kMk1/1 Pd i=1 |Mii| . For a vector v and matrix M , the notations supp ( v ) and supp ( M ) are defined as the sets of their nonzero elements . Given two sequences f ( n ) and g ( n ) , the notation f ( n ) . g ( n ) implies that there exists a constant C < 1 that satisfies f ( n ) Cg ( n ) , and f ( n ) ⇣ g ( n ) implies that f ( n ) . g ( n ) and g ( n ) . f ( n ) . All proofs are deferred to the supplementary file . 1.1 Warm-up : Regularized MLE for Sparsely-changing GMRFs . Consider a multivariate zero-mean Gaussian process { Xt } Tt=0 with distribution P ( Xt ) =exp ⇢ 1 2 h⇥t , XtX > t i+h⌘t , Xti A ( µt , ⇥t ) ( 1 ) for t = 0 , . . . , T where A : Rd⇥d ! R is the log-partition function used to normalize the distribution . Without loss of generality , we assume that the mean is zero . At any given time t , a sequence of data samples { X ( i ) t } Nt i=1 is collected from ( 1 ) . The inference of time-varying GMRFs reduces to estimating the time-varying precision matrix ⇥t from the data samples { X ( i ) t } Nt i=1 . Moreover , the edge set of the Markov graph Gt coincides with the off-diagonal nonzero elements of ⇥t [ 45 ] . We first illustrate the fundamental drawbacks of the ` 1-regularized MLE for time-varying GMRFs with sparsely-changing structures . The sparse precision matrices can be estimated via the following regularized MLE , also known as time-varying Graphical Lasso ( GL ) [ 11 , 16 ] : { b⇥t } Tt=0 = argmin ⇥t TX t=0 ⇣ h⇥t , b⌃ti log det ( ⇥t ) ⌘ + 1 TX t=0 k⇥tko↵ + 2 TX t=1 k⇥t ⇥t 1k1/1 ( 2a ) s.t . ⇥t 0 t = 0 , 1 , . . . , T ( 2b ) where b⌃t 2 Rd⇥d is the sample covariance matrix at time t. Without loss of generality , we assume that the samples have zero mean . Example 1 below shows that ( 2 ) may lead to poor estimates . Example 1 . Consider a scenario where { ⇥t } 4t=0 2 R25⇥25 are randomly generated symmetric and sparse matrices . At each time t = 0 , . . . , 4 , the precision matrix ⇥t has exactly 30 off-diagonal elements with value one in its upper-triangular part , and the remaining off-diagonal entries are set to zero . Moreover , the diagonal entries ⇥t ; ii are chosen as 1+ P j 6=i ⇥t ; ij . At every time , 5 nonzero offdiagonal elements are changed to zero , and 5 zero elements are set to one . The sample covariance b⌃t is obtained by collecting 500 samples from the Gaussian distribution with the constructed precision matrices . Figure 1a illustrates a heatmap of the mismatch error , i.e. , the total number of mismatches in the sparsity patterns of the true and estimated precision matrices and their differences , for different values of the regularization coefficients . It can be seen that after an exhaustive search over the regularization coefficient space , the best achievable mismatch error is in the order of 50 . Thus , the estimated parameters reveal little information about the true structure of the time-varying GMRF . Moreover , Figure 1b depicts the concatena- tion of the nonzero elements in the true pre- cision matrices ( dashed red line ) , and their corresponding values in the estimated matri- ces ( blue curve ) . It can be seen that , even when the sparsity pattern of the elements is correctly recovered , the estimated nonzero entries are “ shrunk ” toward zero , incurring a substantial bias . The above example shows the inferior statistical performance of the time-varying GL as an instance of a regularized MLE method for sparsely-changing GMRFs . In addition to its subpar statistical performance , time-varying GL suffers from expensive computational complexity : a general-purpose interior-point algorithm for solving time-varying GL has a prohibitive per-iteration complexity of O ( Td6 ) [ 10 ] . More recent algorithms for solving time-varying GL have lower per-iteration complexity of O ( Td3 ) [ 16 , 26 , 32 ] . However , these methods suffer from a slow ( sublinear ) convergence rate of O ( 1/✏ ) , increasing the overall complexity to O ( Td3/✏ ) in order to obtain an ✏-accurate solution . Thus , there is an inherent tradeoff between the quality of the solution found and the computational time required , with the performance deteriorating sharply as the number of precision digits increase . Solvers with such computational complexity may fall short of practical use in the large-scale settings . We now discuss the proposed method , which finds optimal solutions to the relevant optimization problems in strongly polynomial time . 2 Proposed Approach . The proposed framework is based on exact solutions to a class of tractable discrete ` 0-problems , thus circumventing bias and other drawbacks of the standard ` 1-approximations , while guaranteeing the scalability of the proposed method . As a general framework , we study the optimization problem : min ( 1 ) TX t=0 k⇥tk0 + TX t=1 k⇥t ⇥t 1k0 ( 3a ) s.t . k⇥t eF ⇤ ( b⌃t ) k1/1 t t = 0 , 1 , . . . , T ( 3b ) ⇥t 2 Rd⇥d t = 0 , 1 , . . . , T ( 3c ) where the optimal solutions { b⇥t } Tt=0 are the estimates of the precision matrices of the sparselychanging GMRF , b⌃t 2 Rd⇥d is the sample covariance matrix at time t , and eF ⇤ ( · ) is an approximate backward mapping of the model . In particular , we use the approximate backward mapping proposed in [ 46 ] , see §4.1 for a formal definition . First , we establish a deterministic guarantee on the estimation error of the optimal solution to ( 3 ) . Theorem 1 ( Estimation error and sparsistency ) . Let 0 < < 1 . For every t = 0 , . . . , T , define St as the set of indices corresponding to the nonzero elements of the true precision matrix ⇥⇤t , and define Dt as the set of indices corresponding to the nonzero elements of ⇥⇤t ⇥ ⇤ t 1 . Assume that • ⇥⇤t eF ⇤ ( b⌃t ) 1/1 < t , 80 t T , • 2 t min ( i , j ) 2St |⇥ ⇤ t ; ij | , 80 t T , • 2 t + 2 t 1 min ( i , j ) 2Dt |⇥ ⇤ t ; ij ⇥ ⇤ t 1 ; ij | , 80 t T . Then , the following statements hold for every 0 t T : Sparsistency supp ⇣ b⇥t ⌘ = supp ( ⇥⇤t ) and supp ⇣ b⇥t b⇥t 1 ⌘ = supp ⇥⇤t ⇥ ⇤ t 1 . Estimation error kb⇥t ⇥⇤t k1/1 2 t and kb⇥t ⇥⇤t k2 2 p |St| t. Theorem 1 presents a set of conditions under which the proposed estimation method achieves sparsistency and small estimation error . The first condition entails that the true precision matrix ⇥⇤t is a feasible solution to ( 3 ) . The second and third conditions imply that there is a non-negligible gap between the zero and nonzero elements of the true parameters and their temporal changes . In §4 we present additional bounds specific to our choice of backward mapping . Theorem 2 ( Computational complexity ) . Given eF ⇤ ( b⌃t ) , the optimization problem ( 3 ) can be solved to optimality in at most O ( ( dT ) 2 ) time and memory on a single thread . Theorem 2 shows that the optimization problem ( 3 ) can be solved efficiently and in strongly polynomial time despite its non-convex nature . As will be explained later , our choice of approximate backwards mapping [ 46 ] requires inverting T + 1 matrices , each with size d⇥ d , thereby increasing the overall complexity to O ( ( dT ) 2 + Td3 ) . If T d , the complexity of the our method is dominated by that of the matrix inversion , which is unavoidable , even if the sample covariance matrix coincides with its true analog ( since we still need to invert them to obtain the true precision matrices ) . Our solution method for ( 3 ) relies on the element-wise decomposability of ( 3 ) : we decompose ( 3 ) into smaller subproblems over different coordinates of { ⇥t } Tt=0 . Then , we show that the optimal solution to each subproblem can be obtained by solving a shortest path problem on an auxiliary weighted directed acyclic graph ( DAG ) . The details of the solution method are presented in §5 . Moreover , the proposed algorithm is easily parallelizable , leading to better runtimes in practice . Example 1 ( continued ) . Figure 2 depicts the performance of the proposed method , compared to the regularized MLE with 1 = 0.14 and 2 = 0.16 ( corresponding to the smallest mismatch error ) for the instances generated in Ex- ample 1 . The regularization parameter in the objective function of ( 3 ) is set to 0.2 . Moreover , for simplicity , we set 0 = · · · = 4 = . Figure 2a demonstrates that the proposed method enjoys a significantly smaller mismatch error , for a wide range of . On the other hand , Fig- ure 2b shows that the synthetic bias caused by the ` 1 penalty in the regularized MLE is allevi- ated via the proposed method .

Summary: In this paper, the authors consider the problem of learning Sparsely changing Gaussian Markov Random Fields. They pose the problem as a non-convex optimization problem that can be solved exactly. The authors provide statistical guarantees. Furthermore, they also provide a computationally efficient algorithm to recover the graphical model.