Efficient methods for Gaussian Markov random fields under sparse linear constraints

1 Introduction . Linearly constrained Gaussian processes ( GPs ) have recently gained attention , especially for Gaussian process regression where the model should obey underlying physical principle such as conservation laws or equilibrium conditions [ 13 , 16 , 17 , 30 , 31 , 36 , 37 ] . A well-known challenge with these models , and GPs in general , is their high computational cost for inference and prediction in the case of big data sets [ 2 , 9 , 14 ] . One way to reduce computational burden is to impose conditional independence assumptions . In fact , conditional independence between random variables is often explicitly or implicitly assumed in large classes of statistical models including Markov processes , hierarchical models , and graphical models . The assumption typically increases the model ’ s interpretability and facilitates computationally efficient methods for inference [ 29 ] . Gaussian variables with conditional independence properties are known as Gaussian Markov random fields ( GMRFs ) , and these are widely used in areas ranging from brain imaging [ 24 , 34 , 20 ] to spatial statistics [ 7 ] and time series analysis [ 28 ] . GMRFs further arise as computationally efficient approximations of certain GPs [ 18 ] , which is a fundamental modeling tool in both machine learning and statistics . In particular , such approximations in combination with the integrated nested Laplace approximation ( INLA ) methodology [ 29 ] made latent GMRFs widely used in the applied sciences [ 3 ] . GMRFs also have connections with convolutional neural networks leading to recent considerations of deep GMRFs [ 33 ] . The key feature of GMRFs that reduces computational cost is sparsity . Specifically , a GMRF X = [ X1 , . . . , Xn ] > ∼ N ( µ , Q−1 ) , ( 1 ) has a sparse precision ( inverse covariance ) matrix Q which enables the use of sparse matrix techniques such as sparse Cholesky factorization [ 8 ] and iterative methods [ 35 , 23 , 34 ] for computationally efficient sampling and statistical inference [ 27 , 28 ] . The sparsity is caused by conditional independence assumptions since Qij = 0 if and only if Xi and Xj are independent conditionally on all other variables in X [ see 28 , Chapter 2 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . For a GMRF X , a set of k linear constraints can be formulated as AX = b , ( 2 ) where each row in the k × n matrix A and the vector b encodes a constraint on X . Note that observations of GMRFs can be formulated as linear constraints , where , e.g. , a point observation of Xi is the simple constraint Xi = xi . These deterministic restrictions are often referred to as hard constraints [ 28 ] . If ( 2 ) is assumed to hold up to Gaussian noise , i.e. , AX ∼ N ( b , σ2I ) , then the constraints are instead referred to as soft constraints . This scenario is common when GMRFs are incorporated in hierarchical models , where the soft constraints represent noisy observations [ 28 ] . Because of the computational advantages of GMRFs , it is of interest to use GMRFs as alternatives to GPs also in the constrained case . Unfortunately , hard constraints can drastically reduce the computational benefits of GMRFs . Specifically , as we will explain in the next section , the current methods for GMRFs with hard constraints in general have a computational cost that scales cubically in the number of constraints k. This is prohibitive when there are many constraints , which is common for applications of constrained GPs . For example , physically constrained GPs such as those in [ 13 ] have k = n and any GMRF model with N observations without measurement noise will have k = N . In the first example the computational cost is cubic in n and in the second it can be cubic in N if the observations are such that A in ( 2 ) is not diagonal , which , e.g. , is the case for area observations [ 21 ] . In both cases , this renders the computational benefits of the unconstrained GMRF irrelevant . Until now , there has been no methods to circumvent this problem . Summary of contributions . The main contribution of this work is the formulation of a novel class of computationally efficient methods for linearly constrained GMRFs in tasks such as parameter estimation and simulation . These methods explore the conditional independence structure to reduce computational costs compared to traditional methods for situations with large numbers of constraints . The focus is in particular on the case , referred to as sparse hard constraints , when each constraint only involves a few number of variables so that A is sparse . This is , e.g. , common for GPs under physical constraints [ 13 ] and for GMRF models with point and area observations [ 21 ] . For this case , the main idea is to perform a change of basis so that the constraints are simpler to enforce in the transformed basis . In particular , we show how the change of basis facilitates using the same efficient methods for sparse matrices that are used in the non-constrained case also for the constrained models . In order to use these models for GP regression , the methods are also generalized to GMRF models with both hard and soft constraints . An important feature of the new class of methods , that previous approaches lack , is its applicability to intrinsic GMRFs , which are improper in the sense that a set of eigenvalues of Q is zero . This makes their distributions invariant to shifts in certain directions , which is a useful property for prior distributions of Bayesian models [ 28 , Chapter 3 ] . Because of this they are often used in areas such as semiparametric regression [ 10 ] , medical image analysis [ 24 , 34 ] , and geostatistics [ 15 , 7 ] , and it is thus of interest to be able to enforce linear constraints also for such models . The final contribution is the derivation of GMRFs for constrained GPs , by combining the proposed methods with the stochastic partial differential equation ( SPDE ) approach by [ 18 ] and the nested SPDE methods by [ 5 ] . The combined approach is highly computationally efficient compared to standard covariance-based methods , as illustrated in two simulation studies . Outline . In Section 2 , the problem is introduced in more detail and the most commonly used methods for sampling and likelihood computations for GMRFs are reviewed . Section 3 introduces the new methods for GMRFs with sparse hard constraints . These methods are extended to the case with both hard and soft constraints in Section 4 , followed by the GMRF methods for constrained GPs in Section 5 . The methods are illustrated numerically in Section 6 . A discussion closes the article , which is supported by a supplementary materials containing proofs and technical details . 2 Standard methods for GMRFs under hard constraints . Hard constraints can be divided into interacting and non-interacting constraints . In the latter , ( 2 ) specifies a constraint on a subset of the variables in X so that AX = b can be written as Xc = bc , where c denotes a subset of the indices . Specifically , let Xu denote the remaining variables , then X = [ Xc Xu ] ∼ N ( [ µc µu ] , [ Qcc Qcu Quc Quu ] −1 ) and X|AX = b ∼ N ( [ bc µu|c ] , [ 0 0 0 Q−1uu ] ) , where µu|c = µu −Q −1 uuQuc ( bc − µc ) . Thus , we can split the variables into two subsets and treat the unconditioned variables separately . The more difficult and interesting situation is the case of interacting hard constraints where a simple split of the variables is not possible . From now on , we will assume that we are in this scenario . Our aim is to construct methods for sampling from the distribution of X|AX = b and for evaluating its log-likelihood function . It is straightforward to show that X|AX = b ∼ N ( µ̂ , Σ̂ ) , where µ̂ = µ−Q−1A > ( AQ−1A > ) −1 ( Aµ−b ) and Σ̂ = Q−1−Q−1A > ( AQ−1A > ) −1AQ−1 . Since Σ̂ has rank n− k , likelihood evaluation and sampling is in general expensive . For example , one way is to use an eigenvalue decomposition of Σ̂ [ see 28 , Chapter 2.3.3 ] . However , this procedure is not practical since it can not take advantage of the sparsity of Q . Also , for intrinsic GMRFs , Σ̂ and µ̂ can not be constructed through the expressions above since Q−1 has unbounded eigenvalues . A commonly used method for sampling under hard linear constraints , sometimes referred to as conditioning by kriging [ 27 ] , is to first sample X ∼ N ( µ , Q−1 ) and then correct for the constraints by using X∗ = X−Q−1A > ( AQ−1A > ) −1 ( AX−b ) as a sample from the conditional distribution . Here the cost of sampling X is CQ + SQ , where CQ denotes the computational cost of a sparse Cholesky factorization Q = R > R and SQ the cost of solving Rx = u for x given u . Adding the cost for the correction step , the total cost of the method is O ( CQ + ( k + 2 ) SQ + k3 ) . Let πAx ( · ) denote the density of Ax ∼ N ( Aµ , AQ−1A > ) , then the likelihood of X|AX = b can be computed through the expression π ( x|Ax = b ) = π∗Ax|x ( b|x ) π ( x ) πAx ( b ) , ( 3 ) where π∗Ax|x ( b|x ) = I ( Ax = b ) |AA > |−1/2 and I ( Ax = b ) denotes the indicator function with I ( Ax = b ) = 1 if Ax = b and I ( Ax = b ) = 0 otherwise . This result is formulated in [ 27 ] and we provide further details in the supplementary materials by showing that ( 3 ) is a density with respect to the Lebesgue measure on the level set { x : Ax = b } . The computational cost of evaluating the likelihood using this formulation is O ( CQ + ( k + 1 ) SQ + k3 ) . Note that these methods only work efficiently for a small number of constraints , because of the term k3 in the computational costs , and for proper GMRFs . In the case of intrinsic GMRFs we can not work with AQ−1A due to the rank deficiency of Q . 3 The basis transformation method . In this section , we propose the new class of methods in two steps . We first derive a change of basis in Section 3.1 , and then use this to formulate the desired conditional distributions in Section 3.2 . The resulting computational costs of likelihood evaluations and simulation are discussed in Section 3.3 . Before stating the results we introduce some basic notation . When working with intrinsic GMRFs the definition in ( 1 ) is inconvenient since the covariance matrix has infinite eigenvalues . Instead one can use the canonical parametrization X ∼ NC ( µC , Q ) , which implies that the density of X is given by π ( x ) ∝ exp ( − 12x > Qx + µ > Cx ) and thus that µC = Qµ . Also , since we will be working with non-invertible matrices we will need the Moore–Penrose inverse and the pseudo determinant . We denote the Moore–Penrose inverse of a matrix B by B† and for a symmetric positive semi definite matrix M we define the pseudo determinant as |M|† = ∏ i : λi > 0 λi where { λi } are the eigenvalues of M. For the remainder of this article , we will assume the following . Assumption 1 . X ∼ NC ( Qµ , Q ) where Q is a positive semi-definite n × n matrix with rank n− s > 0 and null-space EQ . A is a k × n matrix with rank k and we let k0 denote rank ( AEQ ) . Finally , we use the index notation C = ( 1 , . . . , k ) and U = ( k + 1 , . . . , n ) so that , e.g. , vC denotes the sub-vector ( v1 , . . . , vk ) > of the first k elements in the vector v , M· , U ( MU , · ) denotes the submatrix obtained by extracting the columns ( rows ) in M with indices in U , and MU U the submatrix obtained by extracting the columns and rows in M with indices in U .

Summary: This paper formulates a class of computationally efficient methods for Gaussian Markov Random field (GMRF) models with linear constraints. The main idea behind the methods relies on a transformation of basis such that the constraints are easier to enforce upon transformation. The methods outperform existing alternative formulations of GMRFs under scenarios involving large numbers of sparse constraints. Finally, a GMRF for linearly constrained Gaussian process regression is derived by utilizing the proposed methods in conjunction with the stochastic partial differential equation (SPDE) approach.