Efficient methods for Gaussian Markov random fields under sparse linear constraints

1 Introduction . Linearly constrained Gaussian processes ( GPs ) have recently gained attention , especially for Gaussian process regression where the model should obey underlying physical principle such as conservation laws or equilibrium conditions [ 13 , 16 , 17 , 30 , 31 , 36 , 37 ] . A well-known challenge with these models , and GPs in general , is their high computational cost for inference and prediction in the case of big data sets [ 2 , 9 , 14 ] . One way to reduce computational burden is to impose conditional independence assumptions . In fact , conditional independence between random variables is often explicitly or implicitly assumed in large classes of statistical models including Markov processes , hierarchical models , and graphical models . The assumption typically increases the model ’ s interpretability and facilitates computationally efficient methods for inference [ 29 ] . Gaussian variables with conditional independence properties are known as Gaussian Markov random fields ( GMRFs ) , and these are widely used in areas ranging from brain imaging [ 24 , 34 , 20 ] to spatial statistics [ 7 ] and time series analysis [ 28 ] . GMRFs further arise as computationally efficient approximations of certain GPs [ 18 ] , which is a fundamental modeling tool in both machine learning and statistics . In particular , such approximations in combination with the integrated nested Laplace approximation ( INLA ) methodology [ 29 ] made latent GMRFs widely used in the applied sciences [ 3 ] . GMRFs also have connections with convolutional neural networks leading to recent considerations of deep GMRFs [ 33 ] . The key feature of GMRFs that reduces computational cost is sparsity . Specifically , a GMRF X = [ X1 , . . . , Xn ] > ∼ N ( µ , Q−1 ) , ( 1 ) has a sparse precision ( inverse covariance ) matrix Q which enables the use of sparse matrix techniques such as sparse Cholesky factorization [ 8 ] and iterative methods [ 35 , 23 , 34 ] for computationally efficient sampling and statistical inference [ 27 , 28 ] . The sparsity is caused by conditional independence assumptions since Qij = 0 if and only if Xi and Xj are independent conditionally on all other variables in X [ see 28 , Chapter 2 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . For a GMRF X , a set of k linear constraints can be formulated as AX = b , ( 2 ) where each row in the k × n matrix A and the vector b encodes a constraint on X . Note that observations of GMRFs can be formulated as linear constraints , where , e.g. , a point observation of Xi is the simple constraint Xi = xi . These deterministic restrictions are often referred to as hard constraints [ 28 ] . If ( 2 ) is assumed to hold up to Gaussian noise , i.e. , AX ∼ N ( b , σ2I ) , then the constraints are instead referred to as soft constraints . This scenario is common when GMRFs are incorporated in hierarchical models , where the soft constraints represent noisy observations [ 28 ] . Because of the computational advantages of GMRFs , it is of interest to use GMRFs as alternatives to GPs also in the constrained case . Unfortunately , hard constraints can drastically reduce the computational benefits of GMRFs . Specifically , as we will explain in the next section , the current methods for GMRFs with hard constraints in general have a computational cost that scales cubically in the number of constraints k. This is prohibitive when there are many constraints , which is common for applications of constrained GPs . For example , physically constrained GPs such as those in [ 13 ] have k = n and any GMRF model with N observations without measurement noise will have k = N . In the first example the computational cost is cubic in n and in the second it can be cubic in N if the observations are such that A in ( 2 ) is not diagonal , which , e.g. , is the case for area observations [ 21 ] . In both cases , this renders the computational benefits of the unconstrained GMRF irrelevant . Until now , there has been no methods to circumvent this problem . Summary of contributions . The main contribution of this work is the formulation of a novel class of computationally efficient methods for linearly constrained GMRFs in tasks such as parameter estimation and simulation . These methods explore the conditional independence structure to reduce computational costs compared to traditional methods for situations with large numbers of constraints . The focus is in particular on the case , referred to as sparse hard constraints , when each constraint only involves a few number of variables so that A is sparse . This is , e.g. , common for GPs under physical constraints [ 13 ] and for GMRF models with point and area observations [ 21 ] . For this case , the main idea is to perform a change of basis so that the constraints are simpler to enforce in the transformed basis . In particular , we show how the change of basis facilitates using the same efficient methods for sparse matrices that are used in the non-constrained case also for the constrained models . In order to use these models for GP regression , the methods are also generalized to GMRF models with both hard and soft constraints . An important feature of the new class of methods , that previous approaches lack , is its applicability to intrinsic GMRFs , which are improper in the sense that a set of eigenvalues of Q is zero . This makes their distributions invariant to shifts in certain directions , which is a useful property for prior distributions of Bayesian models [ 28 , Chapter 3 ] . Because of this they are often used in areas such as semiparametric regression [ 10 ] , medical image analysis [ 24 , 34 ] , and geostatistics [ 15 , 7 ] , and it is thus of interest to be able to enforce linear constraints also for such models . The final contribution is the derivation of GMRFs for constrained GPs , by combining the proposed methods with the stochastic partial differential equation ( SPDE ) approach by [ 18 ] and the nested SPDE methods by [ 5 ] . The combined approach is highly computationally efficient compared to standard covariance-based methods , as illustrated in two simulation studies . Outline . In Section 2 , the problem is introduced in more detail and the most commonly used methods for sampling and likelihood computations for GMRFs are reviewed . Section 3 introduces the new methods for GMRFs with sparse hard constraints . These methods are extended to the case with both hard and soft constraints in Section 4 , followed by the GMRF methods for constrained GPs in Section 5 . The methods are illustrated numerically in Section 6 . A discussion closes the article , which is supported by a supplementary materials containing proofs and technical details . 2 Standard methods for GMRFs under hard constraints . Hard constraints can be divided into interacting and non-interacting constraints . In the latter , ( 2 ) specifies a constraint on a subset of the variables in X so that AX = b can be written as Xc = bc , where c denotes a subset of the indices . Specifically , let Xu denote the remaining variables , then X = [ Xc Xu ] ∼ N ( [ µc µu ] , [ Qcc Qcu Quc Quu ] −1 ) and X|AX = b ∼ N ( [ bc µu|c ] , [ 0 0 0 Q−1uu ] ) , where µu|c = µu −Q −1 uuQuc ( bc − µc ) . Thus , we can split the variables into two subsets and treat the unconditioned variables separately . The more difficult and interesting situation is the case of interacting hard constraints where a simple split of the variables is not possible . From now on , we will assume that we are in this scenario . Our aim is to construct methods for sampling from the distribution of X|AX = b and for evaluating its log-likelihood function . It is straightforward to show that X|AX = b ∼ N ( µ̂ , Σ̂ ) , where µ̂ = µ−Q−1A > ( AQ−1A > ) −1 ( Aµ−b ) and Σ̂ = Q−1−Q−1A > ( AQ−1A > ) −1AQ−1 . Since Σ̂ has rank n− k , likelihood evaluation and sampling is in general expensive . For example , one way is to use an eigenvalue decomposition of Σ̂ [ see 28 , Chapter 2.3.3 ] . However , this procedure is not practical since it can not take advantage of the sparsity of Q . Also , for intrinsic GMRFs , Σ̂ and µ̂ can not be constructed through the expressions above since Q−1 has unbounded eigenvalues . A commonly used method for sampling under hard linear constraints , sometimes referred to as conditioning by kriging [ 27 ] , is to first sample X ∼ N ( µ , Q−1 ) and then correct for the constraints by using X∗ = X−Q−1A > ( AQ−1A > ) −1 ( AX−b ) as a sample from the conditional distribution . Here the cost of sampling X is CQ + SQ , where CQ denotes the computational cost of a sparse Cholesky factorization Q = R > R and SQ the cost of solving Rx = u for x given u . Adding the cost for the correction step , the total cost of the method is O ( CQ + ( k + 2 ) SQ + k3 ) . Let πAx ( · ) denote the density of Ax ∼ N ( Aµ , AQ−1A > ) , then the likelihood of X|AX = b can be computed through the expression π ( x|Ax = b ) = π∗Ax|x ( b|x ) π ( x ) πAx ( b ) , ( 3 ) where π∗Ax|x ( b|x ) = I ( Ax = b ) |AA > |−1/2 and I ( Ax = b ) denotes the indicator function with I ( Ax = b ) = 1 if Ax = b and I ( Ax = b ) = 0 otherwise . This result is formulated in [ 27 ] and we provide further details in the supplementary materials by showing that ( 3 ) is a density with respect to the Lebesgue measure on the level set { x : Ax = b } . The computational cost of evaluating the likelihood using this formulation is O ( CQ + ( k + 1 ) SQ + k3 ) . Note that these methods only work efficiently for a small number of constraints , because of the term k3 in the computational costs , and for proper GMRFs . In the case of intrinsic GMRFs we can not work with AQ−1A due to the rank deficiency of Q . 3 The basis transformation method . In this section , we propose the new class of methods in two steps . We first derive a change of basis in Section 3.1 , and then use this to formulate the desired conditional distributions in Section 3.2 . The resulting computational costs of likelihood evaluations and simulation are discussed in Section 3.3 . Before stating the results we introduce some basic notation . When working with intrinsic GMRFs the definition in ( 1 ) is inconvenient since the covariance matrix has infinite eigenvalues . Instead one can use the canonical parametrization X ∼ NC ( µC , Q ) , which implies that the density of X is given by π ( x ) ∝ exp ( − 12x > Qx + µ > Cx ) and thus that µC = Qµ . Also , since we will be working with non-invertible matrices we will need the Moore–Penrose inverse and the pseudo determinant . We denote the Moore–Penrose inverse of a matrix B by B† and for a symmetric positive semi definite matrix M we define the pseudo determinant as |M|† = ∏ i : λi > 0 λi where { λi } are the eigenvalues of M. For the remainder of this article , we will assume the following . Assumption 1 . X ∼ NC ( Qµ , Q ) where Q is a positive semi-definite n × n matrix with rank n− s > 0 and null-space EQ . A is a k × n matrix with rank k and we let k0 denote rank ( AEQ ) . Finally , we use the index notation C = ( 1 , . . . , k ) and U = ( k + 1 , . . . , n ) so that , e.g. , vC denotes the sub-vector ( v1 , . . . , vk ) > of the first k elements in the vector v , M· , U ( MU , · ) denotes the submatrix obtained by extracting the columns ( rows ) in M with indices in U , and MU U the submatrix obtained by extracting the columns and rows in M with indices in U .

Summary: The paper proposes a method for evaluating the likelihood and for sampling from the posterior of Gaussian Markov random field $X$ given some linear constraints $X | AX = b$ where $A$ is a sparse matrix. The method is based on the SVD of $A$, which may be computed for a small cost if one can identify rows of $A$ that don't share columns with non-zero elements.   The method is illustrated on two toy examples, where it is shown that it provides massive computational gains. It is however unclear if the competitors considered in the benchmark are the actual alternatives a practitioner would consider.  ----------------------- The author's rebuttal makes me confident that the shortcomings I've identified have been understood by the authors and that they are in a good position to address them at the camera ready stage. I'm thus upgrading my score from 5 to 6.