T-LoHo: A Bayesian Regularization Model for Structured Sparsity and Smoothness on Graphs

1 Introduction . In high-dimensional models such as linear regressions where the number of parameters p may exceed the sample size n , it is often assumed that the parameter vector β ∈ Rp has many zero components , namely the sparsity assumption . This sparsity assumption allows the unknown parameter β to lie on a low-dimensional subspace of Rp , which addresses overfitting and leads to improved predictions and easier interpretations ( Hastie et al. , 2015 ) . In many real-life applications , however , the parameter of interest β should be understood in a certain context where specific data structures exist , such as in time series , spatial , image , and network data analysis . In many such cases , it is desirable to consider another type of low-dimensional structure , where β is assumed to have clustered patterns and many possibly clustered zeros , which we shall call sparse homogeneity assumption . One of the most popular models which assumes sparse homogeneity is the sparse fused lasso ( Tibshirani et al. , 2005 ) in linear regression settings . 1-D sparse fused lasso imposes an ` 1 penalty on the differences of time-neighboring coefficients as well as individual coefficients . The notion of ‘ time-neighboring ’ can be further generalized by considering a graph G= ( V , E ) with |V | = p so that G represents the general context or structure in which β should be interpreted . It leads to the generalized lasso ( Tibshirani et al. , 2011 ) where the penalty term now involves m : = |E| number of pairwise differences of neighboring vertices with respect to G which facilitates βi = βj , where 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . ( i , j ) ∈ E. In other words , the generalized lasso now allows neighboring coefficients with respect to the graph G to be clustered together . Other similar graph-based regularization approaches include graph OSCAR ( Yang et al. , 2012 ) , grouping pursuit ( Zhu et al. , 2013 ) and graph trend filtering ( Wang et al. , 2016 ) . However , when the number of edges , m , is large relative to p , regularization over the whole graph structure often faces severe computational burden . Ke et al . ( 2015 ) and Tang and Song ( 2016 ) addressed this problem by constructing a preliminary ranking of coefficients and performing a segmentation using 1-D fused lasso . Padilla et al . ( 2017 ) used depth-first search ( DFS ) algorithm to establish a chain graph order for constructing generalized lasso . In a spatial setting , Li and Sang ( 2019 ) used a Euclidean minimum spanning tree to form the fused lasso penalty which also enjoys computational benefits . Nevertheless , these fixed chain or tree orders may not be compatible with the true cluster with respect to its context G and hence may lead to over-clustering . Besides , these optimization-based penalized estimators do not usually come with uncertainty measures . Bayesian regularization methods have gained great popularity in high-dimensional models due to its flexibility and convenience in quantifying estimation and prediction uncertainties . There is rich literature in Bayesian high-dimensional regression models under the sparsity assumption , which can be roughly summarized into two categories : ( i ) spike-and-slab priors ( George and McCulloch , 1993 , 1997 ) and ( ii ) global-local shrinkage priors ( see Bhadra et al. , 2019 , and references therein ) . A particular shrinkage prior , called the horseshoe prior ( Carvalho et al. , 2010 ) , has gained a lot of attraction due to its tail-robustness and super-efficiency as well as substantial computational benefits of the global-local shrinkage prior family ( Polson and Scott , 2010 ) . However , relatively few works have been done in the area of Bayesian high-dimensional regression models under the sparse homogeneity assumption . Most approaches put sparsity-inducing priors directly on the pairwise differences of neighboring vertices of a graph to achieve the graph-structured homogeneity of β ( Kyung et al. , 2010 ; Shimamura et al. , 2019 ; Song and Cheng , 2020 ; Kim and Gao , 2020 ; Banerjee , 2021 ) . But similar to the generalized lasso , unless resorting to approximate methods such as EM algorithm , these approaches become computationally expensive to get posterior samples for general graphs with a large number of edges . Another more subtle limitation of these methods is that they fail to incorporate structural assumptions among local shrinkage parameters . In this paper , we propose a Bayesian Tree-based Low-rank Horseshoe ( T-LoHo ) model to identify structured sparsity and smoothness of parameters whose prior structure is represented by a graph . Unlike other existing Bayesian methods which put independent priors on pairwise differences , TLoHo extends the univariate horseshoe shrinkage prior to a multivariate setting where β and its local shrinkage parameters are assumed to be piecewise constants on a graph . This low-rank structured model for local shrinkage parameters allows both clustering and sparsity effects to have strong local adaptivities . A random spanning forest ( RSF ) based graph clustering prior is introduced to adaptively learn a compatible neighboring order to model graph partitions , which extends the recently developed random spanning tree partition models ( Teixeira et al. , 2019 ; Luo et al. , 2021 ) to allow for possibly unconnected graphs . We show that this RSF-based prior retains model richness as its support is flexible enough to accommodate all possible contiguous partitions . We introduce T-LoHo prior to model high-dimensional linear regression coefficients , although it can be flexibly embedded in other high-dimensional models . The resulting cluster estimates not only provide uncertainty measures but also have strong flexibility to accommodate sharp discontinuities . With state of the art computational strategies , we provide a highly efficient MCMC algorithm for posterior inference utilizing tree structures . We also study theoretical aspects of our model including posterior consistency results and its effect on clustering . We demonstrate the efficacy of T-LoHo model by using synthetic data and a real data analysis for anomaly detection on the Manhattan road network . 2 T-LoHo : Tree-based Low-rank Horseshoe Model . Consider a graph G = ( V , E ) with |V | = p and |E| = m which represents the pre-known structure of our parameter of interest β ∈ Rp . For example , when β has a certain order ( e.g . time ) , then we set G as a linear chain graph . When ( vectorized ) β lies on a 2-D image , we let G be a 2-D lattice . A contiguous graph partition Π= { C1 , . . . , CK } is a disjoint partition of V such that each Ck induces a connected subgraph of G ( the term graph partition will always refer to a contiguous graph partition hereafter ) . Under the sparse homogeneity assumption on β , the goal is to find a graph partition Π with unknown size K and corresponding estimate of β which contains many zeros . We introduce the model for β conditional on a partition Π in Section 2.1 and describe the model for Π in Section 2.2 . 2.1 Low-Rank Multivariate Horseshoe Prior . First we review the horseshoe prior ( Carvalho et al. , 2010 ) , which provides a sparse estimate of β by shrinking small effects towards zero while maintaining robustness to large signals : β|σ2 , τ2 , { λj } pj=1 ∼ Np ( 0 , σ 2τ2diag ( λ21 , . . . , λ 2 p ) ) , λj iid∼ C+ ( 0 , 1 ) , τ ∼ C+ ( 0 , τ0 ) , p ( σ2 ) ∝ 1/σ2 , where Np ( m , V ) denotes a p-dimensional multivariate normal distribution with meanm and covariance V , and C+ ( 0 , γ ) denotes a half-Cauchy distribution with density p ( λ ) = 2/ [ πγ ( 1 + λ2/γ2 ) ] for λ > 0 . Here τ is a global shrinkage parameter with hyperparameter τ0 to enforce global shrinkage towards zero , { λj } are local shrinkage parameters with heavy tails which allow some of the βj ’ s to escape the shrinkage , and σ2 is a scaling factor which is often assumed to be the noise variance . Given a graph partition Π of G , we seek a new joint prior distribution of β ∈ Rp to impose sparse homogeneity assumption while incorporating the clustering structure of Π . For example when p = 5 and Π = { { 1 } , { 2 , 4 } , { 3 , 5 } } as shown in Figure 1 ( a ) , we seek a distribution which lies on a 3-dimensional subspace { β ∈ R5 : β2 = β4 , β3 = β5 } . In general , given a graph partition Π with size K , we can construct a K × p matrix Φ from Π : Φkj = 1/ √ |Ck| if j ∈ Ck and 0 otherwise , k = 1 , . . . , K , j = 1 , . . . , p. ( 1 ) This K × p matrix Φ represents a projection from a p-dimensional space to a reduced K-dimensional space . Note that rows of Φ are mutually orthonormal so that ΦΦ > = IK . Using Φ obtained from Π , we introduce the low-rank horseshoe ( LoHo ) prior to impose sparse homogeneity assumption on β : β |σ2 , τ2 , Λ , Π ∼ Np ( 0 , σ2τ2Φ > ΛΦ ) , Λ : = diag ( λ21 , . . . , λ2K ) , ( 2 ) λk iid∼ C+ ( 0 , 1 ) , τ ∼ C+ ( 0 , τ0 ) , p ( σ2 ) ∝ 1/σ2 . ( 3 ) See fig . 1 ( a ) for an example of Φ and a covariance matrix Φ > ΛΦ . Note LoHo introduces a covariance matrix capturing clustering dependence among β and its marginal local shrinkage for simultaneous sparsity and fusion , a key distinction from most existing methods ( Carvalho et al. , 2010 ; Kyung et al. , 2010 ; Shimamura et al. , 2019 ) that assume independence or only dependence among β but not local shrinkage parameters . Since K is assumed to be small relative to p , the covariance matrix Φ > ΛΦ has low rank ( i.e. , rank ( Φ > ΛΦ ) = K < < p ) , and thus LoHo does not have a density with respect to Lebesgue measure on Rp . Instead , its distribution lies on the row space of Φ with dim ( rowsp ( Φ ) ) = K , and we can consider the transformation β̃ = Φβ so that β̃ has a NK ( 0 , σ2τ2Λ ) density with respect to Lebesgue measure on rowsp ( Φ ) . Observe that Φ > is the Moore–Penrose pseudoinverse of Φ , which implies Φ > Φ is a projection onto rowsp ( Φ ) so that we can recover β = Φ > β̃ . By assigning a half-Cauchy prior , global shrinkage parameter τ creates a strong pull towards zero while clusterwise local shrinkage parameters { λk } Kk=1 allow some of the clusterwise β̃k ’ s to escape the shrinkage . Assuming Φ having orthonormal rows is important in the sense that the effect of local shrinkage parameters { λk } Kk=1 remains same across the clusters with varying size . Thus using the projection onto the low-dimensional subspace , LoHo gives more parsimonious estimate of β by forming clusters of zero and non-zero parameters . LoHo can be naturally incorporated into a linear model . With response vector y ∈ Rn and columnstandardized design matrix X ∈ Rn×p so that each column has unit ` 2 norm , we can write y = Xβ + , ∼ Nn ( 0 , σ2In ) Under this formulation , LoHo has a close connection with Bayesian compressed regression ( Guhaniyogi and Dunson , 2015 ) . It randomly projects predictors Xi 7→ ΦXi with a certain family of matrix Φ and performs model averaging , at the cost of losing interpretability of β . But LoHo directly introduces a prior on β using the projection matrix Φ defined as ( 1 ) so that it induces clustered coefficient while maintaining interpretability of β . Also , functional horseshoe ( Shin et al. , 2020 ) shrinks Xβ towards the subspace of colsp ( X ) while LoHo shrinks β towards 0 along the rowsp ( Φ ) . We remark that , although it appears ( τ , Λ ) and Φ handle sparsity and homogeneity separately , shrinkage component in LoHo also affects clustering by facilitating cluster fusion when signal is low and improving cluster identification when signal is high . More details are discussed in Section 3.2 .

Summary: The authors generalize the univariate Bayesian horseshoe shrinkage prior to the multivariate setting and propose a new prior that detects sparsity and encourages smoothness within clusters of variables. The paper illustrates the application of the prior in Bayesian regression and present an MCMC algorithm. Numerical experiments with synthetic data and results with one real dataset illustrate the potential advantages over alternative methods.