T-LoHo: A Bayesian Regularization Model for Structured Sparsity and Smoothness on Graphs

1 Introduction . In high-dimensional models such as linear regressions where the number of parameters p may exceed the sample size n , it is often assumed that the parameter vector β ∈ Rp has many zero components , namely the sparsity assumption . This sparsity assumption allows the unknown parameter β to lie on a low-dimensional subspace of Rp , which addresses overfitting and leads to improved predictions and easier interpretations ( Hastie et al. , 2015 ) . In many real-life applications , however , the parameter of interest β should be understood in a certain context where specific data structures exist , such as in time series , spatial , image , and network data analysis . In many such cases , it is desirable to consider another type of low-dimensional structure , where β is assumed to have clustered patterns and many possibly clustered zeros , which we shall call sparse homogeneity assumption . One of the most popular models which assumes sparse homogeneity is the sparse fused lasso ( Tibshirani et al. , 2005 ) in linear regression settings . 1-D sparse fused lasso imposes an ` 1 penalty on the differences of time-neighboring coefficients as well as individual coefficients . The notion of ‘ time-neighboring ’ can be further generalized by considering a graph G= ( V , E ) with |V | = p so that G represents the general context or structure in which β should be interpreted . It leads to the generalized lasso ( Tibshirani et al. , 2011 ) where the penalty term now involves m : = |E| number of pairwise differences of neighboring vertices with respect to G which facilitates βi = βj , where 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . ( i , j ) ∈ E. In other words , the generalized lasso now allows neighboring coefficients with respect to the graph G to be clustered together . Other similar graph-based regularization approaches include graph OSCAR ( Yang et al. , 2012 ) , grouping pursuit ( Zhu et al. , 2013 ) and graph trend filtering ( Wang et al. , 2016 ) . However , when the number of edges , m , is large relative to p , regularization over the whole graph structure often faces severe computational burden . Ke et al . ( 2015 ) and Tang and Song ( 2016 ) addressed this problem by constructing a preliminary ranking of coefficients and performing a segmentation using 1-D fused lasso . Padilla et al . ( 2017 ) used depth-first search ( DFS ) algorithm to establish a chain graph order for constructing generalized lasso . In a spatial setting , Li and Sang ( 2019 ) used a Euclidean minimum spanning tree to form the fused lasso penalty which also enjoys computational benefits . Nevertheless , these fixed chain or tree orders may not be compatible with the true cluster with respect to its context G and hence may lead to over-clustering . Besides , these optimization-based penalized estimators do not usually come with uncertainty measures . Bayesian regularization methods have gained great popularity in high-dimensional models due to its flexibility and convenience in quantifying estimation and prediction uncertainties . There is rich literature in Bayesian high-dimensional regression models under the sparsity assumption , which can be roughly summarized into two categories : ( i ) spike-and-slab priors ( George and McCulloch , 1993 , 1997 ) and ( ii ) global-local shrinkage priors ( see Bhadra et al. , 2019 , and references therein ) . A particular shrinkage prior , called the horseshoe prior ( Carvalho et al. , 2010 ) , has gained a lot of attraction due to its tail-robustness and super-efficiency as well as substantial computational benefits of the global-local shrinkage prior family ( Polson and Scott , 2010 ) . However , relatively few works have been done in the area of Bayesian high-dimensional regression models under the sparse homogeneity assumption . Most approaches put sparsity-inducing priors directly on the pairwise differences of neighboring vertices of a graph to achieve the graph-structured homogeneity of β ( Kyung et al. , 2010 ; Shimamura et al. , 2019 ; Song and Cheng , 2020 ; Kim and Gao , 2020 ; Banerjee , 2021 ) . But similar to the generalized lasso , unless resorting to approximate methods such as EM algorithm , these approaches become computationally expensive to get posterior samples for general graphs with a large number of edges . Another more subtle limitation of these methods is that they fail to incorporate structural assumptions among local shrinkage parameters . In this paper , we propose a Bayesian Tree-based Low-rank Horseshoe ( T-LoHo ) model to identify structured sparsity and smoothness of parameters whose prior structure is represented by a graph . Unlike other existing Bayesian methods which put independent priors on pairwise differences , TLoHo extends the univariate horseshoe shrinkage prior to a multivariate setting where β and its local shrinkage parameters are assumed to be piecewise constants on a graph . This low-rank structured model for local shrinkage parameters allows both clustering and sparsity effects to have strong local adaptivities . A random spanning forest ( RSF ) based graph clustering prior is introduced to adaptively learn a compatible neighboring order to model graph partitions , which extends the recently developed random spanning tree partition models ( Teixeira et al. , 2019 ; Luo et al. , 2021 ) to allow for possibly unconnected graphs . We show that this RSF-based prior retains model richness as its support is flexible enough to accommodate all possible contiguous partitions . We introduce T-LoHo prior to model high-dimensional linear regression coefficients , although it can be flexibly embedded in other high-dimensional models . The resulting cluster estimates not only provide uncertainty measures but also have strong flexibility to accommodate sharp discontinuities . With state of the art computational strategies , we provide a highly efficient MCMC algorithm for posterior inference utilizing tree structures . We also study theoretical aspects of our model including posterior consistency results and its effect on clustering . We demonstrate the efficacy of T-LoHo model by using synthetic data and a real data analysis for anomaly detection on the Manhattan road network . 2 T-LoHo : Tree-based Low-rank Horseshoe Model . Consider a graph G = ( V , E ) with |V | = p and |E| = m which represents the pre-known structure of our parameter of interest β ∈ Rp . For example , when β has a certain order ( e.g . time ) , then we set G as a linear chain graph . When ( vectorized ) β lies on a 2-D image , we let G be a 2-D lattice . A contiguous graph partition Π= { C1 , . . . , CK } is a disjoint partition of V such that each Ck induces a connected subgraph of G ( the term graph partition will always refer to a contiguous graph partition hereafter ) . Under the sparse homogeneity assumption on β , the goal is to find a graph partition Π with unknown size K and corresponding estimate of β which contains many zeros . We introduce the model for β conditional on a partition Π in Section 2.1 and describe the model for Π in Section 2.2 . 2.1 Low-Rank Multivariate Horseshoe Prior . First we review the horseshoe prior ( Carvalho et al. , 2010 ) , which provides a sparse estimate of β by shrinking small effects towards zero while maintaining robustness to large signals : β|σ2 , τ2 , { λj } pj=1 ∼ Np ( 0 , σ 2τ2diag ( λ21 , . . . , λ 2 p ) ) , λj iid∼ C+ ( 0 , 1 ) , τ ∼ C+ ( 0 , τ0 ) , p ( σ2 ) ∝ 1/σ2 , where Np ( m , V ) denotes a p-dimensional multivariate normal distribution with meanm and covariance V , and C+ ( 0 , γ ) denotes a half-Cauchy distribution with density p ( λ ) = 2/ [ πγ ( 1 + λ2/γ2 ) ] for λ > 0 . Here τ is a global shrinkage parameter with hyperparameter τ0 to enforce global shrinkage towards zero , { λj } are local shrinkage parameters with heavy tails which allow some of the βj ’ s to escape the shrinkage , and σ2 is a scaling factor which is often assumed to be the noise variance . Given a graph partition Π of G , we seek a new joint prior distribution of β ∈ Rp to impose sparse homogeneity assumption while incorporating the clustering structure of Π . For example when p = 5 and Π = { { 1 } , { 2 , 4 } , { 3 , 5 } } as shown in Figure 1 ( a ) , we seek a distribution which lies on a 3-dimensional subspace { β ∈ R5 : β2 = β4 , β3 = β5 } . In general , given a graph partition Π with size K , we can construct a K × p matrix Φ from Π : Φkj = 1/ √ |Ck| if j ∈ Ck and 0 otherwise , k = 1 , . . . , K , j = 1 , . . . , p. ( 1 ) This K × p matrix Φ represents a projection from a p-dimensional space to a reduced K-dimensional space . Note that rows of Φ are mutually orthonormal so that ΦΦ > = IK . Using Φ obtained from Π , we introduce the low-rank horseshoe ( LoHo ) prior to impose sparse homogeneity assumption on β : β |σ2 , τ2 , Λ , Π ∼ Np ( 0 , σ2τ2Φ > ΛΦ ) , Λ : = diag ( λ21 , . . . , λ2K ) , ( 2 ) λk iid∼ C+ ( 0 , 1 ) , τ ∼ C+ ( 0 , τ0 ) , p ( σ2 ) ∝ 1/σ2 . ( 3 ) See fig . 1 ( a ) for an example of Φ and a covariance matrix Φ > ΛΦ . Note LoHo introduces a covariance matrix capturing clustering dependence among β and its marginal local shrinkage for simultaneous sparsity and fusion , a key distinction from most existing methods ( Carvalho et al. , 2010 ; Kyung et al. , 2010 ; Shimamura et al. , 2019 ) that assume independence or only dependence among β but not local shrinkage parameters . Since K is assumed to be small relative to p , the covariance matrix Φ > ΛΦ has low rank ( i.e. , rank ( Φ > ΛΦ ) = K < < p ) , and thus LoHo does not have a density with respect to Lebesgue measure on Rp . Instead , its distribution lies on the row space of Φ with dim ( rowsp ( Φ ) ) = K , and we can consider the transformation β̃ = Φβ so that β̃ has a NK ( 0 , σ2τ2Λ ) density with respect to Lebesgue measure on rowsp ( Φ ) . Observe that Φ > is the Moore–Penrose pseudoinverse of Φ , which implies Φ > Φ is a projection onto rowsp ( Φ ) so that we can recover β = Φ > β̃ . By assigning a half-Cauchy prior , global shrinkage parameter τ creates a strong pull towards zero while clusterwise local shrinkage parameters { λk } Kk=1 allow some of the clusterwise β̃k ’ s to escape the shrinkage . Assuming Φ having orthonormal rows is important in the sense that the effect of local shrinkage parameters { λk } Kk=1 remains same across the clusters with varying size . Thus using the projection onto the low-dimensional subspace , LoHo gives more parsimonious estimate of β by forming clusters of zero and non-zero parameters . LoHo can be naturally incorporated into a linear model . With response vector y ∈ Rn and columnstandardized design matrix X ∈ Rn×p so that each column has unit ` 2 norm , we can write y = Xβ + , ∼ Nn ( 0 , σ2In ) Under this formulation , LoHo has a close connection with Bayesian compressed regression ( Guhaniyogi and Dunson , 2015 ) . It randomly projects predictors Xi 7→ ΦXi with a certain family of matrix Φ and performs model averaging , at the cost of losing interpretability of β . But LoHo directly introduces a prior on β using the projection matrix Φ defined as ( 1 ) so that it induces clustered coefficient while maintaining interpretability of β . Also , functional horseshoe ( Shin et al. , 2020 ) shrinks Xβ towards the subspace of colsp ( X ) while LoHo shrinks β towards 0 along the rowsp ( Φ ) . We remark that , although it appears ( τ , Λ ) and Φ handle sparsity and homogeneity separately , shrinkage component in LoHo also affects clustering by facilitating cluster fusion when signal is low and improving cluster identification when signal is high . More details are discussed in Section 3.2 .

Summary: This paper propose a new prior referred to as Tree-based Low-rank Horseshoe (T-Loho), which can incorporate relationship among variables as a graph. Theoretical characteristics of the clustering effects and posterior concentrations are investigated. In computational experiments based on both synthetic and real-world data, T-Loho turned out to be superior to fused lasso and soft-thresholded Gaussian process in terms of both regression performance and support recovery.