A Comparison of Variable Selection Methods for Blockwise Diagonal Designs

1 INTRODUCTION . Variable selection is one of the core problems in high-dimensional data analysis . Consider a linear regression , where the response y ∈ Rn and the design matrix X = [ x1 , . . . , xp ] ∈ Rn×p satisfy that y = Xβ + z , ‖xj‖ = 1 , z ∼ N ( 0 , σ2In ) . ( 1 ) The goal is estimating the support of β ( Supp ( β ) ) . Lasso ( Tibshirani , 1996 ) is a popular method : β̂lasso = argminβ { ‖y −Xβ‖2/2 + λ‖β‖1 } . ( 2 ) Lasso has good rates of convergence on the Lq-estimation error or prediction error ( Bickel et al. , 2009 ) . However , it can be unsatisfactory for variable selection , especially when the columns in the design matrix are moderately or strongly correlated . Zhao & Yu ( 2006 ) showed that an irrepresentable condition on X is necessary for Lasso to recover Supp ( β ) with high probability , and such a condition is restrictive when p is large ( Fan & Lv , 2010 ) . Ji & Jin ( 2012 ) studied the Hamming error of Lasso and revealed Lasso ’ s non-optimality by lower-bounding its Hamming error rate . Many alternative strategies were proposed for variable selection , such as using non-convex penalties ( Fan & Li , 2001 ; Zhang , 2010 ; Shen et al. , 2012 ) , adding a ridge regularization ( Zou & Hastie , 2005 ) , post-processing on the Lasso estimator ( Zou , 2006 ; Zhou , 2009 ) , and iterative algorithms ( Zhang , 2011 ; Donoho et al. , 2012 ) . In this paper , our main interest is to theoretically compare these different strategies . Existing theoretical studies focused on ‘ model selection consistency ’ ( e.g. , Fan & Li ( 2001 ) ; Zhao & Yu ( 2006 ) ; Zou ( 2006 ) ; Meinshausen & Bühlmann ( 2010 ) ; Loh & Wainwright ( 2017 ) ) , which uses P ( Supp ( β̂ ) = Supp ( β ) ) to measure the performance of variable selection . However , for many real applications , the study of the Hamming error ( i.e. , total number of false positives and false negatives ) is in urgent need . For example , in genome-wide association studies ( GWAS ) or Genetic Regulatory Network , the goal is to identify the genes or SNPs that are truly associated with a given phenotype , and we hope to find a multiple testing procedure that simultaneously controls the FDR and maximizes the power ( for multiple testing ) . This problem can be re-cast as minimizing the Hamming error in a special regression setting ( Efron , 2004 ; Jin , 2012 ; Sun & Cai , 2007 ) . This motivates us to study the Hamming errors of variable selection methods , which were rarely considered in the literature . We adopt the rare and weak signal model ( Donoho & Jin , 2004 ; Arias-Castro et al. , 2011 ; Jin & Ke , 2016 ) , which is often used in theoretical analysis of sparse linear models . Let p be the asymptotic parameter . Given constants ϑ ∈ ( 0 , 1 ) and r > 0 , we assume that βj ’ s are iid generated such that βj = { τp , with probability p , 0 , with probability 1− p , where p = p−ϑ , τp = √ 2r log ( p ) . ( 3 ) As p→∞ , ‖β‖0 ≈ p1−ϑ , and a nonzero βj is at the minimax order √ log ( p ) . The two parameters ( ϑ , r ) capture the sparsity level and signal strength , respectively . We may generalize ( 3 ) to let nonzero βj ’ s take different values in [ τp , ∞ ) , but the current form is more convenient for presentation . The blockwise covariance structure is frequently observed in real applications . In genetic data , there may exist strong correlations between nearby genetic markers , but the long-range dependence is usually negligible ; as a result , the sample covariance matrix is approximately blockwise diagonal ( Dehman et al. , 2015 ) . In financial data , the sample covariance matrix of stock returns ( after common factors are removed ) is also approximately blockwise diagonal , where each block corresponds to an industry group ( Fan et al. , 2015 ) . Motivated by these examples , we consider an idealized setting , where the Gram matrix G = X ′X is block-wise diagonal consisting of 2× 2 blocks G = diag ( B , B , . . . , B , B0 ) , where B = [ 1 ρ ρ 1 ] and B0 = { B , if p is even , 1 , if p is odd . ( 4 ) This is an idealization of the blockwise covariance structures in real applications . We may generalize ( 4 ) to allow unequal block sizes and unequal off-diagonal entries , but we keep the current form for convenience of presentation . Model ( 4 ) is also closely connected to the random designs in compressed sensing ( Donoho , 2006 ) . Write X = [ X1 , X2 , . . . , Xn ] ′ . Suppose X1 , X2 , . . . , Xn are iid generated from N ( 0 , n−1Σ ) , where Σ has the same form as G in ( 4 ) . In a high-dimensional sparse setting , we have ‖β‖0 n p. Then , G = X ′X ≈ Σ , and due to the blessing of sparsity of β , Gβ ≈ Σβ . As a result , X ′y ( sufficient statistic of β ) satisfies that X ′y = Gβ +N ( 0 , G ) ≈ Σβ +N ( 0 , Σ ) , and the right hand side reduces to Model ( 4 ) ( Genovese et al. , 2012 ) . In Section 3.5 , we formally show that this random design setting is asymptotically equivalent to Model ( 4 ) , because they share the same rate of convergence for the optimal Hamming error of variable selection . Now , under model ( 3 ) and model ( 4 ) , we have three parameters ( ϑ , r , ρ ) . They capture the sparsity level , signal strength and design correlations , respectively . Our main results are the explicit convergence rates of Hamming error , as a function of ( ϑ , r , ρ ) , for different methods . We will study six methods : ( i ) Lasso as in ( 2 ) ; ( ii ) Elastic net ( Zou & Hastie , 2005 ) , which adds an additional L2penalty to ( 2 ) , ( iii ) smoothly clipped absolute deviation ( SCAD ) ( Fan & Li , 2001 ) , which replaces the L1-penalty by a non-convex penalty , ( iv ) thresholded Lasso ( Zhou , 2009 ) , which further thresholds the Lasso solution , and two iterative algorithms , ( v ) forward selection and ( vi ) forward backward selection ( Huang et al. , 2016 ) ; see Section 3 for a precise description of each method . To our best knowledge , our results are the first that directly compare Hamming errors of these methods . 2 A PREVIEW OF MAIN RESULTS AND SOME DISCUSSIONS . For any β̂ , its Hamming error is H ( β̂ , β ) = ∑p j=1 1 { β̂j 6= 0 , βj = 0 } + ∑p j=1 1 { β̂j = 0 , βj 6= 0 } . We shall show that , for each of the six methods studied here , there exists a function h ( ϑ , r , ρ ) ∈ [ 0 , 1 ] such that E [ H ( β̂ , β ) ] = Lpp1−h ( ϑ , r , ρ ) , where Lp is a multi-log ( p ) term ( a multi-log ( p ) term is such that Lp · p →∞ and Lp · p− → 0 for any > 0 . Noting that the expected number of true relevant variables is p1−ϑ , we are interested in three cases : • Exact recovery : h ( ϑ , r , ρ ) > 1 . In this case , the expected Hamming error is o ( 1 ) as p→∞ . It follows that model selection consistency holds . • Almost full recovery : ϑ < h ( ϑ , r , ρ ) < 1 . In this case , the expected Hamming error does not vanish as p→∞ , but it is much smaller than the total number of true relevant variables . Variable selection is still satisfactory ( although model selection consistency does not hold ) . • No recovery : h ( ϑ , r , ρ ) ≤ ϑ . In this case , the expected Hamming error is comparable with or much larger than the total number of true relevant variables . Variable selection fails . We call the two-dimensional space ( ϑ , r ) the phase space . For each fixed ρ , the phase space is divided into three regions : Region of Exact Recovery ( ER ) , which is the subset { ( ϑ , r ) : h ( ϑ , r , ρ ) > 1 } , and Region of Almost Full Recovery ( AFR ) and Region of No Recovery ( NR ) defined similarly . This gives rise to a phase diagram for each method . We denote the curve separating ER region and AFR region by r = U ( ϑ ) and the curve separating AFR region and NR region by r = L ( ϑ ) ; they are called the upper and lower phase curves , respectively . The phase diagram and phase curves are convenient ways to visualize the rates of convergence of the Hamming error . Figure 1 shows the phase curves for six methods ( explicit expressions are in the theorems in Section 3 ) . These phase curves depend on the correlation parameter ρ . Under our model , for each diagonal block ( j , j + 1 ) , it holds that E [ x′jy|β ] = βj + ρβj+1 , where βj , βj+1 ∈ { 0 , τp } . Therefore , a positive ρ boosts the signal at each individual site ( i.e. , E [ x′jy|β ] ≥ βj ) , while a negative ρ leads to potential ‘ signal cancellation ’ ( i.e. , E [ x′jy|β ] ≤ βj ) . This is why the phase curves have different shapes for positive and negative ρ . In Figure 1 , we plot the phase curves for ρ = 0.5 and ρ = −0.4 . For other positive/negative value of ρ , the patterns are similar . Discussion of SCAD . SCAD is a representative of the non-convex penalization methods . There were inspiring works that demonstrated the advantage of using a non-convex penalty ( e.g. , Fan & Peng ( 2004 ) ; Loh & Wainwright ( 2017 ) ) . Our results support their insights from a different angle : The phase curve of SCAD is strictly better than that of Lasso , when ϑ < 0.5 and ρ < 0 . Furthermore , our results illustrate where the advantage of SCAD comes from — compared with Lasso , it handles ‘ signal cancellation ’ better . To see this , we recall that ‘ signal cancellation ’ can only happen for ρ < 0 . Moreover , under our model ( 3 ) , for a diagonal block { j , j + 1 } , we get a pair of signals ( i.e. , βj and βj+1 are both nonzero ) with probability 2p = p 1−2ϑ approximately . Therefore , ‘ signal cancellation ’ becomes problematic only when ϑ < 0.5 and ρ < 0 both hold . This explains why the phase curves of SCAD and Lasso are the same for the other values of ϑ and ρ . We note that in previous studies ( e.g. , Loh & Wainwright ( 2017 ) ) , the advantage of a non-convex penalty in handling ‘ signal cancellation ’ is reflected in the conditions of ( X , β ) required for model selection consistency . Our results support this viewpoint from a different angle . The performance of SCAD can be further improved by adding an entry-wise thresholding on β̂ . We are sure that the phase diagrams of thresholded SCAD will be better than those of SCAD , although we do not have the results in hand ( the analysis is too tedious ) . We are also cautious in comparing SCAD and thresholded Lasso . In our settings , Lasso has no model selection consistency because the signals are too weak ( i.e. , r is not sufficiently large ) . In such a case , thresholded Lasso outperforms SCAD in terms of the Hamming error . Alternatively , there are cases where Lasso has no model selection consistency no matter how large the signal strength is ( Zhao & Yu , 2006 ) . For those cases , it is possible that SCAD is better than thresholded Lasso ( see Wainwright ( 2009 ) for a related study ) . Discussion of Elastic net . The phase curve of Elastic net is worse than that of Lasso . As we will explain in Section 3.1 , Elastic net is a ‘ bridge ’ between Lasso and marginal regression . Since the phase curve of marginal regression is always worse than that of Lasso for the blockwise diagonal design , we do not benefit from using Elastic net in the current setting . We must note that Elastic net was motivated by genetic applications where several correlated variables are competing as predictors , and where it is implicitly assumed that groups of correlated variables tend to be all relevant or all irrelevant ( Zou & Hastie , 2005 ) . This is not captured by our model ( 3 ) . Therefore , our result has no contradiction to the benefit of Elastic net known in the literature . Our result actually supports that the advantage of Elastic net comes from ‘ group ’ appearance of signal variables . We conjecture that for a different signal model ( e.g. , a group prior on β ) , Elastic net has a better phase diagram than Lasso . Discussion of thresholded Lasso . Thresholded Lasso is a representative of improving Lasso by post-processing . There have been inspiring works that demonstrate the advantage of such a postprocessing ( van de Geer et al. , 2011 ; Wang et al. , 2020 ; Weinstein et al. , 2020 ) . Our results support these insights from a different angle . It is surprising ( and very encouraging ) that the improvement by post-Lasso thresholding is so significant . We note that Lasso is a 1-stage method , which solves a single optimization to obtain β̂ . In comparison , thresholded Lasso is a 2-stage method . Lasso has only one algorithm parameter λ , while thresholded Lasso has two algorithm parameters λ and t ( the threshold ) . In Lasso , we control false positives and false negatives via the same algorithm parameter λ , and it is sometimes hard to find a value of λ that simultaneously controls the two types of errors well . In thresholded Lasso , the two types of errors are controlled separately by two algorithm parameters . This explains why thresholded Lasso has such a big improvement over Lasso . It inspires us to modify other 1-stage methods , such as SCAD , by adding a post-processing step of thresholding . For example , we conjecture that thresholded SCAD also has a strictly better phase diagram than that of Lasso , even for a positive ρ . On the other hand , thresholding is not a free lunch . It yields one more tuning parameter to decide in practice . Our theoretical results are based on ideal tuning parameters . How to properly choose these tuning parameters in a data-driven way is an interesting question . Weinstein et al . ( 2020 ) proposed a promising approach , where they used cross-validation to select λ and FDR control by knock-off to select t. We leave it to future work to study the phase diagram with data-driven tuning parameters . Discussion of the two iterative algorithms . We consider two iterative algorithms , forward selection ( ‘ Forward ’ ) and forward backward selection ( FB ) . The FB algorithm we analyze is a simplified version in Huang et al . ( 2016 ) , which has only one backward step ( after all the forward steps are completed ) by thresholding the refitted least-squares solution . Our results show that both methods outperform Lasso , and between these two methods , FB is strictly better than Forward . In the literature , there were very interesting theoretical works showing the advantage of iterative algorithms for variable selection ( Donoho et al. , 2012 ; Zhang , 2011 ) . Our results support their insights from a different angle . We discover that , for a wide range of ρ , FB has the best phase diagram among all six methods . This is a very encouraging result . At the same time , we keep in mind that the performance of an iterative algorithm tends to be more sensitive to the form of the design , due to its sequential nature .

Summary: This paper investigates the theoretical performance of a number of variables selection methods when the columns of the design matrix are correlated. They focus specifically on the case where the Gram matrix for the design is block-wise diagonal (2 x 2 blocks) with the off-diagonals within each block given by a correlation parameter and diagonals set to 1. The quality of each estimator is evaluated by using its expected Hamming distance. Extending ideas presented in _UPS delivers optimal phase diagram in high-dimensional variable selection [Ji and Jin]_ and others, phase diagrams are constructed with phase curves (derived in the appendices) establishing the boundary between each region: exact recovery, almost full recovery, and no recovery. A discussion of each variable selection method provides insight and a comparison between them is included.