A Comparison of Variable Selection Methods for Blockwise Diagonal Designs

1 INTRODUCTION . Variable selection is one of the core problems in high-dimensional data analysis . Consider a linear regression , where the response y ∈ Rn and the design matrix X = [ x1 , . . . , xp ] ∈ Rn×p satisfy that y = Xβ + z , ‖xj‖ = 1 , z ∼ N ( 0 , σ2In ) . ( 1 ) The goal is estimating the support of β ( Supp ( β ) ) . Lasso ( Tibshirani , 1996 ) is a popular method : β̂lasso = argminβ { ‖y −Xβ‖2/2 + λ‖β‖1 } . ( 2 ) Lasso has good rates of convergence on the Lq-estimation error or prediction error ( Bickel et al. , 2009 ) . However , it can be unsatisfactory for variable selection , especially when the columns in the design matrix are moderately or strongly correlated . Zhao & Yu ( 2006 ) showed that an irrepresentable condition on X is necessary for Lasso to recover Supp ( β ) with high probability , and such a condition is restrictive when p is large ( Fan & Lv , 2010 ) . Ji & Jin ( 2012 ) studied the Hamming error of Lasso and revealed Lasso ’ s non-optimality by lower-bounding its Hamming error rate . Many alternative strategies were proposed for variable selection , such as using non-convex penalties ( Fan & Li , 2001 ; Zhang , 2010 ; Shen et al. , 2012 ) , adding a ridge regularization ( Zou & Hastie , 2005 ) , post-processing on the Lasso estimator ( Zou , 2006 ; Zhou , 2009 ) , and iterative algorithms ( Zhang , 2011 ; Donoho et al. , 2012 ) . In this paper , our main interest is to theoretically compare these different strategies . Existing theoretical studies focused on ‘ model selection consistency ’ ( e.g. , Fan & Li ( 2001 ) ; Zhao & Yu ( 2006 ) ; Zou ( 2006 ) ; Meinshausen & Bühlmann ( 2010 ) ; Loh & Wainwright ( 2017 ) ) , which uses P ( Supp ( β̂ ) = Supp ( β ) ) to measure the performance of variable selection . However , for many real applications , the study of the Hamming error ( i.e. , total number of false positives and false negatives ) is in urgent need . For example , in genome-wide association studies ( GWAS ) or Genetic Regulatory Network , the goal is to identify the genes or SNPs that are truly associated with a given phenotype , and we hope to find a multiple testing procedure that simultaneously controls the FDR and maximizes the power ( for multiple testing ) . This problem can be re-cast as minimizing the Hamming error in a special regression setting ( Efron , 2004 ; Jin , 2012 ; Sun & Cai , 2007 ) . This motivates us to study the Hamming errors of variable selection methods , which were rarely considered in the literature . We adopt the rare and weak signal model ( Donoho & Jin , 2004 ; Arias-Castro et al. , 2011 ; Jin & Ke , 2016 ) , which is often used in theoretical analysis of sparse linear models . Let p be the asymptotic parameter . Given constants ϑ ∈ ( 0 , 1 ) and r > 0 , we assume that βj ’ s are iid generated such that βj = { τp , with probability p , 0 , with probability 1− p , where p = p−ϑ , τp = √ 2r log ( p ) . ( 3 ) As p→∞ , ‖β‖0 ≈ p1−ϑ , and a nonzero βj is at the minimax order √ log ( p ) . The two parameters ( ϑ , r ) capture the sparsity level and signal strength , respectively . We may generalize ( 3 ) to let nonzero βj ’ s take different values in [ τp , ∞ ) , but the current form is more convenient for presentation . The blockwise covariance structure is frequently observed in real applications . In genetic data , there may exist strong correlations between nearby genetic markers , but the long-range dependence is usually negligible ; as a result , the sample covariance matrix is approximately blockwise diagonal ( Dehman et al. , 2015 ) . In financial data , the sample covariance matrix of stock returns ( after common factors are removed ) is also approximately blockwise diagonal , where each block corresponds to an industry group ( Fan et al. , 2015 ) . Motivated by these examples , we consider an idealized setting , where the Gram matrix G = X ′X is block-wise diagonal consisting of 2× 2 blocks G = diag ( B , B , . . . , B , B0 ) , where B = [ 1 ρ ρ 1 ] and B0 = { B , if p is even , 1 , if p is odd . ( 4 ) This is an idealization of the blockwise covariance structures in real applications . We may generalize ( 4 ) to allow unequal block sizes and unequal off-diagonal entries , but we keep the current form for convenience of presentation . Model ( 4 ) is also closely connected to the random designs in compressed sensing ( Donoho , 2006 ) . Write X = [ X1 , X2 , . . . , Xn ] ′ . Suppose X1 , X2 , . . . , Xn are iid generated from N ( 0 , n−1Σ ) , where Σ has the same form as G in ( 4 ) . In a high-dimensional sparse setting , we have ‖β‖0 n p. Then , G = X ′X ≈ Σ , and due to the blessing of sparsity of β , Gβ ≈ Σβ . As a result , X ′y ( sufficient statistic of β ) satisfies that X ′y = Gβ +N ( 0 , G ) ≈ Σβ +N ( 0 , Σ ) , and the right hand side reduces to Model ( 4 ) ( Genovese et al. , 2012 ) . In Section 3.5 , we formally show that this random design setting is asymptotically equivalent to Model ( 4 ) , because they share the same rate of convergence for the optimal Hamming error of variable selection . Now , under model ( 3 ) and model ( 4 ) , we have three parameters ( ϑ , r , ρ ) . They capture the sparsity level , signal strength and design correlations , respectively . Our main results are the explicit convergence rates of Hamming error , as a function of ( ϑ , r , ρ ) , for different methods . We will study six methods : ( i ) Lasso as in ( 2 ) ; ( ii ) Elastic net ( Zou & Hastie , 2005 ) , which adds an additional L2penalty to ( 2 ) , ( iii ) smoothly clipped absolute deviation ( SCAD ) ( Fan & Li , 2001 ) , which replaces the L1-penalty by a non-convex penalty , ( iv ) thresholded Lasso ( Zhou , 2009 ) , which further thresholds the Lasso solution , and two iterative algorithms , ( v ) forward selection and ( vi ) forward backward selection ( Huang et al. , 2016 ) ; see Section 3 for a precise description of each method . To our best knowledge , our results are the first that directly compare Hamming errors of these methods . 2 A PREVIEW OF MAIN RESULTS AND SOME DISCUSSIONS . For any β̂ , its Hamming error is H ( β̂ , β ) = ∑p j=1 1 { β̂j 6= 0 , βj = 0 } + ∑p j=1 1 { β̂j = 0 , βj 6= 0 } . We shall show that , for each of the six methods studied here , there exists a function h ( ϑ , r , ρ ) ∈ [ 0 , 1 ] such that E [ H ( β̂ , β ) ] = Lpp1−h ( ϑ , r , ρ ) , where Lp is a multi-log ( p ) term ( a multi-log ( p ) term is such that Lp · p →∞ and Lp · p− → 0 for any > 0 . Noting that the expected number of true relevant variables is p1−ϑ , we are interested in three cases : • Exact recovery : h ( ϑ , r , ρ ) > 1 . In this case , the expected Hamming error is o ( 1 ) as p→∞ . It follows that model selection consistency holds . • Almost full recovery : ϑ < h ( ϑ , r , ρ ) < 1 . In this case , the expected Hamming error does not vanish as p→∞ , but it is much smaller than the total number of true relevant variables . Variable selection is still satisfactory ( although model selection consistency does not hold ) . • No recovery : h ( ϑ , r , ρ ) ≤ ϑ . In this case , the expected Hamming error is comparable with or much larger than the total number of true relevant variables . Variable selection fails . We call the two-dimensional space ( ϑ , r ) the phase space . For each fixed ρ , the phase space is divided into three regions : Region of Exact Recovery ( ER ) , which is the subset { ( ϑ , r ) : h ( ϑ , r , ρ ) > 1 } , and Region of Almost Full Recovery ( AFR ) and Region of No Recovery ( NR ) defined similarly . This gives rise to a phase diagram for each method . We denote the curve separating ER region and AFR region by r = U ( ϑ ) and the curve separating AFR region and NR region by r = L ( ϑ ) ; they are called the upper and lower phase curves , respectively . The phase diagram and phase curves are convenient ways to visualize the rates of convergence of the Hamming error . Figure 1 shows the phase curves for six methods ( explicit expressions are in the theorems in Section 3 ) . These phase curves depend on the correlation parameter ρ . Under our model , for each diagonal block ( j , j + 1 ) , it holds that E [ x′jy|β ] = βj + ρβj+1 , where βj , βj+1 ∈ { 0 , τp } . Therefore , a positive ρ boosts the signal at each individual site ( i.e. , E [ x′jy|β ] ≥ βj ) , while a negative ρ leads to potential ‘ signal cancellation ’ ( i.e. , E [ x′jy|β ] ≤ βj ) . This is why the phase curves have different shapes for positive and negative ρ . In Figure 1 , we plot the phase curves for ρ = 0.5 and ρ = −0.4 . For other positive/negative value of ρ , the patterns are similar . Discussion of SCAD . SCAD is a representative of the non-convex penalization methods . There were inspiring works that demonstrated the advantage of using a non-convex penalty ( e.g. , Fan & Peng ( 2004 ) ; Loh & Wainwright ( 2017 ) ) . Our results support their insights from a different angle : The phase curve of SCAD is strictly better than that of Lasso , when ϑ < 0.5 and ρ < 0 . Furthermore , our results illustrate where the advantage of SCAD comes from — compared with Lasso , it handles ‘ signal cancellation ’ better . To see this , we recall that ‘ signal cancellation ’ can only happen for ρ < 0 . Moreover , under our model ( 3 ) , for a diagonal block { j , j + 1 } , we get a pair of signals ( i.e. , βj and βj+1 are both nonzero ) with probability 2p = p 1−2ϑ approximately . Therefore , ‘ signal cancellation ’ becomes problematic only when ϑ < 0.5 and ρ < 0 both hold . This explains why the phase curves of SCAD and Lasso are the same for the other values of ϑ and ρ . We note that in previous studies ( e.g. , Loh & Wainwright ( 2017 ) ) , the advantage of a non-convex penalty in handling ‘ signal cancellation ’ is reflected in the conditions of ( X , β ) required for model selection consistency . Our results support this viewpoint from a different angle . The performance of SCAD can be further improved by adding an entry-wise thresholding on β̂ . We are sure that the phase diagrams of thresholded SCAD will be better than those of SCAD , although we do not have the results in hand ( the analysis is too tedious ) . We are also cautious in comparing SCAD and thresholded Lasso . In our settings , Lasso has no model selection consistency because the signals are too weak ( i.e. , r is not sufficiently large ) . In such a case , thresholded Lasso outperforms SCAD in terms of the Hamming error . Alternatively , there are cases where Lasso has no model selection consistency no matter how large the signal strength is ( Zhao & Yu , 2006 ) . For those cases , it is possible that SCAD is better than thresholded Lasso ( see Wainwright ( 2009 ) for a related study ) . Discussion of Elastic net . The phase curve of Elastic net is worse than that of Lasso . As we will explain in Section 3.1 , Elastic net is a ‘ bridge ’ between Lasso and marginal regression . Since the phase curve of marginal regression is always worse than that of Lasso for the blockwise diagonal design , we do not benefit from using Elastic net in the current setting . We must note that Elastic net was motivated by genetic applications where several correlated variables are competing as predictors , and where it is implicitly assumed that groups of correlated variables tend to be all relevant or all irrelevant ( Zou & Hastie , 2005 ) . This is not captured by our model ( 3 ) . Therefore , our result has no contradiction to the benefit of Elastic net known in the literature . Our result actually supports that the advantage of Elastic net comes from ‘ group ’ appearance of signal variables . We conjecture that for a different signal model ( e.g. , a group prior on β ) , Elastic net has a better phase diagram than Lasso . Discussion of thresholded Lasso . Thresholded Lasso is a representative of improving Lasso by post-processing . There have been inspiring works that demonstrate the advantage of such a postprocessing ( van de Geer et al. , 2011 ; Wang et al. , 2020 ; Weinstein et al. , 2020 ) . Our results support these insights from a different angle . It is surprising ( and very encouraging ) that the improvement by post-Lasso thresholding is so significant . We note that Lasso is a 1-stage method , which solves a single optimization to obtain β̂ . In comparison , thresholded Lasso is a 2-stage method . Lasso has only one algorithm parameter λ , while thresholded Lasso has two algorithm parameters λ and t ( the threshold ) . In Lasso , we control false positives and false negatives via the same algorithm parameter λ , and it is sometimes hard to find a value of λ that simultaneously controls the two types of errors well . In thresholded Lasso , the two types of errors are controlled separately by two algorithm parameters . This explains why thresholded Lasso has such a big improvement over Lasso . It inspires us to modify other 1-stage methods , such as SCAD , by adding a post-processing step of thresholding . For example , we conjecture that thresholded SCAD also has a strictly better phase diagram than that of Lasso , even for a positive ρ . On the other hand , thresholding is not a free lunch . It yields one more tuning parameter to decide in practice . Our theoretical results are based on ideal tuning parameters . How to properly choose these tuning parameters in a data-driven way is an interesting question . Weinstein et al . ( 2020 ) proposed a promising approach , where they used cross-validation to select λ and FDR control by knock-off to select t. We leave it to future work to study the phase diagram with data-driven tuning parameters . Discussion of the two iterative algorithms . We consider two iterative algorithms , forward selection ( ‘ Forward ’ ) and forward backward selection ( FB ) . The FB algorithm we analyze is a simplified version in Huang et al . ( 2016 ) , which has only one backward step ( after all the forward steps are completed ) by thresholding the refitted least-squares solution . Our results show that both methods outperform Lasso , and between these two methods , FB is strictly better than Forward . In the literature , there were very interesting theoretical works showing the advantage of iterative algorithms for variable selection ( Donoho et al. , 2012 ; Zhang , 2011 ) . Our results support their insights from a different angle . We discover that , for a wide range of ρ , FB has the best phase diagram among all six methods . This is a very encouraging result . At the same time , we keep in mind that the performance of an iterative algorithm tends to be more sensitive to the form of the design , due to its sequential nature .

Summary: This paper performs theoretical analyses of several variable selection approaches such as Lasso, Elastic net, SCAD, thresholded Lasso, forward selection, and forward backward selection. While the previous studies focused on the model selection consistency, this paper focused on the impassibleness of the model selection consistency based on Hamming errors. Its theoretical analyses show that SCAD has better performance than Lasso, Elastic net is worse than Lasso, and other approaches are superior to Lasso.