Equivalence of State Equations from Different Methods in High-dimensional Regression

1 INTRODUCTION . Classical statistical methods often failed in the high-dimensional data where the number of features is larger than the number of observed samples . Studies in high dimensional data have attracted lots of attentions in past decades . A set of state equations ( SEs ) were first introduced in approximate message passing ( AMP ) algorithm in ( Donoho et al. , 2009 ) to precisely characterize the meansquare-error ( MSE ) and the phase transition phenomenon for true signal recovery in compressed sensing ( CS ) . Since then , SEs , associated to certain AMP algorithm , have played indispensable role in various high-dimensional problems . For example , ( Donoho et al. , 2011 ) investigated the phase transition phenomenon and the precise MSE of LASSO estimator ; ( Donoho & Montanari , 2016 ) studied the variance of asymptotic distribution of M-estimator ; ( Huang , 2020 ) provided a precise characterization of min-max MSE of l1 penalized robust M-estimator and the corresponding phase transition phenomenon . Though the SEs were first introduced through certain AMP type algorithms , researchers meet them in a variety of models through different methods . For example , the SEs appeared in ( El Karoui et al. , 2013 ) when they performed the leaving-one-out ( LOO ) analysis of M-estimator in high dimensions . They showed that asymptotic normality , asymptotically unbiased property also hold as in the low dimension , nevertheless the variance of asymptotic distribution of M-estimators is higher . ( Sur & Candès , 2019 ) employed the similar idea to analyze the properties of MLE in logistic regression where the SEs were used to show that ( 1 ) asymptotically unbiased property does not hold ; ( 2 ) variance of asymptotic distribution increases ; ( 3 ) likelihood ratio test is not distributed as chi-square . SEs also appeared in another line of researches where Thrompoulidies et al . performed analysis of a family of high dimensional problems through the Convex Gaussian min-max theorem ( CGMT ) . More precisely , ( Thrampoulidis et al. , 2018 ) characterized the MSE precisely for general regularized M-estimator problem in high-dimensions ; ( Salehi et al. , 2019 ) established the correlation and MSE of the resulting estimator of regularized logistic regression ; ( Deng et al. , 2019 ) showed the changing trend of MSE with the growth of features in support vector machine and logistic regression . Lastly , an insightful series of works ( Barbier et al. , 2019 ; Ricci-Tersenghi & Semerjian , 2009 ; Moore , 2014 ; Krzakala et al. , 2016 ; Coja-Oghlan et al. , 2018 ; Mézard & Parisi , 2003 ; Del Ferraro et al. , 2014 ) have utilized the SEs ( named as cavity method in statistical physics ) as a ubiquitous tool when they studied the high dimensional statistical problem through the perspective of statisti- cal physics . Importantly , this tool has exhibited as a powerful weapon in applications of a lot of fields ( Mezard & Montanari , 2009 ; Obuchi & Kabashima , 2016 ; Vuffray , 2014 ; Lesieur et al. , 2015 ; 2016 ) . Though many papers have explicitly written down the corresponding state equations , none of them have shown that these sets of state equations are compatible . To the best of our knowledge , only ( Deng et al. , 2019 ) mentioned there is another set of state equation but without any comparison . Although SEs were proved to be important in high dimensional problems , it is awkward that for one specific problem , the resulting SEs from AMP , CGMT and LOO are different . To be more clear , let us take a look at logistic regression . The SEs derived from CGMT ( 20 ) in ( Deng et al. , 2019 ) are obviously different from the SEs derived from LOO ( 19 ) in ( Sur & Candès , 2019 ) . This is annoying , since the asymptotic performance for a specific high-dimensional problem should be unique no matter which method was used . Therefore , we are interested in the following questions : Are SEs derived from different methods all equivalent in some sense ? If so , from what viewpoint these methods are equivalent and are there more inner equivalence ? Among them , as the most direct , accessible , basic tool , equivalence of SEs is the basis of equivalence of methods and more inner equivalence . Our contributions . We successfully show that for various high dimensional problem , the different sets of SEs derived through different methods are actually equivalent to each other . More precisely , we construct the equivalence between different sets of SEs through explicit parameter transforms for LASSO , M-estimator and logistic regression . These transformations are inspired by the statistical meanings of certain quantities appeared in the SEs . Moreover , we also provide a heuristic explanation on the relation between the different methods : AMP , CGMT and LOO . To the best of our knowledge , this is the first work to clearly clarify the equivalence among SEs derived from different methods and try to establish the equivalence of different methods . Outlines . In section 2 , we show that the SEs for M-estimator from AMP , LOO and CGMT are equivalent to each other . In section 3.1 , we show the equivalence of SEs derived from AMP and CGMT for another example and explain the essential reasons behind this equivalence . In Section 3.2 , we illustrate the similar work regarding the equivalence between CGMT and LOO . Section 4 provides some discussions and future directions . Most proofs are deferred to the appendix . Notations . Let N ( 0 , Id ) , N ( 0 , 1 ) denote the d-dimensional standard Gaussian distribution and 1- dimensional standard Gaussian distribution respectively . For a vector x , we denote ∥x∥p as the lp norm of x . For an integer n we denote [ n ] as { 1 , · · · , n } . We abbreviate independent and identically distributed to i.i.d .. For a function f : R 7→ R , variable x ∈ R and t > 0 , we denote the Moreau envelope associated with f as Mf ( x ; t ) : = min z∈R f ( z ) + 1 2t ( x− z ) 2 ( 1 ) and the proximal operator , which is the solution of this minimization as Proxf ( x ; t ) : = argmin z∈R f ( z ) + 1 2t ( x− z ) 2 . ( 2 ) For multi-dimensional case x = ( x1 , · · · , xd ) T ∈ Rd , Moreau envelope and proximal operator are applied element-wisely : Mf ( x ; t ) : = ( Mf ( xi ; t ) ) ∈ Rd and Proxf ( x ; t ) : = ( Proxf ( xi ; t ) ) ∈ Rd . 2 AN ILLUSTRATIVE EXAMPLE . Suppose that xi i.i.d.∼ N ( 0 , 1dId ) and yi ∈ R satisfying that yi = x T i β ∗ + ϵi , for i ∈ [ n ] ( 3 ) where ϵi are drawn i.i.d . from distribution Pϵ with mean 0 and variance σ2∗ . We assume that the entries β∗i of β ∗ are independently distributed as Π which has finite second moment r2∗ = Eβ∼Πβ2 . Let ρ be a non-negative convex function . We are interested in the the Mean-squared-error ( MSE ) performance limn , p→∞ 1n∥β − β ∗∥2 of the M-estimator : β̂ = argmin β n∑ i=1 ρ ( yi − xTi β ) ( 4 ) when both n and d go to infinity satisfying that limn , d→∞ dn = κ∗ ∈ ( 0 , ∞ ) . This problem first studied by ( El Karoui et al. , 2013 ) where they showed that the MSE of β̂ can be characterized by a set of SEs . More precisely , they proved the following proposition . Proposition 2.1 . ( El Karoui et al. , 2013 ) Given ratio κ∗ < 1 . Consider the following system of nonlinear equations ( SEs ) regarding ( τ1 , γ1 ) : 1− κ∗ = E [ ∂Proxρ ∂x ( W1 + τ1Z1 ; λ1 ) ] κ∗τ 2 1 : = E [ W1 + τ1Z1 − Proxρ ( W1 + τ1Z1 ; λ1 ) ] 2 ( 5 ) where W1 ∼ Pϵ , Z1 ∼ N ( 0 , 1 ) is independent of W1 . If this system of nonlinear equations possesses a unique solution ( τ̄1 , λ̄1 ) , then the τ̄1 is exactly the MSE of β̂ appeared in ( 4 ) . The M -estimator was also studied by ( Donoho & Montanari , 2016 ) where they proved the following proposition . Proposition 2.2 . ( Donoho & Montanari , 2016 ) Given ratio κ∗ < 1 . Consider the following system of nonlinear equations ( SEs ) regarding ( τ2 , γ2 ) : τ22 = 1 κ∗ λ22E [ ∂Mρ ∂x ( W2 + τ2Z2 ; λ2 ) ] 2 κ∗ = λ2E [ ∂2Mρ ∂x2 ( W2 + τ2Z2 ; λ2 ) ] ( 6 ) where W2 ∼ Pϵ , Z2 ∼ N ( 0 , 1 ) is independent of W2 . If this system of nonlinear equations possesses a unique solution ( τ̄2 , λ̄2 ) , then the τ̄2 is exactly the MSE of β̂ appeared in ( 4 ) . Moreover , inspired by the work ( Thrampoulidis et al. , 2014 ) , we employ the CGMT techniques to study the M -estimator and show that the asymptotic MSE can be characterized by the the following SEs . To avoid unnecessary digression , we defer the detailed proof to the appendix A . Proposition 2.3 . Given ratio κ∗ < 1 . Consider the following system of nonlinear equations ( SEs ) regarding ( τ3 , α , µ ) : 0 = α 2 − τ3 √ κ∗ − α µ2 E [ ∂Mρ ∂t ( W3 + τ3Z3 ; α/µ ) ] 0 =− µ √ κ∗ + E [ Z3 ∂Mρ ∂x ( W3 + τ3Z3 ; α/µ ) ] 0 = µ 2 + 1 µ E [ ∂Mρ ∂t ( W3 + τ3Z3 ; α/µ ) ] ( 7 ) where W3 ∼ Pϵ , Z3 ∼ N ( 0 , 1 ) is independent of W3 . If this system of nonlinear equations possesses a unique solution ( τ̄3 , ᾱ , µ̄ ) , then the τ̄3 is exactly the MSE of β̂ appeared in ( 4 ) . On the one hand , these three sets of SEs are different at the first glance . On the other hand , since they are all supposed to describe the MSE of the M -estimators in high dimension , there shall be some relation between these three sets of equations . A striking fact is that we can actually show that all these three set of SEs are equivalent to each other . More precisely , we have the following theorem . Theorem 1 . For M-estimator ( 4 ) , the SEs derived from AMP ( 6 ) , LOO ( 5 ) and CGMT ( 7 ) are equivalent . Specifically , ( 6 ) can be converted into the same form as ( 5 ) after the following parameter transformations : τ1 = τ2 , λ1 = λ2 . ( 8 ) ( 6 ) can be converted into the same form as ( 7 ) after the following parameter transformations : τ1 = τ3 , λ1 = α µ . ( 9 ) The equivalence of these three sets of SEs seems straightforward , however , it suggests us that all the three procedures : AMP , CGMT and LOO might be deeply entangled in some sense . This will be investigated in this manuscript . The proof of this theorem is deferred to the appendix B .

Summary: This paper considers the state evolution equations obtained via different methods (AMP, LOO and CGMT), and it shows that their fixed points are the same for some high-dimensional inference problems (M-estimation, LASSO, and logistic regression).  Most of these state evolutions were obtained in recent work (which is properly cited here). An exception is Proposition 3.2, which gives the state evolution for CGMT in the case of LASSO. However, the proof of this result closely follows the existing literature. Consequently, the main novelty of this contribution is in showing that the fixed points (when unique) are the same for different methods.