Equivalence of State Equations from Different Methods in High-dimensional Regression

1 INTRODUCTION . Classical statistical methods often failed in the high-dimensional data where the number of features is larger than the number of observed samples . Studies in high dimensional data have attracted lots of attentions in past decades . A set of state equations ( SEs ) were first introduced in approximate message passing ( AMP ) algorithm in ( Donoho et al. , 2009 ) to precisely characterize the meansquare-error ( MSE ) and the phase transition phenomenon for true signal recovery in compressed sensing ( CS ) . Since then , SEs , associated to certain AMP algorithm , have played indispensable role in various high-dimensional problems . For example , ( Donoho et al. , 2011 ) investigated the phase transition phenomenon and the precise MSE of LASSO estimator ; ( Donoho & Montanari , 2016 ) studied the variance of asymptotic distribution of M-estimator ; ( Huang , 2020 ) provided a precise characterization of min-max MSE of l1 penalized robust M-estimator and the corresponding phase transition phenomenon . Though the SEs were first introduced through certain AMP type algorithms , researchers meet them in a variety of models through different methods . For example , the SEs appeared in ( El Karoui et al. , 2013 ) when they performed the leaving-one-out ( LOO ) analysis of M-estimator in high dimensions . They showed that asymptotic normality , asymptotically unbiased property also hold as in the low dimension , nevertheless the variance of asymptotic distribution of M-estimators is higher . ( Sur & Candès , 2019 ) employed the similar idea to analyze the properties of MLE in logistic regression where the SEs were used to show that ( 1 ) asymptotically unbiased property does not hold ; ( 2 ) variance of asymptotic distribution increases ; ( 3 ) likelihood ratio test is not distributed as chi-square . SEs also appeared in another line of researches where Thrompoulidies et al . performed analysis of a family of high dimensional problems through the Convex Gaussian min-max theorem ( CGMT ) . More precisely , ( Thrampoulidis et al. , 2018 ) characterized the MSE precisely for general regularized M-estimator problem in high-dimensions ; ( Salehi et al. , 2019 ) established the correlation and MSE of the resulting estimator of regularized logistic regression ; ( Deng et al. , 2019 ) showed the changing trend of MSE with the growth of features in support vector machine and logistic regression . Lastly , an insightful series of works ( Barbier et al. , 2019 ; Ricci-Tersenghi & Semerjian , 2009 ; Moore , 2014 ; Krzakala et al. , 2016 ; Coja-Oghlan et al. , 2018 ; Mézard & Parisi , 2003 ; Del Ferraro et al. , 2014 ) have utilized the SEs ( named as cavity method in statistical physics ) as a ubiquitous tool when they studied the high dimensional statistical problem through the perspective of statisti- cal physics . Importantly , this tool has exhibited as a powerful weapon in applications of a lot of fields ( Mezard & Montanari , 2009 ; Obuchi & Kabashima , 2016 ; Vuffray , 2014 ; Lesieur et al. , 2015 ; 2016 ) . Though many papers have explicitly written down the corresponding state equations , none of them have shown that these sets of state equations are compatible . To the best of our knowledge , only ( Deng et al. , 2019 ) mentioned there is another set of state equation but without any comparison . Although SEs were proved to be important in high dimensional problems , it is awkward that for one specific problem , the resulting SEs from AMP , CGMT and LOO are different . To be more clear , let us take a look at logistic regression . The SEs derived from CGMT ( 20 ) in ( Deng et al. , 2019 ) are obviously different from the SEs derived from LOO ( 19 ) in ( Sur & Candès , 2019 ) . This is annoying , since the asymptotic performance for a specific high-dimensional problem should be unique no matter which method was used . Therefore , we are interested in the following questions : Are SEs derived from different methods all equivalent in some sense ? If so , from what viewpoint these methods are equivalent and are there more inner equivalence ? Among them , as the most direct , accessible , basic tool , equivalence of SEs is the basis of equivalence of methods and more inner equivalence . Our contributions . We successfully show that for various high dimensional problem , the different sets of SEs derived through different methods are actually equivalent to each other . More precisely , we construct the equivalence between different sets of SEs through explicit parameter transforms for LASSO , M-estimator and logistic regression . These transformations are inspired by the statistical meanings of certain quantities appeared in the SEs . Moreover , we also provide a heuristic explanation on the relation between the different methods : AMP , CGMT and LOO . To the best of our knowledge , this is the first work to clearly clarify the equivalence among SEs derived from different methods and try to establish the equivalence of different methods . Outlines . In section 2 , we show that the SEs for M-estimator from AMP , LOO and CGMT are equivalent to each other . In section 3.1 , we show the equivalence of SEs derived from AMP and CGMT for another example and explain the essential reasons behind this equivalence . In Section 3.2 , we illustrate the similar work regarding the equivalence between CGMT and LOO . Section 4 provides some discussions and future directions . Most proofs are deferred to the appendix . Notations . Let N ( 0 , Id ) , N ( 0 , 1 ) denote the d-dimensional standard Gaussian distribution and 1- dimensional standard Gaussian distribution respectively . For a vector x , we denote ∥x∥p as the lp norm of x . For an integer n we denote [ n ] as { 1 , · · · , n } . We abbreviate independent and identically distributed to i.i.d .. For a function f : R 7→ R , variable x ∈ R and t > 0 , we denote the Moreau envelope associated with f as Mf ( x ; t ) : = min z∈R f ( z ) + 1 2t ( x− z ) 2 ( 1 ) and the proximal operator , which is the solution of this minimization as Proxf ( x ; t ) : = argmin z∈R f ( z ) + 1 2t ( x− z ) 2 . ( 2 ) For multi-dimensional case x = ( x1 , · · · , xd ) T ∈ Rd , Moreau envelope and proximal operator are applied element-wisely : Mf ( x ; t ) : = ( Mf ( xi ; t ) ) ∈ Rd and Proxf ( x ; t ) : = ( Proxf ( xi ; t ) ) ∈ Rd . 2 AN ILLUSTRATIVE EXAMPLE . Suppose that xi i.i.d.∼ N ( 0 , 1dId ) and yi ∈ R satisfying that yi = x T i β ∗ + ϵi , for i ∈ [ n ] ( 3 ) where ϵi are drawn i.i.d . from distribution Pϵ with mean 0 and variance σ2∗ . We assume that the entries β∗i of β ∗ are independently distributed as Π which has finite second moment r2∗ = Eβ∼Πβ2 . Let ρ be a non-negative convex function . We are interested in the the Mean-squared-error ( MSE ) performance limn , p→∞ 1n∥β − β ∗∥2 of the M-estimator : β̂ = argmin β n∑ i=1 ρ ( yi − xTi β ) ( 4 ) when both n and d go to infinity satisfying that limn , d→∞ dn = κ∗ ∈ ( 0 , ∞ ) . This problem first studied by ( El Karoui et al. , 2013 ) where they showed that the MSE of β̂ can be characterized by a set of SEs . More precisely , they proved the following proposition . Proposition 2.1 . ( El Karoui et al. , 2013 ) Given ratio κ∗ < 1 . Consider the following system of nonlinear equations ( SEs ) regarding ( τ1 , γ1 ) : 1− κ∗ = E [ ∂Proxρ ∂x ( W1 + τ1Z1 ; λ1 ) ] κ∗τ 2 1 : = E [ W1 + τ1Z1 − Proxρ ( W1 + τ1Z1 ; λ1 ) ] 2 ( 5 ) where W1 ∼ Pϵ , Z1 ∼ N ( 0 , 1 ) is independent of W1 . If this system of nonlinear equations possesses a unique solution ( τ̄1 , λ̄1 ) , then the τ̄1 is exactly the MSE of β̂ appeared in ( 4 ) . The M -estimator was also studied by ( Donoho & Montanari , 2016 ) where they proved the following proposition . Proposition 2.2 . ( Donoho & Montanari , 2016 ) Given ratio κ∗ < 1 . Consider the following system of nonlinear equations ( SEs ) regarding ( τ2 , γ2 ) : τ22 = 1 κ∗ λ22E [ ∂Mρ ∂x ( W2 + τ2Z2 ; λ2 ) ] 2 κ∗ = λ2E [ ∂2Mρ ∂x2 ( W2 + τ2Z2 ; λ2 ) ] ( 6 ) where W2 ∼ Pϵ , Z2 ∼ N ( 0 , 1 ) is independent of W2 . If this system of nonlinear equations possesses a unique solution ( τ̄2 , λ̄2 ) , then the τ̄2 is exactly the MSE of β̂ appeared in ( 4 ) . Moreover , inspired by the work ( Thrampoulidis et al. , 2014 ) , we employ the CGMT techniques to study the M -estimator and show that the asymptotic MSE can be characterized by the the following SEs . To avoid unnecessary digression , we defer the detailed proof to the appendix A . Proposition 2.3 . Given ratio κ∗ < 1 . Consider the following system of nonlinear equations ( SEs ) regarding ( τ3 , α , µ ) : 0 = α 2 − τ3 √ κ∗ − α µ2 E [ ∂Mρ ∂t ( W3 + τ3Z3 ; α/µ ) ] 0 =− µ √ κ∗ + E [ Z3 ∂Mρ ∂x ( W3 + τ3Z3 ; α/µ ) ] 0 = µ 2 + 1 µ E [ ∂Mρ ∂t ( W3 + τ3Z3 ; α/µ ) ] ( 7 ) where W3 ∼ Pϵ , Z3 ∼ N ( 0 , 1 ) is independent of W3 . If this system of nonlinear equations possesses a unique solution ( τ̄3 , ᾱ , µ̄ ) , then the τ̄3 is exactly the MSE of β̂ appeared in ( 4 ) . On the one hand , these three sets of SEs are different at the first glance . On the other hand , since they are all supposed to describe the MSE of the M -estimators in high dimension , there shall be some relation between these three sets of equations . A striking fact is that we can actually show that all these three set of SEs are equivalent to each other . More precisely , we have the following theorem . Theorem 1 . For M-estimator ( 4 ) , the SEs derived from AMP ( 6 ) , LOO ( 5 ) and CGMT ( 7 ) are equivalent . Specifically , ( 6 ) can be converted into the same form as ( 5 ) after the following parameter transformations : τ1 = τ2 , λ1 = λ2 . ( 8 ) ( 6 ) can be converted into the same form as ( 7 ) after the following parameter transformations : τ1 = τ3 , λ1 = α µ . ( 9 ) The equivalence of these three sets of SEs seems straightforward , however , it suggests us that all the three procedures : AMP , CGMT and LOO might be deeply entangled in some sense . This will be investigated in this manuscript . The proof of this theorem is deferred to the appendix B .

Summary: This paper compares the so-called "State Evolution" equations from 3 different derivations for the MSE of  high dimensional M-estimators in the proportional asymptotic regime (number of samples and features go to infinity at a fixed rate). There are currently 3 well-known frameworks for analysis in this regime: 1. Approximate Message Passing (AMP) 2. Convex Gaussian Min-Max Theorem (CGMT) 3. Leave One Out (LOO) The paper shows that there lies an equivalence between the results derived by the 3 methods via a parameter transformation. Two important special cases of LASSO and Binary Logistic Regression are considered and the results are clearly stated.