Ising Model Selection Using $\ell_{1}$-Regularized Linear Regression: A Statistical Mechanics Analysis

1 Introduction . The advent of massive data across various scientific disciplines has led to the widespread use of undirected graphical models , also known as Markov random fields ( MRFs ) , as a tool for discovering and visualizing dependencies among covariates in multivariate data [ 1 ] . The Ising model , originally proposed in statistical physics , is one special class of binary MRFs with pairwise potentials and has been widely used in different domains such as image analysis , social networking , gene network analysis [ 2 , 3 , 4 , 5 , 6 , 7 ] . Among various applications , one fundamental problem of interest is called Ising model selection , which refers to recovering the underlying graph structure of the original Ising model from independent , identically distributed ( i.i.d . ) samples . A variety of methods have been proposed [ 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 ] , demonstrating the possibility of successful Ising model selection even when the number of samples is smaller than that of the variables . Notably , it has been demonstrated that for the ` 1-regularized logistic regression ( ` 1-LogR ) [ 10 , 16 ] and interaction screening ( IS ) [ 14 , 15 ] estimators , M = O ( logN ) samples suffice for an Ising model with N spins under certain assumptions , which is consistent with respect to ( w.r.t . ) previously established ∗Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . information-theoretic lower-bound [ 11 ] . Both ` 1-LogR and IS are ` 1-regularized M -estimators [ 19 ] with logistic and IS objective ( ISO ) loss functions , respectively . In this paper , we focus on one simpler linear estimator called ` 1-regularized linear regression ( ` 1- LinR ) and theoretically investigate its typical learning performance using the powerful replica method [ 20 , 21 , 22 , 23 ] from statistical mechanics . The ` 1-LinR estimator , widely known as least absolute shrinkage and selection operator ( LASSO ) [ 24 ] in statistics and machine learning , is considered here mainly for two reasons . On the one hand , it is one representative example of model misspecification since the quadratic loss of ` 1-LinR does not match the true log-conditional-likelihood as ` 1-LogR , nor does it have the interaction screening property as IS . On the other hand , as one of the most popular linear estimator , ` 1-LinR is more computationally efficient than ` 1-LogR and IS , and thus it is of practical importance to investigate its learning performance for Ising model selection . Since it is difficult to obtain results for general graphs , as a first step we consider the random regular ( RR ) graphs GN , d , K0 in the paramagnetic phase [ 23 ] , where GN , d , K0 denotes the ensemble of RR graphs with constant node degree d and uniform coupling strength K0 on the edges . 1.1 Contributions . The main contributions are summarized as follows . First , we obtain an accurate estimate of the typical sample complexity of ` 1-LinR for Ising model selection for typical RR graphs in the paramagnetic phase , which , remarkably , has the same order as ` 1-LogR . Specifically , for a typical RR graph G ∈ GN , d , K0 , using ` 1-LinR with a regularization parameter 0 < λ < tanh ( K0 ) , one can consistently reconstruct the structure with M > c ( λ , K0 ) logNλ2 samples , where c ( λ , K0 ) = 2 ( 1−tanh2 ( K0 ) +dλ2 ) 1+ ( d−1 ) tanh2 ( K0 ) . The accuracy of our typical sample complexity prediction is verified by its excellent agreement with experimental results . To the best of our knowledge , this is the first result that provides an accurate typical sample complexity for Ising model selection . Interestingly , as λ→ tanh ( K0 ) , a lower bound M > 2 logN tanh2 ( K0 ) of the typical sample complexity is obtained , which has the same scaling as the information-theoretic lower bound M > c ′ logN K20 [ 11 ] for some constant c′ at high temperatures ( i.e. , small K0 ) since tanh ( K0 ) = O ( K0 ) as K0 → 0 . Second , we provide a computationally efficient method to precisely predict the typical learning performance of ` 1-LinR in the non-asymptotic case with moderate M , N , such as precision , recall , and residual sum of square ( RSS ) . Such precise non-asymptotic predictions of ` 1-LinR for Ising model selection have been unavailable even for ` 1-LogR [ 10 , 16 ] and IS [ 14 , 15 ] , nor are they the same as previous asymptotic results of ` 1-LinR assuming fixed α ≡M/N [ 25 , 26 , 27 , 28 ] . Moreover , although our theoretical analysis is based on a tree-like structure assumption , experimental results on two dimensional ( 2D ) grid graphs also show a fairly good agreement , indicating that our theoretical result can be a good approximation even for graphs with many loops . Third , while this paper mainly focuses on ` 1-LinR , our method is readily applicable to a wide class of ` 1-regularized M -estimators [ 19 ] , including ` 1-LogR [ 10 ] and IS [ 14 , 15 ] . Thus , an additional technical contribution is providing a generic approach for precisely characterizing the typical learning performances of various ` 1-regularized M -estimators for Ising model selection . Although the replica method from statistical mechanics is non-rigorous , our results are conjectured to be correct , which is supported by their excellent agreement with the experimental results . Additionally , several technical advances we propose in this paper , e.g. , the entropy term computation by averaging over the Haar measure and the modification of EOS to address the finite-size effect , might be of general interest to those who use the replica method as a tool for performance analysis . 1.2 Related works . There has been some earlier works on the analysis of Ising model selection ( also known as the inverse Ising problem ) using the replica method [ 4 , 5 , 6 , 7 , 29 ] from statistical mechanics . For example , in [ 6 ] , the performance of the pseudo-likelihood ( PL ) method [ 30 ] is studied . However , instead of graph structure learning , [ 6 ] focuses on the problem of parameter learning . Then , [ 7 ] extends the analysis to the Ising model with sparse couplings using logistic regression without regularization . The recent work [ 29 ] analyzes the performance of ` 2-regularized linear regression but the techniques invented there are not applicable to ` 1-LinR since the ` 1-norm breaks the rotational invariance property . Regarding the study of ` 1-LinR ( LASSO ) under model misspecification , the past few years have seen a line of research in the field of signal processing with a specific focus on the single-index model [ 31 , 32 , 27 , 33 , 34 ] . These studies are closely related to ours but there are several important differences . First , in our study , the covariates are generated from an Ising model rather than a Gaussian distribution . Second , we focus on model selection consistency of ` 1-LinR while most previous studies consider estimation consistency except [ 33 ] . However , [ 33 ] only considers the classical asymptotic regime while our analysis includes the high-dimensional setting where M N . As far as we have searched , there is no earlier study of ` 1-LinR estimator for Ising model selection , though some are found for Gaussian graphical models [ 35 , 36 ] . One closely related work [ 15 ] states that at high temperatures when the coupling magnitude is approaching zero , both logistic and ISO losses can be approximated by a quadratic loss . However , their claim is only restricted to the very small magnitude near zero while our analysis extends the validity range to the whole paramagnetic phase . Moreover , they evaluate the minimum number of samples necessary for consistently reconstructing “ arbitrary ” Ising models , which , however , seems much larger than that actually needed . By contrast , we provide the first accurate assessment of typical sample complexity for consistently reconstructing typical samples of Ising models defined over the RR graphs . Furthermore , [ 15 ] does not provide precise predictions of the non-asymptotic learning performance as we do . 2 Background and Problem Setup . 2.1 Ising Model . Ising model is one special class of MRFs with pairwise potentials and each variable takes binary values [ 22 , 23 ] , which is one classical model from statistical physics . The joint probability distribution of an Ising model with N variables ( spins ) s = ( si ) N−1 i=0 ∈ { −1 , +1 } N has the form PIsing ( s|J∗ ) = 1 ZIsing ( J ∗ ) exp ∑ i < j J∗ijsisj , ( 1 ) where ZIsing ( J∗ ) = ∑ s exp { ∑ i < j J ∗ ijsisj } is the partition function and J∗ = ( J∗ij ) i , j are the original couplings , respectively . In general , there are also external fields but here they are assumed to be zero for simplicity . The structure of Ising model can be described by an undirected graph G = ( V , E ) , where V = { 0 , 1 , ... , N − 1 } is a collection of vertices at which the spins are assigned , and E = { ( i , j ) |J∗ij 6= 0 } is a collection of undirected edges , i.e. , J∗ij = 0 for all pairs of ( i , j ) /∈ E. For each vertex i ∈ V , its neighborhood is defined as the subset N ( i ) ≡ { j ∈ V| ( i , j ) ∈ E } . 2.2 Neighborhood-based ` 1-regularized linear regression ( ` 1-LinR ) The problem of Ising model selection refers to recovering the graph G ( edge set E ) , given M i.i.d . samples DM = { s ( 1 ) , ... , s ( M ) } from the Ising model . While the maximum likelihood method has nice properties of consistency and asymptotic efficiency , it suffers from high computational complexity . To tackle this difficulty , several local learning algorithms have been proposed , notably the ` 1-LogR estimator [ 10 ] and IS estimator [ 14 ] . Both ` 1-LogR and IS optimize a regularized local cost function ` ( · ) for each spin i.e. , ∀i ∈ V , Ĵ\i = arg min J\i [ 1 M M∑ µ=1 ` ( s ( µ ) i h ( µ ) \i ) + λ ∥∥J\i∥∥1 ] , ( 2 ) where h ( µ ) \i ≡ ∑ j 6=i Jijs ( µ ) j , J\i ≡ ( Jij ) j ( 6=i ) , and ‖·‖1 denotes the ` 1 norm . Specifically , ` ( x ) = log ( 1 + e−2x ) for ` 1-LogR and ` ( x ) = e−x for IS , which correspond to the minus log conditional distribution [ 10 ] and the ISO [ 14 ] , respectively . Consequently , the problem of recovering the edge set E is equivalently reduced to local neighborhood selection , i.e. , recovering the neighborhood set N ( i ) for each vertex i ∈ V. In particular , given the estimates Ĵ\i in ( 2 ) , the neighborhood set of vertex i can be estimated via the nonzero coefficients , i.e. , N̂ ( i ) = { j|Ĵij 6= 0 , j ∈ V \ i } , ∀i ∈ V. ( 3 ) In this paper , we focus on one simple linear estimator , termed as the ` 1-LinR estimator , i.e. , ∀i ∈ V , Ĵ\i = arg min J\i [ 1 2M M∑ µ=1 ( s ( µ ) i − h ( µ ) \i ) 2 + λ ∥∥J\i∥∥1 ] , ( 4 ) which , recalling that s ( µ ) i ∈ { −1 , +1 } , corresponds to a quadratic loss ` ( x ) = 12 ( x− 1 ) 2 in ( 2 ) . The neighorbood set for each vertex i ∈ V is estimated in the same way as ( 3 ) . Interestingly , the quadratic loss used in ( 4 ) implies that the postulated conditional distribution is Gaussian and thus inconsistent with the true one , which is one typical case of model misspecification . Furthermore , compared with nonlinear estimators ` 1-LogR and IS , the ` 1-LinR estimator is more efficient to implement .

Summary: This paper studies the problem of estimating the edge couplings in a Markov random field using ell-one regularized least-squares estimation. It is assumed throughout that all nodes have the same degree and the nonzero coupling have the same value. Using the replica method, the authors derive a set of couple equations of state (EOS), which depend on the underlying parameters of the model (e.g., the degree and coupling strength),  the ratio between the number of samples and the number of nodes, and the spectral density of the covariance of the resulting MRF. By analyzing the solutions to the equations, the authors provide numerical predictions of the reconstruction performance (e.g, precision and recall) in the limit of high dimension. Further analysis of these expressions gives a sharp predicition for the number of samples needed for exact recovery as a function of the the degree and the coupling strength. Additionally, the authors suggest a modification of the EOS that uses the observed data to produce more accurate predictions. Because the analysis depends on the replica method the results are not rigorous. The accuracy of the theoretical analysis is supported by numerical simulations.