Ising Model Selection Using $\ell_{1}$-Regularized Linear Regression: A Statistical Mechanics Analysis

1 Introduction . The advent of massive data across various scientific disciplines has led to the widespread use of undirected graphical models , also known as Markov random fields ( MRFs ) , as a tool for discovering and visualizing dependencies among covariates in multivariate data [ 1 ] . The Ising model , originally proposed in statistical physics , is one special class of binary MRFs with pairwise potentials and has been widely used in different domains such as image analysis , social networking , gene network analysis [ 2 , 3 , 4 , 5 , 6 , 7 ] . Among various applications , one fundamental problem of interest is called Ising model selection , which refers to recovering the underlying graph structure of the original Ising model from independent , identically distributed ( i.i.d . ) samples . A variety of methods have been proposed [ 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 ] , demonstrating the possibility of successful Ising model selection even when the number of samples is smaller than that of the variables . Notably , it has been demonstrated that for the ` 1-regularized logistic regression ( ` 1-LogR ) [ 10 , 16 ] and interaction screening ( IS ) [ 14 , 15 ] estimators , M = O ( logN ) samples suffice for an Ising model with N spins under certain assumptions , which is consistent with respect to ( w.r.t . ) previously established ∗Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . information-theoretic lower-bound [ 11 ] . Both ` 1-LogR and IS are ` 1-regularized M -estimators [ 19 ] with logistic and IS objective ( ISO ) loss functions , respectively . In this paper , we focus on one simpler linear estimator called ` 1-regularized linear regression ( ` 1- LinR ) and theoretically investigate its typical learning performance using the powerful replica method [ 20 , 21 , 22 , 23 ] from statistical mechanics . The ` 1-LinR estimator , widely known as least absolute shrinkage and selection operator ( LASSO ) [ 24 ] in statistics and machine learning , is considered here mainly for two reasons . On the one hand , it is one representative example of model misspecification since the quadratic loss of ` 1-LinR does not match the true log-conditional-likelihood as ` 1-LogR , nor does it have the interaction screening property as IS . On the other hand , as one of the most popular linear estimator , ` 1-LinR is more computationally efficient than ` 1-LogR and IS , and thus it is of practical importance to investigate its learning performance for Ising model selection . Since it is difficult to obtain results for general graphs , as a first step we consider the random regular ( RR ) graphs GN , d , K0 in the paramagnetic phase [ 23 ] , where GN , d , K0 denotes the ensemble of RR graphs with constant node degree d and uniform coupling strength K0 on the edges . 1.1 Contributions . The main contributions are summarized as follows . First , we obtain an accurate estimate of the typical sample complexity of ` 1-LinR for Ising model selection for typical RR graphs in the paramagnetic phase , which , remarkably , has the same order as ` 1-LogR . Specifically , for a typical RR graph G ∈ GN , d , K0 , using ` 1-LinR with a regularization parameter 0 < λ < tanh ( K0 ) , one can consistently reconstruct the structure with M > c ( λ , K0 ) logNλ2 samples , where c ( λ , K0 ) = 2 ( 1−tanh2 ( K0 ) +dλ2 ) 1+ ( d−1 ) tanh2 ( K0 ) . The accuracy of our typical sample complexity prediction is verified by its excellent agreement with experimental results . To the best of our knowledge , this is the first result that provides an accurate typical sample complexity for Ising model selection . Interestingly , as λ→ tanh ( K0 ) , a lower bound M > 2 logN tanh2 ( K0 ) of the typical sample complexity is obtained , which has the same scaling as the information-theoretic lower bound M > c ′ logN K20 [ 11 ] for some constant c′ at high temperatures ( i.e. , small K0 ) since tanh ( K0 ) = O ( K0 ) as K0 → 0 . Second , we provide a computationally efficient method to precisely predict the typical learning performance of ` 1-LinR in the non-asymptotic case with moderate M , N , such as precision , recall , and residual sum of square ( RSS ) . Such precise non-asymptotic predictions of ` 1-LinR for Ising model selection have been unavailable even for ` 1-LogR [ 10 , 16 ] and IS [ 14 , 15 ] , nor are they the same as previous asymptotic results of ` 1-LinR assuming fixed α ≡M/N [ 25 , 26 , 27 , 28 ] . Moreover , although our theoretical analysis is based on a tree-like structure assumption , experimental results on two dimensional ( 2D ) grid graphs also show a fairly good agreement , indicating that our theoretical result can be a good approximation even for graphs with many loops . Third , while this paper mainly focuses on ` 1-LinR , our method is readily applicable to a wide class of ` 1-regularized M -estimators [ 19 ] , including ` 1-LogR [ 10 ] and IS [ 14 , 15 ] . Thus , an additional technical contribution is providing a generic approach for precisely characterizing the typical learning performances of various ` 1-regularized M -estimators for Ising model selection . Although the replica method from statistical mechanics is non-rigorous , our results are conjectured to be correct , which is supported by their excellent agreement with the experimental results . Additionally , several technical advances we propose in this paper , e.g. , the entropy term computation by averaging over the Haar measure and the modification of EOS to address the finite-size effect , might be of general interest to those who use the replica method as a tool for performance analysis . 1.2 Related works . There has been some earlier works on the analysis of Ising model selection ( also known as the inverse Ising problem ) using the replica method [ 4 , 5 , 6 , 7 , 29 ] from statistical mechanics . For example , in [ 6 ] , the performance of the pseudo-likelihood ( PL ) method [ 30 ] is studied . However , instead of graph structure learning , [ 6 ] focuses on the problem of parameter learning . Then , [ 7 ] extends the analysis to the Ising model with sparse couplings using logistic regression without regularization . The recent work [ 29 ] analyzes the performance of ` 2-regularized linear regression but the techniques invented there are not applicable to ` 1-LinR since the ` 1-norm breaks the rotational invariance property . Regarding the study of ` 1-LinR ( LASSO ) under model misspecification , the past few years have seen a line of research in the field of signal processing with a specific focus on the single-index model [ 31 , 32 , 27 , 33 , 34 ] . These studies are closely related to ours but there are several important differences . First , in our study , the covariates are generated from an Ising model rather than a Gaussian distribution . Second , we focus on model selection consistency of ` 1-LinR while most previous studies consider estimation consistency except [ 33 ] . However , [ 33 ] only considers the classical asymptotic regime while our analysis includes the high-dimensional setting where M N . As far as we have searched , there is no earlier study of ` 1-LinR estimator for Ising model selection , though some are found for Gaussian graphical models [ 35 , 36 ] . One closely related work [ 15 ] states that at high temperatures when the coupling magnitude is approaching zero , both logistic and ISO losses can be approximated by a quadratic loss . However , their claim is only restricted to the very small magnitude near zero while our analysis extends the validity range to the whole paramagnetic phase . Moreover , they evaluate the minimum number of samples necessary for consistently reconstructing “ arbitrary ” Ising models , which , however , seems much larger than that actually needed . By contrast , we provide the first accurate assessment of typical sample complexity for consistently reconstructing typical samples of Ising models defined over the RR graphs . Furthermore , [ 15 ] does not provide precise predictions of the non-asymptotic learning performance as we do . 2 Background and Problem Setup . 2.1 Ising Model . Ising model is one special class of MRFs with pairwise potentials and each variable takes binary values [ 22 , 23 ] , which is one classical model from statistical physics . The joint probability distribution of an Ising model with N variables ( spins ) s = ( si ) N−1 i=0 ∈ { −1 , +1 } N has the form PIsing ( s|J∗ ) = 1 ZIsing ( J ∗ ) exp ∑ i < j J∗ijsisj , ( 1 ) where ZIsing ( J∗ ) = ∑ s exp { ∑ i < j J ∗ ijsisj } is the partition function and J∗ = ( J∗ij ) i , j are the original couplings , respectively . In general , there are also external fields but here they are assumed to be zero for simplicity . The structure of Ising model can be described by an undirected graph G = ( V , E ) , where V = { 0 , 1 , ... , N − 1 } is a collection of vertices at which the spins are assigned , and E = { ( i , j ) |J∗ij 6= 0 } is a collection of undirected edges , i.e. , J∗ij = 0 for all pairs of ( i , j ) /∈ E. For each vertex i ∈ V , its neighborhood is defined as the subset N ( i ) ≡ { j ∈ V| ( i , j ) ∈ E } . 2.2 Neighborhood-based ` 1-regularized linear regression ( ` 1-LinR ) The problem of Ising model selection refers to recovering the graph G ( edge set E ) , given M i.i.d . samples DM = { s ( 1 ) , ... , s ( M ) } from the Ising model . While the maximum likelihood method has nice properties of consistency and asymptotic efficiency , it suffers from high computational complexity . To tackle this difficulty , several local learning algorithms have been proposed , notably the ` 1-LogR estimator [ 10 ] and IS estimator [ 14 ] . Both ` 1-LogR and IS optimize a regularized local cost function ` ( · ) for each spin i.e. , ∀i ∈ V , Ĵ\i = arg min J\i [ 1 M M∑ µ=1 ` ( s ( µ ) i h ( µ ) \i ) + λ ∥∥J\i∥∥1 ] , ( 2 ) where h ( µ ) \i ≡ ∑ j 6=i Jijs ( µ ) j , J\i ≡ ( Jij ) j ( 6=i ) , and ‖·‖1 denotes the ` 1 norm . Specifically , ` ( x ) = log ( 1 + e−2x ) for ` 1-LogR and ` ( x ) = e−x for IS , which correspond to the minus log conditional distribution [ 10 ] and the ISO [ 14 ] , respectively . Consequently , the problem of recovering the edge set E is equivalently reduced to local neighborhood selection , i.e. , recovering the neighborhood set N ( i ) for each vertex i ∈ V. In particular , given the estimates Ĵ\i in ( 2 ) , the neighborhood set of vertex i can be estimated via the nonzero coefficients , i.e. , N̂ ( i ) = { j|Ĵij 6= 0 , j ∈ V \ i } , ∀i ∈ V. ( 3 ) In this paper , we focus on one simple linear estimator , termed as the ` 1-LinR estimator , i.e. , ∀i ∈ V , Ĵ\i = arg min J\i [ 1 2M M∑ µ=1 ( s ( µ ) i − h ( µ ) \i ) 2 + λ ∥∥J\i∥∥1 ] , ( 4 ) which , recalling that s ( µ ) i ∈ { −1 , +1 } , corresponds to a quadratic loss ` ( x ) = 12 ( x− 1 ) 2 in ( 2 ) . The neighorbood set for each vertex i ∈ V is estimated in the same way as ( 3 ) . Interestingly , the quadratic loss used in ( 4 ) implies that the postulated conditional distribution is Gaussian and thus inconsistent with the true one , which is one typical case of model misspecification . Furthermore , compared with nonlinear estimators ` 1-LogR and IS , the ` 1-LinR estimator is more efficient to implement .

Summary: The authors study the Ising model selection problem and compute analytically via the replica method the typical performances of the $\ell_1$-regularized linear regression. They confirm the theoretical result by running a good number of numerical simulations. They demonstrate the sample complexity is such that O(log N) samples are enough to correctly identify the right neighborhood of a generic variable. The theory presented is able to predict the precision and the recall rates also for finite values of the system size and a small number of samples. Moreover, the theory seems to work fine also for graphs with many loops. The methodology of the present work can be extended also to other estimators.