SLOE: A Faster Method for Statistical Inference in High-Dimensional Logistic Regression

1 Introduction . Logistic regression is a workhorse in statistics , machine learning , data science , and many applied fields . It is a generalized linear model that models a binary scalar outcome Y ∈ { 0 , 1 } conditional on observed features X ∈ Rd via E [ Y | X = x ] = g ( β > x ) , with g ( t ) : = 11+exp ( −t ) , ( 1.1 ) with the coefficients β fit using observed data . Logistic regression is popular as a scientific tool because the model is often accurate , and comes with well-established statistical inference procedures for quantifying uncertainty about the parameters β and predictions g ( β > x ) at test inputs x . For example , most statistical software packages not only produce predictions from the model , but also summaries such as confidence intervals ( CIs ) and p-values that enable practitioners to understand the strength of evidence for the prediction in a quantitative way . These widely adopted estimation and ∗Google Research , Brain Team †Google Health 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . statistical inference routines are based on approximations from large-sample asymptotic theory of the maximum likelihood estimator ( MLE ) β̂ . These approximations come from the limiting distribution of β̂ as the sample size n tends toward infinity , but the number of covariates d remains fixed . Unfortunately , these standard approximations perform poorly when the number of covariates d is of a similar order to the number of samples n , even when the sample size is large [ Fan et al. , 2019 ] . Recently , Sur and Candès [ 2019 ] showed that , in this setting , the behavior of β̂ in finite samples is better approximated by its limiting distribution in another regime , where the aspect ratio κ : = d/n > 0 is held fixed as both n and d grow to infinity . They show that the estimated coefficients β̂ ( and therefore , the predictions ) have systematically inflated magnitude and larger variance than the standard approximation predicts . The precise characterization of the limiting distribution of β̂ in Sur and Candès [ 2019 ] and Zhao et al . [ 2020 ] justify a new approximation , which , in principle , facilitates debiasing estimates and constructing CIs and p-values for parameters and test predictions , alike . The goal of this paper is to make the application of these groundbreaking results practical . A major barrier to adoption currently is that the calculation of these bias and variance corrections requires knowing the signal strength γ2 : = Var ( β > X ) , which is challenging to estimate because it is a function of the unknown parameter vector β. Sur and Candès [ 2019 ] proposed a heuristic called ProbeFrontier to estimate this quantity , but this approach is computationally expensive , conceptually complex , and hard to analyze statistically . We propose a simpler estimation procedure . Our approach reparameterizes the problem in terms of the corrupted signal strength parameter η2 : = limn→∞Var ( β̂ > X ) that includes the noise in the estimate β̂ . This is more straightforward ( though non-trivial ) to estimate . We propose the Signal Strength Leave-One-Out Estimator ( SLOE ) , which consistently estimates η2 , and show that using this for inference yields more accurate CIs in finite samples . Importantly , SLOE takes orders of magnitude less computation than ProbeFrontier , having similar runtime to the standard logistic regression fitting routine . 2 Preliminaries . In this section , we revisit some fundamentals of statistical inference with logistic regression , review recent advances in high dimensional settings by Sur and Candès [ 2019 ] and Zhao et al . [ 2020 ] , and discuss the implications of this paradigm shift in terms of better characterizing practice . 2.1 Logistic Regression and Statistical Inference . Estimates of β : = ( β1 , · · · , βd ) in the logistic regression model are usually obtained through maximum likelihood estimation , by maximizing the empirical log-likelihood β̂ : = argmax β∈Rd 1 n n∑ i=1 Yi log ( g ( β > Xi ) ) + ( 1− Yi ) log ( 1− g ( β > Xi ) ) . ( 2.1 ) The log likelihood is concave , and has a unique maximizer whenever the outcomes are not linearly separable in the covariates . We will use logistic regression synonymously with maximum likelihood estimation in the logistic regression model , and call β̂ the MLE . The large-sample asymptotic statistical theory [ Lehmann and Romano , 2005 ] taught in nearly every university Statistics program characterizes the behavior of the estimated coefficients and predictions in the limit as n → ∞ while holding the number of features d fixed . Under this theory , estimates converge to their true value , β̂ p→ β , and the estimation error β̂ − β and prediction error g ( β̂ > x ) − g ( β > x ) will be small to observe , unless amplified by a factor of √ n , in which case √ n ( β̂ − β ) d N ( 0 , I−1β ) , where Iβ : = E [ DβXX > ] is the Fisher information matrix , with Dβ : = g ( β > X ) ( 1− g ( β > X ) ) . Of course , when analyzing real data , data scientists only have access to a finite number of samples , and so this theory serves as an approximation characterization of the behavior expected in practice . If the approximation is good , one can make inferences about the underlying data generating distribution . For example , for δ ∈ ( 0 , 1 ) , we can construct confidence intervals ( CIs ) that contain the true parameters with probability 1−δ . Propagating the uncertainty to predictions gives CIs for the outcome probabilities that can help contextualize machine learning predictions for users such as scientists or clinicians [ Kompa et al. , 2021 ] . The large sample approximation is a good one when the number of predictors is small relative to the number of observations . However , in modern scientific studies and data-driven technology , while many samples may be collected , many features of each sample are also collected . In such settings , the number of features d may be significant compared to the sample size n ; then , the large sample characterization can be quite misleading . We can visualize the extent to which this breakdown in assumptions leads to bias . Figure 1 shows the estimated coefficients and calibration curve for a logistic regression model fit on simulated data with n = 4000 and p = 800 . Despite n being reasonably large , these illustrate significant departures from the behavior predicted by the large sample asymptotics . The magnitude of the estimated coefficients is systematically overestimated and they are noisy . Additionally , the model is overconfident in its predictions . Despite this , the classification accuracy of the resulting model on the in-distribution test data is 0.70 , which is close to the optimal accuracy of 0.77 . Recently , Sur and Candès [ 2019 ] , Mai et al . [ 2019 ] , and Zhao et al . [ 2020 ] showed that the behavior of the MLE in high-dimensional settings is better characterized by a different asymptotic approximation . In this approximation , the aspect ratio κ = d/n , instead of the number of predictors d , remains fixed as n grows to infinity . Formally , they study a sequence of problems in which n , d → ∞ ; d/n→ κ > 0 , γ2 = Var ( β > X ) is fixed as d grows , 3 and the MLE exists asymptotically for ( κ , γ ) . Sur and Candès [ 2019 ] , Zhao et al . [ 2020 ] , and our work assume Xi is marginally Gaussian , and are complementary to the mixture-of-Gaussians generative model studied by Mai et al . [ 2019 ] . The semantics of this asymptotic regime are a bit more complicated than the classical one ; we can no longer think of the argument in terms of a fixed problem with increasing data . However , The characterization from this asymptotic theory better reflects the finite-sample behavior of the MLE β̂ when the data has a moderate aspect ratio κ . For simplicity , we state the key result from this work in the case where the covariates are drawn from a spherical Gaussian , i.e. , X ∼ N ( 0 , Id ) , but note that these results are generalized in Zhao et al . [ 2020 ] to arbitrary covariance structures.4 So far , proof techniques only work for normally distributed features , however empirical results ( Fig . 4b and previous work ) the work for sub-Gaussian features . In this regime , the distribution of the MLE β̂ converges to a normal distribution asymptotically 3This prevents the expectations g ( β > X ) from converging to 0 or 1 as n grows . To satisfy this , the effect of each covariate shrinks as n grows . It is complementary to sparsity models , being more realistic for many problems where predictive performance comes from pooling many weak signals . 4Simulations suggest that our methods generalize to models including an intercept term when using the intercept correction conjectured in Zhao et al . [ 2020 ] . centered around the inflated parameter αβ , for some α > 1 . In particular , for any coefficient βj , √ n ( β̂j − αβj ) d N ( 0 , σ2 ? ) , ( 2.2 ) and for the predicted logit of an input5 x ∈ Rd with asymptotically finite norm and √ nx > β = O ( 1 ) , √ n‖x‖−12 ( β̂ > x− αβ > x ) d N ( 0 , σ2 ? ) , ( 2.3 ) for constants α and σ ? , to be discussed later , that are only functions of κ and γ . The general convergence result when X has an arbitrary covariance is stated in Theorem 3.1 of Zhao et al . [ 2020 ] . If α and σ ? were known , this theory provides corrected approximate CIs and p-values . For example , taking Φ−1 to be the quantile function of a standard normal distribution : ( 1− δ ) CI for βj : ( β̂j α ± Φ−1 ( 1− δ/2 ) σ ? αn1/2 ) ( 2.4 ) p-value under βj = 0 : p = Φ−1 ( Z ) where Z = β̂j/ σ ? αn1/2 . ( 2.5 ) Figure 2 shows how the p-values obtained based on these adjustments have the expected behavior ( i.e. , they are uniform when the null is true ) , whereas those from the standard approximation do not . 2.2 Calculating Dimensionality Corrections . Sur and Candès [ 2019 ] show that the constants α and σ ? are determined by the solution to a system of three non-linear equations , which depend implicitly on the aspect ratio κ and the signal strength γ2 : = Var ( β > X ) : κ2σ2 ? = E [ 2g ( Q1 ) ( λg ( proxλG ( Q2 ) ) ) 2 ] , 0 = E [ g ( Q1 ) Q1λg ( proxλG ( Q2 ) ) ] , 1− κ = E [ 2g ( Q1 ) 1+λg′ ( proxλG ( Q2 ) ) ] , ( 2.6 ) where proxλG ( s ) = argmint λG ( t ) + 1 2 ( s− t ) 2 , for G the anti-derivative of g , and ( Q1 Q2 ) ∼ N ( 0 , [ γ2 −αγ2 −αγ2 α2γ2 + κσ2 ? ] ) . ( 2.7 ) Here , Q1 is a random variable with the same distribution ( asymptotically ) as the population logits β > X , and Q2 is a random variable with the same distribution ( asymptotically ) as the logits with the biased MLE plugged in β̂ > X . The auxiliary parameter λ corresponds to the limiting average eigenvalue of the inverse Hessian at the MLE , ( 1/n ∑n i=1Xig ′ ( β̂ > Xi ) X > i ) −1 , which is useful for making comparisons to asymptotic approximations from standard theory . Sur and Candès [ 2019 ] note that this system has a unique solution so long as ( κ , γ ) take values at which the MLE exists asymptotically . Once set up , these equations are straightforward to solve with numerical methods . The key quantity needed to set up these equations is the signal strength parameter γ , which is not directly observed . Thus , to correct for high dimensions in practice requires that γ be estimated . This is difficult , as γ is itself a function of the unknown parameter β , and is the main focus of this paper . Sur and Candès [ 2019 ] suggest a heuristic method , called ProbeFrontier , for estimating γ in practice . The idea is to search for the ( sub ) sample size n′ < n such that the observations in the subsample are linearly separable . Sub-sampling the data changes the aspect ratio κ without changing the signal strength γ . For a fixed signal strength γ , there is a sharp aspect ratio cutoff κ ? ( γ ) ( the “ frontier ” ) above which the data is separable with high probability . Based on these ideas , ProbeFrontier then inverts κ ? ( γ ) to estimate γ from the empirical aspect ratio p/n′ . This requires subsampling the data repeatedly at various candidate aspect ratios , and for each subsample , checking whether the data are linearly separable using a linear program . Repeatedly checking the separability is computationally expensive , and the statistical behavior near the frontier makes analysis tricky . 5Note here that x is a sequence in n. Up to a 1/ √ d scale factor , these constraints are satisfied for a fixed sequences of examples drawn from an iid test dataset with high probability .

Summary: The paper is based on a work by Sur and Candès (2019) who propose a way to correct the bias of the MLE estimator and correct confidence intervals and p-values for logistic regression. The results in Sur Candes are asymptotic results n \to \infty but assume a fixed aspect ration $p / n = \kappa$ where p is the number of regression coefficients. The results have  been refined in Zhao et al (2020),: in Zhao, the regressors are Gaussian but with an arbitrary covariance matrix whereas the regressors are spherical Gaussian in Sur Candes  (the distribution of the regressor plays a key role to compute the correction factor).   The original algorithm by Sur and Candes require to estimate the signal strength, defined as the $\gamma^2 = Var(\beta^T X)$. This is of course challenging because the parameter is unknown. The authors proposed to use the "corrupted signal strength, defined as $\Var(\hat{\beta}^T X)$ where $\hat{\beta}$ is the MLE (it is claimed that the corrupted signal strength has been introduced in Zhao (2020) but on the arxiv version of this paper I did not find such quantity). The correction factors can be computed by solving an equation to involving the corrupted signal strength. The purpose of this paper is to compute an estimator of the corrupted signal strength.