SLOE: A Faster Method for Statistical Inference in High-Dimensional Logistic Regression
Authors: Steve Yadlowsky
Taedong Yun
Cory Y McLean
Alexander D'Amour
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: high dimensional
logistic regression
statistical inference

Abstract: Logistic regression remains one of the most widely used tools in applied statistics, machine learning and data science. However, in moderately high-dimensional problems, where the number of features $d$ is a non-negligible fraction of the sample size $n$, the logistic regression maximum likelihood estimator (MLE), and statistical procedures based the large-sample approximation of its distribution, behave poorly. Recently, Sur and Candès (2019) showed that these issues can be corrected by applying a new approximation of the MLE's sampling distribution in this high-dimensional regime. Unfortunately, these corrections are difficult to implement in practice, because they require an estimate of the \emph{signal strength}, which is a function of the underlying parameters $\beta$ of the logistic regression. To address this issue, we propose SLOE, a fast and straightforward approach to estimate the signal strength in logistic regression. The key insight of SLOE is that the Sur and Candès (2019) correction can be reparameterized in terms of the corrupted signal strength, which is only a function of the estimated parameters $\widehat \beta$. We propose an estimator for this quantity, prove that it is consistent in the relevant high-dimensional regime, and show that dimensionality correction using SLOE is accurate in finite samples. Compared to the existing ProbeFrontier heuristic, SLOE is conceptually simpler and orders of magnitude faster, making it suitable for routine use. We demonstrate the importance of routine dimensionality correction in the Heart Disease dataset from the UCI repository, and a genomics application using data from the UK Biobank.

SLOE: A Faster Method for Statistical Inference in High-Dimensional Logistic Regression
Steve Yadlowsky* yadlowskyygoogge. com
Taedong Yun tedyun@google.co com
Cory McLean cym@google com
Alexander D) D'mmor"
al alexdamour@googlee.coo com
Abstract
Logistic regression remains of the most widely used tools in applied statisties, machine learning and data science. However, in moderately nigh-dimensional problems, where the number of features d is non-negligible fraction of the sample size n, the logistic regression maximum likelihood estimator (MLE), and statistical procedures based the sarge-sample approximation of its distribution, behave poorly. Recently, Sur and Candès [2019] showed that these issues can be corrected by applying new approximation of the MLE's sampling distribution this high- dimensional regime. Unfortunately, these corrections are difficult to implement in practice, because they require an estimate of the signal strength, which is a function of the underlying parameters 3 of the logistic regression. To address this issue, we propose SLOE, a fast and straightforward approach to estimate the signal strength in logistic regression. The key insight of SLOE is that the Sur and Candès [2019] correction can be reparameterized in terms of the corrupted signal strength. which is only function of the estimated parameters 3. We propose an estimator for this quantity and show that dimensionality correction with it is accurate in finite samples. We demonstrate the importance of routine dimensionality correction in the Heart Disease dataset from the UCI repository, and genomics application using the UK Biobank. Compared to the existing ProbeFrontier heuristic, SLOE is conceptually simpler and orders of magnitude faster, making it suitable for routine use. Additionally, we provide consistency guarantees in the relevant high-dimensional regime.
Introduction
Logistic regression a workhorse in statistics, machine learning, data science, and many applied fields. It is generalized linear model that models a binary scalar outcome Y {0, 1} conditional on observed features X E Rd via
Y = z), with g(t) = 1+exp(-z)
(1.1)
with the coefficients 3 fit using observed data. Logistic regression is popular as a scientific tool because the model is often accurate, and comes with well-established statistical inference procedures for quantifying uncertainty about the parameters 3 and predictions g(3 x) at test inputs æ. For example, most statistical software packages not only produce predictions from the model, but also summaries such as confidence intervals (CIs) and p-values that enable practitioners to understand the strength of evidence for the prediction in quantitative way. These widely adopted estimation and
"Google Research, Brain Team GGogle Health
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

10
1.0
0.4
0.2
100
300
500
800
0.2
0.8
Index
Expected event rate
(b) Calibration curve for predictions iid test set. Figure Coefficients and calibration of the predictions from the MLE from simulated data with 7 4000, = 800. Neither the coefficients, nor predictions are converging to their true values as predicted by standard large-sample asymptotic theory.
(a) True and estimated coefficients
statistical inference routines are based approximations from large-sample asymptotic theory of the maximum likelihood estimator (MLE) 3. These approximations come from the limiting distribution of B as the sample size tends toward infinity, but the number of covariates d remains fixed.
Unfortunately, these standard approximations perform poorly when the number of covariates is of similar order to the number of samples even when the sample size is large [Fan et al., 2019]. Recently, Sur and Candès [2019] showed that, in this setting, the behavior of /3 in finite samples is better approximated by its limiting distribution in another regime, where the aspect ratio : d/n n//n>>. > is held fixed as both n and d grow to infinity. They show that the estimated coefficients (and therefore, the predictions) have systematically inflated magnitude and larger variance than the standard approximation predicts. The precise characterization of the limiting distribution of /3 in Sur approximation, which, in principle, facilitates
and Candès [2019] and Zhao et al. [2020] justify debiasing estimates and constructing CIs and valuue for parameters and test predictions, alike.
The goal of this paper is to make the application of these groundbreaking results practical. A major barrier to adoption currently is that the calculation these bias and variance corrections requires knowing the signal strength 72 == Var(B X), which is challenging to estimate because function of the unknown parameter vector Sur and Candès [2019] proposed a heuris- tic called ProbeFrontier to estimate this quantity, but this approach is computationally expensive, conceptually complex, and hard to analyze statistically. We propose a simpler estimation proce- dure. Our approach reparameterizes the problem in terms of the corrupted signal strength parameter x/2 := lim,- Var(3" X) that includes the noise in the estimate 3. This more straightforward (though non-trivial) to estimate. We propose the Signal Strength Leave-One which consistently estimates 172, and show that using this for inference yields more accurate CIs in finite samples. Importantly, SLOE takes orders of magnitude less computation than ProbeFrontier, having similar runtime to the standard logistic regression fitting routine.
Estimator (SLOE),
Preliminaries
In this section, we revisit some fundamentals of statistical inference with logistic regression, review recent advances in high dimensional settings by Sur and Candès [2019] and Zhao et al. [2020], and discuss the implications of this paradigm shift in terms of better characterizing practice.
2.1 Logistic Regression and Statistical Inference
Estimates of = (B1. ,Ba) in the logistic regression model are usually obtained through maximum likelihood estimation, by maximizing the empirical log-likelihood
8 = argmaxx BERd
- 1og(o(3 X.)) (1 Yo) g(3 X.)).
(2.1)
1=1
The log likelihood is concave, and has unique maximizer whenever the outcomes are not linearly separable in the covariates. We will use logistic regression synonymously with maximum likelihood estimation in the logistic regression model, and call / the MLE.

---

The sagple asymptotic statistical theory [Lehmann and Romano, 2005] taught in nearly every university Statistics program characterizes the behavior of the estimated coefficients and predictions in the limit as - x while holding the number of features d fixed. Under this theory, estimates converge to their true value, 3 3 9, and the estimation error diction error g( z) 2(3 gg.3Tar x) will be small to observe, unless amplified by a factor of vn, in which case vn(ß 3) ~y N(0, Igg) where Z3 == 3 is the Fisher information matrix, with D3 = g(3 X) g(3 X)).
and pre-
Standard p-values
Corrected p-values
Of course, when analyzing real data, data sci- entists only have access a finite number of samples, and so this theory serves as an ap- proximation characterization of the behavior ex- pected in practice. If the approximation is good, one can make inferences about the underlying data generating distribution. For example, for (0, 1), we can construct confidence inter- vals (CIs) that contain the true parameters with probability 1 6. Propagating the uncertainty to predictions gives CIs for the outcome probabili- ties that can help contextualize machine learning predictions for users such as scientists clini- cians [Kompa et al., 2021].
1.4
1.2
Frequency
1.0
Frequee Toquenoy
0.8
0.6
0.4
0.0
0.0
0.5
1.0
0.0
0.5
1.0
p-value
pivalue
Figure 2. Histogram for p-values according to the standard and correct Wald test for the null coefficients in simulated data with K 0.1, 7² = 5, 4000.
The large sample approximation is good one when the number of predictors is small relative to the number of observations. However, in modern scientific studies and data-driven technology, while many samples may be collected, many features of each sample are also collected. such settings, the number of features d may be significant compared to the sample size n; then, the large sample characterization can be quite misleading.
We can visualize the extent to which this breakdown in assumptions leads bias. Figure shows the estimated coefficients and calibration curve for logistic regression model fit on simulated data with 4000 and p = 800. Despite n being reasonably large, these illustrate significant departures from the behavior predicted by the large sample asymptotics. The magnitude of the estimated coefficients systematically overestimated and they are noisy. Additionally, the model is overconfident in its predictions. Despite this, the classification accuracy of the resulting model on the in in-distribution test data 0.70, which close to the optimal accuracy of 0.77.
Recently, Sur and Candès [2019], Mai et al. [2019], and Zhao et al. [2020] showed that the behavior of the MIEE in high-dimensional settings is better characterized by different asymptotic approximation. In this approximation, the aspect ratio k = d/n d/n, instead of the number of predictors d, remains fixed as n grows to infinity. Formally, they study sequence of problems in which n, d oc; d/n + K > 0, 72 Var ( X) is fixed as d grows, 3 and the MLE exists asymptotically for (k, 7). Sur and Candès [2019], Zhao et al. [2020], and our work assume X, marginally Gaussian, and complementary to the mixture-of -f-assssinsss generative model studied by Mai et al. [2019].
The semantics of this asymptotic regime are bit more complicated than the classical one; we no longer think of the argument in terms of a fixed problem with increasing data. However, The characterization from this asymptotic theory better reflects the finite-sample behavior of the MLE B when the data has a moderate aspect ratio K.
For simplicity, we state the key result from this work in the case where the covariates are drawn from spherical Gaussian, i.e., X N(O, I2), but note that these results are generalized in Zhao al. [2020] to arbitrary covariance structures." 4 So far, proof techniques only work for normally distributed features, however empirical results (Fig. 4b and previous work) the work for sub-Gaussian features. In this regime, the distribution of the MLE 3 converges to a normal distribution asymptotically
"This prevents the expectations g(8 X) from converging to 0 or I as grows To satisfy this, the effect of each covariate shrinks as n grows. It is complementary sparsity models, being more realistic for many problems where predictive performance comes from pooling many weak signals.
"Simulations suggest that our methods generalize to models including an intercept term when using the intercept conrection conjectured in Zhao et al. [2020].
3

---

centered around the inflated parameter a;,, for some a >> 1. In particular, for any coefficient Sy.
vn B3 -a83) L o(
(2.2)
and for the predicted logit of an imput x E IRd with asymptotically finite norm and vnx T = O()),
| (3 x ae--881 A N(O,o
(2.3)
for constants a and o., to be discussed later, that are only functions of x. and 7. The general convergence result when X has an arbitrary covariance is stated in Theorem 3.1 of Zhao et al. [2020].
a and ø, were known, this theory provides corrected approximate Cls and p-values. For example, taking db to be the quantile function of standard normal distribution:
B = 1 6/2) 1/2
(1 8) for By:
(2.4)
a
p-value under 3, = 0: p= = 10-1(00 (7) where = B//
O. an
(2.5)
Figure shows how the p-values obtained based on these adjustments have the expected behavior (i.e., they are uniform when the null is true), whereas those from the standard approximation do not.
2.2 Calculating Dimensionality Corrections
Sur and Candès [2019] show that the constants a and a, are determined by the solution to a system of three non-linear equations, which depend implicitly the aspect ratio and the signal strength
y² = Var(33 X):
R²??
E (Ag(Qi)(A(proxx) xG ((=))) (Q2)))²],
0 E Ag (Q2))),
(2.6)
1-K E 20(Q1) (Qa)
where ps) = argmin, AG(t) + [s - -t)?, for G the anti-derivative of g, and
( (8) )8)~m) ->~
S-~NOO, (o.
-a7² 20 -ays [ a²7² + kO? 1)
(2.7)
Here, Q1 is a random variable with the same distribution (asymptotically) as the population logits 3T X, and Q2 is a random variable with the same distribution (asymptotically) as the logits with the biased MLE plugged in A X. The auxiliary parameter À corresponds to the limiting average eigenvalue of the inverse Hessian at the MLE, (1/n E Xzg' (TTX,) X,)x()--, which is useful for making comparisons to asymptotic approximations from standard theory. Sur and Candès [2019] note that this system has unique solution so long as (x, take values at which the MLE exists asymptotically. Once set up, these equations are straightforward to solve with numerical methods.
The key quantity needed to set up these equations is the signal strength parameter y, which is not directly observed. Thus, correct for high dimensions in practice requires that y be estimated. This is difficult, as is itself a function of the unknown parameter B, and is the main focus of this paper.
Sur and Candès [2019] suggest heuristic method, called ProbeFrontier, for estimating 7 in practice. The idea is to search for the (sub)sample size n' < n such that the observations in the subsample linearly separable. Sub-sampling the data changes the aspect ratio without changing the signal strength 7. For a fixed signal strength there is a sharp aspect ratio cutoff k. (2) (the "frontier") above which the data is separable with high probability. Based on these ideas, Probeerrotterr then inverts K., (7) to estimate y from the empirical aspect ratio p/n'. This requires subsampling the data repeatedly at various candidate aspect ratios, and for each subsample, checking whether the data are linearly separable using a linear program. Repeatedly checking the separability is computationally expensive, and the statistical behavior near the frontier makes analysis tricky.
'Note here that x is sequence in n. Up to 1/V4 Vd scale factor, these constraints are satisfied for a fixed sequences of examples drawn from iid test dataset with high probability.

---

3 A Practical Method for Dimensionality-Coreeeee Inference
As we established above, the key to estimating confidence intervals for 3 or the predictions g(x 3) is obtain a consistent estimate of 7 to plug in to Eqs. (2.6) and (2.7) to get estimates of a and a.. Here, we describe our method for obtaining such an estimate. First, we reparameterize the covariance in (2.7) in terms of more easily-estinn eitii-esttmmtedd "corrupted signal strength" parameter, and then, we derive a computationally efficient estimator of by approximating a consistent leave-one-out estimator.
We implemented this estimator in Python, using scikit-learn to perform the MLE, and numpy / scipy [Harris al., 2020, Virtanen al., 2020] for the high dimensional adjustment and inference.
3.1 Reparameterization
We begin by reparameterizing the estimating equations (2.6) and (2.7) in terms of a quantity that is easier estimate. As it appears in Sur and Candès [2019], (2.7) is written in terms of the signal strength 72 = lim Var(BT ( X), but it can also be written in terms of 172 == lim,- Var(ß X). Using the asymptotic properties of 3 from Proposition 2.1, and Lemmas 2.1, 3.1 from Zhao et al. [2020], the variance has simple form: r/² 2²²² + RO?. Thus, we can write (2.7) equivalently as
81 )g²)~ ~~N(
( -a (17² -ar
- ka, [ ko²)
-a (n7 (n2 1PP-Kop ko?)
1)
(3.1)
rp
We can think 17² as the corrupted signal strength of the predictors, corrupted by the fact that only have an estimate 3 of the true signal suggests a natural strategy for estimate 1f2 if Σ were known. Shortly, we will define the (corrupted) Signal Strength Leave-One-Out Estimator (SLOE), TLLE that estimates r7² without knowing Σ.
our predictors. Note that Var(3" X) = E[3 E,3)) which
By plugging this estimate from the data along with the aspect ratio k in to Eqs. (2.6) and (3.1), we can derive estimates a a?? and X of the bias a, variance of and hessian inflation X, respectively. Then, we can construct CIs (2.4) and compute p-values (2.5) using these estimates.
3.2 SLOE Signal Strength Estimator
We now show how to estimate the corrupted signal strength x/2 = lim, Var -im, EE Σ3, using a computationally efficient approximation to ideal, but impractical leave-one-out estimator.
The key_ challenge in estimating 77 is that the predictor covariance Σ must also be estimated. When Σ and 3 are estimated from the same set of observed (X+) these quantities have non-trivial dependence even in large samples, rendering the naive estimate 3 ΣB asymptotically biased.
This dependence can be mitigated by a leave-one-out (LOO) estimator. In particular, lct (3_1) be the set of LOO MLEs, each calculated with all of the data except the i-th example. Define the LOO estimator as the empirical variance of the LOO logits (3 (x)
T..0 - -O
M 3 3T,X, X
(3.2)
This estimator climinates the problematic dependence between 3 and E because 3_, is independent of X,. Theorem shows that Thoo is consistent for 1) (see the Supplementary Materials for proof).
Theorem 1. Assuane that & and are in the range where the MLE exists asymptotically (see Sur and Candès [2019, Theorem 1J), and that for each n, p(n.) satisfies lima- P/n n = K. Let E. be a p x p positive definite matrix with bounded condition number, for all p. and be p-dimensional vector satisfying limm-~oc 3 Ep = =x² /nB i4 II, distribution with second moment and Σ, 33 + En [52]. Assume that X; NO, E.)), and follows the logistic regression model with parameter 3 given X,, independent of the other observations. Then, Thoo 2 17²
tid
Unfortunately, this LOO estimator impractical, as requires refitting the model n times. To address this, we define the Signal Strength Leave-One-Out Estimator (SLOE). SLOE replaces each
In the spirit of the method we leave 'O' out.
5

---

LOO logit __ with an approximation S; that is simple update to the full-data logit B X:
= Σ
Xyg (3
XJXX x; H X
(3.3)
JEI
U; (Yi
S, = 3 X
-9(3" Xi)),
1 1 g'(0 X;)U;
n
2
SLOE
5
8
(3.4)
To derive this estimator,
leave-one-out techniques inspired by the theoretical analyses in El Karoui et al. [2013] and Sur and Candès [2019]. Following along the derivation in Sur and Candès [2019] to approximate 3 B_g. we write out the optimality conditions for 3 and 3_, respectively. Let = [1.....n) n be the indices of all the examples and I=; = (1...., i 1, + 1..... .n be the indices of all but the i-th example. Then, by the first order optimality conditions for the MLE,
Σx,0, - g(3 x2)) = 0, and Σ JEI
X,(, - x(3],,,)) = 0.
JEZ
Taking the difference between these two equations gives
0= X,Y, g(3 x.))+ + E x,(((x1,,))) g(3 X,)). JEI:
(3.5)
We expect the difference between 3_g and 3 to be small, so we can approximate the difference g ,( g( X,) well with Taylor expansion of g around A Xx,
0 X,YY,, 9 x +
E JET
X X,)X, (
(3.6)
When the MLE exists, the Hessian of the log likelihood for the full data H and leave-one-out data,
H-, = Σ Xi {x)x = H + X.g (3 X,)x,,
JEZ
are invertible. Therefore, can solve for 3_
SLOE
get
ăiş estimation error
Mean squared error
SLOE áIă difference
3. - H X,YY, x())..)) X.)).
Then, we can accurately approximate the term 3T X, as
2-
x XX (33 x()).
To estimate all of the matrix inverses efficiently, we can take advantage of the fact that they are each a rank one update away from the full Hes- sian. Applying the Sherman-Morrison inverse formula (Sherman and Morrison, 1950] gives
500
1000
1500
2000
2500
3000
3500
4000
Sample
Figure 3. Empirical convergence of SLOE sample sizes. Shows average squared differ- ences over 1000 simulations of (fisLos ? (3T Σ - r²)), and (i%L A Σ The difference hetween G Σ, = Var(ß X) and its limit 172 appears to be the dominant term.
H X X 1 1+ g XXXX
Therefore, inverting one matrix gives us what we need from all n inverse Hessians. Approximating XL,X; in 7o0 with this gives us the SLOE estimator, TiSLOE- To show that the SLOE estimator V consistent, show the remainders from the Taylor approximation used to derive SLOE, (3.6), vanish in the limit (see the Supplement for proof).
Proposition Under the conditions of Theorem the estimators ii100 and TiSLor are asymptotically equivalent, mooo fisLo + op(()) (1), and therefore, fILLOE P/ 77.

---

Remark 1: The majority of the proof of Theorem I allows us to derive a rate of convergence, - = (n-(), for any e O. However, to show that B T - 172 our proof relies on the approximate message passing (AMP) construction of Sur and Candès [2019] and Zhao et al. [2020]. Deriving rates of convergence for these estimators would require significant new technical results on the analysis of AMP, none of which currently quantify convergence rates. In Figure 3, we show that the estimation error ÍÍLLOE is dominated by this latter difference in simulation, but the empirical error has the shape of a standard n 11/ rate.
4 Simulations
1.0
90% CI Coverage
1.0
90% CI Coverage
10000
0.9
0.9
1000
Coverage
0.8
Coverage
0.9
Time (3)
100
0.7
10
0.6
0.6
0.5
0.3
0.5
2000 4000
18000
Sample size n
Corrected CI
-ww Standard CI
0.1 1.0 5.0
SLOE
ProbeFrontier
(a) Gaussian features
(b) Simulated GWAS features
Figure 5. Comparison of wall clock run time for pa- rameter and varianee es- timation in ProbeFrontier and SLOE
Figure 4. Coverage of true probabilities with 90% standard and corrected CIs over all test examples and over 1000 simulations, with n 4000. Curves are terminated when the observed data are separable in more than 10% of simulations and simulations with numerical instabilities near the separable frontier (vertical dotted lines) are dropped.
100 simu- 0.2 and
lations with 72 1 various sample sizes.
In this section, we evaluate ÍSLOE in simulated data, where ground truth parameter values are known. We show (I) that the asymptotic properties of ĀiLLOE hold in realistic sample sizes; (2) that ÍSLOB can be used to make effective dimensionality corrections even when the covariates X do not have Gaussian distribution, as assumed in the theory; and (3) that ÍiLLOE requires orders of magnitude less computation than the current heuristic ProbeFrontier.
Here, we evaluate approximations in terms of the coverage rate of 90% CIs for the prediction probabilities μ == g(3 x)) = P(Y for each observation in the corresponding test set, using the standard CI and the corrected CI, and compute the fraction of CIs that contain the true μę: Valid 90% CIs will cover the target quantity 90% of the time. This evaluation checks whether the Cls centering and width are appropriately calibrated, and whether the normal approximation adequately captures the tail behavior of the estimator.
X = z). In each simulation, we calculate a 90% CI for pa
data generating process parameterized by the sample size the aspect ratio K, and signal strength 722 We draw n examples of covariate vectors from a given distribution, and, conditional on 2ú, caleulate the μ according to the logistic regression model where B is parameterized by as follows,
In our simulations,
d/8 B, 21/V d/8 < d/4
22/VV
otherwise,
and finally, draw Y; Bernoulli (1z). We consider two different distributions for X. First, generate X ~ N(O, 4p). as studied in the theory. Second, following an experiment Sur and Candès [2019], we draw X from discrete distribution that models feature distributions encountered in Genome- Wide Association Studies (GWAS). Here, each feature X;y takes values in {0, 1, 2, according the Hardy- weiinber equilibrium with parameter Py varying between covariates in the range [0.25. 0.75). This setting is not explicitly covered by the theory, but we expect the approximation work well nonetheless because the features have thin tails.
We show the results of these coverage experiments, with 4000, in Figure 4a. In both the Gaussian and simulated GWAS settings, across all practically feasible values of x and 7, the corrected intervals
7

---

Corrected 90% CI
Standard 90%
Observed Outcome
Example
Example
0.0
0.2
0.4
0.6
0.8
1.0
10 10
0.01
0.10
0.50
0.90
0.991
1011--1---0 101--0
Probability
Probability (logit scale)
Figure 6. Observed outcomes, predictions, and 90% CIs for 8 held-out examples from the Cleveland Clinic Hcart Disease dataset, with probabilities on the (a) absolute scale, and (b) logit scale.
computed with fSLLOE reliably provide 90% coverage. This in stark contrast to CIs generated under standard theory whose coverage begins to drop off at K >> 0.05 (that is, 20 observations per covariate) for all signal strengths. Notably, in the simulated GWAS setting that not covered by theory, the corrected intervals remain reliable while the standard CIs perform worse than in the Gaussian setting.
SLOE is accurate in finite samples and computationally efficient. Figure compares the time to perform estimation and inference with both SLOE and ProbeFrontier. For all dataset sizes, SLOE is 10x faster, and for 3000, SLOE is 100x faster, which is when speed matters the most.
Applications
Here, we present applications of dimensionality corrections with fJSLOR. emphasizing the real-world impact of the differences between standard and corrected statistical inference in logistic regression. See the Supplementary Materials for an additional experiment on the UCI dataset "German Credit Scoring".
Prediction of Heart Disease Reliable uncertainty quantification is essential for supporting effective decision-making clinical applications [Kompa et al., 2021]. We show how the dimensionality conrections substantially changes the uncertainty communicated by CIs for predicting heart discase diagnosis from physiological and demographic information from the Cleveland Clinic Heart Discase dataset [Detrano et al., 1989]. Kompa et al. [2021] showed that logistic regression predictions sensitive to bootstrapping this training data, demonstrating that uncertainty estimation is important.
The Heart Disease dataset (downloadable from the UCI Machine Learning Repository) has 14 predictors, and 303 observations. Converting categorical variables dummy variables, taking interactions between sex and pain categorization given its clinical significance [Mehta et al., 2019], and balancing the classes, gives 136 training examples and predictors (x 0.15). SLOE estimates bias inflation factor â of 1.40 for this data, so the logits in the standard MILE will be inflated by factor of 40%. Correcting these inflated logits significantly changes the center of the CIs for downstream predictions. Figure 6 compares the standard Cls to the corrected CIs for handful of test examples, along with the true labels. Notice that the standard CIs are overconfident, with the top CI being entirely above 99.8% probability of heart disease, when in fact, there wasn't tt The corrected Cls show that in reality, much higher probabilities of misdiagnosis (i.e., in 58 chance of being wrong, rather than in 140) are consistent with the training data. In clinical settings, the uncorrected Cls could communicate an unwarranted sense confidence a diagnosis.
Genomics In medical genomies. it is common to build predictors for clinically relevant binary phenotypes with logistic regression models, using genetic variants as features. Given set of candidate variants (predictors) identified by genome-wide association studies (GWAS) or other related methods, practitioners often use p-values associated with the coefficients of the variants to interpret these models (e.g. Ding et al., 2020). Here, we present a case study with real data, in which the corrected p-values computed with ÍSLOE maintain their intended interpretation, unlike the standard p-values.
The nonparametric bootstrap turns out produce invalid Cls when x 0: see the Supplementary Materials.

---

Standard
Corrected
2.0
False proportion
0.6
1.5
1.0
0.2
0.5
Standard
0.1
Corrected
0.0
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.0
0.2
0.4
0.6
p-values
p-values
threshold
Figure 7. (a) Histogram of p-values for null coefficients (randomly sclected SNPs not known to he associated with glaucoma), from the standard and corrected logistic regression models. (b) Comparison of the Benjamini--Hochbee Q-value threshold and the empirical proportion of false discoveries, using the p-values from the standard and corrected logistic regression models.
We study the association of genetic predictors with glaucoma, eye condition that damages the optic nerve. We build a dataset from UK Biobank [Sudlow al., 2015] 5] with 127 variants directly associated with glaucoma according to recent meta-ana anta--n of GWAS [Gharahkhani et al., 2020] and randomly sample 1873 "null" single nucleotide polymorphisms (SNPs) also present in Hap.Map [International HapMap Consortium et al., 2010]. such that each sampled SNP is at least 100K base pairs away from all associated variants and from each other, and that pairwise Pearson correlation in the training set (see below) is greater than 0.8, avoid correlated features due linkage disequilibrium. This generates 2000 genetic predictors total. Glaucoma status was determined based on touchscreen questionnaire responses and hospital episode statistics as performed previously [Khawaja et al., 2018, Alipanahi et al., 2021]. From this, we construct training set of 5067 cases with glaucoma and 5067 age- and sex-matched controls (x 2000 10134 097) train a logistic regression model, and compute standard and corrected p-values with it.
Figure 7a shows p- paluue of the randomly selected variants whose effects expect to be null. The standard p-values are skewed while the corrected p-values follow the expected uniform distribution closely, as observed in simulated data (Figure 2). Uniform null p-values important for controlling the error rate in procedures applied to select important variants, e.g., for resource-intensie follow-up. In Figure 7b, we investigate how these p-values interact with the Hoohhaeri--Hohherrgg (BH) procedure [Benjamini and Hochberg, 1995] for "discovering" important variants while controlling false discovery rates. To compute approximate empirical false discovery proportion, variants identified by the BH procedure, and compute the proportion of the null variants (as opposed known glaucoma-associated variants) in this The corrected p-values result in better false discovery proportion calibration across range of false discovery rate targets (Figure 7b).
take all
6 Discussion
The groundbreaking work by Sur and Candès [2019] has significant implications for theoretical understanding, uncertainty quantification, and generally, practical statistical inference in large scale data. Connecting this theory to practice highlights some interesting insights and questions.
Bias and Variance In the regime that we analyze, these dimensionality corrections constitute a something of statistical free lunch: the corrections reduce both bias and variance, or, in the case of confidence intervals, they both improve coverage to their nominal level (Figure 4a) while shortening their length the logit scale (Figure 6b). This oceurs because bias is created by overfitting," 9 as opposed to underfitting, which we can correct. This systematic bias in the parameters appears in downstream predictions, and unless corrected, does not vanish asymptotically. Because the bias correction shrinks the coefficient estimates by a factor 1/a, also produces lower-variance predictions, even after correcting for the too-small variance predicted by standard theory.
Cls and Regularized Predictions Cls with valid coverage are an attractive tool for quantifying uncertainty about predictions, but in practice, point predictions are usually made with models that are regularized to reduce the variance the predictions, at the cost of some amount bias. Yet, the CIs
$ reserve 126 individuals (563 cases, 563 controls) sampled the same way test set. To overfit, the model reduces training residuals by amplifying coefficient to produce extreme predictions.

---

around the predictions that we produce are centered around unbiased estimates of the logits; thus, one might ask whether these can produce coherent summaries together. simulations, we find that the 1 -- 5 CIs contain the optimal regularized predictor (selected via leave-one-out cross validation) more fraction of the time. This makes sense, as the LOOCV regularized predictor should be closer to the true probability, which we know is within the CIs 1 - 8 of the time. If one preferred the Cls to always include the regularized predictions, taking the convex hull of the CIs and regularized predictions would give slightly larger, but more interpretable, CIs with higher- nigher-than-nomine than-nominal coverage.
than 1
Broader Implications for ML It would be interesting to further explore the implications of these observations in machine learning. The logistic regression MLE is closely related to the cross entropy / softmax loss used frequently deep learning. The logistic regression model directly related to softmax probabilities through the independence of irrelevant alternatives assumption, potentially enabling multi- extensions of this work. However, the paradigm studied here focuses on the underparameterizzed regime, whereas many deep learning models are overparameterized. Even in large datasets where this may not be true, the signal often strong enough that that the data are separable, and so the MLE does not exist. Salehi et al. [2019] characterizes generalization into regimes where the data are separable by adding regularization, but their results rely more on assuming isotropic Gaussian features, whereas this work and Zhao et al. [2020] generalize well theoretically to elliptical Gaussian features and experimentally to sub-Gaussian features.
Acknowledgments and Disclosure of Funding
This research has been conducted using the UK Biobank Resource under Application Number 17643. The authors thank Babak Alipanahi and Farhad Hormozdiari for their help and feedback on the genomics experiments, Arjun Seshadri for his feedback on the theoretical results and advice on linear programming to implement ProbeFrontier efficiently, and D. Sculley for his support and feedback on the research.
References
B. Alipanahi, F. Hormozdiari, B. Behsaz, Cosentino, Z. R. McCaw, E. Schorsch, D. Sculley, E. H. Dorfman, P. J. Foster, L. H. Peng, S. Phene, N. Hammel, A. Carroll, A. P. Khawaja, and phenotyping significantly improves genomic
C. Y. McLean. Large-scale machine- heatne-lernning-beee discovery for optic nerve head morphology. Am. J. Hum. Genet., 108(7):1217-1230, July 2021.
Y. Benjamini and Y. Hochberg. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society, Series B, 57(1):289 1995.
R. Detrano, A. Janosi, W. Steinbrunn, M. Pfisterer, J.-J. Schmid, S. Sandhu, K. H. Guppy, S. Lee, and V. Froelicher. International application of a new probability algorithm for the diagnosis of coronary artery disease. The American Journal of Cardiology, 64(5):304-310, 1989.
Y. Ding, K. Hou, K. S. Burch, S. Lapinska, Privé, Vilhjälmsson, S. Sankararaman, and B. Pasaniuc. Large uncertainty in individual prs estimation impacts prs-based risk stratification. bioRxiv, 2020. doi: 10. 10.1101/2020.1 1..00000.11......... 1.30.403188. URL https: //www. biorxiv. org.//orr//ooooo content/ wtt.v..rrt/ootee//////oo
early/2000/12/065 2tt////22002/2220 11.11.1.11..... 3...300..8..
N. El Karoui, D. Bean, P. Bickel, C. Lim, and B. Yu. On robust regression with high-dimensional predictors. Proceedings of the National Academy of Sciences, 110:36))1557--1562,, 2013.
Y. Fan, E. Demirkaya, and J. Lv. Nonuniformity of p-values can occur early in diverging dimensions. Journal of Machine Learning Research, 20(77):1 2019. URL http://jmlr happg//aprs////// v20/18 220018-314 html.
Gharahkhani, Jorgenson, P. Hysi, A. P. Khawaja, Pendergrass, X. Han, J. S. Ong, A. W. Hewitt A. Segre, R. P. Igo, H. Choquet, A. Qassim, N. S. Josyula, J. N. Cooke Bailey, P. Bonnemaijer, A. Iglesias, O. M. Siggs, Young, V. Vitart, A. A. Thiadens, Karjalainen, S. Uebe, R. B. Melles, K. S. Nair, R. Luben, M. Simcoe, N. Amersinghe, A. J. Cree, R. Hohn, A. Poplawski, L. J. Chen (CUHK), C.-Y. Cheng, E. Vithana,
G. Tamiya, Y. Shiga, M. Yamamoto, T. Nakazawa, J. Rouhana, H. Currant, E. Birncy, X. Wang, A. Auton, A. Ashaye, O. Olawoye, S. E.
10

---

Williams. S. Akafo. M. Ramsay, K. Hashimoto, Y. Kamatani, M. Akiama, Y. Momozawa, P. J. Foster, P. T. Khaw, E. Morgan, N. G. Strouthidis, P. Kraft, H. Kang, C. C. Pui Pang (CUHK). F. Pasutto, P. Mitchell, A. J. Lotery, A. Palotie, C. van Duijn, J. Haines, C. Hammond, I. R. Pasquale, C. C. Klaver, M. Hauser, C. C. Khor, D. A. Mackey, M. Kubo, T. Aung, Craig, S. MacGregor, and J. Wiggs. A large cross-ancestry meta-analysis of genome-wide association studies identifies 69 novel risk loci for primary open-angle glaucoma and includes a genetic link with alzheimer's s disease. bioRxiv, 2020. doi: 0101100012220001130.27270 // www.. biorxiv org/ httg//ootett///eeeeeeeeeeece content/earrly/
URL https: 00020/22033/02200000 .10.022.001..0000 30. 927882..
C. R. Harris, K. J. Millman, S. J. van der Walt, R. Gommers, P. Virtanen, D. Cournapeau, E. Wieser, Taylor, S. Berg, N. J. Smith, R. Kern, M. Picus, S. Hoyer, M. H. van Kerkwijk, M. Brett, A. Haldane, J. Fernández del Río, M. Wiebe, P. Peterson, P. Gérard-Marchant, K. Sheppard, T. Reddy, W. Weckesser, H. Abbasi, C. Gohlke, and T. E. Oliphant. Array programming with NumPy. Nature, 585:357-362, 2020. doi: 10. 10.1038/s411 020-6-000026---2
International Map 3 Consortium, D. M. Altshuler, R. A. Gibbs, L. Peltonen, D. M. Altshuler, R. A. Gibbs, L.. Peltonen, E. Dermitzakis, S. F. Schaffner, F. Yu, L. Peltonen, E. Dermitzakis, P. E. Bonnen, D. M. Altshuler, R. A. Gibbs, P. I. W. de Bakker, P. Deloukas, S. B. Gabriel, R. Gwilliam, S. Hunt, M. Inouye, X. Jia, A. Palotie, M. Parkin, P. Whittaker, F. Yu, K. Chang,
A. Hawes, L. R. Lewis, Y. Ren, D. Wheeler, R. A. Gibbs, D. M. Muzny, C. Barnes, K. Darvishi, M. Hurles, J. M. Korn, K. Kristiansson, C. Lee, S. A. McCarrol, J. Nemesh, E. Dermitzakis,
A. Keinan, S. B. Montgomery, S. Pollack, A. L. Price, N. Soranzo, P. E. Bonnen, R. A. Gibbs, C. Janzegg--Juureuui, A. Keinan, A. Price, F. Yu, V. Anttila, W. Brodeur, M. J. Daly, S. Leslie, G. Vean. L. Moutsianas, H. Nguyen, S. F. Schaffner, Q. Zhang, M. J. R. Ghori, R. McGinnis, W. McLaren, S. Pollack, A. Price. S. Schaffner, F. Takeuchi, R. Grossman, I. Shlyakhter, E. B. Hostetter, P. C. Sabeti, C. A. Adebamowo, M. W. Foster, D. R. Gordon, J. Licinio, M. C. Manca, A. Marshall, Matsuda, Ngare, V. Wang, D. Reddy, N. Rotimi, C. D. Royal, R. R. Sharp, C. Zeng. I.. D. Brooks, and J. F. McHwwn. Integrating common and rare genetic Sept. 2010.
variation in diverse human populations. Nature, 467(7311):522
A. P. Khawaja. J. N. Cooke Bailey, N. J. Wareham, R. A. Scott, M. Simcoe, R. P. Igo, Jr, Y. E. Song, R. Wojcicchowski, C.-Y. Y. Cheng, P. T. Khaw, L. R. Pasquale, J. L. Haines, P. J. Foster, J. L. Wiggs, C. J. Hammond, P. G. Hysi, UK Biobank Eye and Vision Consortium, and NEIGHBORHOOD Consortium. Geneme-wide analyses identify 68 new loci associated with intraocular pressure and improve risk prediction for primary open-angle glaucoma. Nat. Genet., 50(6):778-78) June 2018.
B. Kompa, J. Snoek, and A. L. Beam. Second opinion needed: communicating uncertainty in medical machine learning. npj Digital Medicine, 4(1):166, 2021.
E. L. Lehmann and J. P. Romano. Testing Statistical Hypotheses, Third Edition. Springer, 2005.
X. Mai, Z. Liao, and R. Couillet. A large scale analysis of logistic regression: Asymptotic performance and new insights. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3357-3361. IEEE, 2019.
P. K. Mehta, C. Bess, S. Elias-Smale, V. Vaccarino, A. Quyyumi, J. Pepine, and C. N. Bairey Merz. Gender in cardiovascular medicine: chest pain and coronary artery disease. European Heart Journal, 40(47):38 19933826 2019.
Salehi, E. Abbasi, and B. Hassibi. The impact of regularization on high-dimensional logistic regression. In Advances in Neural Information Processing Systems, volume 32, pages 12005-12015, 2019. URL attps://proceedinne www.:/rpsss.ccpppppp///////////e 0bde6778888770222221173444538883
pdf.
Sherman and W. Morrison. Adjustment of an inverse matrix corresponding to a change one element of a given matrix. The Annals of Mathematical Statistics, 21(1): 1950.
C. Sudlow, J. Gallacher, N. Allen, V. Beral, P. Burton, J. Danesh, P. Downey, P. Elliott, J. Green, M. Landray, B. Liu, P. Matthews, G. Ong, J. Pell, A. Silman, A. Young, T. Sprosen, T. Peakman, and Collins. UK biobank: an open access resource for identifying the causes of wide range of complex discases of middle and old age. PLoS Med., 12(3) 1001779, Mar. 2015.
11

---

P. Sur and E. J. Candès. A modern maximum-likelibod theory for high-dimensional logistic regression. Proceedings of the National Academy of Sciences, 116(29)114516-1552) 2019.
Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weckesser, J. Bright, S. J. van der Walt, M. Brett, J. Wilson, K. J. Millman, N. Mayorov, A. R. J. Nelson, E. Jones, R. Kern, E. Larson, C. J. Carey, I. Polat, Y. Feng, E. W. Moore, J. VanderPlas, D. Laxalde, J. Perktold. R. Cimrman, I. Henriksen, E. A. Quintero. C. R. Harris, A. M. Archibald, A. H. Ribeiro, F. Pedregosa, P. van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17: 261-272, 2020. doi: 10. 00.1088/41192--099-666
Q Zhao, P. Sur, and E. J. Candes. The asymptotic distribution of the mle in high-dimennional logistic models: Arbitrary covariance. arXiv preprini arXiv:2001.09351, 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [Yes]
Code in Supplement
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
12


Reviews:
[{"review_id": "u0A9mv1UGzP", "reviewer": "Reviewer_JDUF", "summary": "The authors propose to make practical the high dimensional analysis of the logistic regression performed by [Sur and Candes]. Indeed, [Sur and Candes] proved that logistic regression in the classical regime (number of samples going to infinity while the dimensionality is fixed) is biased and must be corrected for modern applications where the number of samples and their size are of the same order of magnitude. However, in [Sur and Candes]'s paper, the asymptotic analysis depends on parameters that are difficult to estimate since they depend on ground truth parameters (regression parameter $\\beta$). The authors of the present article propose a fast estimation method.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**** Overall review *****\n\n1-Strong points\n\nThe article is well written and the motivation is quite clear. The code is also provided for reproducing the results.\n\n2-Weak points\n\nThe originality of the work is quite low.  The large dimensional analysis performed in [Sur and Candes] represents the main novelty and the present paper only provides an estimate for a parameter that will only be valid for Logistic regression. The contributions for a conference like NIPS is either:\n\n(a)-Provide a generic framework for estimating quantities involving quite huge dependence and apply it on more than one application  (more than just logistic regression in binary case) so that the impact is quite important\n\n(b)- provide both the large dimensional+ Estimation procedure as performed in [Sur and Candès] and as is generally the case in the large dimensional analysis of machine learning algorithms. The large dimensional analysis is generally the most contributing part.\n\nDue to the low theoretical part, the authors might compensate by more experiments (in the supplementary materials for example) to show the broader impact of their methods.\n\n**** Detailed comments ******\n\n-In Theorem 1, the notation $p$ has never been used previously. Isn't $d$ instead?\n\n-In theorem 1, Ave_j should be defined.\n\n-In line 188, the hessian should be defined directly after equation (3.3) and not defined in the proofs afterward.\n\n-In line 325, that repeated\n\n-The work did not mention multi-class classification which is of big interest.\n\n-I think it is important to compare the algorithms from a performance point of view to ProbeFrontier.\n\n-The authors should also plot the running time of the classical logistic regression in Figure 5 to justify that the proposed scheme has the same complexity as the classical one.\n\n-The author should also discuss the invertibility of $H$? Does the method work in all the cases? (Specifically in high dimension low sample size regime ($n<d$))\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations of the work are not clearly mentioned. However, the authors could discuss more on the invertibility of H, the conditioning of H, discuss multi-class extension which is missing.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hr"}, "extra_text": {}}, {"review_id": "MYZ8dhUw-Dc", "reviewer": "Reviewer_orH4", "summary": "The paper is based on a work by Sur and Candès (2019) who propose a way to correct the bias of the MLE estimator and correct confidence intervals and p-values for logistic regression. The results in Sur Candes are asymptotic results n \\to \\infty but assume a fixed aspect ration $p / n = \\kappa$ where p is the number of regression coefficients. The results have  been refined in Zhao et al (2020),: in Zhao, the regressors are Gaussian but with an arbitrary covariance matrix whereas the regressors are spherical Gaussian in Sur Candes  (the distribution of the regressor plays a key role to compute the correction factor). \n\nThe original algorithm by Sur and Candes require to estimate the signal strength, defined as the $\\gamma^2 = Var(\\beta^T X)$. This is of course challenging because the parameter is unknown. The authors proposed to use the \"corrupted signal strength, defined as $\\Var(\\hat{\\beta}^T X)$ where $\\hat{\\beta}$ is the MLE (it is claimed that the corrupted signal strength has been introduced in Zhao (2020) but on the arxiv version of this paper I did not find such quantity). The correction factors can be computed by solving an equation to involving the corrupted signal strength. The purpose of this paper is to compute an estimator of the corrupted signal strength. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nOriginality:  The paper follows on a very interesting problem (the corrections to MLE of logistic regression in the limit of large number of variables - large number of samples) that has been theoretically investigated by [1]. The corrections provided by [1] are computationally expensive and impractical. \nThe main contribution of this work is providing a cheaper estimate of those corrections, making it more practical to high dimensional problems. The idea of viewing Var (X^{\\top}\\beta), the true signal strength, as a function of the estimated log-odds variance Var(X^{\\top}\\hat{\\beta}) is powerful. This immediately helps the authors come up with a new estimator for the signal strength based on leave-one-out analysis. The main contribution is to realize that (3.2) is a consistent estimator of \\eta^2. This is a neat idea. \n\nIf considered as an applied paper, I think this paper has potential.  the numerical experiments should be more stressed with applications to multiple datasets to show the relevance of this work in practice, and theoretical computation of computational complexity with respect to [1]. \n\nminore comment: The authors propose to investigate the broader implications to ML but this is left for future works. This is definitely a good research program ! but very ambitious for a  9 pages paper !\n\nClarity: \n- The first half of the paper presents a synthesis of the results of Sur and Candes (in the case of spherical Gaussian regressors). \nThe results are well presented and follow very closely the papers by Candes. \n- The action really starts on page 5 with the reparametrization and definition of the  corrupted signal strength. The paper refers to Lemma 3.1 of Zhao (2020) but Lemma 3.1 establishes the result presented in equations 2.6 and 2.7 and I did not find any reference to the notion of corrupted signal strength in Zhao's paper (or else this notion appears under another name, which is not excluded)\n- The first result obtained by the authors is theorem 1, which establishes the consistency of the leave-one-out (LOO) estimator of the corrupted signal strength (in the case of correlated regressors). This is an interesting result (I did not have time to check the details of the proof). \n- Of course, the LOO estimator is impractical because it requires solving n logistic regression problems. The authors propose a feasible solution by \"approximating\" the LOO. . The approximation adopted and the discussion is classical (note that H does not seem to be defined in eq 3.3) - I do not really understand the meaning of remark 1 - \n\nCorrectness: The paper is well written.  The theorem proofs should follow almost directly from the proofs of earlier results in Sur and Candes (2019). So, I would not say that theoretically there is much innovation. Again, I highlight that realizing Thm 1 is neat.\n\nMinor comments-  authors use expressions such as: apparently (line 215), potentially (line 321) which give the impression of \"guess work\", while these assertions required to be properly justified. It would be better if the authors could rephrase those and take time to write detailed justifications of these claims.\n\nAdditional comments: \n- I would strongly recommend including the original references that introduced the leave-one-out analysis in high-dimensional statistics: https://www.pnas.org/content/110/36/14563 , https://www.jstor.org/stable/42713153 \nThe current lines 190-209 should cite them. \n(By the way the derivation in these lines seems exactly the same as some equations in Sur and Candes (2019) and not inspired by. In this light, I feel the authors should cite the appropriate places a bit better.) \n\n- The authors of the paper are probably not aware of the work [Mai] “A large scale analysis of logistic regression: asymptotic performance and new insights. X. Mai et al (ICASSP’19)”  as it is highly related to their work, and not cited. [Mai] addresses the problems tackled by this work, and even in a more general setting, with regularization and covering under-parametrized and over-parametrized regimes (in the high dimensional,  linearly proportional number of samples-dimension setup). In particular [Mai] consider an elliptical non-centered Gaussian distribution, a loss with regularization, and no restriction on whether (d>n or d<n). More importantly, from Theorem 1 in [Mai], the parameter \\gamma (which the current work goal is to estimate) does not appear in the asymptotic distribution of \\beta. All the parameters in Theorem 1 [Mai] can be estimated consistently and cheaply. Could the authors please provide in which way their work goes beyond [Mai]?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I do not see iany limitations or negative societal impact. The scope is limited (to logistic regression with correlated Gaussian covariate) but at least, the impacts are not negative", "review_text": "", "extra_scores": {"ethical_concerns": "none", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "8wqCqxMyABf", "reviewer": "Reviewer_GWvs", "summary": "Building off recent work by Sur & Candes (2019), this paper proposes a method for computing point estimates and well-calibrated confidence intervals for high-dimensional logistic regression (specifically, when the ratio $d/n$ converges to a constant as $n \\to \\infty$). \n", "questions": "", "limitations": "", "rating": 9, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The method builds on the work of Sur & Candes (2019), but solves a critical issue with that work: how to reliably and quickly estimate a key quantity needed to construct the point estimate and confidence intervals. The approach is simple to use but seems quite original. \n\nQuality: The paper is quite non-trivial from a technical perspective. But the theory isn’t just there for theory’s sake, as it guarantees asymptotic validity of the proposed method. The strengths and limitations of the theory and method are addressed appropriately. The experiments do a good job of validating the correctness via simulations and practical benefits via real-data examples but don’t over-promise. Overall, I found the paper very compelling\n\nClarity: The paper is incredibly clear and well-motivated. \n\nSignificance: Applying logistic regression when the number of predictors is of the same order as the number of observations is extremely common. I could imagine the proposed method being widely adopted in place of standard logistic regression point estimates and CIs. \n\n## Update after author response\n\nI think the authors have done a good job addressing the other reviewers concerns, so I maintain my original score. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "These are appropriately addressed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Following the discussion phase, the reviewers and myself have reached a consensus on this submission, and all agree on its interest to the NeurIPS community, its significance and its potential impact in disseminating earlier ideas to a broader group of practitioners, through an astute algorithm.

To quote from the discussion with the reviewers: the submission builds on the solid foundations of Sur and Candès, and delivers non-trivial theoretical insights on a new (provably correct) algorithm, which we anticipate will be impactful for practitioners of logistic regression in high dimensional settings.
Decision: Accept (Spotlight)