Title: Scale-invariant Optimal Sampling for Rare-events Data and Sparse Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: adaptive lasso

imbalance data

optimality criterion

subsampling
Abstract: Subsampling is effective in tackling computational challenges for massive data with rare events.  Overly aggressive subsampling may adversely affect estimation efficiency, and optimal subsampling is essential to mitigate the information loss. However, existing optimal subsampling probabilities depends on data scales, and some scaling transformations may result in inefficient subsamples. This problem is more significant when there are inactive features, because their influence on the subsampling probabilities can be arbitrarily magnified by inappropriate scaling transformations. We tackle this challenge and introduce a scale-invariant optimal subsampling function in the context of sparse models, where inactive features are commonly assumed. Instead of focusing on estimating model parameters, we define an optimal subsampling function to minimize the prediction error,  using adaptive lasso as an example to outline the estimation procedure and study its theoretical guarantee.  We first introduce the adaptive lasso estimator for rare-events data and establish its oracle properties, thereby validating the use of subsampling. Then we derive a scale-invariant optimal subsampling function that minimizes the prediction error of the inverse probability weighted (IPW) adaptive lasso.  Finally, we present an estimator based on the maximum sampled conditional likelihood (MSCL) to further improve the estimation efficiency. We conduct numerical experiments using both simulated and real-world data sets to demonstrate the performance of the proposed methods.

Scale-invariant Optimal Sampling for Rare-events Data with Sparse Models

Abstract

Subsampling is effective in tackling computational challenges for massive data with rare events. Overly aggressive subsampling may adversely affect estimation efficiency, and optimal subsampling is essential to mitigate the information loss. However, existing optimal subsampling probabilities depends on data scales, and some scaling transformations may result in inefficient subsamples. This problem is more significant when there are inactive features, because their influence on the subsampling probabilities can be arbitrarily magnified by inappropriate scaling transformations. We tackle this challenge and introduce a scale-invariant optimal subsampling function in the context of sparse models, where inactive features are commonly assumed. Instead of focusing on estimating model parameters, we define an optimal subsampling function to minimize the prediction error, using adaptive lasso as an example to outline the estimation procedure and study its theoretical guarantee. We first introduce the adaptive lasso estimator for rare-events data and establish its oracle properties, thereby validating the use of subsampling. Then we derive a scale-invariant optimal subsampling function that minimizes the prediction error of the inverse probability weighted (IPW) adaptive lasso. Finally, we present an estimator based on the maximum sampled conditional likelihood (MSCL) to further improve the estimation efficiency. We conduct numerical experiments using both simulated and real-world data sets to demonstrate the performance of the proposed methods.

1 Introduction

Rare-events data refer to binary-response data that are highly imbalanced, i.e., the number of zeros (a.k.a 'controls' or 'negative instances') are possibly hundreds or thousands of times as large as the number of ones (a.k.a. 'cases' or 'positive instances'). This type of data is common in various fields, such as medicine, natural science, political science, and social science, where examples of rare events can be rare diseases, natural disasters, wars, and financial crises, respectively. Modern technologies also prompt us to pay more attention to rare-events data. For example, in modern online recommendation systems, clicks are usually rare events compared with nonclicks. Statistical analyses, including parameter estimation and inferences, pose unique challenges for rare-events data because of high imbalance. In addition, rare-events data often involve sparse models. For instance,

rare diseases might be linked to a limited number of key genes. Therefore, researchers frequently adopt sparse models in genome-wide association studies for analyzing rare diseases. A different yet related example is the use of deep neural networks to predict click-through rates in modern online recommendation systems. These networks are typically overparameterized, necessitating methods that balance rare-events data with the sparsity of the underlying models. Data balancing is a popular approach to overcome challenges caused by imbalanced data and is usually accomplished through subsampling the zeros [5, 15] or oversampling the ones [3, 12, 16, 4]. In addition, rare-events data are often massive in order to obtain an adequate number of ones, and computation is demanding. Therefore, we focus on the subsampling approach since it addresses the imbalance issue and reduce the computational burden simultaneously.

It is shown in [20] that the efficiency of parameter estimation is essentially determined by the number of ones for rare-events logistic regression, and subsampling does not reduce the estimation efficiency as long as sufficient zeros are kept. In case of excessive removal of zeros, [22] developed an optimal sampling approach to minimize information loss. However, the optimal sampling probabilities in [22] are scale-dependent, which may lead to inefficient results. Figure 1 illustrates the issue using a simulated example, with details in Section D.1 of the appendix. We generate the data from the same logistic regression model and transform one of the covariates with different scales s = 0 . 01 , 0 . 1 , 1 , 10 , and 100 . Then we apply two optimal subsampling methods in [22], labeled with "A-OS" and "L-OS" in Figure 1. It is observed that the prediction errors of A-OS and L-OS are significantly impacted by the data scaling. The A-OS may perform similarly to the Uni (simple random sampling or uniform sampling) in Figure 1a when s = 0 . 01 ; so is the L-OS in Figure 1b when s = 100 . This scaledependent issue is not specific to logistic regression and rare-events data in [22]; it is a wide concern in literature for various data types and models, including but not limited to [1, 29, 21, 14, 26, 25, 24]. In this paper, we propose a scale-invariant optimal subsampling method to overcome the issue. It is labeled "P-OS" in Figure 1.

10

2

10

1

10 0

10 1

10 2

s

10

1.4

10

1.2

prediction error

(a) Non-sparse parameter

10

2

10

1

10 0

10 1

10 2

s

10

1.0

10

0.8

prediction error

(b) Sparse parameter

A-OS

L-OS

P-OS

Uni

The scale-dependence issue can seriously impact variable selection results for sparse models, where true parameters are zero for inactive covariates. In this case, inactive variables may be arbitrarily transformed without changing the underlying model, but the A-OS or L-OS would be highly influenced and may lead to misleading results. To resolve this issue, we investigate scale-invariant optimal subsampling in the context of variable selection, for which one main goal is to distinguish active and inactive features.

Penalty-based feature selection methods are widely used. Specifically, the adaptive lasso is a popular choice due to its oracle properties, convexity, and practical ease of implementation [see 30, 28]. While penalization methods have been used for bias reduction in rare-events analysis [7], variable selection for rare-events data has not been investigated. Conducting effective variable selection is difficult in the context of rare-events data analysis, mainly due to the scarcity of information available for ones. An inaccurate variable selection result can subsequently impact both the effectiveness of optimal subsampling and the efficiency of parameter estimation. In this paper, we address the challenge of variable selection in the context of rare-events data. First, we propose the full data adaptive lasso and study its theoretical properties. Next, we introduce a novel subsampling estimator that seamlessly combines penalty-based variable selection and optimal sampling into one unified framework for rare-events data. The implementation of the adaptive lasso requires a pilot estimator to construct

data-dependent weights for covariates. Given that optimal sampling also relies on pilot estimates [see 23, 1], the adaptive lasso emerges as a natural choice for conducting variable selection method in the context of subsampled rare-events data. We validate the new estimators by proving their oracle properties and also develop an efficient algorithm to facilitate their practical implementation when handling massive real-world data sets. In summary, our main contributions are listed as follows:

We propose scale-invariant optimal subsampling to enhance parameter estimation and variable selection. Existing optimal subsampling methods are scale-dependent, which may lead to unreliable or misleading results.

We define adaptive lasso and establish its oracle properties for rare-events data, which show that the asymptotic variances are determined by the number of ones in the data and the active features in the model.

Wepresent a practical subsampling algorithm based on optimal probabilities that significantly reduces the computational burden and accelerates the optimization for penalty-based feature selection methods.

The rest of the paper is organized as follows. Section 2 introduces the model setup. Section 3 investigates nonuniform sampling and variable selection tailored for rare-events data. We propose new methods to construct scale-invariant optimal probabilities. Section 4 discusses theoretical properties of the MSCL estimator and presents a two-step algorithm to implement the proposed methods. Section 5 conducts numerical experiments on simulated and real data sets. Section 6 concludes the paper. Proofs and mathematical details are presented in the appendix.

2 Background and model setup

We use the subscript t to indicate the true parameters. For a p -dimensional vector x , we use x ( i ) to represent its i -th element. For an index subset A ⊂ { i : 1 , 2 , ..., p } , we use x ( A ) to denote the subvector of x , whose elements correspond to the indexes in A . Furthermore, we use x ⊗ 2 to denote xx T , use ' ⇝ ' to denote convergence in distribution, use ' P -→ ' to denote convergence in probability, and use ' a.s. -→ ' to denote convergence almost surely. We use I to denote an identity matrix of a suitable dimension and use 0 to denote a vector of zeros of a suitable dimension.

Let ( x 1 , y 1 ) , ( x 2 , y 2 ) , ..., ( x N , y N ) denote N sample points from the joint distribution of ( x , y ) , where { x i } N i =1 denote the p -dimensional predictors and { y i } N i =1 the binary responses. Assume that the probability of y being a one ( y = 1 ) given x is

where θ t = ( α t , β T t ) T is the vector of true parameters and f ( x ; β t ) is a smooth function of β t . For rare-events data, N 1 ≪ N 0 , where N 1 = ∑ N i =1 y i is the number of ones (i.e. y i = 1 ) and N 0 = N -N 1 is the number of zeros (i.e. y i = 0 ). Following the model setup used in [22], we assume that α t →-∞ as N →∞ , which implies that, under appropriate moment conditions,

Under this assumption, the asymptotic variance of the full data maximum likelihood estimator (MLE) is of order 1 /N 1 instead of 1 /N , indicating that the estimation efficiency is determined by the number of rare ones. Therefore, we can keep all the ones and sample the zeros to save computational costs. There could be a variance inflation due to aggressive subsampling, and [22] developed optimal subsampling functions to reduce the variance inflation. Specifically, the authors proposed non-uniform optimal sampling functions under the A- and L-optimality criteria, respectively, as follows: φ scale A -OS ( x ) ∝ p ( x ; θ t ) ∥ M -1 ˙ g ( x ; θ t ) ∥ and φ scale L -OS ( x ) ∝ p ( x ; θ t ) ∥ ˙ g ( x ; θ t ) ∥ , where M = E { e f ( x ; β t ) ˙ g ⊗ 2 ( x ; θ t ) } and ˙ g ( x ; θ ) denotes the derivative of g ( x ; θ ) with respect to θ . However, the sampling functions φ scale A -OS ( x ) and φ scale L -OS ( x ) proposed in [22] depend on the scale of x , and may not perform well for certain measurement scale of x . For example, if g ( x ; θ t ) = α t + x T θ t , then φ scale L -OS ( x ) is proportional to 1 + ∥ x ∥ , which will be influenced by the scale of x . Similarly, scale

changes in x may also change φ scale A -OS ( x ) , although the impact may not be in the same direction, as demonstrated in Figure 1. Besides parameter estimation, variable selection is another important topic, which has not been studied in the literature on rare-events data. This work aims to fill this gap.

3 Nonuniform sampling with variable selection for rare-events data

The adaptive lasso [30, 28] is a popular variable selection method because it has oracle properties and is easy to implement. We define the full data adaptive lasso for rare-events data as

where λ N and γ are tuning parameters, and ˆ β pl is a consistent pilot estimator of β t . In practice, it is common to set γ = 1 . In the literature, iterative algorithms such as coordinate descent are commonly used to solve the adaptive lasso [8]. However, their computational demand can become prohibitive when dealing with massive data. It is feasible to alleviate the computational burden by subsampling zeros and create a smaller subset of data for adaptive lasso. To be specific, consider Algorithm 1.

Algorithm 1 Poisson Subsampling algorithm

The inclusion probability in Algorithm 1 for the i th observation is π ( x i , y i ) = y i +(1 -y i ) ρφ ( x i ) , where ρ is the baseline sampling rate for the zeros and φ ( x ) > 0 satisfies E { φ ( x ) } = 1 . Let the subsample from Algorithm 1 be { x sub i , y sub i } N ∗ sub i =1 , which is biased since π ( x i , y i ) 's depend on the responses. We introduce an Inverse Probability Weighting (IPW) adaptive lasso estimator to correct for the bias, defined as

̸

Theorem 1. Let ˆ β pl be a consistent pilot estimate such that λ N / ( √ N 1 | ˆ β pl( j ) | γ ) P -→ ∞ for j ∈ A c . Under Assumptions 1-4, if λ N / √ N 1 → 0 , then the IPW adaptive lasso estimator defined in (3) has the following properties:

To save space, we put the general assumptions used throughout this paper in Section B.1 of the appendix. We use A to denote the set of indexes of active variables, i.e., A = { j : β t( j ) = 0 } and A c to denote the set of indexes of inactive variables, i.e., A c = { j : β t( j ) = 0 } . We first study the asymptotic properties of ˆ θ adp w in the following theorem.

̸

Consistency in variable selection: The estimated active set ˆ A w := { j : ˆ β adp w( j ) = 0 } satisfies that lim N →∞ P ( ˆ A w = A ) = 1 .

Asymptotic normality: The estimator of the active parameter vector satisfies that

lim N →∞ e α t /ρ , and ˙ g ( A ) ( x ; θ t ) consists of the elements of gradient vector ˙ g ( x ; θ t ) with indexes in the active set A .

Remark 1. Theorem 1 shows that the estimation efficiency of ˆ θ adp w( A ) is predominantly determined by the number of ones instead of the full data size. The term c V sub( A ) is the variation inflation due to subsampling. The full data adaptive lasso in (2) correspond to the scenario with ρ = 1 and φ ( x ) = 1 , for which c = lim N →∞ e α t /ρ = 0 . Intuitively, c can be interpreted as the imbalance rate in the subsample. If we include sufficient zeros ( c = 0 ), the subsampling does not reduce the estimation efficiency of ˆ θ adp w( A ) .

̸

From Theorem 1, we see that there maybe information loss reflected as an inflated variance if c = 0 . To minimize the information loss due to sampling, we derive optimal functions as follows, where φ adp A -OS ( x ) corresponds to the A-optimality criterion [17] and φ adp L -OS ( x ) corresponds to the L-optimality criterion [17] in design of experiments. Here, the A-optimality minimizes the trace of the asymptotic variance of ˆ θ adp w( A ) ; the L-optimality focuses on the asymptotic variance of a linearly transformed estimator M ( A ) ˆ θ adp w( A ) , which is proportional to M w( A ) . The A-optimality criterion has a more direct interpretation, while an advantage of the L-optimality criterion is that the resulting optimal function is often faster to calculate.

Proposition 1. The A-optimal function that minimizes tr ( V w( A ) ) is

The L-optimal function that minimizes tr ( M w( A ) ) is

Unlike the optimal sampling function in [22], φ adp A -OS ( x ) (or φ adp L -OS ( x ) ) relies only on the active variables. This implies that a first-step pilot estimator given by the adaptive lasso algorithm can benefit from sparse estimation methods when calculating optimal probabilities. For example, employing the standard lasso can effectively eliminate a large number of inactive variables to facilitate the computation of optimal φ adp A -OS ( x ) and φ adp L -OS ( x ) . However, in practice, pilot estimators are often obtained from a small subsample size, introducing additional uncertainty. Therefore, it becomes crucial to exercise caution and be conservative by over-selecting variables during the first step to prevent the exclusion of important variables. As a consequence, although theoretically φ adp A -OS ( x ) and φ adp L -OS ( x ) do not depend on inactive variables, they are affected by inactive variables in practical implementations.

3.1 Scale invariant optimal function

As discussed in Section 1, scaling dependent optimal probabilities may impact the performance of variable selection in practice. To address the issue, we propose to construct a scale invariant optimal function by focusing on the prediction error of an estimator ˆ θ , defined below.

where P x is the probability measure of x . The probability term p ( x ; θ t ) involves both the covariates x and the parameter vector θ t , and it often does not depend on the scale of x . For example, in the logistic regression model, the value p ( x ; θ t ) is only related to x T β t . If we change the scale of x ( j ) , the value of θ t would change accordingly under the same data-generating model and so p ( x ; θ t ) remains the same. Thus, re-scaling covariates would not affect this criterion. In the following, we give an optimal function that minimizes the prediction error.

Theorem 2. Under the assumptions of Theorem 1, for the IPW adaptive lasso estimator defined in (3) , its prediction error satisfies

where Z ( A ) ∼ N ( 0 , I ) , and Ω ( A ) = E [ e 2 f ( x ; β t ) ˙ g ⊗ 2 ( A ) ( x , θ t ) ] . The optimal function that minimizes the asymptotic mean of the prediction error in (6) is given as

We refer this prediction oriented criterion as P-optimality criterion. As we expect, the optimal function in (7) is unaffected by the scale of x for a class of functions g . The following proposition proves that φ adp P -OS ( x ) is invariant to rescaling of x .

Proposition 2. If g ( x ; θ ) satisfies that for every non-singular matrix A there exists a non-singular matrix B , such that

then, φ adp P -OS ( x ) is invariant to scale changes of x .

Remark 2. The condition in (8) is not restrictive and it is quite easy to satisfy. One simple example of g ( x ; θ ) that satisfies the condition is a linear function g ( x ; θ ) = α + x T β , which corresponds to the logistic regression. The condition is also satisfied by more complex models. For example, consider an L -layer neural network

where W l are the weights and b l are the biases in each layer, l = 1 , 2 , ..., L . If x is rescaled to Ax , we can change W 1 to ( A T ) -1 W 1 so that the value of g does not change. That is

4 Penalized MSCL estimator

The IPW estimator in (3) is not the most efficient estimator, because it assigns smaller weights for more informative data points with larger sampling probabilities. To improve the estimation efficiency, we propose the penalized MSCL estimator for variable selection given as

where l sub i = -log { ρφ ( x sub i ) } . The MSCL estimator introduced in [22] is defined as the minimizer of the objective function in (9), excluding the penalization term. In this paper, we extend this approach by proposing a penalized MSCL estimator to ensure model sparsity. We present the oracle properties of the penalized MSCL estimator in the following theorem.

Theorem 3. Let ˆ β pl be a consistent pilot estimate such that λ N / ( √ N 1 | ˆ β pl( j ) | γ ) P -→ ∞ for j ∈ A c . Under Assumptions 1-3 and 5, if λ N / √ N 1 → 0 , the estimator based on MSCL function with adaptive lasso penalty defined in (9) have the following properties:

̸

Consistency in variable selection: The estimated active set ˆ A mscl := { j : ˆ β adp mscl( j ) = 0 } satisfies that lim N →∞ P ( ˆ A mscl = A ) = 1

Asymptotic normality: The estimator of the active parameter vector satisfies that

The penalized MSCL estimator has the same asymptotic variance as the MSCL estimator under the true model, indicating that it is more efficient than the penalized IPW estimator [22]. We prove this by comparing the asymptotic variances and present the result in the following theorem.

Theorem 4. If the asymptotic variances V w( A ) for ˆ θ adp w( A ) in (2) and V mscl( A ) for ˆ θ adp mscl( A ) in (9), are finite, i.e., 0 < V w( A ) , V mscl( A ) < ∞ , then V mscl( A ) ≤ V w( A ) , where the inequalities hold in the sense of Loewner ordering.

Thus, we give a practical two-step algorithm based on the penalized MSCL estimator. Since the optimal sampling functions contain unknown values and the adaptive lasso penalty also requires a consistent pilot estimator to build weights, it is natural to combine optimal sampling and the adaptive lasso into one unified framework. We recommend to use the lasso for pilot estimation. One reason is that it does estimation and variable selection simultaneously, and excluding some inactive variables improves the estimation accuracy of optimal probabilities. This also reduces the computational burden for subsequent steps. Another reason is that the lasso estimator tends to include more variables in practice and therefore has a low risk of excluding important variables in the pilot step. We present an outline of the practical implementation in Algorithm 2. More details are given in Section C.

Algorithm 2 Two-step subsampling adaptive lasso algorithm

1: First stage screening:

Take a pilot sample of expected sample size N pl using { π ( y i ) = ρ 0 + y i ( ρ 1 -ρ 0 ) } N i =1 and obtain a lasso penalized MSCL pilot estimator and an estimated active set ˆ A pl .

Calculate approximate optimal sampling probabilities { ˆ π ( x i , y i ) = y i + (1 -y i ) ρ ˆ φ ( x i ) } N i =1 based on (4), (5), or (7).

2: Second stage screening: Use Algorithm 1 with the estimated optimal sampling probabilities to obtain a subsample of expected sample size N sub and compute the adaptive lasso penalized MSCL estimator based on ˆ A pl .

5 Numerical experiments

In this section, we use numerical experiments on both simulated and real data to investigate the performances of our proposed optimal subsampling and variable selection procedures.

5.1 Simulation design

We consider a logistic regression with g ( x ; θ ) = α + x T β and the following three true parameters β t of dimension 50. We set different α t so that the proportion of ones is 0 . 005 :

Case A: β t = (0 . 75 , 0 . 75 , 0 T 7 , 0 . 75 , 0 , 0 . 75 , 0 . 75 , 0 T 37 ) T and α t = -5 . 8 .

Case B: β t = (3 , -2 , 0 T 7 , 0 . 85 , 0 , -0 . 75 , 0 T 38 ) T and α t = -6 . 2 .

Case C: β t = (3 , 2 , 0 T 7 , 0 . 85 , 0 T 40 ) T and α t = -7 . 5 .

Here, 0 d denotes the zero vector of dimension d . We use p A and p A c to denote the number of active and inactive variables, respectively, and assume that x is a normal random vector. The active components x ( A ,j ) , 1 ≤ j ≤ p A of x have variances 0.25 and the inactive components x ( A c ,j ) , 1 ≤ j ≤ p A c of x have variances 100 /p 3 A c , 100 / ( p A c -1) 3 , ..., 100 / 3 3 , 100 / 2 3 , 100 / 1 3 . The correlation between the i -th and j -th elements of x is 0 . 5 | i -j | , 1 ≤ i, j ≤ p . We repeat our experiments S = 500 times generating N = 500000 data points in each run and use a pilot sample of size N pl = 500 for obtaining pilot estimates based on the lasso. We consider uniform sampling, the full data lasso, and the full data adaptive lasso for comparison. We use the 5-fold cross-validation and Bayesian information criterion (BIC) to determine the tuning parameter λ for the lasso and the adaptive lasso, and choose γ = 1 for the adaptive lasso.

5.1.1 Estimation and prediction efficiency

We present the empirical median squared error (eMSE) for parameter estimation in Figure 2. All optimal sampling estimators outperform the uniform sampling. As the sampling rate increases, sampling estimators outperform the full data lasso estimator eventually in all of the three cases. Among the three optimal subsampling methods, ˆ β adp P -OS performs better than the other two subsampling methods.

0.004

0.006

0.008

0.010

sampling rate

4.5

4.0

3.5

3.0

log(eMSE)

(a) Case A

0.004

0.006

0.008

0.010

sampling rate

4.5

4.0

3.5

3.0

2.5

2.0

log(eMSE)

(b) Case B

0.004

0.006

0.008

0.010

sampling rate

5.0

4.5

4.0

3.5

3.0

2.5

log(eMSE)

(c) Case C

A-OS

L-OS

P-OS

Uni

A-lasso (full)

0.004

0.006

0.008

0.010

sampling rate

15.0

14.5

14.0

13.5

13.0

log(eMSPE)

(a) Case A

0.004

0.006

0.008

0.010

sampling rate

15.0

14.5

14.0

13.5

13.0

12.5

log(eMSPE)

(b) Case B

0.004

0.006

0.008

0.010

sampling rate

14.5

14.0

13.5

13.0

12.5

12.0

11.5

log(eMSPE)

(c) Case C

A-OS

L-OS

P-OS

Uni

A-lasso (full)

Figure 3 shows the results of the empirical median squared prediction error (eMSPE). Similarly to the results of eMSE, optimal sampling estimators perform better than the uniform sampling, meaning optimal sampling results in less information loss. It is possible that sampling estimators outperform the full data lasso estimator as the sampling rate increases, despite that the latter uses all of the data. In general, ˆ β adp P -OS performs the best among the three optimal subsampling algorithms.

5.1.2 Variable selection and computational complexity

In this section, we discuss the results of variable selection in terms of the first stage screening and the second stage screening. Table 1 presents the mean numbers of selected variables in Case C, where the numbers in the parentheses are the corresponding standard errors. Results for Cases A and B are similar so are put in Table 4 of the appendix.

While the first stage screening significantly reduces the dimension in Table 1, it indeed includes inactive variables as expected. In the second stage screening, the mean numbers of selected variables are close to the true numbers of active variables for all subsampling methods. However, the mean number of selected variables from uniform sampling is smaller than the true number of active variables especially when the sampling rate is low. This indicates that the second-stage screening of uniform sampling may exclude active variable. We present the rates of missing active variables in Table 2 for Case C. It shows that uniform sampling has higher rates of excluding active variables than optimal subsampling procedures, so optimal sampling may be preferable in practice. Results for Cases A and B are similar and are put in Section E.1. We also investgate the rates of selecting the true model in that section.

5.1.3 Computational time

We present the mean computational times of different algorithms in Table 3. Our codes are written in the julia programming language [2] and implemented on a Linux workstation. The lasso pathes are solved with Lasso.jl [13]. As shown in Table 3, subsampling algorithms significantly reduce the computational times compared with full data estimators. Although optimal sampling requires to calculate sampling probabilities, they use only about 0.77% of the computational time that the full data adaptive lasso requires. As we discussed in Section C.2, optimal sampling algorithms reduce both sample size and the data dimension. Therefore, the computational cost of the coordinate decent algorithm, which often requires a large number of iterations, is significantly reduced.

5.2 Real data

We evaluate the performances of proposed estimators on two real data sets.

(i) Covtype data set: It is available at https://archive.ics.uci.edu/ml/datasets/ covertype , with N = 581012 observations and 54 covariates - 10 being quantitative and 44 being qualitative with dummy coding. We drop the 14th and 54th columns to avoid exact colinearity of the dummy variables. Our goal is to classify whether the forest cover type is Cottonwood/Willow (labeled as 1) or not (labeled as 0). The proportion of Cottonwood/Willow is 0.473%, which is highly imbalanced.

(ii) Font data set: It is available at https://archive.ics.uci.edu/ml/datasets/ Character+Font+Images , with 0.50% of the N = 832670 responses being the GADUGI font. The first 10 covariates are about the value, size, and style of the characters and there are additional 400 pixel values of the 20 × 20 images. We remove the 4th, 9th, and 10th covariates because they are constants.

For both data sets, we apply Algorithm 2 on the logarithmic-transformed data. We use pilot samples of size N pl = 1000 for the covtype data and N pl = 1500 for the font data due to its higher dimension. Since we do not know the true parameter for real data, we use area under the curve (AUC) to measure the performances of subsampling algorithms. We repeat the experiment for S = 500 and compute the empirical median AUC using the full data. The results are summarized in Figure 4. As shown in Figure 4, nonuniform sampling outperforms uniform sampling in general. There is one case for font data set that ˆ β adp A -OS is worse than the uniform sampling when the sampling rate is high. For the covtype data set, among the three estimators based on optimal sampling, ˆ β adp P -OS performs the best and ˆ β adp L -OS is worst. For the font data set, ˆ β adp A -OS and ˆ β adp L -OS are similar, and ˆ β adp P -OS based on the scale invariant optimal sampling function is significantly better.

6 Conclusion and limitations

In this paper, we investigated the problem of scale-invariant optimal subsampling in the context of variable selection for rare-events data. We derived optimal probabilities based on the A- and L-

0.0050 0.0075 0.0100 0.0125 0.0150

sampling rate

0.9960

0.9961

0.9962

0.9963

0.9964

0.9965

AUC

0.0050

0.0075

0.0100

0.0125

0.0150

sampling rate

0.824

0.826

0.828

0.830

AUC

(a) AUCs of the covtype data set

(b) AUCs of the font data set

optimality criteria, and discussed their limitations. Furthermore, we proposed scale-invariant optimal probabilities based on prediction errors to overcome the limitations. Both analytical and numerical results show the desirable properties of the proposed methods.

Our investigation has the following limitations.

Our proposed criterion optimizes the probabilities by minimizing the asymptotic mean squared error in estimating rare-event probabilities. While this prioritizes the accuracy of estimation, it puts less emphasis on the quality of variable selection. Further research is needed to devise optimal probabilities that focus on variable selection performance metrics.

Our theoretical analysis is based on asymptotic properties, with optimal probabilities defined through the asymptotic normality. Although our results may hold for sufficiently sparse models, they may not generalize to cases where the model is dense or over-parameterized, because asymptotic normality may no longer be applicable. Therefore, an important direction for future research is to study the non-asymptotic properties of our estimators, such as prediction error bounds. Non-asymptotic behaviors are particularly of interest in highdimensional regimes.

We employ Lasso as the pilot estimator. However, other variable selection methodologies, such as sure independence screening, can also be considered. Exploring the impact of different pilot estimators on our method's performance represents another avenue for future investigations.

We assume that the underlying full model is correctly specified and possesses a sparse structure. Our analysis does not account for model misspecification. Further research is required to address scenarios where the model is possibly misspecified or where the number of features vastly exceeds the number of observations.

Acknowledgments and Disclosure of Funding

The authors are grateful to Professor Kun Chen for the insightful comments and suggestions on the development of the manuscript. Funding in direct support of this work: NEI grant R21EY035710, NSF grant 2105571, UConn CLAS Research Funding in Academic Themes, GPUs donated by NVIDIA.

References

M. Ai, J. Yu, H. Zhang, and H. Wang. Optimal subsampling algorithms for big data regressions. Statistica Sinica , 31(2):749-772, 2021.

J. Bezanson, A. Edelman, S. Karpinski, and V. B. Shah. Julia: A fresh approach to numerical computing. SIAM review , 59(1):65-98, 2017.

N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research , 16:321-357, 2002.

G. Douzas and F. Bacao. Self-organizing map oversampling (somo) for imbalanced data set learning. Expert systems with Applications , 82:40-52, 2017.

C. Drummond, R. C. Holte, et al. C4. 5, class imbalance, and cost sensitivity: why undersampling beats over-sampling. In Workshop on learning from imbalanced datasets II , volume 11, 2003.

J. Fan and J. Lv. Sure independence screening for ultrahigh dimensional feature space. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 70(5):849-911, 2008.

D. Firth. Bias reduction of maximum likelihood estimates. Biometrika , 80(1):27-38, 03 1993.

J. Friedman, T. Hastie, H. Höfling, and R. Tibshirani. Pathwise coordinate optimization. The Annals of Applied Statistics , 1(2):302 - 332, 2007.

J. H. Friedman, T. Hastie, and R. Tibshirani. Regularization paths for generalized linear models via coordinate descent. Journal of Statistical Software , 33(1):1-22, 2010.

W. Fu and K. Knight. Asymptotics for lasso-type estimators. The Annals of statistics , 28(5):13561378, 2000.

C. J. Geyer. On the asymptotics of constrained m-estimation. The Annals of statistics , pages 1993-2010, 1994.

H. Han, W.-Y. Wang, and B.-H. Mao. Borderline-smote: a new over-sampling method in imbalanced data sets learning. In International conference on intelligent computing , pages 878-887. Springer, 2005.

JuliaStats. Lasso.jl. https://github.com/JuliaStats/Lasso.jl , 2022.

N. Keret and M. Gorfine. Analyzing big ehr data-optimal cox regression subsampling procedure with rare events. Journal of the American Statistical Association , 118(544):2262-2275, 2023.

X.-Y. Liu, J. Wu, and Z.-H. Zhou. Exploratory undersampling for class-imbalance learning. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) , 39(2):539-550, 2008.

J. Mathew, C. K. Pang, M. Luo, and W. H. Leong. Classification of imbalanced data by oversampling in kernel space of support vector machines. IEEE transactions on neural networks and learning systems , 29(9):4065-4076, 2017.

F. Pukelsheim. Optimal design of experiments . SIAM, 2006.

J. Shao. Mathematical Statistics, 2nd . Springer-Verlag, New York, 2003.

G. Tripathi. A matrix extension of the cauchy-schwarz inequality. Economics Letters , 63(1):1-3, 1999.

H. Wang. Logistic regression for massive data with rare events. In International Conference on Machine Learning , pages 9829-9836. PMLR, 2020.

H. Wang and Y. Ma. Optimal subsampling for quantile regression in big data. Biometrika , 108(1):99-112, 2021.

H. Wang, A. Zhang, and C. Wang. Nonuniform negative sampling and log odds correction with rare events data. Advances in Neural Information Processing Systems , 34, 2021.

H. Wang, R. Zhu, and P. Ma. Optimal subsampling for large sample logistic regression. Journal of the American Statistical Association , 113(522):829-844, 2018.

J. Wang, J. Zou, and H. Wang. Sampling with replacement vs poisson sampling: a comparative study in optimal subsampling. IEEE Transactions on Information Theory , 68(10):6605-6630, 2022.

Y. Yao and H. Wang. Optimal subsampling for softmax regression. Statistical Papers , pages 585-599, 12 2018.

J. Yu, H. Wang, M. Ai, and H. Zhang. Optimal distributed subsampling for maximum quasilikelihood estimators with massive data. Journal of the American Statistical Association , 0(0):1-12, 2020. DOI:10.1080/01621459.2020.1773832.

G.-X. Yuan, C.-H. Ho, and C.-J. Lin. An improved glmnet for l1-regularized logistic regression. J. Mach. Learn. Res. , 13:1999-2030, 2012.

H. H. Zhang and W. Lu. Adaptive Lasso for Cox's proportional hazards model. Biometrika , 94(3):691-703, 05 2007.

T. Zhang, Y. Ning, and D. Ruppert. Optimal sampling for generalized linear models under measurement constraints. Journal of Computational and Graphical Statistics , 30(1):106-114, 2021.

H. Zou. The adaptive lasso and its oracle properties. Journal of the American Statistical Association , 101(476):1418-1429, 2006.

A Appendix / supplemental material

In this appendix, we present the details of the proof, the practical algorithm and simuation settings in the paper. Details of mathematical proofs are provided in Section B. Details of the practical algorithm are provided in Section C. We present the details of simulation settings in Section D and in Section E, we give some additional simulation results.

B Details of mathematical proofs

In this section, we provide details of mathematical proofs.

B.1 General assumptions in the main paper

We begin with some general assumptions used throughout this paper.

Assumption 1. The first, second and third derivatives of f ( x ; θ ) and e f ( x ; β ) f ( x ; β ) with respect to β are bouned by a square intergrable random variable B ( x ) .

Assumption 2. The matrix E { ˙ g ⊗ 2 ( x ; θ ) } is finite and positive definite.

Assumption 3. The subsampling rate ρ satisfies that c N = e α t /ρ → c , where 0 ≤ c < ∞ is a constant.

Assumption 4. The integral E [{ φ ( x ) + φ -1 ( x ) } B 2 ( x ) ] is finite, where B ( x ) is a squareintegrable function that dominates the first, second, and third derivatives of f ( x ; θ ) and e f ( x ; β ) f ( x ; β ) with respect to β .

Assumption 5. The integral E { e f ( x ; β ) φ -1 ( x ) B ( x ) } is finite.

These assumptions are the same assumptions used in [22]. Here, we remind some notations used in the main paper:

and

To ease the presentation in the following sections, we denote

and a N = √ Ne α t in the appendix. Note that

B.2 Proof of Theorem 1

Proof of Theorem 1. We consider the target of IPW adaptive lasso estimator:

where ˆ w j = 1 / | ˆ β pl( j ) | , 1 ≤ j ≤ p . Then, we have that ˆ u N = a N ( ˆ θ w -θ t ) is the minimizer of

Asymptotic normality: We prove the asymptotic normality part in this paragraph. By Taylor's expansion,

We first consider the limit behavior of the IPW target function by prove the asymptotic normality. In [22], the authors established that under Assumptions 1 to 3,

̸

and

Thus,

Next, we consider the limit behavior of the adaptive lasso penalty. Since we assume ˆ β pl to be a consistent estimator, we know that when j ∈ A , i.e., β t( j ) = 0 ,

and

Therefore, for j ∈ A , we have that

since λ N /a N = λ N / √ Ne α t → 0 . On the other hand, when j ∈ A c , i.e., β t( j ) = 0 , we have that for u ( j ) = 0 ,

̸

since λ N / ( √ Ne α t | ˆ β pl( j ) | γ ) P -→ ∞ . Then, we have that γ N w ( u ) ⇝ γ w ( u ) , where

Note that the unique minimizer of γ N w ( u ) is ( M -1 ( A ) M 1 / 2 w Z T ( A ) , 0 ) T if we put all the indexes of active variables in front. Thus, following the results of [11] and [10], we have the minimizer of γ N w ( u ) , i.e., ˆ u N , satisfies that

Thus,

Since applying Slusky's theorem, we have

Consistency in variable selection We prove the consistency in variable selection in this paragraph. From the result of asymptotic normality, we know that ˆ β w( j ) P -→ β t( j ) for every j ∈ A and therefore P ( j ∈ ˆ A w ) → 1 . Thus, we only consider j ′ ∈ A c . When j ′ ∈ ˆ A w , we know that by K-K-T optimality conditions, we have

We have known that I 1 = ˙ ℓ w ( θ t ) /a N ⇝ Z w . We now prove that proof that I 2 = O P (1) . We apply Taylor expansion to the k -th element of ˙ ℓ w ( ˆ θ w ) and have that

where, which means

and

where ´ θ k is between ˆ θ mle and θ t . First, we prove that ˜ R ( k ) is o P (1) . We have that

where

Therefore, we proved that ˜ R w( k ) = o P (1) . Next, we prove that ˜ ∆ w( k ) = o P (1) . We know that E [ a -2 N ∑ N i =1 δ i /π ( x i , y i ) { y i -p ( x i ; θ t ) } ¨ g ( x i ; θ t ) ] = 0 . We also have that for the every element of a -2 N ∑ N i =1 δ i /π ( x i , y i ) { y i -p ( x i ; θ t ) } ¨ g ( x i ; θ t ) , we have

Thus, due to Chebyshev's inequality, we know that ˜ ∆ w = o P (1) . Since we know that 1 a 2 N ∑ N i =1 δ i /π ( x i , y i ) ϕ ( x i ; θ t ) ˙ g ⊗ 2 ( x i ; θ t ) = H w = O P (1) . Hence, we have that

Thus, we prove the part of consistency of variable selection.

B.3 Proof of Proposition 1

We first give a lemma for general optimal functions.

Lemma 1. Assume that h ( x ) 2 and φ ( x ) are integrable function with E { φ ( x ) } = 1 . The optimal function φ ∗∗ ( x ) that minimize the value E { h 2 ( x ) φ ( x ) } is given as φ ∗∗ ( x ) = h ( x ) E { h ( x ) } .

Proof. Appying Cauchy-Schwartz inequality, we have that

Therefore, we have that E { h 2 ( x ) φ ( x ) } ≥ E { h ( x ) } 2 and the equality holds if and only if √ φ ( x ) = Kh ( x ) / √ φ ( x ) , where K is a constant. Therefore, φ ∗∗ ( x ) = Kh ( x ) , and since E { φ ∗∗ ( x ) } = 1 , we know that φ ∗∗ ( x ) = h ( x ) / E { h ( x ) } .

Now, we prove Proposition 1.

Proof. We first calculate the optimal function that minimizes tr ( V w( A ) ) . We have that

We focus on the values that related to φ ( x ) . We know that

Therefore, we need to minimize

Note that we also have

Therefore,

Appying Lemma 1. We know that the minimizer is given as

Next, we calculate the optimal function that minimize tr ( M w( A ) ) . We have that

Therefore, we need to minimize

Appying Lemma 1. We know that the minimizer is given as

B.4 Proof of Theorem 2

Proof. In the proof of Thereom 1, we know that

To simplify the representation, We define a function

We have that and

Note that | p ( x ; θ ) -p ( x ; θ t ) | ≤ 2 and thus,

and

Hence, due to donimating convergence theorem, we know that the expectation and derivitive are exchangable. Thus, we have that

and

Due to donimating convergence theorem, we also know that the first and second derivitive of h ( θ ) are continous. We have that

Note that e -2 α t E [ ϕ 2 ( x ; θ t ) ˙ g ⊗ 2 ( x ; θ t ) ] → Ω due to donimating covergence theorem. Now applying Theorem 1.12(ii) in [18], we have that

Considering N 1 = a 2 N E { e f ( x ; β t ) } { 1 + o P (1) } , applying Slutsky's theorem, we have that

Since Z w( A ) = N ( 0 , I ) , we have

We focus on the values that related to φ ( x ) . We know that

Therefore, we need to minimize

Appying Lemma 1. We know that the minimizer is given as

B.5 Proof of Proposition 2

Proof. First, we know that g ( x ; θ ) = g ( Ax ; B T θ ) . Since the equation holds for all x and θ , if we take derivitive with respect to θ on both sides, the equation still holds. Thus, we have that

If we scale the whole covariate variable x to ˜ x = Ax , we need to reparameterize θ t to ˜ θ t = B T θ t to remain the problem invariant. We have that for ˜ x and ˜ θ t

Now, we know that

and

Thus, we have that

Therefore, the leveraging term is invariant. For the probability term, we know that is only related to value g ( x ; θ t ) = g (˜ x ; ˜ θ t ) , we know that it does not change after scaling inactive variables. This complete the proof.

B.6 Proof of Theorem 3

Proof of Theorem 3. Consider maximum sampled conditional likeihood function with adaptive lasso penalty:

where ˆ w j = 1 / | ˆ β pl( j ) | γ , 1 ≤ j ≤ p . Then, we have that ˆ u N = a N ( ˆ θ ˆ θ pl mscl -θ t ) is the minimizer of

Asymptotic normality: We prove the asymptotic normality part in this paragraph. By Taylor's expansion,

First, we consider the limit behavior of the MSCL function. In [22], the authors proved that under Assumptions 1 and 3,

and

Thus,

̸

Next, we consider the limit behavior of the adaptive lasso penalty. Since we assume ˆ β pl tp be a consistent estimator, we know that when j ∈ A , i.e., β t( j ) = 0 ,

and

Therefore, for j ∈ A , we have that

since λ N /a N = λ N / √ Ne α t → 0 . On the other hand, when j ∈ A c , i.e., β t( j ) = 0 , we have that for u ( j ) = 0 ,

̸

since λ N / ( √ Ne α t | ˆ β pl( j ) | γ ) P -→ ∞ . Then, we have that γ ˆ θ pl mscl ( u ) ⇝ γ mscl ( u ) , where

Note that the unique minimizer of γ mscl ( u ) is (( Λ pl mscl ) -1 Z T ( A ) , 0 ) T if we put all the indexes of active variables in front. Thus, following the results of [11] and [10], we have the minimizer of γ ˆ θ pl mscl ( u ) , ˆ u N , satisfies that

Thus,

We know that

Hence, applying Slusky's theorem, we have

Therefore, we prove the part of aymptotic normality.

Consistency in variable selection We prove the consistency in variable selection in this paragraph. From the result of asymptotic normality, we know that ˆ β mscl( j ) P -→ β t( j ) for every j ∈ A and therefore P ( j ∈ ˆ A mscl ) → 1 . Thus, we only consider j ′ ∈ A c . When j ′ ∈ ˆ A mscl , we know that by K-K-T optimality conditions, we have

which means

We have known that I 1 = ˙ ℓ ˆ θ pl mscl ( θ t ) /a N ⇝ Z mscl . We now prove that proof that I 2 = O P (1) . We apply Taylor expansion to the k -th element of ˙ ℓ ˆ θ pl mscl ( ˆ θ ˆ θ pl mscl ) and have that

where, and

where ´ θ k is between ˆ θ mscl and θ t . First, we prove that ˜ R mscl( k ) is o P (1) . We have that

where

Therefore, we proved that ˜ R mscl( k ) = o P (1) . Next, we prove that ˜ ∆ mscl( k ) = o P (1) . We know that E [ a -2 N ∑ N i =1 δ ˆ θ pl i { y i -p ˆ θ pl π ( x i ; θ t ) } ¨ g ( x i ; θ t ) ∣ ∣ ∣ ˆ θ pl ] = 0 . We also have that for the every element of a -2 N ∑ N i =1 { y i -p ( x i ; θ t ) } ¨ g ( x i ; θ t ) , we have

Thus, due to Chebyshev's inequality, we know that ˜ ∆ ˆ θ pl mscl = o P (1) . Since we know that 1 a 2 N ∑ N i =1 δ ˆ θ pl i ϕ ˆ θ pl π ( x i ; θ t ) ˙ g ⊗ 2 ( x i ; θ t ) = O P (1) . Hence, we have that

Note that we also have

Therefore,

Thus, we prove the part of consistency of variable selection.

B.7 Proof of Theorem 4

Proof. Letting h = 1 + c { φ ( x ) } -1 e f ( x ; β t ) , v = √ e f ( x ; β t ) ˙ g ( A ) ( x ; ˜ θ ) , f = h 1 2 v , and g = h -1 2 v , we have that

and

Now, applying the matrix form of Cauchy-Schwartz's inequality (see [19]), we have that

Therefore, simple algebra shows that

which complete the proof

C Details about the practical algorithm and its complexity

C.1 Two-step algorithm

We take a pilot sample by uniform sampling with the sampling rate ρ 1 = N pl / 2 N 1 for the ones and ρ 0 = N pl / 2 N 0 for the zeros. Denote a pilot sample of actual sample size N ∗ pl as { ( x pl i , y pl i ) } N ∗ pl i =1 , the pilot estimate of θ as ˆ θ pl , and the pilot estimate of the active set as ˆ A pl = { j : ˆ β pl( j ) = 0 } . We propose the following moment estimators of M ( A ) and Ω ( A ) :

̸

respectively. We also use the following moment estimator to estimate the denominator of (4):

where ω A -OS i = p ( x pl i ; ˆ θ pl ) ∥ ( ˆ M pl ( ˆ A pl ) ) -1 ˙ g ( ˆ A pl ) ( x pl i ; ˆ θ pl ) ∥ . If using (5) or (7), we use

respectively, instead of ω A -OS i in (13). Now, we present the proposed two-step procedure in Algorithm 3 with more details than Algorithm 2 in Section 4.

Algorithm 3 Subsampling adaptive lasso algorithm

1: · Take a pilot sample { ( x pl i , y pl i ) } N ∗ pl i =1 of expected sample size N pl using { π ( y i ) = ρ 0 + y i ( ρ 1 -ρ 0 ) } N i =1 and obtain a pilot estimator

where N ∗ pl is the actual pilot sample size and l = log( N 0 /N 1 ) . We call this first stage screening.

Calculate approximate optimal sampling probabilities { ˆ π ( x i , y i ) = y i + (1 -y i ) ρ ˆ φ ( x i ) } N i =1 by replacing ˆ φ ( x i ) with φ adp A -OS ( x i ; ˆ θ pl ) , φ adp L -OS ( x i ; ˆ θ pl ) , or φ adp P -OS ( x i ; ˆ θ pl ) , based on (4), (5), or (7), respectively. The denominator of (4) is estimated using (13), and we replace ω A -OS i with ω L -OS i or ω P -OS i for the denominator of (5) or (7), respectively. If using π adp A -OS ( x ) or π adp P -OS ( x ) , estimate M ( A ) and Ω ( A ) using the moment estimators in (11) and (12), respectively.

2: Use Algorithm 1 with the estimated optimal sampling probabilities to obtain a subsample { ( x sub i , y sub i ) } N ∗ sub i =1 and compute the adaptive lasso estimator:

where N ∗ sub is the actual subsample size, based on the smaller model obtain from the first stage screening. We call this step the second stage screening.

We consider a coordinate desent method to calculate the estimators defined in Algorithm 3. (see [8], [9] and [27]). In each cycle, we need to find an optimal direction d at a starting point ˜ θ . We consider the quardratic approximation of Q ˆ θ pl mscl ( ˜ θ + d ) -Q ˆ θ pl mscl ( ˜ θ ) , which is where

∑ N i =1 δ ˆ θ pl i ϕ ˆ θ pl π ( x i , ˜ θ ) ˙ g ⊗ 2 ( x i ; ˜ θ ) . Thus, using coordinate desent to obtain the optimal direction, the quadratic approximation for the j -th element is given as

Then, we have the value of z that minimize Q ˆ θ pl mscl ( d + z e j ) -Q ˆ θ pl mscl ( d ) is

which is the same as where

and

For the special form of g ( x ; θ ) = α + f ( x T β ) , we know that

where

and Φ = diag { δ ˆ θ pl i ϕ ˆ θ pl π ( x i , ˜ θ ) } . Thus, we have

Therefore, we can store Gd and keep updating Gd with

Thus, we do not need to obtain the full matrix ¨ ℓ mscl ( ˜ θ ) = G T Φ G . We only need to calculate the diagnoal elements: ¨ ℓ mscl( jj ) ( ˜ θ ) = G T ( j ) Φ G ( j ) , j = 1 , ..., p + 1 and { ˙ ℓ mscl ( ˜ θ ) d } ( j ) = ( Gd ) T Φ G ( j ) , j = 1 , ..., p + 1 . From the analysis above, we can notice that the computaional complexity of one cycle calculating optimal direction d is O ( ζ in Np ) , where ζ in denotes the number of inner iteration.

C.2 Computational complexity

We analyze the computational complexity of the two-step algorithm. To facilitate the presentation, we consider a special case for our model when g ( x ; θ ) = α + f ( x T β ) , and assume that the number of variables selected at the first-stage screening is q . Coordinate descent is a widely used optimization algorithm for solving lasso and adaptive lasso [see 9]. We consider the improved coordinate descent algorithm proposed in [27], which requires inner iterations to determine an optimal direction and outer iterations to update the estimator. Considering the form of g ( x ; θ ) = α + f ( x T β ) , the computational complexity for coordinate descent with data of size N and dimension p is O ( ζ in Np ) per innercycle where ζ in represents the number of inner iterations (detailed derivations of this complexity is presented Section C). Thus, the computational complexity of full data lasso is O ( ζ out × in Np ) , where ζ out × in = ζ out ζ in and ζ out is the number of outer iterations. The computational complexity of the full data adaptive lasso is O ( ζ mle pl Np 2 + ζ out × in Np ) with the MLE as the pilot estimator, and O ( ζ las pl , out × in Np + ζ out × in Nq ) with lasso as the pilot estimator, where ζ MLE pl and ζ las pl , out × in are the iteration numbers in the two pilot estimators, respectively. The coordinate descent algorithm often requires a large ζ out × in or ζ las pl , out × in while Newton's algorithm requires a small ζ mle pl , so it is often the case that ζ mle pl p < ζ out × in . Therefore, the time complexity of the adaptive lasso is O ( ζ out × in Np ) , which is the same as the full data lasso estimator.

Now, we analyze the time complexity of Algorithm 3. We start with the computational complexity of the optimal probabilities, for which the main computational cost is to approximate ∥ M -1 ( A ) ˙ g ( A ) ( x i ; θ ) ∥ or ∥ Ω 1 / 2 ( A ) M -1 ( A ) ˙ g ( A ) ( x i ; θ ) ∥ , respectively, for i = 1 , ..., N . Since ˙ g ( ˆ A pl ) ( x i ; θ ) = (1 , ˙ f ( x T i β ) x T i ( ˆ A pl ) ) T , the computational complexity of calculating ˙ g ( ˆ A pl ) ( x i ; θ ) 's is O ( Nq ) , and the computational complexity of ˆ M pl ( ˆ A pl ) or ˆ Ω pl ( ˆ A pl ) is O { N pl ( q +1) 2 } = O ( N pl q 2 ) . Taking the inverse ( ˆ M pl ( ˆ A pl ) ) -1 and finding the square root ( ˆ Ω pl ( ˆ A pl ) ) 1 / 2 both take O ( q 3 ) time. Thus, the computational complexity of calculating ∥ ( ˆ M pl ( ˆ A pl ) ) -1 ˙ g ( ˆ A pl ) ( x i ; θ ) ∥ 's or ∥ ( ˆ Ω pl ( ˆ A pl ) ) 1 / 2 ( ˆ M pl ( ˆ A pl ) ) -1 ˙ g ( ˆ A pl ) ( x i ; θ ) ∥ 's is O ( Nq + N pl q 2 + q 3 + Nq 2 ) = O ( Nq 2 ) . Therefore, the complexity of approximating the optimal probabilities in (4) or (7) is O ( Nq 2 ) . The computational complexity of approximating the optimal probabilities in (5) is only O ( Nq ) , because there is no need to compute ˆ M pl ( ˆ A pl ) or ˆ Ω pl ( ˆ A pl ) . Next, we analyze the complexity of parameter estimation. The average subsample size with the sampling rate ρ is on average

Using the coordinate descent algorithm, the computational complexity of the two-step algorithm is O { ζ las pl , out × in N pl p + Nq 2 + ζ out × in N ( e α t + ρ ) q } using optimal probabilities in (4) or (7), and it is O { ζ las pl , out × in N pl p + Nq + ζ out × in N ( e α t + ρ ) q } using the optimal probabilities in (5). For optimal probabilities in (4) or (7), when ζ out × in > q/ ( e α + ρ ) , the dominating term of the complexity is ζ out × in N ( e α t + ρ ) q . Remember that ζ out × in = ζ out ζ in and ζ out is usually large for the coordinate descent algorithm. Therefore, ζ out × in > q/ ( e α + ρ ) is often satisfied in practice. The dominating term for the time complexity of optimal probabilities in (5) is also ζ out × in N ( e α t + ρ ) q . Compared with full data estimators, both the sample size and the dimension are reduced. If we set the subsample size to be the same order of N 1 , which is often the case in practice for balancing the ones and zeros, the time complexity of Algorithm 2 is of order O ( ζ out × in Ne α t q ) , which is significantly faster than that of the full data estimator.

D Detalis of simulation settings

In this section, we present more details of the simulation settings in the main paper. In Section D.1, we provide detailed simulation settings of the example in Section 1. In Section D.2, we present detailed settings in Section 5.

D.1 Simulation in Section 1

We first present the detailed settings of the simulations in Section 1, where we illustrate the scaledependent issues of optimal subsampling probabilities. Our simulation based on logistic regression models with the true parameter β t to be 6-dimentional vectors and covariates x ∼ lognormal ( 0 , Σ ) with the ( i, j ) -th element of Σ is given as Σ ij = 0 . 5 | i -j | , 1 ≤ i, j ≤ 6 . We consider two cases of parameters:

(a) Non-sparse parameter : β = ( -1 , -1 , -0 . 01 , -0 . 01 , -0 . 01 , -0 . 01) and α = -4 .

(b) Sparse parameter : β = ( -1 , 0 , 0 , 0 , 0 , 0) and α = -5

We generate full data of size N = 500000 according to the above logistic models. To investigate the effects of scale transformation, we multiply the x (6) with s ( s = 0 . 01 , 0 . 1 , 1 , 10 , 100 ) and divide β t(6) with the same s to remain x T β t to be the same and thus the logistics regression model does not change. We obtain subsamples with optimal subsampling probabilities described in [22] with transformed x under each s and calculate the resultant subsampling estimators. We set the nominal pilot sample size to N pl = 800 and nominal subsample size N sub = 1000 (see details in [22]). We repeat the experiment for 500 times under each scale and compute the mean prediction error.

D.2 Simulations in Section 5

For the estimation procedures in the second step of our two-step algorithm, we choose γ = 1 , which means that the weights in the adaptive lasso penalty are ˆ w j = 1 / | ˆ β pl( j ) | , 1 ≤ j ≤ q , with q being the number of selected variables in the first stage screening. Furthermore, we consider uniform sampling, the full data lasso and the full data adaptive lasso as baselines for comparison. For the uniform sampling method, we use a similar two-step algorithm as presented in Algorithm 2 but set the sampling function in the second step as φ ( x ) = 1 , which means the sampling probabilities are a constant ρ . We use lasso to implement the first stage screening and adaptive lasso with γ = 1 to implement the second stage screening for a fair comparison. For full data lasso, we directly apply the lasso algorithm to the full data. For the full data adaptive lasso, we use the full data MLE estimator as the pilot estimator to construct the weights and then apply the adaptive lasso algorithm to the full data set.

E Additional simuations

In this Section, we give some addtional simulation results. More simulation results of variable selection is provided in Section E.1. We also provide addtional simulation results to compare our approach with standardization to resolve scale dependent issues in Section E.2.

E.1 Addtional variable selection results

It is seen in Table 6 that, no subsampling method dominates others. Table 2 and Table 5 shows that uniform sampling has higher rates of excluding active variables than optimal subsampling procedures.

Although uniform sampling may have a higher rate of selecting the true model in some cases, given that it is more likely to exclude important variables, optimal sampling may be preferable in practice.

E.2 Comparison with standardization

Another approach to avoid scale-dependency is to standardize the data. We compare the proposed scale-independent optimal probabilities with the approach of data standardization here. For the data standardization approach, we standardize the data, calcualte the optimal probabilities, and then implement subsampled adpative lasso algorithm. We used the same pilot estimation methods for fair comparisons.

We first compare the eMSE and eMSPE in Figure 5 and Figure 6, respectively. We use sP-OS to denote the approach with data standardization and use P-OS to denote the approach without data standardization.

0.004

0.006

0.008

0.010

sampling rate

3.5

3.4

3.3

3.2

3.1

3.0

2.9

log(eMSE)

P-OS

sP-OS

(a) Case A

0.004

0.006

0.008

0.010

sampling rate

3.4

3.2

3.0

2.8

log(eMSE)

P-OS

sP-OS

(b) Case B

0.004

0.006

0.008

0.010

sampling rate

4.2

4.0

3.8

3.6

3.4

log(eMSE)

P-OS

sP-OS

(c) Case C

In Figure 5, we notice that the performances of P-OS and sP-OS are similar, this is also true for eMSPE. However, standardization may decrease the rate of selecting the true model. We present results of variable selection in Table 7 and Table 8.

0.004

0.006

0.008

0.010

sampling rate

14.4

14.3

14.2

14.1

14.0

13.9

13.8

log(eMSPE)

P-OS

sP-OS

(a) Case A

0.004

0.006

0.008

0.010

sampling rate

13.9

13.8

13.7

13.6

13.5

13.4

13.3

log(eMSPE)

P-OS

sP-OS

(b) Case B

0.004

0.006

0.008

0.010

sampling rate

13.6

13.4

13.2

13.0

log(eMSPE)

P-OS

sP-OS

(c) Case C

We notice in Table 7 and Table 8 that the rates of selecting true models by ˆ β adp P -OS is higher than ˆ β adp sP -OS without much increase on the rates of excluding active variables. Therefore, although standardization is an approach to solve the scale-dependency issues, it may decrease the rates of selecting true models in practice.

NeurIPS Paper Checklist

[Yes] [No] [NA]

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In Section 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Detailed proofs and required assumptions are provided in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All details for numerical experiments are provided in Section 5 and in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Codes are submitted as supplement for anonymity. They will be released in a public github repository after the review period.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: In Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We provide standard errors in Tables 1 and 2. We perform a large number of repetitions of the simulation experiments to calculate the empirical median squared error, so error bars are not relevant in Figures 1, 2, 3, and 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: In Section 5.1.3.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper is theoretical research.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: URLs for real data provided in Section 5.2.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of learning a sparse model under rare events and scale-invariance. They provide optimal subsampling function to handle scale-invariance for a Lasso-regularized model. They finally provide experimental results for their subsampling scheme.\n\nSTRENGTHS:\n* Well-written\n* Looking at MSPE rather than asymptotic variance seems like a good way to get around the scaling issue.\n\nWEAKNESSES:\n* This problem setting feels very specific: rare events data, sparse models, and scale-invariance. It would be good to motivate why we should care about all of these simultaneously.\n* None of the theoretical results given in the main body seem to have any discussion about proof strategy or novel techniques. It would be interesting to know what technical challenges were posed in developing these results, as well as how they were resolved.\n* In section 4, the paper compares the proposed penalized MSCL estimator to the MSCL estimator. As far as I can tell, this MSCL estimator has not been defined in the paper and so these comparisons and discussions are hard to make sense of.\n\nQUESTIONS:\n* In Proposition 1, why is minimizing $tr(M_{w(A)})$ of interest? It seems like all one would care about is minimizing the asymptotic variance (which as far as I understand is $V_{w(A)}$).\n* In the preamble of section 3.1 (lines 156-158), the MSPE is motivated by the desire for a scale-invariant objective. Is the problem with the previous approach(es) that scaling $x_j$ would inversely scale the corresponding entries of $V_{w(A)}$?\n* In the preamble of section 4 (lines 183 and 184) that discusses the inefficiency of the IPW estimator, the paper says IPW gives less weight to more informative points. Is this a formal (or at least, formalizable) claim, or is it just a heuristic? Either way, can this be elaborated upon?\n* The optimality of this work seems to be the optimal subsampling function towards the objective of MSPE. Is it possible to argue that this estimator is information-theoretically the best that one can do?\n\nLIMITATIONS:\nThe authors addressed their limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe scale-invariant optimal subsampling function proposed in the paper addresses the challenge of inactive features in rare-events data by overcoming the issue of scale-dependence in exitsing optimal subsampling methods. In the context of variable selection for rare-events data, where distinguishing active and inactive features is very important, the scale-invariant optimal subsampling function minimizes prediction error regardless of scaling transformations applied to inactive features.\n\nThe key is to provide reliable and unbiased subsampling probabilities even when inactive variables are transformed without altering the underlying model. This is important because inappropriate scaling transformations of inactive features can significantly impact traditional optimal subsampling methods, leading to unreliable or misleading results. This resluts in improving the accuracy of parameter estimation and variable selection.\n\nSTRENGTHS:\nGood novelty:The paper introduces a novel scale-invariant optimal subsampling function to address the challenge of inactive features in rare-events data, offering a new approach to improving parameter estimation and variable selection in sparse moedls. This novel contribution could advance the field of rare-events data analysis.\n\nStrong theory:The paper establishes a theoretical foundation for the proposed scale-invariant optimal subsampling function, including discussions on assumptions, proofs, and the mathematical framework supporting the method. This strong theoretical basis enhances the credibility and rigor of the proposed approach. WHile i have not gone through all the proofs, i think  the results seem sound.\n\nSufficient empirical evidence: Further, the paper demonstrates the practical relevance of the scale-invariant optimal subsampling function through numerical experiments using both simulated and real-world datasets.The results show non-trivial improvements in prediction error minimization and estimation efficiency.\n\nClarity: The paper is largely well-written,, provides clear and detailed explanations of the methodology, assumptions, and implications of the scale-invariant optimal subsampling function, making it accessible even though there is a lot of notation and theory.\n\nWEAKNESSES:\nI think this is a good paper that should be accepted. Minor gripe:\n\nIt is not clear to me how exactly the methods in section 1 are affected due to x (and thereby making them NOT scale invariant). I understand this would require going through the earlier papers e..g [21], which i did not do. It would make the paper complete if the authors could adda discussion to this effect in the appendix in the final version.\n\nQUESTIONS:\nWhy is the tr(variance) sufficient ot be minimized in the presented results ? In general most of the discussion revolves around assymptotic results, is it interesting to ask questions about non-assymptotic behaviors, such as how would performance of variable selection impact non-assymptotic prediction error? Further, many practical applications could be overparametrized/high-dimensional, how would the results carry over? If not, what would be complications that arise while generalizing them to such settings?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}

comments: SUMMARY:
This paper introduces an optimal subsampling algorithm designed for imbalanced data with inactive features. The primary algorithm can be summarized in three steps:

1. Train a pilot model using a lasso-penalized objective.
2. Subsample the data based on the pilot model.
3. Train the subsampled data using an adaptive lasso-penalized algorithm.

Additionally, the paper proposes two options for the model estimator: the IPW estimator and the more accurate MSCL estimator. Empirical results demonstrate the algorithm's effectiveness, showing that it outperforms the baseline estimator which does not apply variable selection.

STRENGTHS:
1. The paper addresses a key drawback of a previous "optimal estimator" for rare-events data by incorporating a lasso-based estimator.
2. The proposed algorithm is theoretically sound with solid asymptotic guarantees.
3. Empirical results on both synthetic and real data highlight the algorithm's effectiveness.

WEAKNESSES:
No technical weaknesses were observed in this paper.

QUESTIONS:
In the first stage screening of Algorithm 2, we use the Lasso estimator to manage computational costs and noise in the sampled data. If more computational resources are available, is Lasso always the best practical choice for pilot estimation?

LIMITATIONS:
No significant limitations were observed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-15