Nonuniform Negative Sampling and Log Odds Correction with Rare Events Data
Authors: HaiYing Wang
Aonan Zhang
Chong Wang
Venue: NeurIPS 2021 Poster (2021)
Keywords: Imbalanced data
Rare Events Data
Optimal Sampling

Abstract: We investigate the issue of parameter estimation with nonuniform negative sampling for imbalanced data. We first prove that, with imbalanced data, the available information about unknown parameters is only tied to the relatively small number of positive instances, which justifies the usage of negative sampling. However, if the negative instances are subsampled to the same level of the positive cases, there is information loss. To maintain more information, we derive the asymptotic distribution of a general inverse probability weighted (IPW) estimator and obtain the optimal sampling probability that minimizes its variance. To further improve the estimation efficiency over the IPW method, we propose a likelihood-based estimator by correcting log odds for the sampled data and prove that the improved estimator has the smallest asymptotic variance among a large class of estimators. It is also more robust to pilot misspecification. We validate our approach on simulated data as well as a real click-through rate dataset with more than 0.3 trillion instances, collected over a period of a month. Both theoretical and empirical results demonstrate the effectiveness of our method.

Nonuniform Negative Sampling and Log Odds Correction with Rare Events Data
HaiYing Wang Department of Statistics University of Connecticut haiying. wang@uconn. edu
Aonan Zhang ByteDance Inc. aonan www..aabtttdddanee.coo
Chong Wang
ByteDance Inc.
chong. wang@bytedance.com
Abstract
We investigate the issue of parameter estimation with nonuniform negative sam- pling for imbalanced data. We first prove that, with imbalanced data, the available information about unknown parameters is only tied to the relatively small number positive instances, which justifies the usage of negative sampling. However, if the negative instances subsampled the same level of the positive cases, there is information loss. To maintain more information, we derive the asymptotic distribution of a general inverse probability weighted (IPW) estimator and obtain the optimal sampling probability that minimizes its variance. To further improve the IPW method, propose likelihood-based
the estimation efficiency estimator by correcting log odds for the sampled data and prove that the improved estimator has the smallest asymptotic variance among large class of estimators. It is also more robust to pilot misspecification. We validate our approach on simu- lated data as well as real click-through rate dataset with more than 0.3 trillion instances, collected over period of month. Both theoretical and empirical results demonstrate the effectiveness of our method.
Introduction
Imbalanced data are ubiquitous in scientific fields and applications with binary response outputs, where the number of instances in the positive class is much smaller than that in the negative class. For example, in online search or recommendation systems where billions of impressions appear each day, non-click impressions usually dominate. Using these non-click impressions as negative instances is prevalent and proved to be useful especially in modern machine learning systems [21, 12, 4. 9, 14]. common approach address imbalanced data to balance by subsampling the negative class [6. 19] and/or up-sampling the positive class [2, 10, 20, 5], and great deal of attentions have been drawn this problem, see [15, 17, 3, 7, 23, 27, 25, 8, 18, 30]. and the references therein.
In this paper, we focus on subsampling the negative class, i.e., negative sampling, while keeping all rare positive instances. This is beneficial for modern online learning systems, where negative sampling significantly reduces data volume distributed storage while saving training time for multiple downstream models. Rare events data and negative sampling are studied in [30], but it focused on linear logistic regression and only considered uniform sampling. We approach this problem under general binary response models from the perspective of optimal subsampling, which aims to minimize the asymptotic variance of the IPW estimator [32, 28, 31, 33]. This topic was not well investigated neither for imbalanced data nor for negative sampling. In addition, existing optimal
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

probability formulation minimizes the conditional asymptotic variance and the variations due to the data randommess are ignored. We fill these gaps by deriving the asymptotic distributions for both full data and subsample estimators under general binary response models, and we find the unconditional optimal probability for negative sampling. In addition, since IPW may inflate the variance [13, 22], we develop a more efficient likelihood-bart based estimator through nonuniform log odds correction to avoid IPW. Our main contributions are summarized as follows.
Under a general binary response model with rare events data, we find that the difference between the full data estimator and the true parameter converges normal distribution at a rate that is tied to the number of rare positive instances only. This indicates the possibility to throw away the majority of negative instances without information loss, ie, it justifies the usage of negative sampling.
We show that there no asymptotic information loss with aggressive negative sampling if the negative instances kept dominates the positive instances. However there is information loss expressed as a variance inflation if the negative instances are brought down to the same level of the positive instances. For this case, obtain optimal subsampling probabilities by minimizing the unconditional asymptotic variance of a general IPW estimator.
We develop a likelihood estimator through nonuniform log odds correction for sampled data, and prove that it has the smallest asymptotic variance among a large class of estimators.
We apply the proposed method to a real online streaming click-through rate (CTR) dataset with more than 0.3 trillion instances and demonstrate its effectiveness.
The rest of paper is organized as follows. Section 2 defines the problem this paper focuses on and shows the full data results. Section 3 presents the optimal probability for the IPW estimator. Section proposes the likelihood- baaseedd--badd estimator and establishes its asymptotic optimality. Section discusses practical implementation and the theoretical properties of the resulting estimators. Section 6 presents numerical experiments with simulated data, and applies the proposed method to real online streaming CTR dataset with more than 0.3 trillion instances. Section 7 concludes the paper and points out limitations of our investigation. Proofs and additional experiments with misspecifications are in the supplementary materials.
2 Problem setup and full data results
Let {(x; M:) be training data of size N that satisfies the binary response model,
Pr(y x) p(x; 0),
(1)
where y E {0, 1) is the binary class label (y for the positive and y = 0 for the negative). x is the vector of features, and 0 is the unknown d-dimensional parameter.
Let N₁ be the number of positive instances (N1 = =E vi) and No be the number of negative instances (N, = N N1). We consider the scenario of massive imbalanced data, i.e., N₁ < Ng- For asymptotic investigations when N, is much smaller than No. it is more appropriate to assume that N2 increases a slower rate compared with No. i.e., / - 0 in probability as N - oo [30]. This requires Pr(y- 1) = E[p(x; })} + 0 as N - co on the model side. We assume that the parameter 0 contains two components 0 = (a,, FT and the log odds can be written as
g(x: 0) = log { - p(: 0) = f(x;
p(x; 0)
a
where the true parameter, denoted as 0" = (a*, (a*, satisfies that a - -00 as N + oc and 3 is fixed. Here f(x; (3) is a smooth function of (3, such as a linear function or nonlinear neural network model. If it is linear, the model reduces to the logistic regression model. A diverging a* and fixed 3 indicate that both the marginal and conditional probabilities for a positive instance are small. Heuristically, this implies that a change in the feature value does make a rare event become a large probability event. We also allow 3 to change with N, but long f has a finite limit, the situation is essentially the same as a fixed 3*. So here we assume 3 to be fixed to simplify the presentation.
Throughout this paper, we use (x;; 0) and (xx; 0) to denote the gradient and Hessian of g(x; f) with respect to (w.r.t) 0, respectively. Let ||v|| be the Frobenius norm and 02 vvT for a vector or

---

matrix v. We denote [x, y) as the sampling probability for an instance (x, y). For negative cases, we sometimes use a shorthand notation (xx) denote its sampling probability. We use (1( to represent random quantity that converges to 0 in probability as N 00.
2.1 It is the number of positive instances that matters
We show that for rare events data, the estimation error rate for 0 related to N instead of N. Using the full data, a commonly used estimator of 0 is the maximum likelihood estimator (MLE)
0: arg max Σ
[x:g(x;;)) 0) log {1 cs(x;:0))]
To investigate the theoretical properties of the estimator 0;, make the following assumptions.
Assumption The first, second, and third derivatives of f(x; (3) and ef(x;)) f(x; (3) w.r.t any components of B are bounded by square intergrable random variable (xx).
Assumption 2. The matrix { (x: )²) is finite and positive definite.
Assumption imposes constraints on the smoothness of f(x; 0) w.r.t and the tightness of the distribution of x. Assumption 2 ensures that all components of 0 are estimable.
Theorem 1. Let VF E(e (x::8) MM,' where Mr B (x20))))))))))))))))))))))) g0 )²) Under Assump- tions and 2, as N + 00,
(0: 0") - N(O, Vi), in distribution.
This result shows that even if all the N instances are used for model training, the resulting estimator O converges to the true parameter 0" at the rate of NT which is much slower than the usual rate of N-1/2 for regular cases. This indicates that with rare events data, the available information about unknown parameters
the scale of N1. Although N - oo much faster than Nj - oo (in terms of NN/N - 0), N does not explicitly show in the asymptotic normality result of Theorem 1. For the specific case of linear logistic regression with gx; 0) a + x² 3, our Theorem reduces to Theorem of [30].
2.2 Negative sampling
Since the available information ties to the number of positive instances instead the full data size, one can keep all the positive instances and significantly subsmaple the negative instances to reduce the computational cost. For better estimation efficiency, we consider nonuniform sampling. Let xp(x) > ( be integrable function and p be the sampling rate the negative class. Without loss of generality, assume Ef (xx) so that (x;) := (x,) is the sampling probability for the i-th data point if y, 1. We present the nonuniform negative sampling procedure in Algorithm 1.
Algorithm 1 Negative sampling
Vi, For i 1: if = 1, include {x,, Vi, क (x, yi) 1} in the sample;
y/ calculate xx(x;) and generate u, U(00, 1); if uf, S pp(x;), include (x;, Us. (X Us) = px((x;) } in the sample.
Weighted estimation and its optimal negative sampling probability
Let = 1 if the i-th data point is selected and 6, O otherwise. Given subsample taken according to क (x,, yi) y/ + ys (x,,), 1,... N, the IPW estimator
N 8. (x v.) [ 0) log {1 e(xx0) 1)
arg max
(2)
We need an assumption on the subsampling rate to investigate the subsample estimators.

---

Assumption 3. The subsampling rate p satisfies that ON 1= e= / c for a constant c such that
0 S c < 00.
Note that ey cannot be zero but its limit can be exactly zero. It can be shown that eN B Bee N1 (Nop) (1 + op(1)), so (E(( is the asymptotic positive/negative ratio for the sample. The theorem below shows the asymptotic distribution of 0w.
Theorem 2. Under Assumptions 1-3, if (()(((( + x (x) < oo then N +
(1 - 0-) - N(O, Vw),
in distribution,
where V = Vr + Suu and Suu = clE(e JM, E( (x)) (x;00)
The subsampled estimator One have the same convergence rate as the full data estimator. However, the asymptotic variance Vv. may be inflated by the term Vaub from subsampling the negative cases. Here V sub is zero if c 0. If we keep much more negative instances than the positíve instances in the subsample (c = 0), then there is no asymptotic information loss due to subsampling (V = Vy). In this scenario, the number of negative instances can still be aggressively reduced (p + so that the computational efficiency is significantly improved, and the sampling function xx() does not play an significant role. If we have to reduce the negative instances to the same level of the positive instances (0 < oo), then the variance inflation V is not negligible. In this scenario, well designed sampling function xxx) it more relevant and critical.
Theorem shows that the distribution of the estimation error err := (N - 0*) is approximated by normal distribution N(0, V.). Thus for any > O the probability of excess error Pr(err > c) is approximated by [ V.,)ll > e} Pr PEi= \₃² > +), where Xÿ's are eigenvalues of V. w and 2j 's are independent standard normal random variables. Therefore, smaller trace of V means a smaller probability of excess error of the same level since tr (((w) ww) = Σÿ= Az-
The following theorem gives the optimal negative sampling function for the IPW estimator. Theorem 3. For given sampling rate p. the asymptotically optimal (p(x) that minimizes (r(Vw) is
min mix; 0*), 8.).T)
Pos (x)
E. Emiin (1(x; 0¹) ),])]
p(x: 0) | (xx: 0)1l and T is the maximum number so that ølmin [1(x; {)) E[min (1(x; 0" T}] with probability one.
where t(x: 9)
Remark 1. p + 0 then pxx: x 0*) < B[1(x; } almost surely, so T can be dropped (i.e., T = oo). If lim P > 0 then
0, so the variance inflation due to subsampling is negligible. Thus, the truncation term I' can be ignored in practice with imbalanced data: it only plays an role for not very imbalanced data when the sampling ratio is not very small.
In (x), f(x; 0*) consists of two components, p(x; 0*) and | 0') The term p(x; 0*) is from the binary response model structure, and gives a higher preference for a data point with larger 0*) @²] corresponds to the A optimality criterion [24]. For computational benefit in optimal sampling, the A optimality is often replaced by the L optimality [e.g., 32, 28]. Under the I. optimality, ||M, (x; 0*)] || is replaced by ()ğ 0*) and this minimizes the asymptotic mean squared error (MSE) of M, u... In case the gradient is difficult to obtain, one could ignore the gradient term and simply use p(x; 0*) to replace t(x: 0*). Although this does not give an optimal sampling probability, it outperforms the uniform subsampling because takes into account the information of the model structure like the local case-control (LCC) sampling.
[x:
p(x: 0*) in the negative class. The term ||,, [xx
4 More efficient estimation based on the likelihood with log odds correction
4.1 Nonuniform log odds correction
The optimal function Fos (x) assigns larger probabilities to more informative instances. However, the IPW estimator in (2) assigns smaller weights to more informative instances in estimation, so the resulting estimator can be improved to have higher estimation efficiency. A naive unweighted

---

estimator is biased. and the bias correction approach in [8. 29] does not work for the sampling procedure in Algorithm 1 even when the underlying model is the logistic regression. We adopted the idea of [11] and seek to define a more efficient estimator through finding the corrected likelihood of the sampled data based on any negative sampling probability π (x).. For data included in the subsample (where d = 1), the conditional probability of y = is
Ps (x: 0) = Pr(y = 1 x, 8 1) 1+ t 9(x:0)-
(4)
where I - log/ ( (x)). Please see the detailed derivation of (4) in the supplement. To avoid the IPW, we propose the estimator based on the log odds corrected likelihood in (4), namely.
N Σ
mak arg max (n)(O),
where
(
be [v 0) log {1 e
(5)
With 0. in (2), T(x;) is in the inverse. an instance with much smaller x =(x;) is selected in the sample, it dominates the objective function, making the resulting estimator unstable. With 0ak this problem is ameliorated because r(x,) in the logarithm in the ongllikelhhod of (5) and log(v) goes to infinity much slower than v as O.
4.2 Theoretical analysis of the likelihood- estimator
The following shows the asymptotic distribution of 0ik-
Theorem 4. Under Assumptions 1-3, (e - (x) oc, then as N -> oo,
( 0) N(O, VIn) in distribution,
where Vink ((e((x:"")) Amm and Auk
E
e ((0i (x.000)) Itop - (xx)er((.)))
Theorem 3 shows that the proposed estimator Onk is asymptotic normal with variance Vik. We compare Vik with V. show that One has a higher estimation efficiency than Q..
Theorem 5. V w and V nie are finite and positive definite, then Vikk Van, ie., V. non-negative definite. The equality holds when c = 0 and in this case V V Vy.
Vik is
Since the estimator Ink in Eg. (5) is based on the conditional log- likelihood of the sampled data, it actually has the highest estimation efficiency among a class of asymptotically unbiased estimators.
Theorem 6. Let X (X1,..., "N)) be the feature matrix and Ds denote the sampled data. Let du be a subsample estimator with the following asymptotic representation:
OU U(O"; Ds) + N op((),
(6)
where the variance V(U(O";Ds) exists, and U(O"; Da) satisfies that E(U(O": D3) X] 0" and TE (O0 XX = with ; = a0(0"; (D)0":Da)aa0- If ( Vu in probability, then Vu 2 Vikk
Remark Theorem tells us that Onk is statistically the most efficient among a class of estimators, which includes both O., and O as special cases. The condition EE (E" Dr)) X} 0" ensures that the estimator is asymptotically unbiased. The condition E (0" Db) X} = 0 can be intuitively interpreted the requirement that the derivative of the (1(1) term in (6) is also negligible. Clearly,
this is satisfied by all unbiased estimators for which the (1()) term in (6) is O.
Practical considerations
In this section, we first show how we design a more practical estimator based on the previous results and then present the theoretical analysis behind these improvements.
5

---

5.1
Making estimators more practical
Like the LCC sampling [8] and the A-optimal sampling [29], the (os( in (3) depends on 0, and thus a pilot value of 0, say 0, is required in practice. Here, 0 can be constructed from a pilot sample, say (K,, Di), i 1, ..., nt taken by uniform sampling from both classes with equal expected sample sizes. With obtained, calculate L p(x;; 0) M ; 0)l for i = 1,... n, where Mr is the Hessian matrix of the pilot sample objective function. In case that the Hessian matrix and gradients are hard to record, the gradient term can be dropped, i.e., use & p(:; 0). As mentioned in Remark 1, T can be ignored for very imbalanced data. The expectation the denominator (3) can be approximated by mimicking the method of moment estimator, namely by
W 2N (A.N ( L (oo(NN)-1E
In practice, is common instances have positive probabilities to be selected. This is more critical for the IPW estimator while the likelihood estimator is not very sensitive very small sampling probabilities.
lower threshold, say e. on sampling probabilities, ensure that all
Denote the pilot value as 1 (0, ()) The following probabilities be practically implemented
Te xx: 0) min (pPos e), IJ, where (os(X) = 5 5--:--1xx3 0),
and we use this in our experiments in Section 6. The @ in (7) is essentially a normalizing term so that the expected subsample size is For practical systems such online models, W is often treated as a tuning parameter. We will illustrated this in our real data example in Section 6.2.
5.2 Theoretical analysis of practical estimators
Denote 8 and lik the IPW estimator and the likelihood estimator, respectively, with practically estimated optimal probability (8). We have the following results.
Theorem 7. Assume that the pilot estimator - is independent of the data, (x) - Pplli (x) in probability, and p_ g - > O. Under Assumptions 1-3, as N co, the @ satisfies that,
( - N(O, Velt) in distribution,
where Vplt has the same expression of V.., m except that sp(x) is replaced by max (x), ci). If Pr[ppu (x) 2 c} = 1 and the pilot estimator is consistent, then Vpl achieves the optimal variance.
Theorem 8. Assume that the pilot estimator Ô is independent of the data, Fxx(x) + (ppu(()) in probability, and e o(p). Under Assumptions 1-3, N xc, 0 satisfies that,
N (0 0) - NO, Vpil) in distribution,
where Vikk has the same expression of V lik except that p(x) is replaced by (Ppl(x)) If the pilor estimator is consistent, then Vplt achieves the variance with the optimal probability.
Remark 3. Theorems 7 and do not require to be consistent, i.e., the pilot estimator can be misspecified. However à has to be consistent in order to achieve the asymptotic variances with the optimal probability. Compared with the likelihood estimator 0 the IPW estimator 0 requires stronger condition on e to have asymptotic normality, because x (x; 8 the denominator of the objective function for 0² the turbulence of i is amplified. The requirement a 0 means that cannot be too small in practice. The likelihood estimator does not have this constraint.
Numerical experiments
We present simulated results and report the performance on a CTR dataset with more than 0.3 trillion instances. More experiments with model and pilot misspecifications available in the supplement.

---

6.1 Experiments on simulated data
We implemented the following negative sampling methods for logistic regression. 1) uniW: uniform sampling with the IPW estimator. 2) uniLik: uniform sampling with the likelihood estimator. 3) optW: optimal sampling with the IPW estimator. 4) optl.ik optimal sampling with the likelihood estimator. We also implement the full data MLE (Full) and the LCC for comparisons.
In each repetition of the simulation, we generate full data of size N 5 10 from the logistic regression with g(x: 0) = + x² B. We set the true 3 to be a 6 x 1 vector of ones and set different values for a for different feature distributions so that the positive/negative ratio is close to 1:400. We consider the following feature distributions: (a) Normal distribution which is symmetric with light tails; the true intercept a 7.65. (b) Log- Log-normal distribution which is asymmetric and positively skewed; the true intercept is a = -00-00000 0.5. (c) ty distribution which is symmetric with heavier tails; the true intercept a 77
We set the sampling rate as p= 0.002, 0.004, 0.006, 0.01, and 0.02 for all sampling methods. In each repetition, uniform samples of average size 100 are selected from each class to calculate the pilot estimates, so the uncertainty due to the pilot estimates are taken into account. We also consider pilot misspecification by adding a uniform random number from U(0, 1.5) to the pilot so that the pilot is systematically different from the true parameter. We repeat the simulation for It = 1000 times to calculate the MSE as R- E/= ||0(+) 011 where ((+) is the estimate at the r-th repetition.
-0.5
Consistent Plat
-1.
Consistent Pilo
Consistent Piot
Unnuk
IngIMSE)
1.0
1.5
apilic LOC
opow
(cg(MSE)
-1.
-2.0
220
-2.5%
2.5+
-30
0.005
0.010
0.515
0. 0.021
0.005
0,010
0.015
0.02:
0.
0.005
0.016
0.15
0.02
0.5
Misspociffed Piio
Misspecrfed Pilot
Misspecrted Pitot
1100
-1.5
lng(MSE)
og(MSE)
-2.0
-2.5
-3.0
0.005
0,010
0.015
0.021
0.005
0.010
0 015
0.02(
0.005
0.010
O 0015
0.02c
sampling rate
samoling rate
samplina rate
(a). x's are normal
(b) x's are lognormal
(c) x's 'ss are t3
Figure 1: Log(MSEs) for different estimators (the smaller the better). The top row uses consistent
pilot estimators; the bottom uses misspecified pilot estimators.
Results are presented in Figure 1. The MSES of all estimators decrease as the sampling rate p increases. The optLik outperforms other sampling methods in general, and its performance is close to the full data MLE when p -= 0.02, especially if the pilot is consistent. The advantage of optLik over optW is more evident for smaller p. If the pilot is misspecified, optLik is quite robust but optLik and LCC significantly deteriorate. Since uniLik and uniW do not require a pilot, they are not affected by pilot misspecification, but they have much larger MSE than optl iik and optW when the pilot is consistent, especially wniW. Different feature distributions also affect the performances of different sampling methods.
To verify Theorem numerically, we let the full data size increase at a faster rate than the average number of positive instances Nf B(NV 1) so that the probability Pr(y 1) decreases towards zero. As N increases, we fixed the value of 3 and set decreasing values to a* to mimic our scaling regime. We considered two covariate distributions: 1) multivariate normal distribution for which logistic regression model is correct model; and 2) multivariate normallal distribution for which a logistic regression model is misspecified. We simulated for 100 times to calculate the empirical variance V, of the full data MLE, and report the results in Table 1. It is seen that tr tr(V,) 5) is decreasing towards zero According to Theorem 1, ir(V,) should converge in a rate of 1 1/N1, so NTur(V,) should be relatively stable as N increases. This is indeed the case as seen in the third and sixth
7

---

columns of Table 1. On the other hand, Ntr(VV (V gets large dramatically as N increases. The aforementioned observations confirm the theoretical result in Theorem 1. Furthermore, Table 1 shows that the convergence rate in Theorem may also be true for some misspecified models.
Table 1: Empirical variances of the full data MLE for different full data size and average number of positive instances combinations.
Correct model
Mis-sprcified model
(N, M
tr tr(V,)
Nftr(V,)
Ntr(V,)
tr(V,)
Nftr(V,)
Ntr(V) y
(10", 32)
0.169
5.41
169.17
0.969
30.99
968.70
(104, 64)
0.097
6,20
969.29
0.322
20.59
3217.12
(105, 128)
0.045
5.76
4497.24
0.135
17.32
13527.60
(10% 256)
0.018
4.62
18048.40
0.046
11.74
45847.40
6.2 Experiments on real data
We conduct experiments on an internal CTR dataset from our products with more than 0.3 trillion instances. There are 299 input features, including user and ad profiles together with rich context features. We concatenate all features as a single nigh-dimensional vector x, whose length greater than 5, 000. We use a 3-layer fully connected deep neural network as a feature extractor that maps into a 256-dimensional dense features lx).. The downstream model is linear logistic regression model taking h(x) input to predict the binary output whether the user clicks the ad or not.
The positive/negative ratio is around 1:80. Due to limited storage, we first uniformly drop 80% negative instances and try various negative sampling algorithms on the rest 60 billion instances. On this pre-processed dataset, we then split out percent the data for testing, and do negative sampling on the rest data. The entire training procedure scans the subsampled data set from begin to end in one-pass according to the timestamp. The testing done all the way through training. We calculate the area under the ROC curve (AUC) for testing instances. The entire model is trained on distributed machine learning platform using 1 1,200 CPU cores and can finish within two days.
We adopt the sampling probability 28 (x: Q demonstrated in (8) and use the empirical sampling rate to calibrate the normalizing term W. As mentioned before, we remove the gradient term for implementation tractability so ((x; 0) p(x; 0). As the result will show, this approximate score produces consistent results with the simulation experiments. We compare four methods: 1) uniW and 2) uniLik are the same as in simulated experiments; and 3) optW and 4) optLik represent nonuniform sampling probability using <x;(x; O) with the IPW estimator and with the likelihood estimator. We use "opt" refer both optW and optLik when focusing on the sampling.
We study two sets of negative sample rates (w.r.t. the original 0.3 trillion data): (i) [0.01, 0.05). In this case, negative instances still outnumber positive instances. (ii) [0.001, 0.008). In this case, negative subsampling is too aggressive that positive instances dominates negative instances. We demonstrate the result in Figure When the sample rate is moderate (Figure 2.(b)), opt nearly optimal and can significantly outperform uniform subsampling. There is still a small gap between the IPW estimator and the likelihood estimator. When the sample rate is extremely small (Figure (.()), opt becomes sub-optimal and its gap w.r.t. uniform sampling is closer. Still we see a clear difference between the IPW estimator and the likelihood estimator. This due to a larger asymptotic variance for uniform sampling when the sampling rate goes to zero. Note that on this huge data set, a small relative AUC gain (e.g., 0.0005) usually could bring about significant revenue gain in products.
6.3 Sensitivity analysis
In practice, we find it necessary to use validation find the optimal @ (8) given a fixed sample rate. As demonstrated in Figure 3, for each sampling rate we tune from a very small value (in this case, 10 -) all the way to its largest value. When achieves its largest value, تع
and - T (x; 9) reduces to uniform negative sampling. We observe that T = (x; 9) with a moderate e achieves the best results for various sampling rates. Note that is the lower truncation level for negative sampling probabilities. Thus, larger g means more "simple" negative instances (pō-1p(x;;
; θ) is smaller than

---

0.812
0,815
AUC
0.100
0.814
untw uniLik optW
uniy
0.808
uniLik
0.813
optW
0.806
optLKK
opLLik
0,812 :4
0.002
0.004 sampling rate
0.008
0.01
0.02
0.03
0.04
0.05
sampling rate
(a). Extremely small sampling rates.
(b) Moderate sampling rates.
Figure 2: Empirical testing AUC subsample estimators for different sample sizes (the larger the better).
2) will have better chances be selected. This coineides with recent empirical results for negative subsampling in large search systems [14], where they mix "simple" (smaller p(x;; 0)) and "hard" (larger (x;; f)) negative instances. also observe an empirical variance of around 0.0001 for each setup, demonstrating that the relative improvement is consistent.
Semole Rats = 8.05
Sample Rate = 0.04
Sample Rabe m 0.03
Sample Rate 8.62
Sampa Rabe 0.03
wos
Sample Rate - 0.009
Sumple Rats 6,006
Sample Rate 0.004
Sumple Rats + 8.052
Sampl Rate - 8.00L
cands
1.000
$150
Figure 3: AUC for IPW and likelihood estimators when tuning the truncation lower bound Ø.
7 Conclusions, discussion, and limitations
In this paper, we have derived asymptotic distributions for full data and subsample estimators with rare events data. We have also found the optimal sampling probability that minimizes the unconditional asymptotic variance of the IPW estimator, and proposed the likelihood estimator through nonuniform log odds correction to further improve the estimation efficiency. The optimal probability depends on unknown parameters so have discussed practical implementations and investigated the convergences of the resultant estimators. Experiments on both simulated and real big online streaming data confirm our theoretical findings. Heuristically, our nonuniform negative sampling method gives preference to data points that are harder observe, so it would give preference to sub-populations that are difficult observe. We do not think this would bring any negative societal impacts. We have also examined our real data application and did not notice discrimination against any sub-populations.
This paper has the following limitations: 1) We assume that the model is correctly specified. Theoret- ical properties with model misspecification is not considered, and this important question requires future studies. 2) Oversampling the rare positive instances another common practice to deal with imbalanced data. Its application together with negative sampling is not considered. 3) The assumption of a fixed 3 means that both the marginal and conditional probabilities for positive instance to occur small. If the proportion of the positive instances for given feature in dataset large,

---

this assumption may not be appropriate. Further investigation is need to see if our results is still applicable. In the following. we present three scenarios that the scaling regime may or may not fit in:
Scenario (phone call while driving) Car crashes occur with small probabilities, and making phone calls while driving significantly increases the probability of car crash. However, the probability of car crashes among people making phone calls while driving is still small, so for these types of features, our scaling regime is appropriate model the rare events.
Scenario 2 (anti-causal learning) Anti-causal learning [26, 16] assumes that label (y) causes observa- tions (z:). Thus Pr(x y) represents the causal mechanism and is fixed. One standard example is that diseases (y) cause symptoms (x)) Our scaling regime fits the framework of anti-causal learning. To see this, using Bayes Theorem write the log odds
Pr(v 1) + log Pr(x y f(x: = log Pr(y 0)
Pr(x y = 1)
In anti-causal learning, the marginal distribution of y changes while the conditional distribution of x given y is fixed. Thus only the scale factor a changes, and fx: (3) is fixed.
Scenario 3 (local transmission the COVID-19) Our scaling regime has some limitations and does apply all types of rare events data. For example, although the COVID-19 infection rate for the whole population is low, the infection rate for people whose family members are the same house with positive test results high. This means the change of family member's test result converts snaal-probabilit--eeeett to a ange-probability-eeenn, and our scaling regime would not be appropriate.
Acknowledgments and Disclosure of Funding
HaiYing Wang's 's research was partially supported by NSF grant CCF 2105571.
References
[1] N. V. Chawla. Data mining for imbalanced datasets: An overview. In Data Mining and Knowledge Discovery Handbook, pages 875-886. Springer, 2009.
[2] N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer. Smote: synthetic minority over sem-sanlling technique. Journal of Artificial Intelligence Research, 16:321-357, 2002.
[3] N. Chawla, N. Japkowicz, and A. Kotez. Editorial: special issue on learning from imbalanced data sets. ACM SIGKDD Explorations Newsletter, 6(1):166, 2004.
[4] J. Chen, B. Sun, H. Li, H. Lu, and X.-S. Hua. Deep ctr prediction in display advertising. In Proceedings of the 24th ACM International Conference on Multimedia (MM), pages 81 811-820, 2016.
[5] G. Douzas and F. Bacao. Self-organizing map oversampling (somo) for imbalanced data set learning. Expert Systems with Applications, 82:40-52, 2017.
[6] C. Drummond, R. C. Holte, et al. C4. 5, class imbalance, and cost sensitivity: why under- sampling beats over-sampling. Workshop on Learning from Imbalanced Datasets II, vol- ume 11, pages 1-8. Citeseer, 2003.
[7] A. Estabrooks, T. Jo, and N. Japkowicz. A multiple resampling method for learning from imbalanced data sets. Computational intelligence, 20(1):18-36 2004.
[8] W. Fithian and T. Hastie. Local case-control sampling: Efficient subsampling in imbalanced data sets. Annals of statistics, 42(5):1693, 2014.
[9] H. Guo, R. Tang, Y. Ye, Z. Li, and X. He. Deepfm: a factorization-macla machine based neural network for etr prediction. arXiv preprint arXiv:1703.04247 2017.
[10] H. Han, W.Y Wang, and B.-H. Mao. Borderline-smote: A new over-sampling method in imbalanced data sets learning. D.-S. Huang, X.P. Zhang, and G.-B. Huang, editors, Advances in Intelligent Computing, pages 878, Berlin, Heidelberg, 2005. Springer Berlin Heidelberg.
10

---

[11] L. Han, K. M. Tan, T. Yang, T. Zhang, et al. Local uncertainty sampling for large-scale multiclass logistic regression. Aunals of Statistics, 48(3):1770-1788. 2020.
[12] X. He, J. Pan, O. Jin, T. Xu, B. Liu, T. Xu, Y. Shi, A. Atallah, R. Herbrich, S. Bowers, et al. Practical lessons from predicting clicks on ads at facebook. In Proceedings of the Eighth International Workshop on Data Mining for Online Advertising, pages 1-9, 2014.
[13] T. Hesterberg. Weighted average importance sampling and defensive mixture distributions. Technometries, 37(2):185-194, 1995.
[14] J.-T. Huang, A. Sharma, S. Sun, L. Xia, D. Zhang, P. Pronin, J. Padmanabhan, G. Ottaviano, and L. Yang. Embedding- -bsed retrieval in facebook search. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2553-2561 2020.
[15] N. Japkowicz. Learning from imbalanced data sets: Papers from the AAAI workshop, AAAI, 2000. Technical Report WS-00-05, 2000.
[16] N. Kilbertus, G. Parascandollo and B. Schölkopf. Generalization in anti-causal learning. arXiv preprint arXiv:1812..00524, 2018.
[17] G. King and L. Zeng. Logistic regression in rare events data. Political analysis, 9(2):137-133, 2001.
[18] G. Lemaître, F. Nogueira, and C. K. Aridas. Imbalanced-learn: A python toolbox to tackle the curse of imbalanced datasets in machine learning. The Journal of Machine Learning Research, 18(1)5599-663 2017.
[19] X. Liu, J. Wu, and Z. Zhou. Exploratory undersampling for class-imbalance learning, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 39(2):539 550, 2009.
[20] J. Mathew, C. K. Pang, M. Luo, and W. H. Leong. Classification of imbalanced data by oversampling kernel space of support vector machines. IEEE Transactions on Neural Networks and Learning Systems, 29(9):4065-4076, 2017.
[21] H. B. McMahan, G. Holt, Sculley, M. Young, D. Ebner, J. Grady, L. Nie, Phillips. E. Davydov, D. Golovin, et al. Ad click prediction: view from the trenches. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. pages 1222-1230, 2013.
[22] A. Owen and Y. Zhou. Safe and effective importance sampling. Journal of the American Statistical Association, 95(449):135-1M) 143, 2000.
[23] A. B. Owen. Infinitely imbalanced logistic regression. The Journal of Machine Learning Research, 8:761-773, 2007.
[24] F, Pukelsheim. Optimal design of experiments. SIAM, 2006.
[25] M. M. Rahman and D. Davis. Addressing the class imbalance problem in medical datasets. International Journal of Machine Learning and Computing, 3(2):224, 2013.
[26] B. Schölkopf, Janzing, J. Peters, E. Sgouritsa, K. Zhang, and J. Mooij. On causal and anticausal learning. In ICML 2012: Proceedings of the 29th International Conference on Machine Learning, Edinburgh, Scotland, June 26-July 1, 2012, pages 1255-1262, 2012.
[27] Y. Sun, M. S. Kamel, A. K. Wong, and Y. Wang. Cost-sensitive boosting for classification of imbalanced data. Pattern Recognition, 40(12):338--338, 2007.
[28] D. Ting and E. Brochu. Optimal subsampling with influence functions. In Advances in neural information processing systems, pages 3650-3659, 2018.
[29] H. Wang- More efficient estimation for logistic regression with optimal subsamples. Journal of Machine Learning Research, 20(132):1-59, 2019.

---

[30] H. Wang. Logistic regression for massive data with rare events. In H. D. III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 9829-9836, PMLR, 13-18 Jul 2020.
[31] H. Wang and Y. Ma. Optimal subsampling for quantile regression in big data. Biometrika, 108(1):99-11 112, 2021.
[32] H. Wang. Zhu, and P. Ma. Optimal subsampling for large sample logistic regression. Journal of the American Statistical Association, 113(522):829-844 2018.
[33] J. Yu, H. Wang, M. Ai, and H. Zhang. Optimal distributed subsampling for maximum quasi- likelihood estimators with massive data. Journal of the American Statistical Association, 000):1-12, 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section 2.1 and Section 3
(b) Did you include complete proofs of all theoretical results? [Yes] In the supplement.
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] We provide all the codes that were used for the experiments on simulated data. However, the code and the real data are proprietary so are not provided.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 6.
(c) Did you report error bars (e.g., with respect to the random seed after running ex- periments multiple times)? [No] We perform large number of repetitions of the simulation experiments to calculate the empirical MSE error bars not relevant in Section 6.1. For the real data experiments in Section 6.2, error bars would be too computationally expensive obtain. In addition, experiments because both the training and testing data have extremely large sample sizes. Furthermore, we have performed systematic sensitivity analysis in Section 6.3. (d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 6.2.
are confident with the real data
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
12

---

(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "yo5fzk-ff0v", "reviewer": "Reviewer_VDHH", "summary": "The paper discusses the sampling strategy for imbalanced binary-label datasets. For a binary-label dataset where the number of negative samples $N_0$ is much larger than that of positive samples $N_1$, we often \"subsample\" a part of negative samples in order to balance two labels.\n\nThe paper discusses the asymptotic convergence rate of the optimal model parameter against the number of samples in the dataset, and improves the rate from conventional model parameter estimation method.\n-   Theorem 1 proved that the convergence rate is proportional to $N_1^{-1/2}$, that is, determined not by the number of all instances but of positive instances if $N_0\\gg N_1$.\n-   Theorems 2 and 3 composes a function that provides the best strategy to subsample negative samples.\n-   Theorems 4 through 6 modifies the result of Theorems 2 and 3 for improving the efficiency by incorporating the sampling strategy.\n-   Theorems 7 and 8 discusses the effect of the \"pilot\" model parameter and provides the construction of it for better convergence rate.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Since we often encounter imbalanced datasets, subsampling strategy is very important for them. The proposed method is theoretically proved to be effective, including the setups for practical uses. Experimental results supported the results.\n\nThe reviewer felt that the paper was not well organized in the sense that the paper's contribution is not clear. The reviewer hopes the paper that the following points to be clearer: what is based on conventional methods, and what is new in the proposed method. (In the appendix there are references if existing analyses are employed for the proposed method, but the reviewer hopes references in the main paper in order to discuss the novelty of the paper.)\n\n-   Section 1, the part \"Rare events data and negative sampling are studied in [28], but it focused on linear logistic regression and only considered uniform sampling. We approach this problem under general binary response models from the perspective of optimal subsampling, which aims to minimize the asymptotic variance of the IPW estimator [30, 26, 29, 31]. This topic was not well investigated neither for imbalanced data nor for negative sampling.\"\n    -   What is main updates of the proposed method from the negative sampling strategy in [28] and the IPW? Does the paper just combines them (as well as providing with some new analyses)?\n-   Section 2.1\n    -   The reviewer felt that the result of Theorem 1 for simpler case seem to be proved by someone else. Please refer to conventional related analyses and their relationships to Theorem 1 in the main paper (not only the appendix) if any: For example, balanced case with (without) Assumptions 1 or 2, or imbalanced case with different assumptions.\n-   Section 2.1\n    -   Please provide, if any, the intuitive interpretation of $M^{-1}_f$ since it is essential to discuss the performance of the proposed method.\n-   Section 6\n    -   Since main theorems discuss the convergence rate against the number of samples in Theorem 1, perhaps experiment on it will support the paper's analysis more.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The impact is mainly stated in Section 1 and 7, theoretical assumptions are stated before respective theorems, and limitations are stated in Section 7. The reviewer thinks that they are adequately stated.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "GrlpgpZQNX", "reviewer": "Reviewer_j78J", "summary": "This paper considers the estimation of the parameters of a binary response model when the positive instances are rare. Rareness is modeled via a scaling regime on the logits, with the key restriction being that the scaling occurs _uniformly over features_. The paper first shows that the fraction of positive instances bottlenecks estimation and uses this to motivate subsampling the negative instances, to save on computation. The paper considers two negative subsampling approaches: importance weighting the original likelihood and maximizing it (IPW), or maximizing the log-likelihood under subsampling (Lik). Under correct model specification, asymptotic analysis is provided for both methods, with the latter enjoying many advantages, including stability and smaller asymptotic variance. The approach is tested on synthetic and click-through-rate data. The experiments support the theory and go beyond it, by showing that the methods work well even under model misspecification.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n\n+ The paper gives a very comprehensive modeling and treatment of rare positive instances in binary response: motivating sub-sampling, revisiting key sub-sampling methods, optimizing these methods to perform well, analyzing the presented variants, giving practical implementation tweaks, analyzing those tweaks, and finally testing on meaningful experiments. Each of these steps is a worthwhile contribution to share with the community. _(originality, significance, quality)_\n\n### Weaknesses\n\n- The scaling regime is somewhat limiting, because it requires the logits to decay uniformly over all features. This is definitely an acceptable and useful simplifying assumption for theoretical work, but it’s not clear why this is the right model to capture rare positives. Some discussion on this matter would be appreciated. _(significance)_\n\n- The paper is theory-intense, which is not a problem, but this is presented in a dry and dense fashion which makes it a heavy read. More signpost paragraphs to maintain the high-level picture can make this a more pleasant read. _(clarity)_\n\n- Some claims need a little care for nuance. For example, on line `line 86`, does the MLE remain the most efficient estimator (among asymptotically unbiased ones), even in this scaling regime? Saying the number of positive instances is what matters `line 98`, is obviously not true, because the result hinges on $N_1/N\\to 0$. Of course, down the line with the subsampling results, it’s clear we need to include \"enough\" negative samples to get the same asymptotic behavior as the full sample.  On `line 117` saying that will happen at $c=0$ should be emphasized to be just conceptual, since at that point Theorem 2 no longer holds. _(quality)_\n\n### Suggestions\n\n* It’s worthwhile to do a grammatical pass on the text, since there are several incorrect English phrases. These don’t deter from the quality of the work, but the polish would be appreciated. \n\n[Edit: Thank you authors for your response. My evaluation is unchanged.]", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper adequately discusses the limitations of the work. Societal impact is not discussed. While there may be some implications, considering that a big motivation is eliciting marketing response, which could negatively impact some sub-populations, it's not immediately apparent whether the proposed approaches could actively lead to discrimination. ", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns. Some of the experiments are rather large and impossible to reproduce with small resources, but there are also synthetic experiments that are (likely, did not try) reproducible.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4.5"}, "extra_text": {}}, {"review_id": "GM5ZcykizS", "reviewer": "Reviewer_SFTU", "summary": "This paper considers the question of parameter estimation in a binary response model (e.g., logistic regression), in the imbalanced case where the fraction of positive instances is small/vanishing.\nThis question is formulated by considering a well-specified model, and adding an intercept to the log-odds ratio which diverges as the sample size goes to infinity (with the other parameters kept fixed).\n\nThe main results are the following:\n\n- First, it is shown (in Theorem 1) that the convergence rate of the MLE (computed on the full dataset) only depends on the number N_1 of positive instance, rather than on the sample size (which also includes negative instances). MLE also shown to be asymptotically normal, and the asymptotic covariance is provided.\n\n- Since the convergence rate of MLE only depends on N_1 (but not on the total number N of samples), it can make sense to perform \"negative sampling\", namely to reduce N by only retaining a small fraction of negative instances, in order to reduce computations. Of course, removing negative instances creates a bias, so the authors considers inverse probability weighted (IPW) estimators, where negative samples are re-weighted (inversely proportionally to the deletion probability). An asymptotic normality result is obtained, with a covariance larger than MLE (as expected), by an additional covariance depending on the rejection probability function (Theorem 2). The optimal rejection function (minimizing asymptotic variance) is deduced (Theorem 3).\n\n- Finally, the authors consider a different estimator, based on log-odds correction. This estimator corresponds to MLE on the selected data, based on adjusting the likelihood of the selected data by accounting for the deletion mechanism. This leads to a correction which is less sensitive to small probability instances than the IPW method.\nThe asymptotic normality and covariance of this estimator is provided (Theorem 4). Interestingly, it is shown (Theorem 5) that this scheme is always better than the IPW (with optimal rejection function), in that it leads to a smaller asymptotic covariance. An optimality result of this estimator among (asymptotically) unbiased estimators is then provided in Theorem 6.", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I enjoyed reading this paper. It is well-written, considers a practically meaningful question, and provides precise answers. It also does not contain any spurious or unsubstantiated claims or decorative results. I appreciate the interesting adjusted MLE estimator based on the likelihood of selected data, and in particular its comparison with the IPW. The results also provide a coherent picture and naturally complement each other.\n\nI am therefore strongly in favor of acceptance of this paper, and would also suggest considering it for a spotlight or oral presentation, as it could potentially be of interest to many NeurIPS attendees. I would also encourage the authors to consider publishing a final version of this work to a suitable statistics journal.\n\nI should mention, however, that I am not familiar with the literature on the topic, in the sense that I have read none of the papers listed in the references. I am therefore not in a very good position to gauge the level of originality (though I somehow doubt that related work could warrant rejection, unless the same results were obtained before). I might also be slightly biased by the fact that many of the ideas are new to me, although some might be more routine in this topic. (Due to a high reviewing load, I did not check the proofs in the supplementary material.)\nI have accounted for this both in my low confidence score and in the overall score.\n\nBelow are some comments:\n\n1. I would suggest (maybe in a final version) to put the results (asymptotic covariance matrices) in more explicit or simple form in the specific case of logistic regression, which is arguably one of the most standard and relevant ones.\n\n2. It would perhaps be helpful to comment the results in connection with the asymptotic distribution of the MLE (given by the Fisher distribution) or the Cramer-Rao lower bound/asymptotic optimality of MLE. It actually seems that these standard result do not apply, as the data distribution changes with N (since the fraction of positive instances vanishes), which justifies a specific analysis as in this paper.\n\n3. It seems that, as noted in the paper, the misspecified case could also be considered in future work. Asymptotic normality should also hold there (with a covariance depending on the distribution), with the true parameter replaced by the best KL approximation.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "an afternoon"}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, an optimal negative sampling strategy from imbalanced data for rare event prediction is studied. The problem of class imbalance in rare event prediction is of considerable importance in various applied problems. The theoretical properties of the proposed method are clearly presented, and its advantages are clearly demonstrated by experimental validation on artificial and real data. All reviewers found the paper useful for the ML community. The authors are encouraged to consider updating the paper based on the reviewers' comments and suggestions.
Decision: Accept (Poster)