Nonuniform Negative Sampling and Log Odds Correction with Rare Events Data

1 Introduction . Imbalanced data are ubiquitous in scientific fields and applications with binary response outputs , where the number of instances in the positive class is much smaller than that in the negative class . For example , in online search or recommendation systems where billions of impressions appear each day , non-click impressions usually dominate . Using these non-click impressions as negative instances is prevalent and proved to be useful especially in modern machine learning systems [ 21 , 12 , 4 , 9 , 14 ] . A common approach to address imbalanced data is to balance it by subsampling the negative class [ 6 , 19 ] and/or up-sampling the positive class [ 2 , 10 , 20 , 5 ] , and a great deal of attentions have been drawn to this problem , see [ 15 , 17 , 3 , 7 , 23 , 27 , 1 , 25 , 8 , 18 , 30 ] , and the references therein . In this paper , we focus on subsampling the negative class , i.e. , negative sampling , while keeping all rare positive instances . This is beneficial for modern online learning systems , where negative sampling significantly reduces data volume in distributed storage while saving training time for multiple downstream models . Rare events data and negative sampling are studied in [ 30 ] , but it focused on linear logistic regression and only considered uniform sampling . We approach this problem under general binary response models from the perspective of optimal subsampling , which aims to minimize the asymptotic variance of the IPW estimator [ 32 , 28 , 31 , 33 ] . This topic was not well investigated neither for imbalanced data nor for negative sampling . In addition , existing optimal 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . probability formulation minimizes the conditional asymptotic variance and the variations due to the data randomness are ignored . We fill these gaps by deriving the asymptotic distributions for both full data and subsample estimators under general binary response models , and we find the unconditional optimal probability for negative sampling . In addition , since IPW may inflate the variance [ 13 , 22 ] , we develop a more efficient likelihood-based estimator through nonuniform log odds correction to avoid IPW . Our main contributions are summarized as follows . • Under a general binary response model with rare events data , we find that the difference between the full data estimator and the true parameter converges to a normal distribution at a rate that is tied to the number of rare positive instances only . This indicates the possibility to throw away the majority of negative instances without information loss , i.e , it justifies the usage of negative sampling . • We show that there is no asymptotic information loss with aggressive negative sampling if the negative instances kept dominates the positive instances . However there is information loss expressed as a variance inflation if the negative instances are brought down to the same level of the positive instances . For this case , we obtain optimal subsampling probabilities by minimizing the unconditional asymptotic variance of a general IPW estimator . • We develop a likelihood estimator through nonuniform log odds correction for sampled data , and prove that it has the smallest asymptotic variance among a large class of estimators . • We apply the proposed method to a real online streaming click-through rate ( CTR ) dataset with more than 0.3 trillion instances and demonstrate its effectiveness . The rest of paper is organized as follows . Section 2 defines the problem this paper focuses on and shows the full data results . Section 3 presents the optimal probability for the IPW estimator . Section 4 proposes the likelihood-based estimator and establishes its asymptotic optimality . Section 5 discusses practical implementation and the theoretical properties of the resulting estimators . Section 6 presents numerical experiments with simulated data , and applies the proposed method to a real online streaming CTR dataset with more than 0.3 trillion instances . Section 7 concludes the paper and points out limitations of our investigation . Proofs and additional experiments with misspecifications are in the supplementary materials . 2 Problem setup and full data results . Let { ( xi , yi ) } Ni=1 be training data of size N that satisfies the binary response model , Pr ( y = 1 | x ) = p ( x ; θ ) , ( 1 ) where y ∈ { 0 , 1 } is the binary class label ( y = 1 for the positive and y = 0 for the negative ) , x is the vector of features , and θ is the unknown d-dimensional parameter . Let N1 be the number of positive instances ( N1 = ∑N i=1 yi ) and N0 be the number of negative instances ( N0 = N −N1 ) . We consider the scenario of massive imbalanced data , i.e. , N1 N0 . For asymptotic investigations when N1 is much smaller than N0 , it is more appropriate to assume that N1 increases in a slower rate compared with N0 , i.e. , N1/N0 → 0 in probability as N → ∞ [ 30 ] . This requires Pr ( y = 1 ) = E { p ( x ; θ ) } → 0 as N → ∞ on the model side . We assume that the parameter θ contains two components θ = ( α , βT ) T and the log odds can be written as g ( x ; θ ) : = log { p ( x ; θ ) 1− p ( x ; θ ) } = α+ f ( x ; β ) , where the true parameter , denoted as θ∗ = ( α∗ , β∗T ) T , satisfies that α∗ → −∞ as N → ∞ and β∗ is fixed . Here f ( x ; β ) is a smooth function of β , such as a linear function or a nonlinear neural network model . If it is linear , the model reduces to the logistic regression model . A diverging α∗ and a fixed β∗ indicate that both the marginal and conditional probabilities for a positive instance are small . Heuristically , this implies that a change in the feature value does make a rare event become a large probability event . We can also allow β∗ to change with N , but as long as β∗ has a finite limit , the situation is essentially the same as a fixed β∗ . So here we assume β∗ to be fixed to simplify the presentation . Throughout this paper , we use ġ ( x ; θ ) and g̈ ( x ; θ ) to denote the gradient and Hessian of g ( x ; θ ) with respect to ( w.r.t . ) θ , respectively . Let ‖v‖ be the Frobenius norm and v⊗2 = vvT for a vector or a matrix v. We denote π ( x , y ) as the sampling probability for an instance ( x , y ) . For negative cases , we sometimes use a shorthand notation π ( x ) to denote its sampling probability . We use oP ( 1 ) to represent a random quantity that converges to 0 in probability as N →∞ . 2.1 It is the number of positive instances that matters . We show that for rare events data , the estimation error rate for θ is related to N1 instead of N . Using the full data , a commonly used estimator of θ is the maximum likelihood estimator ( MLE ) θ̂f : = argmax θ N∑ i=1 [ yig ( xi ; θ ) − log { 1 + eg ( xi ; θ ) } ] . To investigate the theoretical properties of the estimator θ̂f , we make the following assumptions . Assumption 1 . The first , second , and third derivatives of f ( x ; β ) and ef ( x ; β ) f ( x ; β ) w.r.t . any components of β are bounded by a square intergrable random variable B ( x ) . Assumption 2 . The matrix E { ġ⊗2 ( x ; θ∗ ) } is finite and positive definite . Assumption 1 imposes constraints on the smoothness of f ( x ; θ ) w.r.t . θ and on the tightness of the distribution of x . Assumption 2 ensures that all components of θ are estimable . Theorem 1 . Let Vf = E { ef ( x ; β ∗ ) } M−1f where Mf = E { ef ( x ; β ∗ ) ġ⊗2 ( x ; θ∗ ) } . Under Assumptions 1 and 2 , as N →∞ , √ N1 ( θ̂f − θ∗ ) −→ N ( 0 , Vf ) , in distribution . This result shows that even if all the N instances are used for model training , the resulting estimator θ̂f converges to the true parameter θ∗ at the rate of N −1/2 1 , which is much slower than the usual rate of N−1/2 for regular cases . This indicates that with rare events data , the available information about unknown parameters is at the scale of N1 . Although N →∞ much faster than N1 →∞ ( in terms of N1/N → 0 ) , N does not explicitly show in the asymptotic normality result of Theorem 1 . For the specific case of linear logistic regression with g ( x ; θ ) = α + xTβ , our Theorem 1 reduces to Theorem 1 of [ 30 ] . 2.2 Negative sampling . Since the available information ties to the number of positive instances instead of the full data size , one can keep all the positive instances and significantly subsmaple the negative instances to reduce the computational cost . For better estimation efficiency , we consider nonuniform sampling . Let ϕ ( x ) > 0 be an integrable function and ρ be the sampling rate on the negative class . Without loss of generality , assume E { ϕ ( x ) } = 1 so that π ( xi ) : = ρϕ ( xi ) is the sampling probability for the i-th data point if yi = 1 . We present the nonuniform negative sampling procedure in Algorithm 1 . Algorithm 1 Negative sampling For i = 1 , ... , N : if yi = 1 , include { xi , yi , π ( xi , yi ) = 1 } in the sample ; if yi = 0 , calculate ϕ ( xi ) and generate ui ∼ U ( 0 , 1 ) ; if ui ≤ ρϕ ( xi ) , include { xi , yi , π ( xi , yi ) = ρϕ ( xi ) } in the sample . 3 Weighted estimation and its optimal negative sampling probability . Let δi = 1 if the i-th data point is selected and δi = 0 otherwise . Given a subsample taken according to π ( xi , yi ) = yi + ( 1− yi ) ρϕ ( xi ) , i = 1 , ... , N , the IPW estimator of θ is θ̂w = argmax θ N∑ i=1 δiπ −1 ( xi , yi ) [ yig ( xi ; θ ) − log { 1 + eg ( xi ; θ ) } ] . ( 2 ) We need an assumption on the subsampling rate to investigate the subsample estimators . Assumption 3 . The subsampling rate ρ satisfies that cN : = eα ∗ /ρ → c for a constant c such that 0 ≤ c < ∞ . Note that cN can not be zero but its limit c can be exactly zero . It can be shown that cNE { ef ( x ; β ∗ ) } = N1 ( N0ρ ) −1 { 1 + oP ( 1 ) } , so cE { ef ( x ; β ∗ ) } is the asymptotic positive/negative ratio for the sample . The theorem below shows the asymptotic distribution of θ̂w . Theorem 2 . Under Assumptions 1-3 , if E [ { ϕ ( x ) + ϕ−1 ( x ) } B2 ( x ) ] < ∞ then as N →∞ , √ N1 ( θ̂w − θ∗ ) −→ N ( 0 , Vw ) , in distribution , where Vw = Vf + Vsub and Vsub = cE { ef ( x ; β ∗ ) } M−1f E { ϕ−1 ( x ) e2f ( x ; β ∗ ) ġ⊗2 ( x ; θ∗ ) } M−1f . The subsampled estimator θ̂w have the same convergence rate as the full data estimator . However , the asymptotic variance Vw may be inflated by the term Vsub from subsampling the negative cases . Here Vsub is zero if c = 0 . If we keep much more negative instances than the positive instances in the subsample ( c = 0 ) , then there is no asymptotic information loss due to subsampling ( Vw = Vf ) . In this scenario , the number of negative instances can still be aggressively reduced ( ρ→ 0 ) so that the computational efficiency is significantly improved , and the sampling function ϕ ( x ) does not play an significant role . If we have to reduce the negative instances to the same level of the positive instances ( 0 < c < ∞ ) , then the variance inflation Vsub is not negligible . In this scenario , a well designed sampling function ϕ ( x ) it is more relevant and critical . Theorem 2 shows that the distribution of the estimation error err : = √ N1 ( θ̂w − θ∗ ) is approximated by a normal distribution N ( 0 , Vw ) . Thus for any > 0 the probability of excess error Pr ( err > ) is approximated by Pr { ‖N ( 0 , Vw ) ‖ > } = Pr ( ∑d j=1 λjz 2 j > ) , where λj ’ s are eigenvalues of Vw and zj ’ s are independent standard normal random variables . Therefore , a smaller trace of Vw means a smaller probability of excess error of the same level since tr ( Vw ) = ∑d j=1 λj . The following theorem gives the optimal negative sampling function for the IPW estimator . Theorem 3 . For a given sampling rate ρ , the asymptotically optimal ϕ ( x ) that minimizes tr ( Vw ) is ϕos ( x ) = min { t ( x ; θ∗ ) , T } E [ min { t ( x ; θ∗ ) , T } ] , ( 3 ) where t ( x ; θ ) = p ( x ; θ ) ‖M−1f ġ ( x ; θ ) ‖ and T is the maximum number so that ρ [ min { t ( x ; θ∗ ) , T } ] ≤ E [ min { t ( x ; θ∗ ) , T } ] with probability one . Remark 1 . If ρ→ 0 then ρt ( x ; θ∗ ) ≤ E { t ( x ; θ∗ ) } almost surely , so T can be dropped ( i.e. , T =∞ ) . If limN→∞ ρ > 0 then c = 0 , so the variance inflation due to subsampling is negligible . Thus , the truncation term T can be ignored in practice with imbalanced data ; it only plays an role for not very imbalanced data when the sampling ratio is not very small . In ϕos ( x ) , t ( x ; θ∗ ) consists of two components , p ( x ; θ∗ ) and ‖M−1f ġ ( x ; θ∗ ) ‖ . The term p ( x ; θ∗ ) is from the binary response model structure , and it gives a higher preference for a data point with larger p ( x ; θ∗ ) in the negative class . The term ‖M−1f ġ ( x ; θ∗ ) ‖ corresponds to the A optimality criterion [ 24 ] . For computational benefit in optimal sampling , the A optimality is often replaced by the L optimality [ e.g. , 32 , 28 ] . Under the L optimality , ‖M−1f ġ ( x ; θ∗ ) ‖ is replaced by ‖ġ ( x ; θ∗ ) ‖ , and this minimizes the asymptotic mean squared error ( MSE ) of Mf θ̂w . In case the gradient is difficult to obtain , one could ignore the gradient term and simply use p ( x ; θ∗ ) to replace t ( x ; θ∗ ) . Although this does not give an optimal sampling probability , it outperforms the uniform subsampling because it takes into account the information of the model structure like the local case-control ( LCC ) sampling .

Summary: This paper considers the question of parameter estimation in a binary response model (e.g., logistic regression), in the imbalanced case where the fraction of positive instances is small/vanishing. This question is formulated by considering a well-specified model, and adding an intercept to the log-odds ratio which diverges as the sample size goes to infinity (with the other parameters kept fixed).  The main results are the following:  - First, it is shown (in Theorem 1) that the convergence rate of the MLE (computed on the full dataset) only depends on the number N_1 of positive instance, rather than on the sample size (which also includes negative instances). MLE also shown to be asymptotically normal, and the asymptotic covariance is provided.  - Since the convergence rate of MLE only depends on N_1 (but not on the total number N of samples), it can make sense to perform "negative sampling", namely to reduce N by only retaining a small fraction of negative instances, in order to reduce computations. Of course, removing negative instances creates a bias, so the authors considers inverse probability weighted (IPW) estimators, where negative samples are re-weighted (inversely proportionally to the deletion probability). An asymptotic normality result is obtained, with a covariance larger than MLE (as expected), by an additional covariance depending on the rejection probability function (Theorem 2). The optimal rejection function (minimizing asymptotic variance) is deduced (Theorem 3).  - Finally, the authors consider a different estimator, based on log-odds correction. This estimator corresponds to MLE on the selected data, based on adjusting the likelihood of the selected data by accounting for the deletion mechanism. This leads to a correction which is less sensitive to small probability instances than the IPW method. The asymptotic normality and covariance of this estimator is provided (Theorem 4). Interestingly, it is shown (Theorem 5) that this scheme is always better than the IPW (with optimal rejection function), in that it leads to a smaller asymptotic covariance. An optimality result of this estimator among (asymptotically) unbiased estimators is then provided in Theorem 6.