Efficient Truncated Linear Regression with Unknown Noise Variance

1 Introduction . A common challenge facing statistical estimation is the systematic omission of relevant data from the data used to train models . Data omission is a prominent form of dataset bias , which may occur for a variety of reasons , including incorrect experimental design or data collection campaigns , which prevent certain sub-populations from being observed , instrument errors or saturation , which make certain measurements unreliable , societal biases , which may suppress the realization of certain samples , as well as legal or privacy constraints , which might prevent the use of some data . In turn , it is well-understood that failure to account for the systematic omission of data in statistical inference may lead to incorrect models , and this has motivated a long line of research in statistics , econometrics , and a range of other theoretical and applied fields , targeting statistical inference that is robust to missing data ; see e.g . Galton ( 1897 ) ; Pearson ( 1902 ) ; Pearson & Lee ( 1908 ) ; Fisher ( 1931 ) ; Tukey ( 1949 ) ; Tobin ( 1958 ) ; Amemiya ( 1973 ) ; Hausman & Wise ( 1977 ) ; Heckman ( 1979 ) ; Breen et al . ( 1996 ) ; Hajivassiliou & McFadden ( 1998 ) ; Mohan et al . ( 2013 ) ; Balakrishnan & Cramer ( 2014 ) . In this paper , we revisit the classical challenge of truncated linear regression . As in the standard linear regression setting , we assume that the world produces pairs ( 𝑥 ( 𝑖 ) , 𝑦 ( 𝑖 ) ) 𝑖 , where each 𝑥 ( 𝑖 ) ∈ R𝑘 * CD and PS were supported by NSF Awards IIS-1741137 , CCF-1617730 , and CCF-1901292 , by a Simons Investigator Award , by the Simons Collaboration on the Theory of Algorithmic Fairness , by a DSTA grant , 11 and by the DOE PhILMs project ( No . DE-AC05-76RL01830 ) †MZ was supported by NSF DMS-2023505 ( FODSI ) 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . is a feature vector drawn from some distribution 𝒫 and its corresponding label 𝑦 ( 𝑖 ) is linearly related to 𝑥 ( 𝑖 ) according to 𝑦 ( 𝑖 ) = 𝑤 * 𝑇𝑥 ( 𝑖 ) + 𝜀 ( 𝑖 ) , for both some unknown coefficient vector 𝑤 * ∈ R𝑘 and random noise 𝜀 ( 𝑖 ) ∼ 𝒩 ( 0 , 𝜎 * 2 ) , where 𝜎 * 2 is unknown and 𝜀 ( 𝑖 ) is 𝑖.𝑖.𝑑 over all 𝑖 . The difference to the standard setting is that we do not get to observe all pairs ( 𝑥 ( 𝑖 ) , 𝑦 ( 𝑖 ) ) that the world produces . Rather , a pair ( 𝑥 ( 𝑖 ) , 𝑦 ( 𝑖 ) ) is only observed if 𝑦 ( 𝑖 ) ∈ 𝑆 , where 𝑆 ⊂ R is a fixed “ observation set , ” that is known or given to us via oracle access , i.e . we can query an oracle about whether or not some point belongs to 𝑆 . If 𝑦 ( 𝑖 ) /∈ 𝑆 , then ( 𝑥 ( 𝑖 ) , 𝑦 ( 𝑖 ) ) is removed from observation . Our goal is to estimate 𝑤 * and 𝜎 * 2 from the pairs that survived removal , called “ truncated samples. ” Overview of our results . While both truncated and censored linear regression are fundamental problems , which commonly arise in practice and have been studied in the literature since at least the 1950s ( Tobin , 1958 ; Amemiya , 1973 ) , there are no known statistical inference methods that are computationally efficient and enjoy statistical rates that scale with the dimension 𝑘 of the problem . Recent work of Daskalakis et al . ( 2019 ) has obtained such methods in the simpler setting where the noise variance 𝜎 * 2 is known . Our goal here is to extend those results in two important ways . First , we study settings where 𝜎 * 2 is unknown , providing computationally and statistically efficient estimators for the joint estimation of 𝑤 * and 𝜎 * 2 . Moreover , we establish the asymptotic normality of our estimators , and use this to derive confidence regions for our estimators . To achieve these stronger results we make some additional assumptions compared to Daskalakis et al . ( 2019 ) . Namely , we assume that the covariates are sampled from some prior distributions , and we also make a stronger assumption about the survival probability of our model as we discuss below . Other works that consider learning and regression problems with censored or truncated samples are Plevrakis ( 2021 ) ; Fotakis et al . ( 2020 ) ; Moitra et al . ( 2021 ) ; Ilyas et al . ( 2020 ) . We next provide informal statements of our results , postponing formal statements to Sections 3 and 5 . It is important to note that , as shown in prior work ( Daskalakis et al. , 2018 , 2019 ) , for the model parameters to be identifiable from truncated samples , we need to place some assumptions that limit how aggressively the data is truncated . The assumptions we make are listed below , and stated more precisely in Section 2 . They combine normalization assumptions ( Assumption 1.1 ) , assumptions that are needed even without truncation ( Assumption 1.3 ) , and assumptions that are needed for identifiability purposes in the presence of truncation ( Assumption 1.2 ) . Assumption 1.1 . We assume that ‖𝑤 * ‖22 ≤ 𝛽 and ⃦⃦ 𝑥 ( 𝑖 ) ⃦⃦2 2 ≤ 1 for all 𝑖 . Assumption 1.2 . For all ( 𝑥 ( 𝑖 ) , 𝑦 ( 𝑖 ) ) in the truncated dataset , conditioning on 𝑥 ( 𝑖 ) , the probability that 𝑤 * 𝑇𝑥 ( 𝑖 ) + 𝜀 ∈ 𝑆 , with respect to the randomness in 𝜀 ∼ 𝒩 ( 0 , 𝜎 * 2 ) , is larger than some 𝑎 . Assumption 1.3 . If ( 𝑥 ( 𝑖 ) , 𝑦 ( 𝑖 ) ) 𝑛𝑖=1 is the truncated dataset , we assume that the feature covariance matrix , 𝑋 = 1𝑛 ∑︀𝑛 𝑖=1 𝑥 ( 𝑖 ) 𝑥 ( 𝑖 ) 𝑇 , satisfies : E𝑥∼𝒫 [ 𝑋 ] = E𝑥∼𝒫 [ 𝑥𝑥𝑇 ] ⪰ 𝑏 · 𝐼 for some 𝑏 . With these assumptions we prove our main results for the joint estimation of 𝑤 * and 𝜎 * 2 , listed as Informal Theorems 1.4–1.5 below , and respectively Theorems 3.2 and 5.1 in later sections . Informal Theorem 1.4 ( Estimation ) . Suppose we are given 𝑛 samples ( 𝑥 ( 𝑖 ) , 𝑦 ( 𝑖 ) ) 𝑛𝑖=1 from the truncated linear regression model with parameters 𝑤 * , 𝜎 * 2 , and suppose Assumptions 1.1–1.3 hold . Then running Projected Stochastic Gradient Descent ( Algorithm 1 ) on the conditional negative log-likelihood of the sample , using the projection set described in Definition 3.1 , we obtain with probability at least 2/3 ( which can be boosted to as large a constant as desirable ) estimations ? ̂ ? , ? ̂ ? 2 such that ‖ ? ̂ ? − 𝑤 * ‖2 + ⃒⃒ ? ̂ ? 2 − 𝜎 * 2 ⃒⃒ ≤ 𝜀 , as long as 𝑛 ≥ 𝐶1 log 4 ( 1/𝜀 ) 𝜀4 or 𝑛 ≥ 𝐶2 𝜀2 ( depending on the learning rate ) , where 𝐶1 depends polynomially on 𝛽 , 1/𝑎 and 1/𝑏 , and 𝐶2 depends linearly in 1/𝑏 and exponentially in 𝛽/𝑎 , where 𝛽 , 𝑎 and 𝑏 are respectively the constants in Assumptions 1.1 , 1.2 and 1.3 . Informal Theorem 1.5 ( Asymptotic Normality ) . Let ? ̂ ? , ? ̂ ? 2 be the estimates of Informal Theorem 1.4 . There exists a matrix 𝐴 that depends on ? ̂ ? , ? ̂ ? 2 ( and is explicitly provided in Theorem 5.1 ) such that the random vector √ 𝑛 ( ( ︀ ? ̂ ? ? ̂ ? 2 ) ︀ − ( ︀ 𝑤 * 𝜎 * 2 ) ︀ ) converges asymptotically to a normal distribution with zero mean and covariance matrix 𝐴 . Hence the 1−𝛼 confidence region of ( ? ̂ ? , ? ̂ ? 2 ) can be computed using the matrix 𝐴 and the quantiles of the chi-squared distribution . Techniques . Our estimates are computed by running Projected Stochastic Gradient Descent ( PSGD ) on the population conditional negative log-likelihood of the truncated sample ; a convex function of the parameters , after an appropriate reparameterization . We provide two analyses of PSGD resulting in two different bounds that are more effective in a different range of parameters , as we discuss next . Our first analysis hinges on establishing a lower bound on the convexity of the conditional negative log-likelihood function . We can ’ t however , expect that a strong bound holds over the whole parameter domain . For this reason , we add a projection step to our method , identifying a projection set wherein we show that the true parameters lie and wherein the conditional negative log-likelihood is strongly convex . This approach is similar to that of Daskalakis et al . ( 2019 ) , except that the unknown noise variance introduces many challenges in identifying an appropriate projection set . The main result bounding the strong convexity of the conditional negative log likelihood function within the projection set is stated as Theorem 4.1 . The main disadvantage of this analysis is that , when the variance is unknown , the bound on the strong convexity of the conditional negative log-likelihood becomes small and the resulting dependence of the sample size on the parameters 𝛽 and 1/𝑎 of the problem is exponential . On the other hand , the dependence of the sample size on 1/𝑏 is linear and its dependence on the required estimation accuracy is Θ ( 1/𝜀2 ) , which is optimal . Our other analysis follows an approach that has not been pursued in prior work , and in particular does not rely on a global lower bound on the strong convexity of the conditional negative log-likelihood . Instead , we only bound the strong convexity at the optimum and use an upper bound on the smoothness of the objective function to prove the convergence of PSGD in terms of function value . Again , it is impossible to prove a bound on the smoothness over the whole domain , because large noise variance can lead to very bad smoothness bounds . Thus we use the same projection set as above and show that , within this set , we can get an upper bound on the smoothness that depends polynomially on all parameters , 𝛽 , 1/𝑎 , 1/𝑏 , of the problem . The main drawback of this method is that the dependence on the accuracy is Θ ( log4 ( 1/𝜀 ) /𝜀4 ) . It is worth noting that obtaining an algorithm whose sample complexity has an optimal Θ ( 1/𝜀2 ) dependence on the estimation accuracy , and also scales polynomially in 1/𝑎 is a fundamental bottleneck , which is already present in the much simpler special case of our problem of estimating the variance of a 1-dimensional truncated Gaussian , studied in Daskalakis et al . ( 2018 ) .3 Finally , to help build intuition around the intricacies in estimating truncated linear regression models with unknown noise variance , let ’ s consider why natural approaches of reducing this problem to estimating truncated linear regression models with known noise variance – which can be done using the algorithm of Daskalakis et al . ( 2019 ) – fail . A simple such reduction might compute the empirical variance 𝜎20 of the truncated sample , and plug that into the algorithm of Daskalakis et al . ( 2019 ) to estimate the linear model , pretending that 𝜎20 is the true noise variance . A more sophisticated approach might grid over candidate noise variances , plug each of these candidates into the algorithm of Daskalakis et al . ( 2019 ) to estimate a linear model , and then select the ( linear model , noise variance ) pair that is most consistent , e.g . the pair whose noise variance is closest to the empirical variance of the residuals between the linear model ’ s predictions and the observed labels . Both of these approaches fail , as suggested by the example below . Suppose that the ground truth is 𝑦 = 𝑤 * 𝑥 + 𝑤 * 0 + 𝜀 , where 𝑥 is one-dimensional , ( 𝑤 * , 𝑤 * 0 ) = ( 0 , 0 ) , and the noise variance is 𝜎 * 2 = 1 . Suppose also that we create a truncated dataset by , repeatedly , sampling 𝑥𝑖 from 𝒩 ( 0 , 1 ) , generating 𝑦𝑖 according to the ground truth , and truncating the resulting pair ( 𝑥𝑖 , 𝑦𝑖 ) unless 𝑦𝑖 ≥ 𝜏 , where 𝜏 is some threshold . It is clear that , if we guess the true noise variance to be the empirical one , then as 𝜏 gets bigger , the empirical variance gets smaller , thus the linear model that would be estimated by treating the empirical variance as the true noise variance would have intercept that is close to 𝜏 rather than 0 . So this naive approach would fail . Let us consider the more sophisticated one that grids over candidate noise variances . In particular , let us compare what happens when ( 1 ) we guess the noise variance to be 𝜎21 = 𝜎 * 2 , and run the algorithm of Daskalakis et al . ( 2019 ) with this guess to get an estimate ( ? ̂ ? , ? ̂ ? 0 ) ; and ( 2 ) when we guess the noise variance to be 𝜎22 ≡ 𝜎20 ( the empirical variance ) and run the algorithm of Daskalakis et al . ( 2019 ) with this guess to get an estimate ( ? ̂ ? ′ , ? ̂ ? ′0 ) . It is easy to see that , as 𝜏 gets large , ⃒⃒⃒ 𝜎 * 2− 1𝑁 ∑︀ 𝑖 ( 𝑦𝑖− ? ̂ ? 𝑥𝑖− ? ̂ ? 0 ) 2 ⃒⃒⃒ ≫ ⃒⃒⃒ 𝜎20 − 1𝑁 ∑︀ 𝑖 ( 𝑦𝑖− ? ̂ ? ′𝑥𝑖− ? ̂ ? ′0 ) 2 ⃒⃒⃒ ( here , 𝑁 is the number of samples . This holds even if 𝑁 goes to infinity ) . Thus the selection criterion proposed above will fail to prefer ( 1 ) over ( 2 ) , even though the guess of the noise variance under ( 1 ) is the correct one . Our approach in this paper amounts to using a different criterion , namely selecting the pair with the largest conditional log-likelihood . But analyzing when and how well this works , as well as obtaining an efficient algorithm to find the best pair is the main contribution of this work . 3While Daskalakis et al . ( 2018 ) does not explicitly analyze the dependence of the sample complexity on 𝛼 ; by tracing through their bounds , one can see that the dependence of the sample complexity on 𝛼 is exponential . Roadmap . We present the truncated linear regression setting and our assumptions in Section 2 . We state and prove our main estimation results Section 3 and 4 , and prove the asymptotic normality of our estimators in Section 5 . Finally , in Section 6 we assess the performance of our methods .

Summary: # Summary  The paper studies truncated linear regression when the additive error follows a zero-mean normal distribution with unknown variance.  This setting generalizes the work of Daskalakis-Gouleakis-Tzamos-Zampetakis-2019, who assumed the knowledge of the variance of error. This work provides two statistical rates for the joint estimation of the regressor (w) and the noise variance (\sigma) in Theorem 3.2. The rates in Theorem 3.2 are achieved using a SGD-without-replacement algorithm on the negative log likelihood after a suitable re-parameterization, which is also used in Daskalakis-Gouleakis-Tzamos-Zampetakis-2019.