Robust Regression in R An Appendix to An R Companion to Applied Regression, Second Edition

Linear least-squares regression can be very sensitive to unusual data. In this appendix to Fox and Weisberg (2011), we describe how to fit several alternative robust-regression estimators, which attempt to down-weight or ignore unusual data: M -estimators; bounded-influence estimators; MM -estimators; and quantile-regression estimators, including L1 regression. All estimation methods rely on assumptions for their validity. We say that an estimator or statistical procedure is robust if it provides useful information even if some of the assumptions used to justify the estimation method are not applicable. Most of this appendix concerns robust regression, estimation methods typically for the linear regression model that are insensitive to outliers and possibly high leverage points. Other types of robustness, for example to model misspecification, are not discussed here. These methods were developed between the mid-1960s and the mid-1980s. With the exception of the L1 methods described in Section 5, they are not widely used today. 1 Breakdown and Robustness The finite-sample breakdown point of an estimator or procedure is the smallest fraction of “bad” data values such that if the [n ] bad values → ∞ then the estimator or procedure also becomes infinite. For example, the sample mean of x1, . . . , xn can be written as an explicit function of one of the observations in the sample as x̄n = 1 n n ∑ i=1 xi = 1 n [ n−1 ∑ i=1 xi + xn ] = n− 1 n x̄n−1 + 1 n xn and so if xn is large enough then x̄n can be made as large as desired regardless of the other n− 1 values. Unlike the mean, the sample median, as an estimate of a population median, can tolerate up to 50% bad values. In general, the breakdown point cannot exceed 50%. (Reader: Why is that?) 2 M -Estimation In linear regression, the breakdown of the ordinary least squares (OLS) estimator is analogous to the breakdown of the sample mean: A few extreme observations can largely determine the value of 1 the OLS estimator. In Fox and Weisberg (2011, Chap. 6), methods for detecting potentially influential points are presented, and these provide one approach to dealing with such points. Another approach is to replace ordinary least squares with an estimation method that is less affected by the outlying and influential points and can therefore produce useful results by accommodating the non-conforming data. The most common general method of robust regression is M -estimation, introduced by Huber (1964). This class of estimators can be regarded as a generalization of maximum-likelihood estimation, hence the “M.” We consider only the linear model yi = 0 + 1xi1 + 2xi2 + ⋅ ⋅ ⋅+ pxip + "i = xi + "i for the ith of n independent observations. We assume that the model itself is not at issue, so E(y∣x) = xi , but the distribution of the errors may be heavy-tailed, producing occasional outliers. Given an estimator b for , the fitted model is ŷi = b0 + b1xi1 + b2xi2 + ⋅ ⋅ ⋅+ bpxip + ei = xib and the residuals are given by ei = yi − ŷi = yi − xib In M -estimation, the estimates b are determined by minimizing a particular objective function over all b, n ∑ i=1 (ei) = n ∑ i=1 (yi − xib) (1) where the function gives the contribution of each residual to the objective function. A reasonable should have the following properties:  always-non negative, (e) ≥ 0;  equal to 0 when its argument is 0, (0) = 0;  symmetric, (e) = (−e), although in some problems one might argue that symmetry is undesirable; and  monotone in ∣ei∣, (ei) ≥ (ei′) for ∣ei∣ > ∣ei′ ∣. For example, the least-squares -function (ei) = e 2 i satisfies these requirements, as do many other functions. 2.1 Computing M -Estimates The minimum value of Equation 1 can be found by differentiating with respect to the argument b, and setting the resulting partial derivatives to 0: 0 = ∂ ∂b n ∑ i=1 (yi − xib) = n ∑ i=1 (yi − xib)x ′ i