# On the Upper Bounds of Test Statistics for a Single Outlier Test in Linear Regression Models

Authors: Tobias Ejiofor Ugah; Emmanuel Ikechukwu Mba; Micheal Chinonso Eze; Kingsley Chinedu Arum; Ifeoma Christy Mba; Henrietta Ebele Oranye
Journal: Journal of Applied Mathematics
 (2021)

---

## Abstract

A bewildering large number of test statistics have been found for testing the presence of an outlier in multiple linear regression models. Exact critical values of these test statistics are not available, and approximate ones are usually obtained by the first-order Bonferroni upper bound or large-scale simulations. In this paper, we show that the upper bound values of two of these test statistics are algebraically the same. An application to real data for multiple linear regression is used to demonstrate the procedure.


---

## Body

## 1. Introduction

An outlier is a discordant observation. It is an observation that does not fit in with the pattern of the remaining observations. It differs markedly not only from other members of the set from which it occurs, but also from its fitted value. Such an observation usually has a large residual. Outliers meet data analysts at the point of data analysis and in data mining. Reference [1] pointed out that there various causes of outliers such as human errors, erroneous operation of computer systems, sampling errors, or standardization failures. Excellent books on outliers include [2–4].Outliers usually have a major influence on the resulting parameter estimates, and their presence impacts adversely on the results of the statistical inference concerning the models. They can reduce the power of statistical tests during analysis. Reference [5] advised that there is the need for the analyst to identify outliers if they exist so that appropriate measures might be taken.Outliers need to be identified and corrected or eliminated. The process of identification and correction of outliers is not a straightforward thing; rather, it requires marked ability, competence, circumspection, and a strict adherence to scientific objectivity (impartiality) of high degree. If identified outliers cannot be remedied, they need to be removed because they contaminate the information contained by the remainder of that set of data (see [1, 6]).Test for an outlying observation in the response variable is usually based on the use of test statistics that depend on the standardized residuals. Different test statistics have been developed for testing of an outlier in a least squares analysis of linear regression models. However, exact critical values of some of these test statistics are not available and are not easy to obtain. The available approximate ones are based on the first-order Bonferroni upper bound or large-scale simulations.Upper bounds for the critical values of test statistics for detecting the presence of a single outlier in linear regression have been developed by [7, 8]. Although formal distinctions exist in the principles invoked by [7, 8] in deriving these upper bounds, we show in this paper that these upper bounds derived by [7, 8] are algebraically the same.The multiple linear regression model is(1)Y=Xβ+ε,where Y is the n×1 observation vector, X is an n×p matrix of constants, β is a p×1 vector of unknown parameters to be estimated, and ε is an n×1 vector of normally distributed errors. Assuming that Eε=0 and Varε=σ21, the least squares estimator of β in (1) is given by
(2)β^=X′X−1X′Y,and the vector of residuals is
(3)e=Y−Xβ^=I−XX′X−1X′ε.The variance-covariance matrixe(4)Vare=I−XX′X−1X′σ2.Ifσ2 is estimated using σ∧2=e′e/n−p, then the estimated variance-covariance matrix of e becomes
(5)Var^e=I=XX′X−1X′σ∧2.Residuals are important diagnostic tools in regression analysis as no regression analysis is complete without a thorough examination of them. They are versatile as most regression diagnostics are written in terms of them. They are used in checking model adequacy and the validity of model assumptions. A thorough examination of the residuals therefore provides valuable information concerning the appropriateness of assumptions that underlie statistical models and helps in pinpointing an appropriate model. Different types of graphic plots (representations) of residuals are used for diagnostic purposes.Ordinary residuals are not all that suitable for diagnostic purposes, and a standardized version of them is usually preferred. This is because the variances of the residuals are not homogeneous, and this makes them intractable. A standardized residual has a representation of the form(6)Ri=yi−y^iσ∧21−hii,where y^i is the predicted value of yi and hii is the ith element of matrix XX′X−1X′, called the hat matrix. The ith transformed residual Ri is often called an internally studentized residual. They are tractable and are more versatile. They are used as a replacement of the ordinary residuals in regression diagnostics. Numerous graphical and numerical techniques for checking model assumptions using standardized residuals can be found in the regression literature. They are also fundamental building blocks for most of the known test statistics studied in the literature for outlier detection in linear models (see [9, 10]).The test statistic(7)Rn=maxyi−y^iσ∧21−hii=maxRi,is called the maximum absolute internally studentized residuals. Reference [11], following the suggestion of [12], used a large-scale simulation study involving many thousands of sampling experiments to obtain approximate critical values of (7) for a simple linear regression. The approximate values obtained by [11] are almost the same with the values obtained by [13].Reference [7] considered the test statistic
(8)Rn∗=maxeiσ¯ei,where σ¯ei2 is the estimated average variance of the ordinary residuals. Reference [9] showed that the variance of the residuals is n−pσ2/n, so that the estimated variance of the ordinary residuals σ¯ei2=n−pσ∧2/n.Therefore,(9)Rn∗=n1/2maxei∑ei21/2.Reference [7] showed that the corresponding percentage point R0∗ of Rn∗ is bounded above by
(10)U=n−pFn−p−1+F,where F is the 1001−α/n percentage point of the F distribution with degrees of freedom 1 and n−p−1, n is the number of observations, and p is the number parameters estimated. Reference [7] results for simple linear regression were found to be almost identical to those of [11]. Reference [7] also suggested the use of (10) to obtain other critical values that are not in Table 1. The reference [7] result was not elaborate and extensive enough as the result in [8] because of the unavailability of the needed values of the F-distribution (see [8]).Table 1
Upper bounds of the critical values ofRn for detecting a single outlier in a simple linear regression model.
Sample sizeα=0.10α=0.05α=0.01RnRnRn41.41311.41390.414251.69741.71471.728661.88381.92701.975172.01422.07992.166782.11252.19612.317892.19112.28832.4398102.25622.36432.5407122.36022.48402.6988142.44142.57602.8186162.50792.65022.9136182.56412.71212.9919202.61262.76513.0575302.78692.95163.2812603.05083.22473.5869Define(11)ξi=Rin−p.Reference [14] showed that the joint distribution of ξi′s has a multivariate Inverted-Students Function and that the probability density function for any ξi is a univariate Inverted-Students Function with probability density function given by
(12)fξi=C1−ξi2n−p−3/2,ξi2≤1,where
(13)CΓn−p/2Γ1/2Γn−p−1/2.Reference [8], following the suggestion of [7], made use of the results of [14]. Reference [8] used the first-order Bonferroni inequality to obtain the upper bounds R0 of the critical values of Rn. Reference [8] obtained ξ0 from
(14)∫ξ012nfξidξi=α,where ξ0=R0/n−p and then obtained R0 using the relationship between R0 and ξ0 given by the equation
(15)R0=ξ0n−p,for sample sizes up to n=100, regression parameters p=25 and α=0.10, 0.05, and 0.01. With that, [8] is claimed to have produced the most elaborate upper bound values R0. For p=2, the upper bounds R0∗ computed by [7] using (10) and the upper bound values R0 computed by [8] using (14) are extremely very close.


## 2. Demonstration of the Sameness of Upper Bounds

In this section, we show that the upper boundsR0∗ and R0 are algebraically identical. From (10), we let
(16)U=n−pFn−p−1+F.We determine the distribution ofU as follows:
(17)PrU<u=Prn−pFn−p−1+F<u=PrF<u2n−p−1n−p−u2,so that
(18)fUu=fFu2n−p−1n−p−u2,1,n−p−12n−p−1n−pup+u2−n2,with distribution domain or range given by 0,n−p. Explicitly, we have
(19)fUu=H1−u2n−pn−p−3/20<u<n−p,where
(20)H=2Γn−p/2Γ1/2n−pΓn−p−1/2.Then, using the first Bonferroni inequality, one can obtain the upper boundsR0∗ by solving
(21)∫R0∗n−pnfUudu=α.Now, from (11), we have
(22)PrRi<r=Prξin−p<r=Prξi<rn−p,so that
(23)fRir=fξrn−p1n−p,with distribution domain or range given by −n−p,n−p. Explicitly, we have
(24)fRir=D1−r2n−pn−p−3/2−n−p<r<n−p,where
(25)D=Γn−p/2Γ1/2n−pΓn−p−1/2.Let(26)Yi=Ri,PrYi<y=PrRi<y=Pr−y<Ri<y.Because of the symmetry of the distribution ofRi in (24), we obtain the distribution of Yi=Ri as follows:
(27)PrYi<y=2PrRi<y.Explicitly, we have(28)fYiy=H1−y2n−pn−p−3/20<y<n−p.Then, using the first Bonferroni inequality, one can obtainR0 by solving
(29)∫R0n−pnfYiydy=α.The sameness of (21) and (29) means that
(30)∫R0∗n−pnfUiudu=α⇒∫R0n−pnfYiydy=α,implying that R0=R0∗. This also means that Rn and Rn∗ have distributions that are bounded by the same distribution.Using (21) to obtain the upper bounds R0∗ averts the problem encountered by [7]. This is because (10) depends on the tabulated percentage points of F-distribution while (21) does not. Reference [8] remarked that implementation of the suggestion made by [7] was very difficult because the needed percentage points of the F-distribution were not available. Therefore, for any value of α, R0∗ can easily be obtained using (21) without recourse to a tabulated value of the F-distribution. It is also preferable to use (29) to obtain R0 instead of (14). This is because, using (14) involves a kind of transformation from ξ0 to R0 as indicated in (15), but using (29) does not. The use of approximate critical values for detecting a single outlier in linear regression can be found in the work of [7, 8].


## 3. Table Construction

We use the Bonferroni inequality to obtain upper bound values for the 10 percent, 5 percent, and 1 percent critical values of the test statisticRn. A table of the upper bounds of the critical values of Rn is presented in Table 1 for a simple linear regression and sample sizes up to 60. These were obtained by solving (29) using the Mathematica Software. It is to demonstrate numerically that the upper bound values of the two test statistics (7) and (8) are the same as what (30) shows. Equation (29) produces precise and accurate values of upper bound values of these test statistics ((7) and (8)). The values in Table 1 compare favorably with values obtained by [7] by solving (10), the values obtained by [8] by solving (14), and the approximate values obtained by [11] via simulation.


## 4. Application to Real Data

We now show that the upper bounds of the two test statistics are the same with an application to a real data set. The data in Table2 is from [15]. Reference [15] carried out an investigation concerning the source from which corn plants obtain their phosphorus. It was carried out by chemically determining the concentrations of inorganic (X1) and organic (X2) phosphorus in the soils. They used eighteen soil samples in the experiment and measured the phosphorus content Y of the corn grown on n=18 Iowa soils. The phosphorus content Y was used as the dependent variable in a multiple regression analysis with X1 and X2 as the independent variables. The content Y (phosphorus content) of the corn in soil sample number 17 was found to be considerably larger than the phosphorus content of the corn grown in the other soil samples (no explanation was given for its size) and produced a standardized residual of 3.18. Multiple linear regression analysis of the data set produced the result in Table 2.Table 2
Data set and standardized residuals for multiple regression of plant available phosphorus (Y) on inorganic phosphorus (X1) and organic phosphorus (X2).
Soil samplenYX1X2Ri1640.4530.133312600.4230.043883713.1190.401694610.6340.029055544.724-0.684816771.7650.793267819.4440.1989389310.1310.8042599311.6290.68538105112.658-1.7289511769.437-0.02256129623.146-0.29759137723.150-1.29842149321.644-0.30278159523.156-0.3954216541.936-0.458701716826.8583.17401189929.951-0.85219We now show that the upper bound valuesR0∗ and R0 are the same. We compute the upper value R0∗ of Rn∗ using equation (10) which is given by
(31)n−pFn−p−1+F,where F is the 1001−α/n percentage point of the F distribution with degrees of freedom 1 and n−p−1, n is the number of observations, and p is the number parameters estimated. For α=0.01, n=18, p=3, and F=19.76, the upper bound for their critical value of Rn∗ is
(32)R0∗=15×19.7614+19.76=2.96.To obtain the upper bound for the critical value ofRn, we make use of equation (14). For α=0.01, n=18, and p=3, we apply equation (14) using the Mathematica software to obtain R0=2.96. Thus, the upper bound values of the two test statistics are the same. Equation (21) or (29) gives the same value.Finally, the observed value of 3.18 is found to be significant at the one percent level (3.18>2.97). Thus, the phosphorus content of the corn grown in soil sample number 17 should be regarded as an outlier, and the null hypothesis of no outlier in the data set is rejected at the 0.01 level.


## 5. Conclusions

In this article, we have shown that the upper bound valuesR0 of the test statistic (7) and the upper bounds R0∗ of the test statistic (8) are identical. Although formal distinctions exist in the principles used by [7] in deriving R0∗ and those employed by [8] in deriving R0, we have herein shown that they are algebraically the same. Having shown this, we recommend the use of (29) to compute the upper bounds of (7) or (8). It is more tractable than (10) and (14). Since (14) borders on some kind of transformation and (10) makes use of tabulated values of F-distribution, accuracy and precision may be lost when using them.



---

*Source: 1478843-2021-09-22.xml*