# Robust Wild Bootstrap for Stabilizing the Variance of Parameter Estimates in Heteroscedastic Regression Models in the Presence of Outliers

Authors: Sohel Rana; Habshah Midi; A. H. M. R. Imon
Journal: Mathematical Problems in Engineering
 (2012)

---

## Abstract

Nowadays bootstrap techniques are used for data analysis in many other fields like engineering, physics, meteorology, medicine, biology, and chemistry. In this paper, the robustness of Wu (1986) and Liu (1988)'s Wild Bootstrap techniques is examined.  The empirical evidences indicate that these techniques yield efficient estimates in the presence of heteroscedasticity problem. However, in the presence of outliers, these estimates are no longer efficient. To remedy this problem, we propose a Robust Wild Bootstrap for stabilizing the variance of the regression estimates where heteroscedasticity and outliers occur at the same time. The proposed method is based on the weighted residuals which incorporate the MM estimator, robust location and scale, and the bootstrap sampling scheme of Wu (1986) and Liu (1988). The results of this study show that the proposed method outperforms the existing ones in every respect.


---

## Body

## 1. Introduction

Bootstrap technique was first proposed by Efron [1]. It is a computer intensive method that can replace theoretical formulation with extensive use of computer. The attractive feature of the bootstrap technique is that it does not rely on the normality or any other distributional assumptions and is able to estimate standard error of any complicated estimator without any theoretical calculations. These interesting properties of the bootstrap method have to be traded off with computational cost and time.  There are considerable papers that deal with bootstrap methods in the literatures (see [2–5]). The classical bootstrap methods are known to be a good general procedure for estimating a sampling distribution under the independent and identically distributed (i.i.d.) models. Let us consider a standard linear regression model:(1.1)Y=Xβ+ε,
where Y=(y1,y2,…,yn)T,X=(x1,x2,…,xn)T, and ε=(ε1,ε2,…,εn)T. In this equation β is a k×1 vector of unknown parameters, Y is an n×1 vector, X is an n×k data matrix of full rank k≤n, and ε is an n×1 vector of unobservable random errors with E (ε) = 0 and V(ε)=σ2I. In practice the i.i.d. set-up is often violated, as, for example, the homoscedastic assumption of Var(εi)=σ2I is often violated. Wu [6] proposed a weighted bootstrap technique which gives better performance under both the homoscedastic and heteroscedastic models. However, a better alternative approximation is developed by Liu [7] following the suggestions of Liu [7] and Beran [8]. This type of weighted bootstraps is called the wild bootstrap in the literature. Several attempts have been made to use the Wu and Liu wild bootstrap techniques to remedy the problem of heteroscedasticity (see [6, 7, 9, 10]).Salibian-Barrera and Zamar [11] pointed out that the problem of classical bootstrap is that the proportion of outliers in the bootstrap sample might be greater than that of the original data. Hence, the entire inferential procedure of bootstrap would be erroneous in the presence of outliers.  As an alternative, robust bootstrap technique has been drawn a greater attention to the statisticians (see [11–15]).  However, not much work is devoted to bootstrap technique when both outliers and heteroscedasticity are present in a data. Those wild bootstrap techniques can only rectify the problem of heteroscedasticity and not resistant to outliers.  Moreover, these procedures are based on the OLS estimate which is very sensitive to outliers. We introduce the classical wild bootstrap in Section 2. In Section 3, we discuss the newly proposed robust wild bootstrap methods. A numerical example and a simulation study are presented in Sections 4 and 5, respectively. The conclusion of the study is given in Section 6.


## 2. Wild Bootstrap Techniques

In regression analysis, the most popular and widely used bootstrap technique is the fixed-x resampling or bootstrapping the residuals [2]. This bootstrapping procedure is based on the ordinary least squares (OLS) residuals summarized as follows.Step 1.
Fit a modelyi=f(xi,βols) by the OLS method to the original sample of observations to get β̂ols and hence the fitted model is ŷi=f(xi,β̂ols).Step 2.
Compute the OLS residualsε̂i= yi-ŷi and each residual ε̂i has equal probability, 1/n.Step 3.
Draw a random sampleε1*,ε2*,...,εn* from ε̂i with simple random sampling with replacement and attached to ŷi for obtaining fixed-x bootstrap values yi*b whereyi*b=f(xi,β̂ols)+εi*b.Step 4.
Fit the OLS to the bootstrapped valuesyi*b on the fixed-x to obtain β̂ols*b.Step 5.
Repeat Steps3 and 4 for B times to get β̂ols*b1,...,β̂ols*bB where B is the bootstrap replications.We call this bootstrap scheme Bootols since it is based on the OLS method.When heteroscedasticity is present in the data, the variances of the data are different and neither of these bootstrap schemes can yield efficient estimates of the parameters. Wu [6] showed that they are inconsistent and asymptotically biased under the heteroscedasticity. Wu [6] proposed a wild bootstrap (weighted bootstrap) that can be used to obtain the standard error which is asymptotically correct under heteroscedasticity of unknown form. Wu slightly modified Step 3 of the OLS bootstrap and kept the other steps unchanged. For each i, draw a value ti*, with replacement, from a distribution with zero mean and unit variance and attached to ŷi for obtaining fixed-x bootstrap values yi*b, where yi*b=f(xi,β̂ols)+ti*ε̂i/1-hii and hii=xiT(XTX)xi  is the ith leverage. Note that the variance of ti*ε̂i is not constant when the original errors are not homoscedastic. Therefore, this bootstrap scheme takes into consideration the nonconstancy of the error variances. As an alternative [6], ti* can be chosen, with replacement, from   a1,a2,...,an, where(2.1)ai=ε̂i-ε̂̅in-1∑i=1n(ε̂i-ε̂̅)2
with ε̂̅=n-1∑i=1nε̂i. For a regression model with intercept term, ε̂̅i approximately equals zero. This is nonparametric implementation of Wu’s bootstrap since the resampling is done from the empirical distribution function of the (normalized) residuals. We call this method Wu’s bootstrap and denote it by Bootwu.Following the idea of Wu [6], another wild bootstrap technique was proposed by Liu [7] in which ti* is randomly selected from a population that has third central moment equal to one with zero mean and unit variance. Such kind of selection is used to correct the skewness term in the Edgeworth expansion of the sampling distribution of ITβ̂, where I is an n-vector of ones. Liu’s bootstrap can be conducted by drawing random numbers ti* in the following two ways.(1)
ti*=Zi-E(Zi),i=1,2,...,n, and Z1,Z2,...,Zn are independently and identically distributed having density gZ(x)=[αβ/(β-1)!]xβ-1e-axI(x>0), where α=2 and β=4.(2)
ti*=HiDi-E(Hi)E(Di),i=1,2,...,n, where H1,H2,...,Hn are independently and identically distributed normal distribution with mean (1/2)(17/6)+1/6 and variance 1/2. D1,D2,...,Dn are also independently and identically distributed normal distribution with mean (1/2)(17/6)-1/6 and variance 1/2. Hi’s and Di’s are independent.It is worth mentioning that selecting random numbersti* by procedure 1 or procedure 2 of Liu [7] will produce third central moment equal to one. Following Cribari-Neto and Zarkos [16], we consider the second procedure of drawing the random sample ti*. We call this bootstrap scheme as Bootliu.


## 3. Proposed Robust Wild Bootstrap Techniques

We have discussed the classical wild bootstrap procedures which are based on the OLS residuals. It is now evident that the OLS suffers a huge setback in the presence of outliers since it has 0% breakdown [17]. Since the wild bootstrap samples are based on the OLS residuals, it is not resistant to outliers. Hence, in this article we propose to use the high-breakdown and high-efficiency robust MM estimator [18] to obtain the robust residuals. It is expected that for good data point, the residuals of the MM estimator are approximately the same as the OLS residuals. On the other hand, the residuals of the MM estimator would be larger for outlier observation. We assign weights to the MM residuals. The standardized residuals |ε̂iMM|/σMM are computed, where σMM is the square root of the mean squares error of the residuals of the MM estimates (see [19]). Following the idea of Furno [20], weights equal to one and c/(|ε̂iMM|/σMM) are assigned to |ε̂iMM|/σMM≤c and |ε̂iMM|/σMM>c, respectively, where c is an arbitrary constant which is chosen between 2 and 3. We multiply the new weights with the residuals of the MM estimates and the resultants are denoted by ε̂iWMM. It is now expected that not only the residuals corresponding to the good data points but also the residuals corresponding to the bad data point of the MM residuals tend to be similar to the OLS residuals with no outliers. Based on the new weighted residuals ε̂iWMM, we propose to robustify Bootols, Bootwu, and  Bootliu. We call the resulting robust bootstraps RBootols, RBootwu, and RBootliu.We propose to replace the OLS residuals byε̂iWMM in Step 3 of the Bootols. That is, the bootstrap sample ε1*,ε2*,...,εn* is drawn from ε̂iWMM with simple random sampling and the other steps remain unchanged. We call this bootstrap scheme Rbootols. Now we will discuss the formulation of robust wild bootstrap based on Wu’s procedures. The algorithm is summarized as follows.Step 1.
Fit a modelyi=xiβ+εi by the MM estimator to the original sample of observations to get the robust parameters β̂MM and hence the fitted model is ŷi=xiβ̂MM.Step 2.
Compute the residuals of the MM estimateε̂iMM=yi-ŷi. Then assign weight to each residual, ε̂iMM, such that the weight equals 1  if  |ε̂iMM|/σMM≤c and equals c/(|ε̂iMM|/σMM) if |ε̂iMM|/σMM>c.Step 3.
The final weighted residuals of the MM estimates denoted byε̂iWMM are formulated by multiplying the weights obtained in Step 2 with the residuals of the MM estimates. That is, ε̂iWMM=1×ε̂iMM if the observation corresponds to good data point (no outliers) and ε̂iWMM=c/(|ε̂iMM|/σMM)×ε̂iMM if the observation corresponds to outliers.Step 4.
Construct a bootstrap sample(yi*,X), where
(3.1)yi*=xiβ̂MM+ti*ε̂iWMM(1-hii),
and  t* is a random sample following Wu [6] procedure.Step 5.
The OLS procedure is then applied to the bootstrap sample(yi*,X), and the resultant estimate is denoted by β̂*R=(XTX)-1XTy*. Here, the robust estimates are very reliable since the bootstrap sample is constructed based on the robust weighted residuals, ε̂iWMM.Step 6.
Repeat Steps4 and 5 for B times, where B is the bootstrap replications.As discussed earlier, in the classical scheme of Wu’s bootstrap, the quantityti* is drawn from a population that has mean zero and variance equal to one or, ti* can be drawn from normalized residualsa1,a2,...,an,that is, (3.2)ai=ε̂i-ε̂̅in-1∑i=1n(ε̂i-ε̂̅)2.
However, following Maronna et al. [21], we suggest computing the robust normalized residuals based on median and normalized  median absolute deviations(NMADs) instead of mean and standard deviation which are not robust.  Thus,(3.3)aiR=ε̂iWMM-median(ε̂iWMM)NMADnorm⁡(ε̂iWMM),
where NMADnorm=median{|ε̂iWMM-median(ε̂iWMM)|}/0.6745. We call this proposed robust nonparametric bootstrap as RBootwu.In this paper we also want to robustify the wild bootstrap based on the Liu [7] algorithm. It is important to note that the only difference between the Wu and Liu implementation of wild bootstrap is the choice of the random sample ti*. In the proposed robust bootstrap based on the Liu wild bootstrap, we choose the random sample ti* exactly the same manner as the classical Liu bootstrap. We call this bootstrap scheme as RBootliu.


## 4. Numerical Example

In this section, a numerical example is presented to assess the performance of the robust wild bootstrap methods. In order to compare the robustness of the classical and robust wild bootstrap in the presence of outliers, the Concrete Compressive Strength data is taken from Yeh [22]. Concrete is the most important material in civil engineering. The concrete compressive strength is a function of the eight output such as cement (Kg/m3), blast furnace slag (Kg/m3), fly ash (Kg/m3), water (Kg/m3), superplasticizer (Kg/m3), coarse aggregate (Kg/m3), fine aggregate (Kg/m3), and age of testing (days). The residuals versus fitted values are plotted in Figure 1 that show a funnel shape suggesting a  heterogeneous error variances for the data (see [19]).Figure 1
Residuals versus Fitted values plot of Concrete Compressive Strength data.We checked whether this data set contain any outliers or not by using Least trimmed of Squares (LTSs) residuals. It is found that 61 observations (about 6% of the sample of size 1030) appear to be outliers. The robust and non-robust (Classical) wild bootstrap methods were then applied to the data by considering two types of situations, namely, the data with outliers and data without outliers (omitted the outlying data points). The results are based on 500 bootstraps and are given in Table1.Table 1
Wild bootstrap standard errors of the parameters for the Concrete Compressive Strength data.
Standard error (Se)Classical wild bootstrapRobust wild bootstrapBootolsBootwuBootliuRBootolsRBootwuRBootliuData with outliersIntercept29.003727.067019.455927.195519.234316.5458Cement0.008990.008740.006120.008570.006410.00503Blast Furnace Slag0.010900.010220.007400.010470.007570.00605Fly Ash0.012830.012960.008640.011800.009710.00694Water0.045840.041290.029140.043010.029860.02566Superplasticizer0.098240.095640.066400.102530.071040.05706Coarse Aggregate0.010190.009560.006870.009280.006660.00559Fine Aggregate0.011130.010560.007720.010620.007720.00653Age0.007960.005500.005560.003010.004190.00175Average Se3.245533.029052.177083.043872.153051.85116Data without outliersIntercept26.394420.648317.558225.583820.736716.9330Cement0.008560.006870.005540.008100.006660.00531Blast Furnace Slag0.010000.008050.006790.009840.007850.00648Fly Ash0.011880.010140.007570.011430.010220.00736Water0.040680.030510.026460.038370.030800.02580Superplasticizer0.093840.071510.059400.094980.071460.05974Coarse Aggregate0.009090.007520.006020.009030.007390.00575Fine Aggregate0.010470.008230.006990.010250.008260.00679Age0.009920.008420.006900.010160.008370.00619Average Se2.954312.311061.964872.864002.320861.89516The standard errors of the parameter estimates from robust and nonrobust wild bootstrap methods are exhibited in Table1. The average standard errors of the parameter estimates are also shown. When there are no outliers, the standard errors of the classical wild bootstrap are reasonably closed to the standard errors of the robust wild bootstrap. It is interesting to note that the classical wild bootstrap methods provide larger standard errors compared to the wild bootstrap methods when outliers are present in the data.We cannot make a final conclusion yet, just by observing the results of the real data, but a reasonable interpretation up to this stage is that the classical wild bootstrap is affected by outliers.


## 5. Simulation Study

In this section, the performances of the proposed robust wild bootstrap estimators are evaluated based on a simulation study. At first we generate some artificial data to see the performance of proposed bootstrap techniques. The final investigation of the performance of the proposed estimators is verified by the simulation approach on bootstrap samples.


### 5.1. Artificial Data

We follow the data generation technique of Cribari-Neto and Zarkos [16] and MacKinnon and White [23]. The design of this experiment involves a linear model with two covariates:(5.1)yi=β0+β1x1i+β2x2i+σiεi.
We consider the sample sizes n= 20, 60, 100. For n = 20 the covariate values x1i were obtained from U(0,1) and the covariate values x2i were obtained from N(0,1). These observations were replicated three and five times for creating the sample of size n = 60 and   n = 100, respectively. The data generation was performed using β0=β1=β2=1. For all i under the homoscedasticity, σi=1. However, the main interest here is to find the heteroscedastic model. In this respect, we create a heteroscedastic generating mechanism following Cribari-Neto [24]’s work, where
(5.2)σi2=exp⁡(3.2x1i).
The degree of heteroscedasticity was measured by
(5.3)℘=max⁡(σi2)min⁡(σi2),i=1,2,...,n,
The degree of heteroscedasticity remains constant for different sample sizes since the covariate values are replicated for generating different sample sizes. In our study the degree of heterogeneity was approximately ℘=4. We focus on the situation where regression design would include outliers. To generate a certain percentages of outliers in Model (5.1), some i.i.d. normal errors εi’s were replaced by N(5, 10). Hence the contaminated heteroscedastic model becomes(5.4)yi=β0+β1x1i+β2x2i+σiεi(cont.),
where εi(cont.)=αN(0,1)+(1-α)N(5,10) and α is chosen according to level of percentage of outliers. In this study we choose the 5%, 10%, 15%, and 20% outliers in the model; that is, α is 0.95, 0.80, 0.85, and 0.80, respectively. Now for each sample size, the OLS, the classical, and the proposed robust wild bootstrap were then applied to the data. The replications of the bootstrap were 500 in each model for the different sample sizes. It is noteworthy that the bootstrap is extremely computer intensive, and S-plus programming language was used for computing the bootstrap estimates.The wild bootstrap standard errors of the estimates for different sample sizes and different percentage of contaminations are computed.  The bootstrap standard errors of Bootols, Bootwu, and Bootliu are obtained by taking the square root to the main diagonal of the covariance matrix:(5.5)(B-1)-1∑b=1B(β̂*b-β̅*)(β̂*b-β̅*)T,
where β̅*=(1/B)∑b=1Bβ̂*b. On the other hand, the bootstrap standard errors of RBootols, RBootwu, and RBootliu are obtained by taking the square root to the main diagonal of the covariance matrix as given in (5.5); the only essential difference is, however, we replace the usual bootstrap estimates by the robust bootstrap estimates.The influences of outliers on the standard errors of the estimates are visible in Figures2, 3, and 4. In these plots, the average standard errors of the parameters estimates are plotted at different levels of outliers for different bootstrap methods.  The results presented in Figures 2–4 show that the performances of the wild bootstrap estimates are fairly close to the classical estimates at the 0% level of contamination. It emerges that the average standard errors of the RBootwu and RBootliu are closer to the average standard errors of the classical Bootwu and Bootliu, respectively, in “clean” data, regardless of the percentage of outliers. However, at the 5%, 10%, 15%, and 20% levels of contaminations, the classical standard errors of the bootstrap estimates become unduly large. On the contrary, it is interesting to see that not much influence is visible for the robust wild bootstrap techniques of RBootwu and RBootliu, at the different percentage levels of outliers. It is also observed that the performance of RBootliu  is the best overall followed by RBootwu.Figure 2
The average effect of outliers on standard errors of parameters for sample sizen=20.Figure 3
The average effect of outliers on standard errors of parameters for sample sizen=60.Figure 4
The average effect of outliers on standard errors of parameters for sample sizen=100.


### 5.2. Simulation Approach on Bootstrap Sample

In the previous section, we used artificial data sets for different sample sizes. Now we would like to investigate the performances of different bootstrap estimators where data sets are generated by Monte Carlo simulations. Let us consider a heteroscedastic model which is given by(5.6)yi=β0+β1x1i+β2x2i+σiεi.
The covariate values of x1i and x2i are generated fromU(0,1) for sample sizes 20, 60, and 100. We have also considered β0=β1=β2=1 as the true parameters in this model and the heteroscedasticity generating function was σi2=exp(0.4x1i+0.4x2i). In this study the level of heteroscedasticity is set as ℘=max(σi2)/min(σi2)=4.In each simulation run and for the different sample size,εi’s were generated from N(0, 1) for the data with no outliers. However, for generating the 5% and 10% outliers, the 95% and 90% ofεi’s were generated from N(0, 1) and the 5% and 10% were generated from N(0, 20). It is worth mentioning that although such simulations are extremely computer intensive, the simulation for each sample size entails a total of 250000 replications with 500 replications and 500 bootstrap samples each. This simulation procedure was performed following the design of Cribari-Neto and Zarkos [16] and Furno [20].The simulation results for the different bootstrap methods are presented in Tables2–4. Table 2 shows the biasness measures of the non-robust and robust wild bootstrap techniques. It is observed that for the different sample sizes, the biasness of the Bootols, the Bootliu, and the Bootwu increases with the increase in the percentage of outliers. On the other hand, the RBootwu, and the RBootliu are slightly biased with the increase in the percentage of outliers. We can draw the same conclusion from the mean of the biasness of the estimates. The standard errors of the non-robust and robust wild bootstrap are presented in Table 3. It is observed that the standard errors of the classical bootstrap estimates increase with the increase in the percentage of outliers for different sample sizes. However, the robust bootstrap estimates are slightly affected by these outliers. By investigating the average standard errors of the estimates, it is also observed that the robust wild bootstrap techniques provide less standard error of the estimates in the presence of outliers. Finally, the robustness of different bootstrapping techniques are evaluated based on robustness measures defined in (5.5). Here the percentage robustness measure, that is, the ratio of the RMSEs of the estimators compared with the RMSEs of the OLS estimator for good data is presented in Table 4. From this table we see that the OLS and the classical bootstrap methods perform poorly. In the presence of outliers, the efficiency of the classical bootstrap estimates is very low. However, the efficiency of the robust bootstrap estimates is fairly closed to 100%.Table 2
Biasness measures of the non-robust and robust wild bootstrap.
% outliersCoeff.BootolsBootwuBootliuRBootwuRBootliuSample Sizen= 200%β00.0115-0.0246−0.0070−0.0701−0.0799β1−0.5604−0.5005−0.4813−0.3338−0.2796β20.37530.35260.29680.35760.3348Mean0.31570.29260.26170.25380.23145%β0−0.9419-1.1981-1.37020.03570.0821β12.95882.85703.1427−0.3130−0.3450β20.87421.30831.47160.41640.4077Mean1.59161.78781.99480.25500.278310%β0-5.6740-5.3968-5.5439-0.2166-0.215β16.78266.90836.95000.24500.2714β27.44016.87147.05770.41900.4073Mean6.63226.39216.51720.29350.2979Sample Sizen= 600%β00.01970.06120.0458-0.0123-0.0056β10.0159−0.01890.02180.04000.0345β20.0080−0.0209−0.02470.04620.0174Mean0.01450.03370.03080.03280.01925%β0-0.1451-0.1466-0.0648-0.0190-0.0045β10.43420.40110.29540.09210.0921β20.11100.14520.05260.03580.0073Mean0.23010.23100.13760.04900.034610%β0-0.7390-0.7896-0.74780.0164-0.0033β10.99160.98750.963020.07930.0794β20.89741.02921.00040.01810.0005Mean0.87600.93540.90370.03790.0277Sample Sizen= 1000%β00.02260.02160.0240−0.0505-0.0554β1−0.1086−0.1037−0.1099−0.0514−0.0457β2−0.0448−0.0422−0.04330.03710.0387Mean0.05870.05580.05910.04630.04665%β00.12180.13960.10950.02050.0048β1−0.1854−0.3399−0.2944−0.1880−0.1699β2−0.1944−0.1522−0.1174−0.0163−0.0038Mean0.16720.21060.17380.07490.059510%β00.75460.64920.8318-0.2921-0.2748β1−1.1835−1.0402−1.18090.32500.3294β2−0.8359−0.7686−1.04360.28000.2558Mean0.92470.81931.01870.29900.2867Table 3
Standard errors of the non-robust and robust wild bootstrap.
% outliersCoeff.BootolsBootwuBootliuRBootwuRBootliuSample Sizen= 200%β01.14911.04140.95401.05680.9774β11.21061.32201.02941.27391.1894β21.62571.48661.38451.48271.3737Mean1.32851.28331.12261.27111.18025%β04.47484.47003.25550.85030.9795β16.74268.09926.55371.12911.2546β25.43434.98123.26801.46561.6396Mean5.55065.85014.35911.14831.291210%β011.289812.535410.44261.14930.9913β112.938615.090412.54331.32961.15814β215.513216.873914.12091.48121.1552Mean13.247214.833212.36891.32001.1015Sample Sizen= 600%β00.75000.72960.64830.74550.6800β10.66320.67110.56440.72130.6072β20.97790.98650.83580.99790.8787Mean0.79700.79570.68280.82160.72205%β02.92502.38792.09640.76770.7043β14.64235.32094.68520.76590.7000β23.43062.12421.79460.98330.9184Mean3.66603.27772.85870.83900.774210%β06.35127.93356.53330.81640.7583β16.70288.54946.95650.88800.8150β28.729710.91029.11011.11441.0058Mean7.26129.13107.53330.93960.8597Sample Sizen= 1000%β00.46580.49000.37780.43260.3500β10.63400.65310.57120.64410.5901β20.63550.67170.50550.56810.4619Mean0.57840.60490.48480.54830.46735%β02.65261.96891.87330.46160.3887β14.66674.72854.70070.68050.6310β23.15971.59211.35640.58430.4947Mean3.49302.76322.64350.57550.504810%β05.71907.20035.92570.49990.4379β15.92507.49916.14140.53240.4687β27.86839.81718.08010.79990.7220Mean6.50418.17226.71570.61070.5429Table 4
Robustness measure of RMSE of the non-robust and robust wild bootstrap.
Robustness measure of RMSE% outliersCoeff.BootolsBootwuBootliuRBootwuRBootliuSample Sizen= 200%β0—110.3099120.4542102.0019117.1796β1—94.37567117.395194.5929109.1911β2—109.20311117.8290101.1577118.0043Mean—104.6296118.559499.25083114.79175%β025.1307424.8323232.53479109.03777135.0219β118.1188315.5340918.35560103.90257113.8576β230.3130432.3967646.5529494.03987109.5074Mean24.5208724.2543932.48111102.3267119.462310%β09.09498.42039.719998.2525106.2831β19.13248.03869.303598.6716105.1563β29.69779.157810.5692108.3857109.2154Mean9.30838.53899.8642101.7699120.5487Sample Sizen= 600%β0—102.47133115.4533100.52476110.3378β1—98.81731117.450991.85584109.0760β2—99.10685116.952597.81249111.2705Mean—100.1318116.618996.73103110.22815%β025.6212631.3631335.7753492.89488106.5277β114.2297912.4339314.1329480.0158393.96979β228.4912245.9306954.4704990.03794106.4720Mean22.7807629.9092534.7929287.64955102.323210%β011.73519.411511.410591.90098.8490β19.79197.70939.447474.419781.2251β211.14378.923410.670587.751897.1719Mean10.890238.681410.5094784.690592.41533Sample Sizen= 1000%β0—95.07261123.1937100.26721131.6080β1—97.26561110.577997.95562108.6698β2—94.65821125.5736100.14950137.4324Mean—95.66548119.781799.45744125.90345%β017.563323.627824.853789.4943119.9566β113.773313.568813.657687.991098.4320β220.126139.835546.797590.0007128.7815Mean17.154225.677428.436389.1620115.723410%β08.08486.45107.794080.542690.1938β110.64658.496510.2857103.1180112.2725β28.05216.47027.820275.176783.1701Mean8.92787.13938.633386.279195.2121


## 5.1. Artificial Data

We follow the data generation technique of Cribari-Neto and Zarkos [16] and MacKinnon and White [23]. The design of this experiment involves a linear model with two covariates:(5.1)yi=β0+β1x1i+β2x2i+σiεi.
We consider the sample sizes n= 20, 60, 100. For n = 20 the covariate values x1i were obtained from U(0,1) and the covariate values x2i were obtained from N(0,1). These observations were replicated three and five times for creating the sample of size n = 60 and   n = 100, respectively. The data generation was performed using β0=β1=β2=1. For all i under the homoscedasticity, σi=1. However, the main interest here is to find the heteroscedastic model. In this respect, we create a heteroscedastic generating mechanism following Cribari-Neto [24]’s work, where
(5.2)σi2=exp⁡(3.2x1i).
The degree of heteroscedasticity was measured by
(5.3)℘=max⁡(σi2)min⁡(σi2),i=1,2,...,n,
The degree of heteroscedasticity remains constant for different sample sizes since the covariate values are replicated for generating different sample sizes. In our study the degree of heterogeneity was approximately ℘=4. We focus on the situation where regression design would include outliers. To generate a certain percentages of outliers in Model (5.1), some i.i.d. normal errors εi’s were replaced by N(5, 10). Hence the contaminated heteroscedastic model becomes(5.4)yi=β0+β1x1i+β2x2i+σiεi(cont.),
where εi(cont.)=αN(0,1)+(1-α)N(5,10) and α is chosen according to level of percentage of outliers. In this study we choose the 5%, 10%, 15%, and 20% outliers in the model; that is, α is 0.95, 0.80, 0.85, and 0.80, respectively. Now for each sample size, the OLS, the classical, and the proposed robust wild bootstrap were then applied to the data. The replications of the bootstrap were 500 in each model for the different sample sizes. It is noteworthy that the bootstrap is extremely computer intensive, and S-plus programming language was used for computing the bootstrap estimates.The wild bootstrap standard errors of the estimates for different sample sizes and different percentage of contaminations are computed.  The bootstrap standard errors of Bootols, Bootwu, and Bootliu are obtained by taking the square root to the main diagonal of the covariance matrix:(5.5)(B-1)-1∑b=1B(β̂*b-β̅*)(β̂*b-β̅*)T,
where β̅*=(1/B)∑b=1Bβ̂*b. On the other hand, the bootstrap standard errors of RBootols, RBootwu, and RBootliu are obtained by taking the square root to the main diagonal of the covariance matrix as given in (5.5); the only essential difference is, however, we replace the usual bootstrap estimates by the robust bootstrap estimates.The influences of outliers on the standard errors of the estimates are visible in Figures2, 3, and 4. In these plots, the average standard errors of the parameters estimates are plotted at different levels of outliers for different bootstrap methods.  The results presented in Figures 2–4 show that the performances of the wild bootstrap estimates are fairly close to the classical estimates at the 0% level of contamination. It emerges that the average standard errors of the RBootwu and RBootliu are closer to the average standard errors of the classical Bootwu and Bootliu, respectively, in “clean” data, regardless of the percentage of outliers. However, at the 5%, 10%, 15%, and 20% levels of contaminations, the classical standard errors of the bootstrap estimates become unduly large. On the contrary, it is interesting to see that not much influence is visible for the robust wild bootstrap techniques of RBootwu and RBootliu, at the different percentage levels of outliers. It is also observed that the performance of RBootliu  is the best overall followed by RBootwu.Figure 2
The average effect of outliers on standard errors of parameters for sample sizen=20.Figure 3
The average effect of outliers on standard errors of parameters for sample sizen=60.Figure 4
The average effect of outliers on standard errors of parameters for sample sizen=100.


## 5.2. Simulation Approach on Bootstrap Sample

In the previous section, we used artificial data sets for different sample sizes. Now we would like to investigate the performances of different bootstrap estimators where data sets are generated by Monte Carlo simulations. Let us consider a heteroscedastic model which is given by(5.6)yi=β0+β1x1i+β2x2i+σiεi.
The covariate values of x1i and x2i are generated fromU(0,1) for sample sizes 20, 60, and 100. We have also considered β0=β1=β2=1 as the true parameters in this model and the heteroscedasticity generating function was σi2=exp(0.4x1i+0.4x2i). In this study the level of heteroscedasticity is set as ℘=max(σi2)/min(σi2)=4.In each simulation run and for the different sample size,εi’s were generated from N(0, 1) for the data with no outliers. However, for generating the 5% and 10% outliers, the 95% and 90% ofεi’s were generated from N(0, 1) and the 5% and 10% were generated from N(0, 20). It is worth mentioning that although such simulations are extremely computer intensive, the simulation for each sample size entails a total of 250000 replications with 500 replications and 500 bootstrap samples each. This simulation procedure was performed following the design of Cribari-Neto and Zarkos [16] and Furno [20].The simulation results for the different bootstrap methods are presented in Tables2–4. Table 2 shows the biasness measures of the non-robust and robust wild bootstrap techniques. It is observed that for the different sample sizes, the biasness of the Bootols, the Bootliu, and the Bootwu increases with the increase in the percentage of outliers. On the other hand, the RBootwu, and the RBootliu are slightly biased with the increase in the percentage of outliers. We can draw the same conclusion from the mean of the biasness of the estimates. The standard errors of the non-robust and robust wild bootstrap are presented in Table 3. It is observed that the standard errors of the classical bootstrap estimates increase with the increase in the percentage of outliers for different sample sizes. However, the robust bootstrap estimates are slightly affected by these outliers. By investigating the average standard errors of the estimates, it is also observed that the robust wild bootstrap techniques provide less standard error of the estimates in the presence of outliers. Finally, the robustness of different bootstrapping techniques are evaluated based on robustness measures defined in (5.5). Here the percentage robustness measure, that is, the ratio of the RMSEs of the estimators compared with the RMSEs of the OLS estimator for good data is presented in Table 4. From this table we see that the OLS and the classical bootstrap methods perform poorly. In the presence of outliers, the efficiency of the classical bootstrap estimates is very low. However, the efficiency of the robust bootstrap estimates is fairly closed to 100%.Table 2
Biasness measures of the non-robust and robust wild bootstrap.
% outliersCoeff.BootolsBootwuBootliuRBootwuRBootliuSample Sizen= 200%β00.0115-0.0246−0.0070−0.0701−0.0799β1−0.5604−0.5005−0.4813−0.3338−0.2796β20.37530.35260.29680.35760.3348Mean0.31570.29260.26170.25380.23145%β0−0.9419-1.1981-1.37020.03570.0821β12.95882.85703.1427−0.3130−0.3450β20.87421.30831.47160.41640.4077Mean1.59161.78781.99480.25500.278310%β0-5.6740-5.3968-5.5439-0.2166-0.215β16.78266.90836.95000.24500.2714β27.44016.87147.05770.41900.4073Mean6.63226.39216.51720.29350.2979Sample Sizen= 600%β00.01970.06120.0458-0.0123-0.0056β10.0159−0.01890.02180.04000.0345β20.0080−0.0209−0.02470.04620.0174Mean0.01450.03370.03080.03280.01925%β0-0.1451-0.1466-0.0648-0.0190-0.0045β10.43420.40110.29540.09210.0921β20.11100.14520.05260.03580.0073Mean0.23010.23100.13760.04900.034610%β0-0.7390-0.7896-0.74780.0164-0.0033β10.99160.98750.963020.07930.0794β20.89741.02921.00040.01810.0005Mean0.87600.93540.90370.03790.0277Sample Sizen= 1000%β00.02260.02160.0240−0.0505-0.0554β1−0.1086−0.1037−0.1099−0.0514−0.0457β2−0.0448−0.0422−0.04330.03710.0387Mean0.05870.05580.05910.04630.04665%β00.12180.13960.10950.02050.0048β1−0.1854−0.3399−0.2944−0.1880−0.1699β2−0.1944−0.1522−0.1174−0.0163−0.0038Mean0.16720.21060.17380.07490.059510%β00.75460.64920.8318-0.2921-0.2748β1−1.1835−1.0402−1.18090.32500.3294β2−0.8359−0.7686−1.04360.28000.2558Mean0.92470.81931.01870.29900.2867Table 3
Standard errors of the non-robust and robust wild bootstrap.
% outliersCoeff.BootolsBootwuBootliuRBootwuRBootliuSample Sizen= 200%β01.14911.04140.95401.05680.9774β11.21061.32201.02941.27391.1894β21.62571.48661.38451.48271.3737Mean1.32851.28331.12261.27111.18025%β04.47484.47003.25550.85030.9795β16.74268.09926.55371.12911.2546β25.43434.98123.26801.46561.6396Mean5.55065.85014.35911.14831.291210%β011.289812.535410.44261.14930.9913β112.938615.090412.54331.32961.15814β215.513216.873914.12091.48121.1552Mean13.247214.833212.36891.32001.1015Sample Sizen= 600%β00.75000.72960.64830.74550.6800β10.66320.67110.56440.72130.6072β20.97790.98650.83580.99790.8787Mean0.79700.79570.68280.82160.72205%β02.92502.38792.09640.76770.7043β14.64235.32094.68520.76590.7000β23.43062.12421.79460.98330.9184Mean3.66603.27772.85870.83900.774210%β06.35127.93356.53330.81640.7583β16.70288.54946.95650.88800.8150β28.729710.91029.11011.11441.0058Mean7.26129.13107.53330.93960.8597Sample Sizen= 1000%β00.46580.49000.37780.43260.3500β10.63400.65310.57120.64410.5901β20.63550.67170.50550.56810.4619Mean0.57840.60490.48480.54830.46735%β02.65261.96891.87330.46160.3887β14.66674.72854.70070.68050.6310β23.15971.59211.35640.58430.4947Mean3.49302.76322.64350.57550.504810%β05.71907.20035.92570.49990.4379β15.92507.49916.14140.53240.4687β27.86839.81718.08010.79990.7220Mean6.50418.17226.71570.61070.5429Table 4
Robustness measure of RMSE of the non-robust and robust wild bootstrap.
Robustness measure of RMSE% outliersCoeff.BootolsBootwuBootliuRBootwuRBootliuSample Sizen= 200%β0—110.3099120.4542102.0019117.1796β1—94.37567117.395194.5929109.1911β2—109.20311117.8290101.1577118.0043Mean—104.6296118.559499.25083114.79175%β025.1307424.8323232.53479109.03777135.0219β118.1188315.5340918.35560103.90257113.8576β230.3130432.3967646.5529494.03987109.5074Mean24.5208724.2543932.48111102.3267119.462310%β09.09498.42039.719998.2525106.2831β19.13248.03869.303598.6716105.1563β29.69779.157810.5692108.3857109.2154Mean9.30838.53899.8642101.7699120.5487Sample Sizen= 600%β0—102.47133115.4533100.52476110.3378β1—98.81731117.450991.85584109.0760β2—99.10685116.952597.81249111.2705Mean—100.1318116.618996.73103110.22815%β025.6212631.3631335.7753492.89488106.5277β114.2297912.4339314.1329480.0158393.96979β228.4912245.9306954.4704990.03794106.4720Mean22.7807629.9092534.7929287.64955102.323210%β011.73519.411511.410591.90098.8490β19.79197.70939.447474.419781.2251β211.14378.923410.670587.751897.1719Mean10.890238.681410.5094784.690592.41533Sample Sizen= 1000%β0—95.07261123.1937100.26721131.6080β1—97.26561110.577997.95562108.6698β2—94.65821125.5736100.14950137.4324Mean—95.66548119.781799.45744125.90345%β017.563323.627824.853789.4943119.9566β113.773313.568813.657687.991098.4320β220.126139.835546.797590.0007128.7815Mean17.154225.677428.436389.1620115.723410%β08.08486.45107.794080.542690.1938β110.64658.496510.2857103.1180112.2725β28.05216.47027.820275.176783.1701Mean8.92787.13938.633386.279195.2121


## 6. Concluding Remarks

This paper examines the performance of classical wild bootstrap techniques which were proposed by Wu [6] and Liu [7] in the presence of heteroscedasticity and outliers. Both the artificial example and simulation study show that the classical bootstrap techniques perform poorly in the presence of outliers in the heteroscedastic model although they perform superbly for “clean” data. We attempt to robustify those classical bootstrap techniques to gain better efficiency in the presence of outliers. The numerical results show that the newly proposed robust wild bootstrap techniques, namely, the RBootwu and RBootliu outperform the classical wild bootstrap techniques when both outliers and heteroscedasticity are present in the data. RBootliu performs slightly better than RBootwu. Another advantage of using the RBootwu and the RBootliu is that no diagnosis for the data is required before the application of these methods.



---

*Source: 730328-2012-02-26.xml*