# Robust AIC with High Breakdown Scale Estimate

Authors: Shokrya Saleh
Journal: Journal of Applied Mathematics
 (2014)

---

## Abstract

Akaike Information Criterion (AIC) based on least squares (LS) regression minimizes the sum of the squared residuals; LS is sensitive to outlier observations. Alternative criterion, which is less sensitive to outlying observation, has been proposed; examples are robust AIC (RAIC), robust Mallows Cp (RCp), and robust Bayesian information criterion (RBIC). In this paper, we propose a robust AIC by replacing the scale estimate with a high breakdown point estimate of scale. The robustness of the proposed methods is studied through its influence function. We show that, the proposed robust AIC is effective in selecting accurate models in the presence of outliers and high leverage points, through simulated and real data examples.


---

## Body

## 1. Introduction

Akaike Information Criterion (AIC) [1] is a powerful technique for model selection, and it has been widely used for selecting models in many fields of study.Consider a multiple linear regression model:(1)yi=μ+xiTβ+ϵi,
where xi=(xi1,…,xip)T is a vector containing p explanatory variables, yi is the response variable, β is a vector of p parameters, μ is the intercept parameter, and ϵi is the error component, which is independent and identically distributed (iid), with mean 0 and variance σ2. The classical AIC is defined as
(2)AIC=ln⁡(SSEn)+2p,
where SSE=∑ri2, with ri=yi-μLS-xiTβ^LS.Since theLS estimator is vulnerable in presence of outliers, it is not surprising that AIC inherits this problem. Several robust AIC alternatives to the AIC have been proposed in the literature (see [2–4]). For an example Ronchetti [3] has proposed and investigated the properties of a robust version of AIC, with respect to M-estimation. A similar idea was used by Martin [2] for autoregressive models. Furthermore, very recently Tharmaratnam and Claeskens [4] have proposed a robust AIC with respect to S-estimation and MM-estimation, to generalize the information criteria, using the full likelihood models. In spite of all these complicated approaches of deriving robust AIC, we introduce a straightforward approach to derive robust AIC, which focuses on modifying the estimate of the scale.To set the idea, the influence of outlier onAIC is illustrated through the presence of outliers in the Y-direction (called a vertical outlier) or in the X-direction (called a leverage point). For this, a point with coordinates (0,y10) is added, where the value of y ranges between (−1.5 and 3). A similar approach is done for leverage points, by replacing the value x with (x10,0) (Figure 1). Table 1 and Figure 2 show that the value of AIC increases as the size of contamination in (x10,y10) increases, as expected, and if x or y is extremely large, the AIC is unbounded; that is, it will tend to infinity.Table 1
AIC for different values ofy10 and x10.
y
10
AIC
x
10
AIC
3
3.2
4.5
3.3
2.5
2.9
4
3.2
2
2.4
3.5
3.0
1.5
1.9
3
2.8
1
1.2
2.5
2.6
0.5
0.2
0
−0.5
0
−0.5
−2.5
2.6
−0.5
0.3
−3
2.9
−1
1.2
−3.5
3.1
−1.5
1.9
−4
3.2Figure 1
Data and positions fory10 and x10 points.Figure 2
Effect of adding one observation(0,y10) or (x10,0) on the values of AIC.The remainder of the paper has been organized as follows. Section1.1 reviews some robust regression estimate methods. In Section 1.2 we review a robust version of AIC; we discuss the robustness problem from the viewpoint of model selection and point out the sensitivity of robust AIC, based on M-estimator to leverage points. We derive the influence function of AIC and study its properties in Section 2. The performance of robust AIC is evaluated and compared to the commonly used classical AIC in Section 3. Finally, concluding remark is presented in Section 4.


### 1.1. A Robust Regression Estimate

TheM-estimator [5] β^M is the value of β that minimizes the following function:
(3)∑i=1i=n‍ρ(yi-μ-xiTβ),
where ρ is symmetric and nondecreasing on [0,∞). Furthermore, ρ(0)=0, and ρ is almost continuously differentiable, anywhere. Furthermore, ρ is a function, which is less sensitive to outliers than squares, yielding the estimating equation:
(4)∑i=1i=n‍ψ(yi-μ-xiTβ^)xi=0,
where ψ=ρ′. If we choose ρ function in (3) as Tukey Biweight function
(5)ρ(r;d)={3(rd)2-3(rd)4+(rd)6,if|r|≤d,1,elsewhere,
where d=1.5476 yields b=EΦ[ρ(Z;d)]=2(1-F0(d)), with Φ the standard normal cumulative distribution function and Z~N(0,1), the resulting estimator is then Biweight S-estimator. M-estimators are efficient and highly robust to unusual value of Y, but one rogue leverage point can break them down completely. For this reason, generalized M-estimators were introduced, which solve
(6)∑i=1i=n‍w(xi)ψ(yi-μ-xiTβ^)xi=0,
where w is a weight function [6].In recent years, a good deal of attention in the literature has been focused on high-breakdown methods; that is, one method must be resistant to even multiple severe outliers. Many methods are based on minimizing a more robust scale estimate than sum of squared residuals. For example, Rousseeuw [7] proposed LMS, a high-breakdown method that minimizes the median of the squared residuals, rather than their sum. In addition, Rousseeuw [8] proposed least trimmed squares (LTS), which minimizes the sum of the H smallest squared residuals, defined as
(7)β^LTS,H,N=argmin⁡∑i=1i=H‍ri2(β),
based on the ordered absolute residuals |r(1)|⩽|r(2)|⩽⋯⩽|r(n)|. LTS converges at the rate of n1/2 with the same asymptotic efficiency under normality, as Huber’s skip estimator. The convergence rate of LMS is n1/3, and its objective function is less smooth than LTS.


### 1.2. The Robust Version of AIC

Consider scale estimate of errors defined by(8)S=SSE(n-p),
with SSE=∑i=1i=n‍ri2 and ri=yi-μ^-xiTβ^. By replacing the value of SSE in (2) in terms of S, AIC in (2) can be expressed as follows:
(9)AICS=ln⁡((n-p)S2n)+2p.
Notice that the largest values of AIC indicate that the model (and hence the explanatory variables) is less successful in explaining the variations in the response, while a small value of AIC indicates an excellent fit (i.e., model) to the response data.Ronchetti [3] proposed a robust counterpart of the AIC statistic. The extension of AIC to RAIC is inspired by the extension of maximum likelihood estimation to M-estimation. The author derived RAIC for an error distribution with density function f(ϵ)=exp⁡(-ρ(ϵ)). For a given constant α and a given function ρ, the author chose the model that minimizes
(10)RAIC(p,α,ρ)=∑i=1i=n‍ρ((yi-μ-xiTβ^M)σ^)+αp,
where σ^ is some robust estimate of σ and β^M is the M-estimator defined as in (3). Huber [9] suggested α=2E([ψ2(((yi-μ-xiTβ^))/σ^)])/(E[ψ′(((yi-μ-xiTβ^))/σ^)]) and k=1.345.We introduce an alternative robust version ofAIC, by replacing S in (8) to the robust estimator of scale which attains a 50% breakdown point. When H=n/2, (8) finds the estimates corresponding to the half samples, having the smallest sum of squares of residuals. As such, as expected, the breakdown point is 50%, and the estimated scale from LTS is
(11)σ^(LTS,H,N)2=1H∑i=1i=H‍r(i)2(β^(LTS,H,N)).
For other robust estimations, the M-estimator and the Biweight S-estimator are compared to least trimmed of squares. Based on the results shown in Table 2, it is evident that the M-estimator is much more robust than LS but suffers from leverage points. The Biweight S-estimator and LTS show robust behavior: the AICBS is stable, even though the size of the outliers increases. In the next section, we generalize these findings, by computing the associated influence functions.Table 2
Robust AIC for different values ofy10 (vertical) and x10 (leverage).
y
10
AI
C
M
AI
C
BS
AI
C
LTS
x
10
AI
C
M
AI
C
BS
AI
C
LTS
3
3.7
3.9
3.4
4.5
3.9
3.9
3.4
2.5
3.7
3.9
3.4
4
3.7
3.9
3.4
2
3.7
3.9
3.4
3.5
3.6
3.9
3.4
1.5
3.7
3.9
3.4
3
3.6
3.9
3.4
1
3.7
3.9
3.4
2.5
3.7
3.9
3.4
0.5
3.7
3.9
3.4
0
3.5
3.6
3.2
0
3.6
3.6
3.2
−2.5
3.7
3.9
3.4
−0.5
3.7
3.9
3.4
−3
3.7
3.9
3.4
−1
3.7
3.9
3.4
−3.5
3.9
3.9
3.4
−1.5
3.7
3.9
3.4
−4
4
3.9
3.4


## 1.1. A Robust Regression Estimate

TheM-estimator [5] β^M is the value of β that minimizes the following function:
(3)∑i=1i=n‍ρ(yi-μ-xiTβ),
where ρ is symmetric and nondecreasing on [0,∞). Furthermore, ρ(0)=0, and ρ is almost continuously differentiable, anywhere. Furthermore, ρ is a function, which is less sensitive to outliers than squares, yielding the estimating equation:
(4)∑i=1i=n‍ψ(yi-μ-xiTβ^)xi=0,
where ψ=ρ′. If we choose ρ function in (3) as Tukey Biweight function
(5)ρ(r;d)={3(rd)2-3(rd)4+(rd)6,if|r|≤d,1,elsewhere,
where d=1.5476 yields b=EΦ[ρ(Z;d)]=2(1-F0(d)), with Φ the standard normal cumulative distribution function and Z~N(0,1), the resulting estimator is then Biweight S-estimator. M-estimators are efficient and highly robust to unusual value of Y, but one rogue leverage point can break them down completely. For this reason, generalized M-estimators were introduced, which solve
(6)∑i=1i=n‍w(xi)ψ(yi-μ-xiTβ^)xi=0,
where w is a weight function [6].In recent years, a good deal of attention in the literature has been focused on high-breakdown methods; that is, one method must be resistant to even multiple severe outliers. Many methods are based on minimizing a more robust scale estimate than sum of squared residuals. For example, Rousseeuw [7] proposed LMS, a high-breakdown method that minimizes the median of the squared residuals, rather than their sum. In addition, Rousseeuw [8] proposed least trimmed squares (LTS), which minimizes the sum of the H smallest squared residuals, defined as
(7)β^LTS,H,N=argmin⁡∑i=1i=H‍ri2(β),
based on the ordered absolute residuals |r(1)|⩽|r(2)|⩽⋯⩽|r(n)|. LTS converges at the rate of n1/2 with the same asymptotic efficiency under normality, as Huber’s skip estimator. The convergence rate of LMS is n1/3, and its objective function is less smooth than LTS.


## 1.2. The Robust Version of AIC

Consider scale estimate of errors defined by(8)S=SSE(n-p),
with SSE=∑i=1i=n‍ri2 and ri=yi-μ^-xiTβ^. By replacing the value of SSE in (2) in terms of S, AIC in (2) can be expressed as follows:
(9)AICS=ln⁡((n-p)S2n)+2p.
Notice that the largest values of AIC indicate that the model (and hence the explanatory variables) is less successful in explaining the variations in the response, while a small value of AIC indicates an excellent fit (i.e., model) to the response data.Ronchetti [3] proposed a robust counterpart of the AIC statistic. The extension of AIC to RAIC is inspired by the extension of maximum likelihood estimation to M-estimation. The author derived RAIC for an error distribution with density function f(ϵ)=exp⁡(-ρ(ϵ)). For a given constant α and a given function ρ, the author chose the model that minimizes
(10)RAIC(p,α,ρ)=∑i=1i=n‍ρ((yi-μ-xiTβ^M)σ^)+αp,
where σ^ is some robust estimate of σ and β^M is the M-estimator defined as in (3). Huber [9] suggested α=2E([ψ2(((yi-μ-xiTβ^))/σ^)])/(E[ψ′(((yi-μ-xiTβ^))/σ^)]) and k=1.345.We introduce an alternative robust version ofAIC, by replacing S in (8) to the robust estimator of scale which attains a 50% breakdown point. When H=n/2, (8) finds the estimates corresponding to the half samples, having the smallest sum of squares of residuals. As such, as expected, the breakdown point is 50%, and the estimated scale from LTS is
(11)σ^(LTS,H,N)2=1H∑i=1i=H‍r(i)2(β^(LTS,H,N)).
For other robust estimations, the M-estimator and the Biweight S-estimator are compared to least trimmed of squares. Based on the results shown in Table 2, it is evident that the M-estimator is much more robust than LS but suffers from leverage points. The Biweight S-estimator and LTS show robust behavior: the AICBS is stable, even though the size of the outliers increases. In the next section, we generalize these findings, by computing the associated influence functions.Table 2
Robust AIC for different values ofy10 (vertical) and x10 (leverage).
y
10
AI
C
M
AI
C
BS
AI
C
LTS
x
10
AI
C
M
AI
C
BS
AI
C
LTS
3
3.7
3.9
3.4
4.5
3.9
3.9
3.4
2.5
3.7
3.9
3.4
4
3.7
3.9
3.4
2
3.7
3.9
3.4
3.5
3.6
3.9
3.4
1.5
3.7
3.9
3.4
3
3.6
3.9
3.4
1
3.7
3.9
3.4
2.5
3.7
3.9
3.4
0.5
3.7
3.9
3.4
0
3.5
3.6
3.2
0
3.6
3.6
3.2
−2.5
3.7
3.9
3.4
−0.5
3.7
3.9
3.4
−3
3.7
3.9
3.4
−1
3.7
3.9
3.4
−3.5
3.9
3.9
3.4
−1.5
3.7
3.9
3.4
−4
4
3.9
3.4


## 2. Influence Function

Consider the linear model in (1), for i=1,…,n. Assume the distribution of errors satisfying Fσ(x)=F0(x/σ), where σ is the residual scale parameter and F0 is symmetric with valid probability density function.Letx and y be independent stochastic variables with distribution H. The functional T is Fisher-consistent for the parameters (μ,β) at the model distribution H, as follows:
(12)T(H)=[a(H)b(H)]=[μβ,].
For a Fisher-consistent scale estimator, Fσ(x)=F(x/σ), for all σ>0. In general, the influence function of T at the distribution F is defined as
(13)IF((x,y),T,H)=lim⁡ϵ→0⁡T((1-ϵ)H+ϵΔ(x,y)-T(H))ϵ=∂∂ϵ(T(Δ(x,y))),
where T(H) is the functional defined as the solution of the objective model and Δ(x,y) is the distribution which contains outliers. The following theorem gives the influence function of AIC with any scale S.Theorem 1.
LetH be some distribution other than F. Take (X,Y)~H and denote by ϵ the error term of the model. Assume that S has the property that is differentiable with partial derivatives equal to zero at the origin (0,0). Then,
(14)IF((x,y),AICS,H)=2n(n-p)IF(ripσp,S,F0),
where rip=yi-μ^-xiTβ^p and σ^p2=SSE/(n-p) are computed from sum model. (The proof is in the Appendix.)It is clear that the influence function in (13) is bounded if the IF(rip/σp,S,F0) is also bounded. It is evident that AIC is nonrobust, since the LS estimate has unbounded influence function. The influence function of M-estimation with respect to Y is bounded by choice of ψ, but it is unbounded with respect to the X direction. That is,
(15)IF((x,y),T,F)=M-1(ψ,F)xψ(y-xTT(F)),
where ψ=ρ′ and M is a certain p×p matrix given by
(16)M(ψ,F)=∫ψ′(y-xTT(F))xxTdF(x,y).
The influence function of the AIC using LTS estimators, following Theorem 1, takes the form
(17)IF((x,y),AICLTS,H)=2n(n-p)·IF(ripσp,σ^LTS2,F0).
We have noted that the influence function of AICLTS is bounded in both Y and X directions, as IF(rip/σp,σ^LTS2,F0) is bounded. Moreover, we conclude that AIC with high breakdown point estimator provides reliable fits to the data in presence of leverage points.


## 3. Numerical Examples

In this section, AICLTS, AICLMS, and AICBS are compared with AIC and RAIC. In this study, 50 independent replicates of 3 independent uniform random variables on [-1,1] of xi1,xi2, and xi3 and 50 independent normally distributed errors ϵi~N(0,9) were generated. The true model is given by yi=xi1+xi2+ϵi, for i=1,…,50, using two variables xi1 and xi2. We have computed the following AIC, based on the respective estimation criterion: (i) LS estimation; (ii) M-estimator; (iii) BS estimator; (iv) LTS estimator; and (v) LMS estimator. In order to illustrate the robustness to outliers, we consider the following cases:(a)
vertical outliers (outliers in they only),(b)
good leverage points (outliers in they and x),(c)
bad leverage points (outliers in thex only).For all situations, we randomly generate (0%, 5%, 10%, 20%, 30%, and 40%) of outliers fromN(50,0.12) and N(100,0.52) distributions, respectively. For each of these settings we simulate 1000 samples.


### 3.1. Simulation Result

The resulting fit to the data is classified as one of the following:(i)
correct fit (true model);(ii)
overfit (models containing all the variables in the true model plus other variables that are redundant);(iii)
under fit (models with only a strict of the variables in true model);(iv)
wrong fit (model that are neither of the above).Tables3, 4, and 5 show detailed simulation result for different versions of AIC methods. For uncontaminated datasets, the classical AIC performs best, compared to robust AIC. By introducing vertical outliers, the classical AIC selects a large proportion of wrong fit models and, as expected, the robust AIC will usually (i.e., with higher proportion) select the correct model.Table 3
Percentage of select models from classical AIC, robust RAIC,AICLTS, AICLMS, and AICBS, with vertical outliers.
ϵ   %
AIC
RAIC
AI
C
LTS
AI
C
LMS
AI
C
BS
0
Correct fit
84.6%
54.4%
57.6%
65.2%
45.2%
Overfit
15.4%
0%
0%
0%
0%
Under fit
0%
43.6%
41.2%
34%
54.6%
Wrong fit
0%
2.0%
1.2%
0.8%
0.2%
5
Correct fit
2.8%
49.6%
56.8%
62.6%
45.4%
Overfit
2%
0%
0%
0%
0%
Under fit
22%
51%
42.8%
36.6%
54.6%
Wrong fit
73.2%
1.4%
0.4%
0.8%
0%
10
Correct fit
3.2%
45.0%
51%
56%
39.8%
Overfit
0.8%
0%
0%
0%
0%
Under fit
20.2%
52.4%
48%
43.8%
55.2%
Wrong fit
75.8%
2.6%
1.0%
0.2%
0%
20
Correct fit
4.2%
30.8%
49.4%
58%
34.2%
Overfit
0.2%
0%
0%
0%
0%
Under fit
22.8%
67.8%
49.6%
41.4%
65.8%
Wrong fit
72.8%
1.4%
1.0%
0.6%
0%
30
Correct fit
1.6%
0%
44.0%
51.2%
23.2%
Overfit
0.2%
0%
0%
0%
0%
Under fit
22.6%
73.8%
55.4%
48.2%
76.8%
Wrong fit
72.8%
26.2%
0.6%
0.6%
0%
40
Correct fit
2.2%
0%
37.6%
41.2%
12.4%
Overfit
0.6%
0%
0%
0%
0%
Under fit
22.8%
70.8%
62.4%
58.4%
87.4%
Wrong fit
74.4%
29.2%
0%
0.4%
0.2%Table 4
Percentage of select models from classical AIC, robust RAIC, and robustAICLTS, AICLMS, and AICBS with bad leverage points.
ϵ  %
AIC
RAIC
AI
C
LTS
AI
C
LMS
AI
C
BS
5
Correct fit
0%
0%
54.6%
60.8%
43.8%
Overfit
70.4%
0%
0%
0%
0%
Under fit
0%
64.4%
44%
38.4%
55.4%
Wrong fit
29.6%
35.6%
1.4%
0.8%
0.8%
10
Correct fit
0%
0%
63.8%
67.8%
51.0%
Overfit
63%
0%
0%
0%
0%
Under fit
0%
54.6%
34.6%
31.4%
48.8%
Wrong fit
37%
42.4%
1.6%
0.8%
0.2%
20
Correct fit
0%
0%
56.6%
63.4%
49.2%
Overfit
54.8%
0%
0%
0%
0%
Under fit
0.2%
60.8%
42.4%
35.8%
50.6%
Wrong fit
44.8%
39.2%
1.0%
0.8%
0.2%
30
Correct fit
0%
0%
56.6%
61.4%
46%
Overfit
37.6%
0%
0%
0%
0%
Under fit
0.25%
29.6%
42.6%
36%
53.8%
Wrong fit
60.4%
42.6%
0.8%
2.4%
0.2%
40
Correct fit
1.0%
0%
55.2%
64.6%
51.4%
Overfit
13.8%
0%
0%
0%
0%
Under fit
1.2%
54.4%
43.8%
33.8%
48.6%
Wrong fit
81%
45.4%
1.0%
1.6%
0%Table 5
Percentage of select models from classical AIC, robust RAIC, and robustAICLTS, AICLMS, and AICBS, with good leverage points.
ϵ   %
AIC
RAIC
AICLTS
AICLMS
AICBS
5
Correct fit
0.2%
47.0%
53.6%
58.4%
42.6%
Overfit
99.8%
0%
0%
0%
0%
Under fit
0%
50.4%
46%
41.2%
57.4%
Wrong fit
0%
2.6%
0.4%
0.4%
0%
10
Correct fit
0%
44.2%
54.6%
59.4%
40.4%
Overfit
99.6%
0%
0%
0%
0%
Under fit
0.2%
53.8%
44.6%
39.8%
59.6%
Wrong fit
0.2%
2.0%
0.8%
0.8%
0%
20
Correct fit
0.8%
32.4%
50.4%
56.2%
33.2%
Overfit
97.6%
0%
0%
0%
0%
Under fit
0.8%
66.6%
48.4%
43%
66.4%
Wrong fit
0.8%
1.0%
1.2%
0.8%
0.2%
30
Correct fit
1.8%
0%
46.0%
50.6%
27.0%
Overfit
97.8%
96.8%
0%
0%
0%
Under fit
2.8%
2.8%
53.6%
49.2%
72.8%
Wrong fit
0.6%
0.4%
0.4%
0.2%
0.2%
40
Correct fit
0.2%
0%
37.4%
37%
13.8%
Overfit
97.4%
100%
0%
0%
0%
Under fit
2.2%
0%
62.6%
62.6%
86.2%
Wrong fit
0.2%
0%
0%
0.4%
0%For bad leverage points, we observe thatAIC tends to produce overfit and with high level of contamination it takes a higher proportion of wrong fit. However, AICM tend to produce either an under fit or wrong fit model. However, the robust estimate produces comparable power in the presence of bad leverage points.For good leverage points,AIC tends, also, to produce overfit. On the other hand, the robust AIC tend to produce either correct fit or under fit model.


### 3.2. Example 2 (Stack Loss Data)

Stack loss data was presented by [10]. This data set consists of 21 observations on three independent variables, and it contains four outliers (cases 1, 3, 4, and 21) and high leverage points (cases 1, 2, 3, and 21). The data are given in Table 6.Table 6
Stack loss data set.
x
1
x
2
x
3
y
80
27
89
42
80
27
88
37
75
25
90
37
62
24
87
28
62
22
87
18
62
23
87
18
62
24
93
19
62
24
93
20
58
23
87
15
58
18
80
14
58
18
89
14
58
17
88
13
58
18
82
11
58
19
93
12
50
18
89
8
50
18
86
7
50
19
72
8
50
19
79
8
50
20
80
9
56
20
82
15
70
20
91
15We applied the traditional and robust versions ofAIC methods on the data. Table 7 shows that the classical method selects the full model, and robust RAIC method ignored one of the important variables (xi2), whereas robust AIC methods, based on high break down points estimators, agreed with the importance of the two variables, xi1 and xi2.Table 7
The result variable selection of stack loss data.
Selected variables
AIC
RAIC
AI
C
LTS
AI
C
LMS
AI
C
BS
x
1
6.7
8.0
4.7
4.3
5.5
x
2
7.1
6.5
5.9
5.4
7.0
x
3
8.4
7.3
7.0
6.4
7.3
x
1
,
x
2
8.2
9.0
5.5
4.7
6.9
x
1
,
x
3
8.7
8.9
6.9
6.3
7.6
x
2
,
x
3
9.1
9.0
8.1
7.3
8.8
x
1
,
x
2
,
x
3
4.7
10.6
7.6
6.7
9.1


## 3.1. Simulation Result

The resulting fit to the data is classified as one of the following:(i)
correct fit (true model);(ii)
overfit (models containing all the variables in the true model plus other variables that are redundant);(iii)
under fit (models with only a strict of the variables in true model);(iv)
wrong fit (model that are neither of the above).Tables3, 4, and 5 show detailed simulation result for different versions of AIC methods. For uncontaminated datasets, the classical AIC performs best, compared to robust AIC. By introducing vertical outliers, the classical AIC selects a large proportion of wrong fit models and, as expected, the robust AIC will usually (i.e., with higher proportion) select the correct model.Table 3
Percentage of select models from classical AIC, robust RAIC,AICLTS, AICLMS, and AICBS, with vertical outliers.
ϵ   %
AIC
RAIC
AI
C
LTS
AI
C
LMS
AI
C
BS
0
Correct fit
84.6%
54.4%
57.6%
65.2%
45.2%
Overfit
15.4%
0%
0%
0%
0%
Under fit
0%
43.6%
41.2%
34%
54.6%
Wrong fit
0%
2.0%
1.2%
0.8%
0.2%
5
Correct fit
2.8%
49.6%
56.8%
62.6%
45.4%
Overfit
2%
0%
0%
0%
0%
Under fit
22%
51%
42.8%
36.6%
54.6%
Wrong fit
73.2%
1.4%
0.4%
0.8%
0%
10
Correct fit
3.2%
45.0%
51%
56%
39.8%
Overfit
0.8%
0%
0%
0%
0%
Under fit
20.2%
52.4%
48%
43.8%
55.2%
Wrong fit
75.8%
2.6%
1.0%
0.2%
0%
20
Correct fit
4.2%
30.8%
49.4%
58%
34.2%
Overfit
0.2%
0%
0%
0%
0%
Under fit
22.8%
67.8%
49.6%
41.4%
65.8%
Wrong fit
72.8%
1.4%
1.0%
0.6%
0%
30
Correct fit
1.6%
0%
44.0%
51.2%
23.2%
Overfit
0.2%
0%
0%
0%
0%
Under fit
22.6%
73.8%
55.4%
48.2%
76.8%
Wrong fit
72.8%
26.2%
0.6%
0.6%
0%
40
Correct fit
2.2%
0%
37.6%
41.2%
12.4%
Overfit
0.6%
0%
0%
0%
0%
Under fit
22.8%
70.8%
62.4%
58.4%
87.4%
Wrong fit
74.4%
29.2%
0%
0.4%
0.2%Table 4
Percentage of select models from classical AIC, robust RAIC, and robustAICLTS, AICLMS, and AICBS with bad leverage points.
ϵ  %
AIC
RAIC
AI
C
LTS
AI
C
LMS
AI
C
BS
5
Correct fit
0%
0%
54.6%
60.8%
43.8%
Overfit
70.4%
0%
0%
0%
0%
Under fit
0%
64.4%
44%
38.4%
55.4%
Wrong fit
29.6%
35.6%
1.4%
0.8%
0.8%
10
Correct fit
0%
0%
63.8%
67.8%
51.0%
Overfit
63%
0%
0%
0%
0%
Under fit
0%
54.6%
34.6%
31.4%
48.8%
Wrong fit
37%
42.4%
1.6%
0.8%
0.2%
20
Correct fit
0%
0%
56.6%
63.4%
49.2%
Overfit
54.8%
0%
0%
0%
0%
Under fit
0.2%
60.8%
42.4%
35.8%
50.6%
Wrong fit
44.8%
39.2%
1.0%
0.8%
0.2%
30
Correct fit
0%
0%
56.6%
61.4%
46%
Overfit
37.6%
0%
0%
0%
0%
Under fit
0.25%
29.6%
42.6%
36%
53.8%
Wrong fit
60.4%
42.6%
0.8%
2.4%
0.2%
40
Correct fit
1.0%
0%
55.2%
64.6%
51.4%
Overfit
13.8%
0%
0%
0%
0%
Under fit
1.2%
54.4%
43.8%
33.8%
48.6%
Wrong fit
81%
45.4%
1.0%
1.6%
0%Table 5
Percentage of select models from classical AIC, robust RAIC, and robustAICLTS, AICLMS, and AICBS, with good leverage points.
ϵ   %
AIC
RAIC
AICLTS
AICLMS
AICBS
5
Correct fit
0.2%
47.0%
53.6%
58.4%
42.6%
Overfit
99.8%
0%
0%
0%
0%
Under fit
0%
50.4%
46%
41.2%
57.4%
Wrong fit
0%
2.6%
0.4%
0.4%
0%
10
Correct fit
0%
44.2%
54.6%
59.4%
40.4%
Overfit
99.6%
0%
0%
0%
0%
Under fit
0.2%
53.8%
44.6%
39.8%
59.6%
Wrong fit
0.2%
2.0%
0.8%
0.8%
0%
20
Correct fit
0.8%
32.4%
50.4%
56.2%
33.2%
Overfit
97.6%
0%
0%
0%
0%
Under fit
0.8%
66.6%
48.4%
43%
66.4%
Wrong fit
0.8%
1.0%
1.2%
0.8%
0.2%
30
Correct fit
1.8%
0%
46.0%
50.6%
27.0%
Overfit
97.8%
96.8%
0%
0%
0%
Under fit
2.8%
2.8%
53.6%
49.2%
72.8%
Wrong fit
0.6%
0.4%
0.4%
0.2%
0.2%
40
Correct fit
0.2%
0%
37.4%
37%
13.8%
Overfit
97.4%
100%
0%
0%
0%
Under fit
2.2%
0%
62.6%
62.6%
86.2%
Wrong fit
0.2%
0%
0%
0.4%
0%For bad leverage points, we observe thatAIC tends to produce overfit and with high level of contamination it takes a higher proportion of wrong fit. However, AICM tend to produce either an under fit or wrong fit model. However, the robust estimate produces comparable power in the presence of bad leverage points.For good leverage points,AIC tends, also, to produce overfit. On the other hand, the robust AIC tend to produce either correct fit or under fit model.


## 3.2. Example 2 (Stack Loss Data)

Stack loss data was presented by [10]. This data set consists of 21 observations on three independent variables, and it contains four outliers (cases 1, 3, 4, and 21) and high leverage points (cases 1, 2, 3, and 21). The data are given in Table 6.Table 6
Stack loss data set.
x
1
x
2
x
3
y
80
27
89
42
80
27
88
37
75
25
90
37
62
24
87
28
62
22
87
18
62
23
87
18
62
24
93
19
62
24
93
20
58
23
87
15
58
18
80
14
58
18
89
14
58
17
88
13
58
18
82
11
58
19
93
12
50
18
89
8
50
18
86
7
50
19
72
8
50
19
79
8
50
20
80
9
56
20
82
15
70
20
91
15We applied the traditional and robust versions ofAIC methods on the data. Table 7 shows that the classical method selects the full model, and robust RAIC method ignored one of the important variables (xi2), whereas robust AIC methods, based on high break down points estimators, agreed with the importance of the two variables, xi1 and xi2.Table 7
The result variable selection of stack loss data.
Selected variables
AIC
RAIC
AI
C
LTS
AI
C
LMS
AI
C
BS
x
1
6.7
8.0
4.7
4.3
5.5
x
2
7.1
6.5
5.9
5.4
7.0
x
3
8.4
7.3
7.0
6.4
7.3
x
1
,
x
2
8.2
9.0
5.5
4.7
6.9
x
1
,
x
3
8.7
8.9
6.9
6.3
7.6
x
2
,
x
3
9.1
9.0
8.1
7.3
8.8
x
1
,
x
2
,
x
3
4.7
10.6
7.6
6.7
9.1


## 4. Conclusions

The least trimmed squares (LTS) and the least median squares (LMS) are robust regression methods, frequently used in practice. Nevertheless, they are not commonly used for selecting models. This paper introduced the Akaike Information Criterion (AIC) based on LTS and LMS scales, which are robust against outliers and leverage point. Our simulation result illustrated excellent performance of AICLTS and AICLMS for contaminated data sets. This paper focused on the AIC variable selection criteria; one might be interested in extending other robust model selection criteria to advanced robust breakdown point estimation methods, such as LTS, LMS, or BS estimators. In addition, this paper has considered regression model with continuous variables; however, future studies might consider mixed variables (i.e., continuous and dummy).



---

*Source: 286414-2014-09-08.xml*