# Data Transformation Technique to Improve the Outlier Detection Power of Grubbs’ Test for Data Expected to Follow Linear Relation

Authors: K. K. L. B. Adikaram; M. A. Hussein; M. Effenberger; T. Becker
Journal: Journal of Applied Mathematics
 (2015)

---

## Abstract

Grubbs test (extreme studentized deviate test, maximum normed residual test) is used in various fields to identify outliers in a data set, which are ranked in the order ofx1≤x2≤x3≤⋯≤xn(i=1,2,3,…,n). However, ranking of data eliminates the actual sequence of a data series, which is an important factor for determining outliers in some cases (e.g., time series). Thus in such a data set, Grubbs test will not identify outliers correctly. This paper introduces a technique for transforming data from sequence bound linear form to sequence unbound form (y=c). Applying Grubbs test to the new transformed data set detects outliers more accurately. In addition, the new technique improves the outlier detection capability of Grubbs test.  Results show that, Grubbs test was capable of identifing outliers at significance level 0.01 after transformation, while it was unable to identify those prior to transforming at significance level 0.05.


---

## Body

## 1. Introduction

Grubbs test [1] is a statistical test used to detect outliers which was introduced in 1950 and extended in 1969 [2] and 1972 [3] by the same author. Grubbs test locates outliers that exist in a univariate data set using mean, standard deviation, and tabulated criterion. Grubbs test is also known as maximum normed residual test or “extreme studentized deviate” (ESD) test, and the data set is assumed to be normally distributed. The test is defined as
(1)G=max⁡i=1,…,NYi-Y-s,
where s is standard deviation and Y- is the sample mean. If the maximum G related to the ith element is greater than the relevant tabulated criterion, then the element is considered an outlier. The testing procedure is continuing until no more outliers are detected. However Grubbs test is not recommended for detecting outliers for sample size of six or less. When the sample size is six or less, most of the times Grubbs test identified nonoutliers as outliers [4].During the last decades Grubbs test was used to identify outliers in different disciplines [5–9]. Also, during the last decades pros and cons of  Grubbs test were identified and were improved as well. In 1975 Rosner showed that Grubbs test (ESD) performs much better than studentized range methods and performs equally as Kurtosis and R-statistic methods [10]. In 1983 Rosner introduced an improved version of ESD as generalized extreme studentized deviate (GESD) test [11]. However, GESD does not work well when the sample size is less than 25 [11]. Brant in 1990 stated that the combination ESD rules and boxplot provide comparable performance [12]. On the other hand, it was shown that the standard deviation and mean are affected by two or more outliers; Grubbs test does not detect outliers [13] correctly. Also, if the standard deviation of the data set is too large or too small, the test will tend to detect false outliers and vice versa. This was overcome by setting a threshold value for standard deviations for the specific considered data domain [13]. Meanwhile, some publications show that Grubbs test is robust against the effect of intraclass correlation structure [14] and data that have Baldessari’s structure [15, 16].Form the definition of Grubbs test, it locates outliers in a data set which are ranked in the order ofx1≤x2≤x3≤⋯≤xn. This implies that Grubbs test considers only the value of data points but not the real order of the data. In other words, Grubbs test treats sorted series and unsorted series in the same manner. Thus, Grubbs test is valid only for those data domains where the occurrence order is of no importance. However, with respect to outliers, the order of the data points is a very important factor for data series that are expected to have gradual increment or decrement over time. Thus, applying Grubbs test to data which has a relation with the occurrence order will not give a correct output. This is particularly important in the area of process control where the order of the data points has a very high impact on data interpretation. Therefore, Grubbs test is not a reliable method for detecting outliers in time series, because in time series occurrence order is a critical factor.Grubbs test is capable of checking whether a certain suspected data point is an outlier. By default suspected points are the minimum and the maximum of the data set. If the most suspected data points are not outliers, Grubbs test does not identify other data points as outliers. Figure1 shows two artificial data sets (data sets 1 and 2) with one outlier in each data set. Table 1 shows results of Grubbs test for two significance levels (α) of 0.05 and 0.01. In both data sets, the test does not detect outliers for any considered significance level. In data set 1, the outlier is not significant enough to be identified by Grubbs test. Although the outlier in data set 2 (190) deviated significantly in relation to its position, after ranking it moves to the end of the series and becomes an insignificant outlier.Table 1
Results of Grubbs’ test (two-sided) for data set 1 and data set 2 for significance levels (α) of 0.05 (critical value of G=2.29) and  α=0.01 (G = 2.48).
Data set 1
G
Outlier?
Data set 2
G
Outlier?
α
=
0.05
α
=
0.01
α
=
0.05
α
=
0.01
1
30
1.22
No
No
30
1.34
No
No
2
35
0.98
No
No
190
2.24
No
No
3
40
0.73
No
No
50
0.89
No
No
4
45
0.49
No
No
60
0.67
No
No
5
50
0.24
No
No
70
0.45
No
No
6
55
0
No
No
80
0.22
No
No
7
60
0.24
No
No
90
0
No
No
8
65
0.49
No
No
100
0.22
No
No
9
70
0.73
No
No
110
0.45
No
No
10
100
2.20
No
No
120
0.67
No
NoFigure 1
Two data sets, each containing one outlier which cannot be located by Grubbs test. The outlier in data set 1 is the last element of the series and considerably deviates from the expected value. The outlier in data set 2 is the maximum of the series, but it is not the first or last element.The aim of this paper is to introduce a method for transforming “sequence bound" data into a “sequence unbound” form. Since the transformed series is totally independent of the sequence, applying Grubbs test could produce more robust results. Furthermore, the transformation increases the outlier detection capability of Grubbs test for data which are expected to have linear or nearly linear relation.


## 2. Methodology

Data transformation techniques are used to convert data status that is closer to the requirements of the technique or method to be applied [17]. The transformation process converts each data point of xi into the transformed value yi by means of a function f, where yi=f(xi). Since Grubbs test is not suitable for detecting outliers in sequence bound series such as time series, one solution is to transform the sequence bound series into a sequence unbound series. In the domain of linear regression, any curve with the form of y=c, value of any data point is always constant. Therefore, any curve with the form of y=c is a curve that is independent of the sequence.Lemma 1.
If it is possible to find a proper reference curvefR  for any curve fA which has the same domain as fR, it is possible to transfer fA into a constant.Proof.
IffA represents the curve of actual data and d is a constant, the function fA′=fA+d has the same domain. However, fA′ has a different range than fA. If the curve of fA-fA′=fD then fD=d. Since fA′ is f(fA), fD is also f(fA). Then, fD can be considered as a transformation form of fA, which is equal to a constant, and fA′ can be considered as the fR, which is the reference curve.
The curvefD, which is the transformed form of fA, has a simpler form than fA. Also, fD can be used to describe the behaviour of fA. Because fD=d, then fD is independent of the sequence of the data. Therefore, because Grubbs test gives correct detections with the data sets that are independent of the occurrence sequence, fD is a suitable data set, which can be tested with Grubbs test. Figures 2 and 3 illustrate usage of the above-mentioned concept for outlier detection.Figure 2
IffA represents the curve of actual data and if it is possible to find a curve fD=fA-fR≈c where fR is a reference curve that represents actual behaviour of the real data, fD reflects the existing outliers of fA. When there are no outliers, difference curve (fD) is always nearly a constant.Figure 3
For an outlier, difference curve (fD) shows significant deviation in relation to the outlier.In the real world, it is not always possible to find the exactfR for a certain data set in advance. Thus, if fR has an approximate relation to the behaviour of the real data, then fD≈c for all data elements (Figure 2). If the actual curve has abnormal data (outlier), fD shows higher deviation from c (Figure 3). Applying Grubbs test to fD the suspected element can be checked for an outlier.WhenfR is known in advance it is possible to apply this method for any data set of any form. fR can be known in advance theoretically or by means of preknowledge of data. If these two options are not available, one possibility is to derive fR from existing data of original data (fA). This paper shows a method of deriving fR for a data set that is expected to have linear (y=mx+c) form, using the original data (fA).For anyfA=mx+c, the curve fA′=mx is a curve which has the same gradient as fA, where m is the gradient of fA. Then,
(2)fA-fA′=mx+c-mx=c.
According to (2) the curve fA-fA′ is a constant. Therefore, for any linear function fA=mx+c the function y=mx can be considered as the reference function fR. In other words fR is the curve, which goes through the origin with the same gradient as fA. As shown in Figure 4, y=mx form (fR) can be considered as the transformation of y=mx+c form (fA). Because fA-fR=c (or y≈c), then fD=fA-fR. Since the gradient of fR:m=tanθ, it can be calculated either by using known theoretical and/or practical information or by deriving it from existing data. We focus on deriving the gradient of fR using a part of original data (fA). When deriving any information from existing original data (fA), the influence of outliers introduces distortions to the derived value. Outlier detections methods are used to remove such data points. However, when detecting outliers this is not a feasible solution. Therefore, when detecting outliers, the best solution is to exclude all suspected data points to minimize the influence of outliers to identify outliers.Figure 4
Transformation technique: if it is possible to findfR which has the form of y=mx in relation to fA, then it is possible to find fD as fA-fR.Unlike most of outlier detection methods, Grubbs test always considers the maximum and the minimum as most suspected data points. Thus, we excluded the maximum and the minimum from the calculations. After removing the maximum and the minimum, the original series splits into a maximum of three small series (Figure4: Segment 1, Segment 2, and Segment 3 of fA). If there are equal maximum values and minimum values, the value with low index is considered as the maximum and the value with high index is considered as the minimum for a data series with increment. For a data series with decrement the value with high index is considered as the maximum and the value with low index is considered as the minimum.Then the series with the highest number of consecutive items (longest series) was considered for calculating the gradient offR. If the longest series of fA is fAL and GAL≡(XAL,YAL) then
(3)YAL=∑i=n1n2yin2-n1+1,XAL=n1+n22,
where n1 is the starting index of the fAL and n2 is the end index of the fAL.IfPi≡(xAi,yAi) is any point of fAL, then
(4)mi=(yAi-YAL)(xAi-XAL),
where mi is the gradient of fAL at point i.AllPi’s on fAL are not suspected data points and candidates for calculating the gradient of fAL. However, among all the data points of fAL, still it is not possible to determine the most suitable point for calculating mi. If the selected Pi is a bias data point (e.g., point B1 in Figure 4), it may introduce distortions to the calculated mi even though it is not suspected. Therefore, it is necessary to have a more reliable method for calculating the gradient of fAL. If the average of all gradients at all Pi’s is considered, it will provide much better approximation for gradient instead of a gradient derived by referring to a certain single point. Therefore, if the resultant gradient of fAL is mfAL, then mfALcan be defined as the mean of all mi’s. Then,
(5)mfAL=∑i=n1n2mi(n2-n1+1).
From (4),
(6)mfAL=∑i=n1n2yAi-YAL/xAi-XALn2-n1+1.
Because fAL is the longest segment of fA, mfAL(gradient of fAL) is considered as the gradient of fA. For the linear relation y=mx form is the reference function (fR). Therefore, gradient of fR is the same as the gradient of fAL. Then fR = mfAL*xi for all i and fD=fA-fR. According to Lemma 1  fD is a constant and has the form of y=d, where d is a constant. Then the function fD is the final transformation form, which is suitable for applying Grubbs test.Finally, we applied Grubbs test onfD and checked for outliers in the fD. The existence of outliers in fD confirms the existence of outliers in fA. If the kth item of fD is identified as an outlier, the kth item of fA is considered as an outlier. Since fR depends on x which is the index of the data point, fD is also a function of x. This modification establishes a relation between data points and their index and eliminates the major problem identified for  Grubbs test. After transformation Grubbs test can be applied repeatedly on fD until no outliers were detected.


### 2.1. Evaluation Using Artificial Data

Four artificial data sets with one outlier in each data set (which cannot be identified by Grubbs test) were tested with the new method. Each data set consists of 10 elements with an outlier of different type, as mentioned in Table2. Data sets 1 and 2 are the same data sets as in Table 1.Table 2
Type of outlier contained in the data sets.
Data set name
Number of elements/outliers
Type of outlier
Data set 1
10/1
Last element of data set, considerably deviated
Data set 2
10/1
Maximum element of data set, not the last element
Data set 3
10/1
Neither the maximum nor the minimum; deviation is very small
Data set 4
10/1
Neither the first nor the last; data set is not continuously incrementing or decrementing


### 2.2. Evaluation Using Real Data

Real data sets collected from a biogas plant over a period of 60 days with a frequency of one data point per day were tested using both our transformation technique and standard Grubbs test. Among the different parameters, the counter reading of the electricity generator and the volumetric percentage of methane (CH4) in the biogas were selected for testing. During the stable situation, the counter reading of the electricity generator (operating hours) is continuously increasing, while the percentage of CH4 is fluctuating around a certain value. Both data sets were tested with the new technique and standard Grubbs test for the significance level of 0.05 and window sizes of 4, 5, 6, and 10 without overlapping. Also, Grubbs test was repeatedly applied until there were no outliers detected.


## 2.1. Evaluation Using Artificial Data

Four artificial data sets with one outlier in each data set (which cannot be identified by Grubbs test) were tested with the new method. Each data set consists of 10 elements with an outlier of different type, as mentioned in Table2. Data sets 1 and 2 are the same data sets as in Table 1.Table 2
Type of outlier contained in the data sets.
Data set name
Number of elements/outliers
Type of outlier
Data set 1
10/1
Last element of data set, considerably deviated
Data set 2
10/1
Maximum element of data set, not the last element
Data set 3
10/1
Neither the maximum nor the minimum; deviation is very small
Data set 4
10/1
Neither the first nor the last; data set is not continuously incrementing or decrementing


## 2.2. Evaluation Using Real Data

Real data sets collected from a biogas plant over a period of 60 days with a frequency of one data point per day were tested using both our transformation technique and standard Grubbs test. Among the different parameters, the counter reading of the electricity generator and the volumetric percentage of methane (CH4) in the biogas were selected for testing. During the stable situation, the counter reading of the electricity generator (operating hours) is continuously increasing, while the percentage of CH4 is fluctuating around a certain value. Both data sets were tested with the new technique and standard Grubbs test for the significance level of 0.05 and window sizes of 4, 5, 6, and 10 without overlapping. Also, Grubbs test was repeatedly applied until there were no outliers detected.


## 3. Results and Discussion

The results from the test with artificial data show that applying Grubbs test on the transformed data set using our proposed method is capable of locating outliers at a significance level of 0.01 (Tables3, 4, 5, and 6). When applied on the original data set, Grubbs test was unable to locate the outliers even with significance level of 0.05. The outlier in Table 5 deviates very little and is also neither the maximum nor the minimum, which is the worst case situation for single outlier domain. However, after transformation, Grubbs test identifies the outlier with a high level of confidence. Even though the data set in Table 6 has no continuous increment or decrement, Grubbs test located the outlier after transformation.Table 3
Data set 1 (n=10): last element is the outlier and deviates considerably. With new transformation technique, Grubbs’ test on fD is capable of identifying the outlier with significance level of 0.01. However, Grubbs’ test on fA is capable of identifying the outlier with even significance level of 0.05.
x
i
f
A
=
y
i
f
A
L
m
i
=
y
i
-
Y
A
L
x
i
-
X
A
L
f
R
=
m
f
A
L
*
x
i
f
D
=
f
A
-
f
R
Grubbs’ test onfD(α=0.01)
Grubbs’ test onfA(α=0.05)
1
30
—
—
5
25
No
No
2 (n1)
35
35
5
10
25
No
No
3
40
40
5
15
25
No
No
4
45
45
5
20
25
No
No
5
50
50
5
25
25
No
No
6
55
55
5
30
25
No
No
7
60
60
5
35
25
No
No
8
65
65
5
40
25
No
No
9 (n2)
70
70
5
45
25
No
No
10
100
—
—
50
50
Yes
No
X
A
L
=
n
1
+
n
2
2
=
5.5
Y
A
L
=
∑
i
=
n
1
n
2
y
i
n
2
-
n
1
+
1
=
52.5
m
f
A
L
=
∑
i
=
n
1
n
2
m
i
n
2
-
n
1
+
1
=
5Table 4
Data set 2 (n=10): maximum element of the data set is the outlier, but it is not the first or the last element. With new transformation technique, Grubbs’ test on fD is capable of identifying the outlier with significance level of 0.01. However, Grubbs’ test on fA is capable of identifying the outlier with even significance level of 0.05.
x
i
f
A
=
y
i
f
A
L
m
i
=
y
i
-
Y
A
L
x
i
-
X
A
L
f
R
=
m
f
A
L
*
x
i
f
D
=
f
A
-
f
R
Grubbs’ test onfD(α=0.01)
Grubbs’ test onfA(α=0.05)
1
30
—
—
10
20
No
No
2
20
—
—
20
0
No
No
3
50
—
—
30
20
No
No
4
190
—
—
40
150
Yes
No
5 (n1)
70
70
10
50
20
No
No
6
80
80
10
60
20
No
No
7
90
90
10
70
20
No
No
8
100
100
10
80
20
No
No
9
110
110
10
90
20
No
No
10 (n2)
120
120
10
100
20
No
No
X
A
L
=
n
1
+
n
2
2
=
7.5
Y
A
L
=
∑
i
=
n
1
n
2
y
i
n
2
-
n
1
+
1
=
95
m
f
A
L
=
∑
i
=
n
1
n
2
m
i
n
2
-
n
1
+
1
=
10Table 5
Data set 3 (n=10): neither the maximum nor the minimum is the outlier; deviation is very small. With new transformation technique, Grubbs’ test on fD is capable of identifying the outlier with significance level of 0.01. However, Grubbs’ test on fA is capable of identifying the outlier with even significance level of 0.05.
x
i
f
A
=
y
i
f
A
L
m
i
=
y
i
-
Y
A
L
x
i
-
X
A
L
f
R
=
m
f
A
L
*
x
i
f
D
=
f
A
-
f
R
Grubbs’ test onfD(α=0.01)
Grubbs’ test onfA(α=0.05)
1
30.0
—
—
9.999996
20.00000
No
No
2 (n1)
40.0001
40.0001
9.999975
19.999993
20.00011
Yes
No
3
50.0
50.0
10.000005
29.999989
20.00001
No
No
4
60.0
60.0
10.000008
39.999986
20.00001
No
No
5
70.0
70.0
10.000025
49.999982
20.00002
No
No
6
80.0
80.0
9.999975
59.999979
20.00002
No
No
7
90.0
90.0
9.9999917
69.999975
20.00003
No
No
8
100.0
100.0
9.999995
79.999971
20.00003
No
No
9 (n2)
110.0
110.0
9.9999964
89.999968
20.00003
No
No
10
120.0
—
—
99.999964
20.00004
No
No
X
A
L
=
n
1
+
n
2
2
=
5.5
Y
A
L
=
∑
i
=
n
1
n
2
y
i
n
2
-
n
1
+
1
=
75
mfAL=∑i=n1n2min2-n1+1=9.999996Table 6
Data set 4 (n=10): data set is not continuously incrementing or decrementing. Outlier is neither the first nor the last element. With new transformation technique, Grubbs’ test on fD is capable of identifying the outlier with significance level of 0.01.
x
i
f
A
=
y
i
f
A
L
m
i
=
y
i
-
Y
A
L
x
i
-
X
A
L
f
R
=
m
f
A
L
*
x
i
f
D
=
f
A
-
f
R
Grubbs’ test onfD(α=0.01)
Grubbs’ test onfA(α=0.05)
1
30
—
—
4.667
25.333
No
No
2
28
—
—
9.333
18.667
No
No
3
40
—
—
14
26
No
No
4
76
—
—
18.667
57.333
Yes
No
5 (n1)
51
51
4.8
23.333
27.667
No
No
6
54
54
6
28
26
No
No
7
62
62
2
32.667
29.333
No
No
8
66
66
6
37.333
28.667
No
No
9
69
69
4
42
27
No
No
10 (n2)
76
76
5.2
46.667
29.333
No
No
X
A
L
=
n
1
+
n
2
2
=
7.5
Y
A
L
=
∑
i
=
n
1
n
2
y
i
n
2
-
n
1
+
1
=
63
m
f
A
L
=
∑
i
=
n
1
n
2
m
i
n
2
-
n
1
+
1
=
4.667For the real data sets, the results show that our transformation technique is capable of identifying the outliers depending on the selected window size (Figures5 and 6). The data points in most of the selected windows of Figure 5 consist of values that slightly deviated from the actual value. After transformation, Grubbs test was able to locate those data points. However, standard Grubbs test was unable to locate any of those points as outliers from both data sets for the same window sizes.Outlier detection of the counter reading of electricity generator (operating hours) of the biogas plant with new transformation technique for significance level of 0.05 for different window sizes without overlap. Plots (a), (b), (c), and (d) correspond to window sizes 4, 5, 6, and 10, respectively.
(a)
(b)
(c)
(d)Outlier detection of volumetric percentage of CH4 in biogas with new transformation technique for significance level of 0.05 for different window sizes without overlap. Plots (a), (b), (c), and (d) correspond to window sizes 4, 5, 6, and 10, respectively.
(a)
(b)
(c)
(d)The data series shown in Figure6 is not a linear series. However, application of windowing technique allowed locating outliers in each window using new transformation method. However, standard Grubbs test was unable to locate the outliers in the same windows. Furthermore, the data points shown in different window sizes in Figure 6 have different forms (increment, decrement, or constant). After transformation Grubbs test located the outliers despite the behaviour of data in the selected window. Another important fact is that the located outliers were outliers in relation to the selected window size and linear relation. Finally, the results show the capability of  applying  Grubbs test after transforming the series with new transformation and suitable windowing technique. Thus, the method can be used for locating outliers in time series regardless of the fact that the series is linear or nonlinear. However, still each window is considered as a window containing a linear segment of the curve.According to the generally accepted idea, Grubbs test is not suitable for locating outliers in a data set with six or fewer terms [4]. However, the results show that after transforming with new method, Grubbs test was capable of locating outliers in the data sets with four and six terms (window sizes four and six). This is in disagreement with the generally accepted idea. In particular, when applying Grubbs test on nonlinear data series it is necessary to apply suitable windowing technique for having data windows which has better approximation for linearity (Figure 6). Therefore, we can state that the new transformation technique eliminates one of the major drawbacks that prevent applying  Grubbs test on small windows.The accuracy and the reliability of the transformation totally depend on the gradient(m) of fR. Therefore, applying better method could give much better approximation for m. We considered other statistical properties such as mode and the median of the series as well as the longest segment for deriving m. We excluded the median because it is a single data point. The problem of any single data point is that it is not a reliable data point as a reference data point. Even though the considered single point is neither the maximum nor the minimum, it can be a deviated data point such as B1 in Figure 4. Not like median the mode of a series represents multiple data points. Therefore, the mode can be considered as a good alternative for a data set expected to follow the form of y=c. Unfortunately, the mode cannot be used for a data series expected to follow the form of y=mx+c (increasing or decreasing), because in such a data series it is not possible to expect multiple equal or nearly equal values. Finally, in general we decided to use mean for deriving m. However, if the considered domain ensures the reliability, it is possible to use any other method for deriving m, rather than the method we used. For example, if there is a guarantee of accuracy of a certain data point, even any single data point (such as the first data point of the series) can be used for deriving m.In this paper we used the longest segment of the data set for derivingm. However, if there are considerable numbers of data points in other segments it is possible to calculate the gradient of other segments and get the average gradient of considered segments as the gradient. On the other hand, if the outliers were clustered and located in the longest segment, the method we mentioned in this paper will not give a better approximation for m due to the influence of outliers. However, if the considered domain is having or expected to have clustered outlier, then excluding the whole cluster before calculating m will give a better approximation for m. One possibility is to remove k nearest neighbours of the maximum and the minimum including the maximum and the minimum. This will provide much better data set for deriving m.


## 4. Conclusion

The results for artificial and real data show that our new transformation technique improves the outlier detection power of Grubbs test. The transformation is independent of already existing reference data sets and derived reference set from the part of the original data set. This is the main advantage of the new method. After transformation, Grubbs test was capable of detecting outliers at the significance level 0.01 which were not identified without transformation, even at the significance level 0.05. Also, after transformation, Grubbs test was capable of locating outliers in a data set that is not in ranked order, since the new technique transforms data from the formy=mx+c to the form y=c which is independent of the sequence.



---

*Source: 708948-2015-01-14.xml*