# Variation-Oriented Data Filtering for Improvement in Model Complexity of Air Pollutant Prediction Model

Authors: Chi Man Vong; Weng Fai Ip; Pak Kin Wong
Journal: Mathematical Problems in Engineering
 (2014)

---

## Abstract

Accurate prediction models for air pollutants are crucial for forecast and health alarm to local inhabitants. In recent literature,discrete wavelet transform (DWT) was employed to decompose a series of air pollutant levels, followed by modeling using support vector machine (SVM). This combination of DWT and SVM was reported to produce a more accurate prediction model for air pollutants by investigating different levels of frequency bands. However, DWT has a significant demand in model complexity, namely, the training time and the model size of the prediction model. In this paper, a new method called variation-oriented filtering (VF) is proposed to remove the data with low variation, which can be considered as noise to a prediction model. By VF, the noise and the size of the series of air pollutant levels can be reduced simultaneously and hence so are the training time and model size. The SO2 (sulfur dioxide) level in Macau was selected as a test case. Experimental results show that VF can effectively and efficiently reduce the model complexity with improvement in predictive accuracy.


---

## Body

## 1. Introduction

Rapid urban development if inappropriately managed may lead to increase in pollution. Many studies [1–3] reported that the amount of sulfur dioxide (SO2) is associated with adverse human health. Moreover, WHO (World Health Organization) [4] reported that the health problems in turn may increase the burden of the health-care systems in the long run. Therefore, one possibility for reducing pollutant related sick leave is an early warning. Authorities may provide general public with an early warning using a reliable short-term prediction during adverse pollution conditions.There are two mainstream methods for air pollution prediction: deterministic and statistical based method. Deterministic models [5] are, however, costy to develop (establishment of various inventory) and difficult to operate in real time. Even when adequate data and resources were to become available to implement the deterministic approach, Gardner and Dorling [6] and Kukkonen et al. [7] pointed out that the complexity of a problem in general increases when the spatial interactions betweensystems (regional and urban backgrounds) are ill defined. Since statistical methods usually are derived from empirical relationships between air pollution and other related parameters, these methods are simple to develop and have been widely used in short-term prediction of air pollution [8]. Statistical methods applied to air pollution prediction includemultiple linear regression,nonlinear regression,autoregressive moving average (ARMA), andartificial neural networks (ANNs). Among these statistical methods, ANNs are regarded as a cost-effective and reliable method for air pollution forecasting. Comrie [9] and Ando et al. [10] showed that ANNs are better suited for air pollution forecasting than linear and nonlinear models. Recent approaches had ANNs combined with other machine learning methods, for example,genetic algorithm(GA) [11], in order to improve the predictability of ANN models [12–15]. However, ANNs suffer from some inherent drawbacks [16, 17] such as local minima, overfitting, poor generalization, and long training time. Recently,support vector machine (SVM) has attracted considerable attention for prediction problems [18–20]. This approach is based on the structural risk minimization to provide a good generalization capability [21]. SVM is therefore very resistant to the underfitting and overfitting that are common when ANNs are used. SVM is therefore expected to improve generalization of ANNs and obtain global solutions simultaneously. For example, Lu and Wang [22] reported that SVM model provided better prediction of air pollution than that from ANNs model.In [23], various data filtering techniques, such asmoving average, exponential smoothing, and total variation(TV)denoising, can be applied to further improve the accuracy of a prediction model by filtering or smoothing the noise in the training dataset. However, these data filtering techniques usually filtered or smoothed acute values which are considered as noise or outliers. In many categories of time-series prediction [24], such as financial data, environmental parameter estimation (current application belongs to this aspect), electric utility load, and machine reliability, the acute values are those of great significance and interest. The filtering of these acute values is likely to incur critical information loss in a warning system in particular and hence deterioration in prediction accuracy as shown in Figure 1. It can be seen that significant proportion of information with high variation become lost after TV denoising process. Therefore, the class of the above-mentioned data filtering techniques that attenuate the acute values is not applicable to current application.Figure 1
A sample series of SO2 level in Macau applied after TV denoising; large λ indicates higher filtering rate.Furthermore, time-frequency decomposition tool such asdiscrete wavelet transform (DWT) has been reported [25] to aid time-series prediction. For example, Osowski and Garanty [26] attempted to integrate DWT with SVM for a higher prediction accuracy for the application of air pollution forecasting. Moreover, many studies [27–30] in different applications have also reported that DWT improves the performance of prediction models by decomposing the time series into various levels of frequency bands for forecasting. However, DWT decomposes a time series into several subseries corresponding to various frequency bands. The frequency bands with relatively low variations are usually considered as insignificant because they carry little or even no information that hardly contribute to forecasting. Hence the subseries corresponding to these frequency bands can be filtered out. For every remaining subseries, a corresponding SVM submodel is constructed. Juhos et al. [31] reported that SVM model led to higher accuracy in their study for forecasting NO and NO2 after optimizing the SVM hyperparameters through a time-consuming grid search. Nevertheless the time-consuming grid search can be replaced by efficient, direct search methods such as genetic algorithms (GA). There is, however, usually a degeneration of prediction accuracy because GA easily returns local suboptimal hyperparameters.To achieve the best prediction accuracy, every SVM submodel requires a time-consuming optimization process of hyperparameters such that the training process using DWT and SVM can easily take several hours (e.g., in current application). The current application focuses ondaily forecasting of SO2 level whilehourly forecasting in practice may become necessary for dominant diurnal activities (i.e., every hour, the dataset is updated and the prediction model is retrained) and therefore time is another critical issue. The time issue becomes even more critical when more associated factors (i.e., input variables) to the pollutant level are available. In short, DWT significantly increases the model complexity (i.e., training time and model size) of the prediction model for air pollution forecasting. From this viewpoint, the objective of this study is to design an efficient algorithm that can improve the prediction accuracy while it does not significantly increase the model complexity as in DWT.To design such efficient algorithm, direct manipulation on the time series is preferred to the time-frequency decomposition as in DWT. A possible solution to this algorithm may involve the reduction of the number of training data by clustering similar data points into various clusters. However, the application of SO2 level prediction has a stochastic nature, that is, two similar inputs can produce two highly different outputs. Hence, clustering of data points may even cause performance degeneration and thus does not necessarily work in the current application.In fact, the reduction of training data can be based on the determination of variation from a statistical viewpoint. It can be observed that a series of data points (x
i
,
y
i) carries little information if there is low variation among y
i of its successive points, where x
i is the vector of input variables and y
i indicates the corresponding SO2 level for i
=
1 to N. For example, a series of N
=
5 data points is represented as S
=
[
(
x
1
,
10
)
,
(
x
2
,
12
)
,
(
x
3
,
11
)
,
(
x
4
,
10
)
,
(
x
5
,
21
)
]. The first four points of S only carry little information because y
1 to y
4 are close to each other. Hence, the second and third data points (with 12 and 11, resp.) can be discarded because they hardly contribute any extra information. Finally N can be reduced to 3 while S becomes [
(
x
1
,
10
)
,
(
x
4
,
10
)
,
(
x
5
,
21
)
]. Furthermore, the low variation among these data points may be even considered as noise. For a real example, Figure 2(a) shows the trend of SO2 levels in Macau from year 2003 to year 2008. It can be seen that high daily variation of SO2 levels occurred in some days (Figure 2(b)). The input data from these days carry more valuable information for SO2 level prediction. In Figure 2(c), some successive days are with low daily variation of SO2 levels. Such data carry little practical information for a warning system of adverse air quality. Therefore, these data can be discarded without significantly affecting the accuracy of SO2 prediction models under a statistical perspective, where data or dimensions with low variations can be discarded (as mentioned in the theory ofprinciple components analysis (PCA) [32]). With a daily variation threshold δ of 0.15 (to be explained in Section 2), about 30% (662) of days from year 2003 to year 2008 are with low daily variation of SO2 level and can be discarded. Based on this idea, we propose a novel algorithm calledvariation-oriented filtering (VF) which can effectively and efficiently reduce the number of data points of low variation while most of the intrinsic information (i.e., high variation and acute values) of the training data can be retained. Therefore, compared to DWT, VF can reduce the model complexity for SO2 level prediction without obvious sacrifice of the prediction accuracy.(a) The series of SO2 levels in Macau from year 2003 to year 2008. (b) A representative case with high variation. (c) A representative case with low variation.
(a)
(b)
High variation
(c)
Low variationMoreover, since using VF the number of training data points can be reduced, the number of support vectors #SV (selected from the training data points) of the final SVM regression model [21] as shown in the following is very likely to reduce or at least remained the same:
(1)
y
=
∑
i
=
1
#
SV
(
α
i
-
α
i
′
)
K
(
x
i
,
x
)
+
b
,
where α
i and α
i
′ are the support values corresponding to the ith support vector, K is a kernel function, and b is a bias constant. With less training data points, the training time can be shortened while, with less support vectors, less memory is necessary for storage of the model and also faster execution time. Since DWT and SVM are well-known techniques in the past two decades, interested readers may refer to the classical textbooks [21, 25] for technical details.In Section2, the algorithm of VF and the framework of modeling for SO2 level prediction are described. In Section 3, an illustrative application of SO2 level prediction is presented, followed by the simulation results and discussion in Section 4. Finally a conclusion (and future work) is drawn in Section 5.


## 2. Methods

In this section, the description of the proposed algorithm is presented, followed by the framework of constructing SO2 prediction models.


### 2.1. Algorithm of Variation-Oriented Filtering (VF)

The trend of SO2 levels consists of a series of data points (x
i
,
y
i), where x
i is the vector of normalized input variables and y
i indicates the corresponding output variable (normalized SO2 level), for i
=
1 to N (number of days). Data normalization is mentioned in Section 3.2. VF interprets a series as different segments of data points. During the interpretation of segments of data points, a data point (x
i
,
y
i) is compared with its previous point (x
i
-
1
,
y
i
-
1). If the difference between y
i
-
1 and y
i exceeds or equals a user-defined variation threshold δ, that is,
(2)
|
y
i
-
y
i
-
1
|
≥
δ
,
then (x
i
,
y
i) is considered to be significant and is assigned the high variation class.Otherwise, (x
i
,
y
i) belongs to the low variation class and is discarded. After the interpretation, a reduced series of data points with only the high variation class is produced. The procedure is described in detail as follows.A data point (x
i
,
y
i) is assumed to be the beginning of a segment. It is compared with the next data point (x
i
+
1
,
y
i
+
1) using (1) under an optimized threshold of daily variation (δ
*). If (2) is false, the data point (x
i
+
1
,
y
i
+
1) belongs to the low variation class and is discarded. The comparison then continues with (x
i
+
2
,
y
i
+
2) and so on until either (2) is true with a data point (x
j
,
y
j) for i
<
j
<
N, or the end of the series of data points is reached; that is, j
=
N. In order to retain the trend of a segment, the preceding and current data points, namely, (x
j
-
1
,
y
j
-
1) and (x
j
,
y
j), are both retained instead of (x
j
,
y
j) alone. Finally (x
j
,
y
j) is marked as the end point of the current segment and simultaneously the beginning of next segment, where the interpretation of next segment begins.An example of applying VF is shown in Figure3. Assume the data point (x
1
,
y
1) at Day 1 is the beginning of Segment 1. It is compared with its following data point (x
2
,
y
2) at Day 2 using (2) under δ. Since (2) is true, (x
2
,
y
2) is retained and marked as the end of Segment 1 and the beginning of Segment 2. For Segment 2, the data point (x
3
,
y
3) at Day 3 is compared with (x
2
,
y
2) and (2) is false. Then, (x
3
,
y
3) is discarded. Similarly, the data points at Day 4 and Day 5 are discarded. The comparison with (x
2
,
y
2) continues until the data point (x
6
,
y
6) at Day 6 is reached where (2) becomes true. In order to maintain the trend of the segment, the data point (x
6
,
y
6) and its previous one (x
5
,
y
5) are retained. Then (x
6
,
y
6) is marked as the end of Segment 2 and also the beginning of Segment 3. The dash dot line in Figure 3 shows the trend of SO2 levels, which is not significantly deformed, after applying VF. In addition, the data points of Day 3 and Day 4 carry little information and can be considered as noise. The algorithm of VF described above is outlined as in Algorithm 1.Algorithm 1
inti
=
1;       //initialize the beginning of the segment
intj
=
2;       //initialize the end of the segment
select (x
1, y
1);      //select the data at the first day of the first segment
forj
=
2 to N do   //loop from 2 to N, i.e., the end of the series
ifi
=
(
j
-
1
) and  (2) is true {
//if theith day and the jth day are successive days and the variation of
the SO2 levels in these two days is equal to or greater than δ
select (x
j, y
j)    //select the data from the jth day
i
=
j;      //change the end of the segment to the beginning of the segment
j++;
//change the end of the segment to the next day
}
elseif  (2) is true {//if the ith day and the jth day are not on successive days but
their variation is equal to or greater thanδ
select (x
j
-
1, y
j
-
1)  //select data at the (
j
-
1
)th day
select (x
j, y
j)      //select data at the jth day
i
=
j;       //set the beginning of the next segment
j++; }      //change the end of the segment to the next day
elsej++; }      //discard the data point and move to next segmentFigure 3
An example of variation-oriented filtering.


### 2.2. Workflow of Modeling

The workflow of modeling employs the techniques of VF, SVM, and GA. For accurate SO2 level prediction, VF is proposed to filter out the noise in the input data (i.e., the data points with low variation). In fact, the variation threshold δ for VF is difficult to define and is subject to different training data. If δ is set too high, some informative data points for prediction will be discarded. On contrary, if δ is set too low, the redundant data points cannot be effectively filtered out. Hence, an optimization for δ is required.SVM was employed as the modeling technique in this study. Mostly radial basis function (RBF) kernel is selected in SVM for modeling problems. Under this setup, SVM includes two hyperparametersc and g, which are the regularization factor and the RBF kernel parameter, respectively. Therefore, there are three hyperparameters (δ, c,  and  g) in total to be optimized in the current modeling framework. Although there are numerous optimization techniques in the literature, GA is regarded as one of the most powerful optimization techniques and has been widely used in many problems [12, 13, 33] to optimize the model performance. In this study, GA is employed as the optimization tool and the details of GA setup can be found in Section 3.4. Other optimization techniques may be tried in the future.The workflow of modeling is depicted in Figure4. The first objective is to determine the optimal hyperparameters (δ
*, c
*, and g
*) using GA, as shown in Figure 4(a). An initial population of hyperparameters (δ, c,  and  g) is randomly generated. For every instance of (δ, c,  and  g) from the population, the steps of VF and SVM are applied as follows. Given a training dataset D, a set of data points of high variation class D
VF is selected by applying VF with the variation threshold δ. Then SVM with the c and g is used to build a provisional SVM model. If there are 100 instances of (δ, c,  and  g) in the population, there are 100 provisional SVM models. Every provisional SVM model is then evaluated using a fitness function over an independent validation setVALID (more details can be found in Section 3.4). The above procedure repeats until the stopping criterion is satisfied. Finally the optimal hyperparameters (δ
*, c
*, and g
*) are returned. The second step is simply the construction of a prediction model of SO2 levels from D using the optimal hyperparameters as shown in Figure 4(b).Workflow of modeling in the current study: (a) determination of optimal hyperparameters; (b) SVM model construction.
(a)
(b)


## 2.1. Algorithm of Variation-Oriented Filtering (VF)

The trend of SO2 levels consists of a series of data points (x
i
,
y
i), where x
i is the vector of normalized input variables and y
i indicates the corresponding output variable (normalized SO2 level), for i
=
1 to N (number of days). Data normalization is mentioned in Section 3.2. VF interprets a series as different segments of data points. During the interpretation of segments of data points, a data point (x
i
,
y
i) is compared with its previous point (x
i
-
1
,
y
i
-
1). If the difference between y
i
-
1 and y
i exceeds or equals a user-defined variation threshold δ, that is,
(2)
|
y
i
-
y
i
-
1
|
≥
δ
,
then (x
i
,
y
i) is considered to be significant and is assigned the high variation class.Otherwise, (x
i
,
y
i) belongs to the low variation class and is discarded. After the interpretation, a reduced series of data points with only the high variation class is produced. The procedure is described in detail as follows.A data point (x
i
,
y
i) is assumed to be the beginning of a segment. It is compared with the next data point (x
i
+
1
,
y
i
+
1) using (1) under an optimized threshold of daily variation (δ
*). If (2) is false, the data point (x
i
+
1
,
y
i
+
1) belongs to the low variation class and is discarded. The comparison then continues with (x
i
+
2
,
y
i
+
2) and so on until either (2) is true with a data point (x
j
,
y
j) for i
<
j
<
N, or the end of the series of data points is reached; that is, j
=
N. In order to retain the trend of a segment, the preceding and current data points, namely, (x
j
-
1
,
y
j
-
1) and (x
j
,
y
j), are both retained instead of (x
j
,
y
j) alone. Finally (x
j
,
y
j) is marked as the end point of the current segment and simultaneously the beginning of next segment, where the interpretation of next segment begins.An example of applying VF is shown in Figure3. Assume the data point (x
1
,
y
1) at Day 1 is the beginning of Segment 1. It is compared with its following data point (x
2
,
y
2) at Day 2 using (2) under δ. Since (2) is true, (x
2
,
y
2) is retained and marked as the end of Segment 1 and the beginning of Segment 2. For Segment 2, the data point (x
3
,
y
3) at Day 3 is compared with (x
2
,
y
2) and (2) is false. Then, (x
3
,
y
3) is discarded. Similarly, the data points at Day 4 and Day 5 are discarded. The comparison with (x
2
,
y
2) continues until the data point (x
6
,
y
6) at Day 6 is reached where (2) becomes true. In order to maintain the trend of the segment, the data point (x
6
,
y
6) and its previous one (x
5
,
y
5) are retained. Then (x
6
,
y
6) is marked as the end of Segment 2 and also the beginning of Segment 3. The dash dot line in Figure 3 shows the trend of SO2 levels, which is not significantly deformed, after applying VF. In addition, the data points of Day 3 and Day 4 carry little information and can be considered as noise. The algorithm of VF described above is outlined as in Algorithm 1.Algorithm 1
inti
=
1;       //initialize the beginning of the segment
intj
=
2;       //initialize the end of the segment
select (x
1, y
1);      //select the data at the first day of the first segment
forj
=
2 to N do   //loop from 2 to N, i.e., the end of the series
ifi
=
(
j
-
1
) and  (2) is true {
//if theith day and the jth day are successive days and the variation of
the SO2 levels in these two days is equal to or greater than δ
select (x
j, y
j)    //select the data from the jth day
i
=
j;      //change the end of the segment to the beginning of the segment
j++;
//change the end of the segment to the next day
}
elseif  (2) is true {//if the ith day and the jth day are not on successive days but
their variation is equal to or greater thanδ
select (x
j
-
1, y
j
-
1)  //select data at the (
j
-
1
)th day
select (x
j, y
j)      //select data at the jth day
i
=
j;       //set the beginning of the next segment
j++; }      //change the end of the segment to the next day
elsej++; }      //discard the data point and move to next segmentFigure 3
An example of variation-oriented filtering.


## 2.2. Workflow of Modeling

The workflow of modeling employs the techniques of VF, SVM, and GA. For accurate SO2 level prediction, VF is proposed to filter out the noise in the input data (i.e., the data points with low variation). In fact, the variation threshold δ for VF is difficult to define and is subject to different training data. If δ is set too high, some informative data points for prediction will be discarded. On contrary, if δ is set too low, the redundant data points cannot be effectively filtered out. Hence, an optimization for δ is required.SVM was employed as the modeling technique in this study. Mostly radial basis function (RBF) kernel is selected in SVM for modeling problems. Under this setup, SVM includes two hyperparametersc and g, which are the regularization factor and the RBF kernel parameter, respectively. Therefore, there are three hyperparameters (δ, c,  and  g) in total to be optimized in the current modeling framework. Although there are numerous optimization techniques in the literature, GA is regarded as one of the most powerful optimization techniques and has been widely used in many problems [12, 13, 33] to optimize the model performance. In this study, GA is employed as the optimization tool and the details of GA setup can be found in Section 3.4. Other optimization techniques may be tried in the future.The workflow of modeling is depicted in Figure4. The first objective is to determine the optimal hyperparameters (δ
*, c
*, and g
*) using GA, as shown in Figure 4(a). An initial population of hyperparameters (δ, c,  and  g) is randomly generated. For every instance of (δ, c,  and  g) from the population, the steps of VF and SVM are applied as follows. Given a training dataset D, a set of data points of high variation class D
VF is selected by applying VF with the variation threshold δ. Then SVM with the c and g is used to build a provisional SVM model. If there are 100 instances of (δ, c,  and  g) in the population, there are 100 provisional SVM models. Every provisional SVM model is then evaluated using a fitness function over an independent validation setVALID (more details can be found in Section 3.4). The above procedure repeats until the stopping criterion is satisfied. Finally the optimal hyperparameters (δ
*, c
*, and g
*) are returned. The second step is simply the construction of a prediction model of SO2 levels from D using the optimal hyperparameters as shown in Figure 4(b).Workflow of modeling in the current study: (a) determination of optimal hyperparameters; (b) SVM model construction.
(a)
(b)


## 3. Application

In this section, the environment and monitoring sites in Macau are briefly introduced. The data preparation and representation in this study are described. In addition, the detail of experimental setup is described.


### 3.1. The Environment and Monitoring Sites in Macau

Macau, located on the southern coast of China with merely 26.8 square miles land area, comprises three land zones: Macau peninsula, Taipa, and Coloane (Figure5). Similar to many coastal cities in Mainland China, Macau has experienced rapid urban development over the past decades. Therefore, the air pollution data and meteorological data in Macau were used as a case study.Figure 5
The three land zones of Macau and the location of the meteorological stations and the power station.Macau government meteorological center (DSMG) [34] has established general and roadside meteorological stations for collecting air pollution data, such as suspended particulate matters (SPM), nitrogen dioxide (NO2), sulfur dioxide (SO2), and ozone (O3), and meteorological data, such as temperature, wind direction, wind speed, and relative humidity. Since the land area of Macau is relatively small, the data obtained at the general meteorological station at Taipa Grande (at an elevation of approximately 150 m above sea level) may be considered as representative for the entire region of Macau. Therefore, air pollution data and meteorological data form Taipa Grande general meteorological station were used in this study.


### 3.2. Data Preparation and Representation

The daily average values for air pollution data and meteorological data from year 2003 to year 2008 were extracted from the website of DSMG. These data were considered representative measures and were adopted in this study of SO2 modeling. The choice of the study periods was based primarily on the completeness of both air pollution data and meteorological data available at the time of the experiment. In the study periods, the percentage of missing data, possibly due to maintenance or calibration work, is less than 3%. Outside of these periods, either air pollution data or meteorological data are unavailable.It is necessary to provide commensurate data ranges so that the SO2 model will not be dominated by variables with large values. Moreover, the normalization procedure usually leads to more stable and accurate prediction results. In this study, the training data sets (TRAIN) and test data sets (TEST), including input variables x and output variable y, were normalized based on zero mean and unit variance, as shown in the following equation:(3)
a
⟵
a
-
a
-
σ
a
,
where a is either an input variable in x or the output variable y, and a
- and σ
a are the mean and the standard deviation of the variable, respectively.In order to maintain parsimony for the input variables in the model so that the model does not become overgeneralized, it is necessary to sort out and select the highly related input variables only. The selection of input variables is based on Pearson correlation coefficient [35]. In short, the Pearson correlation coefficient is a measure of the linear dependence between two variables X and Y, giving a value between +1 and −1 inclusive. A value close to +1 indicates positive correlation while a value close to −1 indicates negative correlation. A value close to zero indicates no dependence between the two variables.Pearson’s correlation coefficient between two variablesX and Y is commonly denoted by r
X
,
Y:
(4)
r
X
,
Y
=
∑
i
=
1
N
(
X
i
-
X
-
)
(
Y
i
-
Y
-
)
∑
i
=
1
N
(
X
i
-
X
-
)
2
∑
i
=
1
N
(
Y
i
-
Y
-
)
2
.
In this study, variables X and Y are time series, containing a series of data point. X
i and Y
i are the ith data points of X and Y, respectively, i
=
1 to N, where N is the number of data. X
- and Y
- are the means of X and Y, respectively.In this case study, the candidates of input variables are the air pollutants (SPM, SO2, NO2, and O3) and available meteorological data (atmospheric pressure (AP), temperature (TEMP), mean relative humidity (mRH), wind speed (WS), rainfall (RF), and sunshine hour (SHr)) on previous days (e.g., current day d, previous day d
-
1); the output variable is the SO2 level on the following day d
+
1. Pearson’s correlation coefficients among the output variables SO2(d
+
1) and the candidates of input variables on previous days are calculated based on (4) and summarized in Table 1. For example, in order to find Pearson’s correlation coefficient (r
X
,
Y
=
0
.
64) between X and Y, namely, NO2(d) and SO2(d
+
1), respectively, d is set as 1 January 2003. Then the series of NO2 levels from 1 January 2003 to 30 December 2008 is retrieved and N is set to the number of days in this period. Each NO2 level in this period is considered as X
i, i
=
1 to N. Similarly, the series of SO2 levels from 2 January 2003 to 31 December 2008 is selected because one day is shifted for d
+
1, and each SO2 level in the period is Y
i. The corresponding data are put into (3) to calculate r
X
,
Y.Table 1
Pearson correlation coefficients between the output variablesSO
2
(
d
+
1
) and input variables candidates on pervious days in the time period from year 2003 to year 2008.
Relation toSO
2
(
d
+
1
)
Day
Input variables
d
d
-
1
d
-
2
d
-
3
d
-
4
SPM
0.50
0.40
0.37
0.35
0.36
SO2
0.72
0.57
0.50
0.45
0.44
NO2
0.64
0.52
0.48
0.44
0.43
O3
−0.05
−0.03
−0.03
−0.01
0.00
AP
0.57
0.56
0.54
0.52
0.49
TEMP
−0.60
−0.59
−0.55
−0.52
−0.48
mRH
−0.36
−0.31
−0.27
−0.23
−0.21
WS
0.21
0.27
0.26
0.22
0.20
RF
−0.15
−0.14
−0.14
−0.14
−0.14
SHr
−0.08
−0.08
−0.07
−0.08
−0.07The level of significance is assumed to be 0.50; that is, two variables have positive correlation if the corresponding Pearson correlation coefficient is greater than 0.50 and negative correlation if the coefficient is smaller than −0.50; otherwise, the two variables have no relationship. Hence, with the level of significance set to 0.50, the followings can be summarized from Table1.(i)
SO2(d
+
1) seems positively correlated with SPM, SO2, NO2, and AP, and negatively correlated with TEMP, on different days (d, d
-
1, and so on). Therefore the variables SPM, SO2, NO2, AP, and TEMP on different days were selected as input variables in this study.
(ii)
SO2(d
+
1) seems poorly correlated with O3, mean relative humidity (mRH), wind speed (WS), rainfall (RF), and sunshine hours (SHr) on all days. These input variables candidates have weak influence on the prediction of SO2 level at the following day d
+
1 and thus are not selected.Furthermore, Macau is characterized as a subtropical monsoon climate. Air-borne industrial pollutants may be carried by north-ward prevailing wind from Mainland China through Macau in winter, while the south-eastern prevailing wind from the sea in summer usually carries pollutants away. Wind direction (WD), a significant factor to SO2 levels in Macau, is therefore included in the SO2 prediction model. In the current study, WD is separated into 16 discrete directions {
N
,
NNE
,
NE
,
ENE
,
E
,
…
}. In order to make WD more applicable for the generalization and to avoid unnecessary input variables, only the relevant wind directions were selected. Out of the 16 wind directions, those from North- and South-ward, namely, {
NNW
,
N
,
NNE
,
NE
,
E
,
ESE
,
SE
,
SSE
}, are related to SO2 levels prediction in Macau and thus selected as input variables as well. To represent these wind directions without incurring any bias, a set of Boolean variables WD
i
∈
{
0
,
1
} was used for i
=
1 to 8, instead of a number WD
∈
{
1,2
,
…
,
8
}. In addition, the wind direction from the current day and the previous day, namely, WD
i
(
d
) and WD
i
(
d
-
1
), can adequately be represented in the input, rather than using wind direction for more than one day ago.Finally the data representation in this study is defined as a pair (x
,
y), where x is the vector of input variables and y indicates the corresponding SO2 level. According to Table 1, the representation of x is defined as
(5)
x
=
〈
SPM
(
d
)
,
SO
2
(
d
-
2
)
,
SO
2
(
d
-
1
)
,
SO
2
(
d
)
,
NO
2
(
d
-
1
)
,
NO
2
(
d
)
,
AP
(
d
-
3
)
,
AP
(
d
-
2
)
,
AP
(
d
-
1
)
,
AP
(
d
)
,
TEMP
(
d
-
3
)
,
TEMP
(
d
-
2
)
,
TEMP
(
d
-
1
)
,
TEMP
(
d
)
,
WD
i
(
d
-
1
)
,
WD
i
(
d
)
〉
for
i
=
1
to
8
.
Note that the output y
=
SO
2
(
d
+
1
) is defined as the SO2 level at the following day d
+
1.


### 3.3. Construction of Prediction Models

In this study, four SO2 models using combinations of SVM, VF, GA, and DWT (shown in Table 2) were constructed to evaluate the effectiveness and efficiency of VF against DWT. The hyperparameters c and g for SVM model and SVM-DWT model were optimized using exhaustive grid search, while the optimization of c and g for SVM-GA and SVM-VF was done using GA. The ranges of both c and g are defined as 0.0~9.9 for a demonstrative trial. Practically wider ranges of these hyperparameters are preferred. Since VF employs GA to search for hyperparameters, SVM-VF works much faster than SVM, which uses exhaustive grid search. In order to illustrate fair comparison between the models SVM and SVM-VF, a model SVM-GA is therefore required to assess the efficiency of VF.Table 2
Combinations of SVM models.
Model
SVM
Grid Search
GA
VF
DWT
SVM
✓
✓
SVM-GA
✓
✓
SVM-VF
✓
✓
✓
SVM-DWT
✓
✓
✓Each model was independently trained 10 times to generate more reliable results. For a more comprehensive comparison for model stability, experiment data (from year 2003 to year 2008) were divided into three groups ofTRAIN andTEST (as shown in Table 3). Each group of data was employed to construct four SO2 models. Therefore, there are totally 12 SO2 models in this study.Table 3
Three groups of experiment data.
Group
TRAIN
TEST
1
Year 2003 to 2005
Year 2006
2
Year 2004 to 2006
Year 2007
3
Year 2005 to 2007
Year 2008


### 3.4. GA Setup

As mentioned in Section2.2, the hyperparameters δ, c, and g are optimized using GA. The ranges of δ, c, and g are defined as 0.00~0.99, 0.0~9.9, and 0.0~9.9, respectively. Hence, it is enough to employ 2 digits (0 to 9) for each of the hyperparameters; that is, an individual of GA is a real-code string of 6 digits. An example of an individual with a string of “865407” is shown in Figure 6, which means δ
=
0.86, c
=
5.4, and g
=
0.7, respectively.Figure 6
An example of an individual.The generation of populations of the hyperparameters is controlled by the operations of selection, crossover, and mutation. The selection procedure in this study was divided into two parts. Firstly, in order to imply the elitism strategy [36] which retains the best individuals, two individuals with the best fitness were directly selected from the current population without further processing. Secondly, the other 98 individuals were selected from the current population by using tournament selection and passed to crossover and mutation. The predefined probability of crossover and mutation is 0.8 and 0.05 according to [37].The evaluation of the fitness of an individual is the accuracy of the SVM model over an independent validation dataset (VALID) using the individual (i.e., the hyperparameters). The accuracy of the SVM model is evaluated usingcomplementary Willmott’s index of agreement (CWIA), which is further discussed in Section 4.1.VALID consists of 36 data points randomly selected fromTEST. The stopping criterion for GA process is that no better fitness can be achieved for 50 successive iterations. Finally, the individual with the best fitness is returned as the optimal hyperparameters (δ
*, c
*, and g
*).


### 3.5. DWT Setup

As a comparison with VF, DWT was also applied to the training dataset. The family of Daubechies (db) wavelets is one of the most popular mother wavelets and was employed in this study. Ten kinds of Daubechies wavelets (from db1 to db10) were applied to construct different SVM models as illustrated in Figure7. Given a training dataset T
R
A
I
N
=
{
(
x
i
,
y
i
)
} for i
=
1 to N, DWT decomposesTRAIN into several levels (
T
R
A
I
N
1
,
T
R
A
I
N
2
,
…
,
T
R
A
I
N
J
+
1
). For k
=
1 to J
+
1, every T
R
A
I
N
k
=
{
(
x
i
,
y
i
k
)
} is used to construct a submodel SVMk, respectively, where y
i
k is thekth decomposed value of y
i. In other words, the series of SO2 levels, i
=
1 to N, is decomposed into J
+
1 series {
y
i
k
} for k
=
1 to J
+
1. The final SO2 prediction is simply the summation of the prediction by all submodels SVMk.Figure 7
Representation of SVM-DWT model.The SVM model accuracy using different kinds of Daubechies wavelets can be evaluated through CWIA (discussed in Section4.1) over an independent validation set. From simulation results (not shown here), db7 was found to be the most suitable for this study. In addition, it is also important to decide the decomposition level J. The number of the decomposition levels of each model can be estimated by using the function “wmaxlev” provided in the MATLAB wavelet toolbox. In this study, J was found to be 6.


## 3.1. The Environment and Monitoring Sites in Macau

Macau, located on the southern coast of China with merely 26.8 square miles land area, comprises three land zones: Macau peninsula, Taipa, and Coloane (Figure5). Similar to many coastal cities in Mainland China, Macau has experienced rapid urban development over the past decades. Therefore, the air pollution data and meteorological data in Macau were used as a case study.Figure 5
The three land zones of Macau and the location of the meteorological stations and the power station.Macau government meteorological center (DSMG) [34] has established general and roadside meteorological stations for collecting air pollution data, such as suspended particulate matters (SPM), nitrogen dioxide (NO2), sulfur dioxide (SO2), and ozone (O3), and meteorological data, such as temperature, wind direction, wind speed, and relative humidity. Since the land area of Macau is relatively small, the data obtained at the general meteorological station at Taipa Grande (at an elevation of approximately 150 m above sea level) may be considered as representative for the entire region of Macau. Therefore, air pollution data and meteorological data form Taipa Grande general meteorological station were used in this study.


## 3.2. Data Preparation and Representation

The daily average values for air pollution data and meteorological data from year 2003 to year 2008 were extracted from the website of DSMG. These data were considered representative measures and were adopted in this study of SO2 modeling. The choice of the study periods was based primarily on the completeness of both air pollution data and meteorological data available at the time of the experiment. In the study periods, the percentage of missing data, possibly due to maintenance or calibration work, is less than 3%. Outside of these periods, either air pollution data or meteorological data are unavailable.It is necessary to provide commensurate data ranges so that the SO2 model will not be dominated by variables with large values. Moreover, the normalization procedure usually leads to more stable and accurate prediction results. In this study, the training data sets (TRAIN) and test data sets (TEST), including input variables x and output variable y, were normalized based on zero mean and unit variance, as shown in the following equation:(3)
a
⟵
a
-
a
-
σ
a
,
where a is either an input variable in x or the output variable y, and a
- and σ
a are the mean and the standard deviation of the variable, respectively.In order to maintain parsimony for the input variables in the model so that the model does not become overgeneralized, it is necessary to sort out and select the highly related input variables only. The selection of input variables is based on Pearson correlation coefficient [35]. In short, the Pearson correlation coefficient is a measure of the linear dependence between two variables X and Y, giving a value between +1 and −1 inclusive. A value close to +1 indicates positive correlation while a value close to −1 indicates negative correlation. A value close to zero indicates no dependence between the two variables.Pearson’s correlation coefficient between two variablesX and Y is commonly denoted by r
X
,
Y:
(4)
r
X
,
Y
=
∑
i
=
1
N
(
X
i
-
X
-
)
(
Y
i
-
Y
-
)
∑
i
=
1
N
(
X
i
-
X
-
)
2
∑
i
=
1
N
(
Y
i
-
Y
-
)
2
.
In this study, variables X and Y are time series, containing a series of data point. X
i and Y
i are the ith data points of X and Y, respectively, i
=
1 to N, where N is the number of data. X
- and Y
- are the means of X and Y, respectively.In this case study, the candidates of input variables are the air pollutants (SPM, SO2, NO2, and O3) and available meteorological data (atmospheric pressure (AP), temperature (TEMP), mean relative humidity (mRH), wind speed (WS), rainfall (RF), and sunshine hour (SHr)) on previous days (e.g., current day d, previous day d
-
1); the output variable is the SO2 level on the following day d
+
1. Pearson’s correlation coefficients among the output variables SO2(d
+
1) and the candidates of input variables on previous days are calculated based on (4) and summarized in Table 1. For example, in order to find Pearson’s correlation coefficient (r
X
,
Y
=
0
.
64) between X and Y, namely, NO2(d) and SO2(d
+
1), respectively, d is set as 1 January 2003. Then the series of NO2 levels from 1 January 2003 to 30 December 2008 is retrieved and N is set to the number of days in this period. Each NO2 level in this period is considered as X
i, i
=
1 to N. Similarly, the series of SO2 levels from 2 January 2003 to 31 December 2008 is selected because one day is shifted for d
+
1, and each SO2 level in the period is Y
i. The corresponding data are put into (3) to calculate r
X
,
Y.Table 1
Pearson correlation coefficients between the output variablesSO
2
(
d
+
1
) and input variables candidates on pervious days in the time period from year 2003 to year 2008.
Relation toSO
2
(
d
+
1
)
Day
Input variables
d
d
-
1
d
-
2
d
-
3
d
-
4
SPM
0.50
0.40
0.37
0.35
0.36
SO2
0.72
0.57
0.50
0.45
0.44
NO2
0.64
0.52
0.48
0.44
0.43
O3
−0.05
−0.03
−0.03
−0.01
0.00
AP
0.57
0.56
0.54
0.52
0.49
TEMP
−0.60
−0.59
−0.55
−0.52
−0.48
mRH
−0.36
−0.31
−0.27
−0.23
−0.21
WS
0.21
0.27
0.26
0.22
0.20
RF
−0.15
−0.14
−0.14
−0.14
−0.14
SHr
−0.08
−0.08
−0.07
−0.08
−0.07The level of significance is assumed to be 0.50; that is, two variables have positive correlation if the corresponding Pearson correlation coefficient is greater than 0.50 and negative correlation if the coefficient is smaller than −0.50; otherwise, the two variables have no relationship. Hence, with the level of significance set to 0.50, the followings can be summarized from Table1.(i)
SO2(d
+
1) seems positively correlated with SPM, SO2, NO2, and AP, and negatively correlated with TEMP, on different days (d, d
-
1, and so on). Therefore the variables SPM, SO2, NO2, AP, and TEMP on different days were selected as input variables in this study.
(ii)
SO2(d
+
1) seems poorly correlated with O3, mean relative humidity (mRH), wind speed (WS), rainfall (RF), and sunshine hours (SHr) on all days. These input variables candidates have weak influence on the prediction of SO2 level at the following day d
+
1 and thus are not selected.Furthermore, Macau is characterized as a subtropical monsoon climate. Air-borne industrial pollutants may be carried by north-ward prevailing wind from Mainland China through Macau in winter, while the south-eastern prevailing wind from the sea in summer usually carries pollutants away. Wind direction (WD), a significant factor to SO2 levels in Macau, is therefore included in the SO2 prediction model. In the current study, WD is separated into 16 discrete directions {
N
,
NNE
,
NE
,
ENE
,
E
,
…
}. In order to make WD more applicable for the generalization and to avoid unnecessary input variables, only the relevant wind directions were selected. Out of the 16 wind directions, those from North- and South-ward, namely, {
NNW
,
N
,
NNE
,
NE
,
E
,
ESE
,
SE
,
SSE
}, are related to SO2 levels prediction in Macau and thus selected as input variables as well. To represent these wind directions without incurring any bias, a set of Boolean variables WD
i
∈
{
0
,
1
} was used for i
=
1 to 8, instead of a number WD
∈
{
1,2
,
…
,
8
}. In addition, the wind direction from the current day and the previous day, namely, WD
i
(
d
) and WD
i
(
d
-
1
), can adequately be represented in the input, rather than using wind direction for more than one day ago.Finally the data representation in this study is defined as a pair (x
,
y), where x is the vector of input variables and y indicates the corresponding SO2 level. According to Table 1, the representation of x is defined as
(5)
x
=
〈
SPM
(
d
)
,
SO
2
(
d
-
2
)
,
SO
2
(
d
-
1
)
,
SO
2
(
d
)
,
NO
2
(
d
-
1
)
,
NO
2
(
d
)
,
AP
(
d
-
3
)
,
AP
(
d
-
2
)
,
AP
(
d
-
1
)
,
AP
(
d
)
,
TEMP
(
d
-
3
)
,
TEMP
(
d
-
2
)
,
TEMP
(
d
-
1
)
,
TEMP
(
d
)
,
WD
i
(
d
-
1
)
,
WD
i
(
d
)
〉
for
i
=
1
to
8
.
Note that the output y
=
SO
2
(
d
+
1
) is defined as the SO2 level at the following day d
+
1.


## 3.3. Construction of Prediction Models

In this study, four SO2 models using combinations of SVM, VF, GA, and DWT (shown in Table 2) were constructed to evaluate the effectiveness and efficiency of VF against DWT. The hyperparameters c and g for SVM model and SVM-DWT model were optimized using exhaustive grid search, while the optimization of c and g for SVM-GA and SVM-VF was done using GA. The ranges of both c and g are defined as 0.0~9.9 for a demonstrative trial. Practically wider ranges of these hyperparameters are preferred. Since VF employs GA to search for hyperparameters, SVM-VF works much faster than SVM, which uses exhaustive grid search. In order to illustrate fair comparison between the models SVM and SVM-VF, a model SVM-GA is therefore required to assess the efficiency of VF.Table 2
Combinations of SVM models.
Model
SVM
Grid Search
GA
VF
DWT
SVM
✓
✓
SVM-GA
✓
✓
SVM-VF
✓
✓
✓
SVM-DWT
✓
✓
✓Each model was independently trained 10 times to generate more reliable results. For a more comprehensive comparison for model stability, experiment data (from year 2003 to year 2008) were divided into three groups ofTRAIN andTEST (as shown in Table 3). Each group of data was employed to construct four SO2 models. Therefore, there are totally 12 SO2 models in this study.Table 3
Three groups of experiment data.
Group
TRAIN
TEST
1
Year 2003 to 2005
Year 2006
2
Year 2004 to 2006
Year 2007
3
Year 2005 to 2007
Year 2008


## 3.4. GA Setup

As mentioned in Section2.2, the hyperparameters δ, c, and g are optimized using GA. The ranges of δ, c, and g are defined as 0.00~0.99, 0.0~9.9, and 0.0~9.9, respectively. Hence, it is enough to employ 2 digits (0 to 9) for each of the hyperparameters; that is, an individual of GA is a real-code string of 6 digits. An example of an individual with a string of “865407” is shown in Figure 6, which means δ
=
0.86, c
=
5.4, and g
=
0.7, respectively.Figure 6
An example of an individual.The generation of populations of the hyperparameters is controlled by the operations of selection, crossover, and mutation. The selection procedure in this study was divided into two parts. Firstly, in order to imply the elitism strategy [36] which retains the best individuals, two individuals with the best fitness were directly selected from the current population without further processing. Secondly, the other 98 individuals were selected from the current population by using tournament selection and passed to crossover and mutation. The predefined probability of crossover and mutation is 0.8 and 0.05 according to [37].The evaluation of the fitness of an individual is the accuracy of the SVM model over an independent validation dataset (VALID) using the individual (i.e., the hyperparameters). The accuracy of the SVM model is evaluated usingcomplementary Willmott’s index of agreement (CWIA), which is further discussed in Section 4.1.VALID consists of 36 data points randomly selected fromTEST. The stopping criterion for GA process is that no better fitness can be achieved for 50 successive iterations. Finally, the individual with the best fitness is returned as the optimal hyperparameters (δ
*, c
*, and g
*).


## 3.5. DWT Setup

As a comparison with VF, DWT was also applied to the training dataset. The family of Daubechies (db) wavelets is one of the most popular mother wavelets and was employed in this study. Ten kinds of Daubechies wavelets (from db1 to db10) were applied to construct different SVM models as illustrated in Figure7. Given a training dataset T
R
A
I
N
=
{
(
x
i
,
y
i
)
} for i
=
1 to N, DWT decomposesTRAIN into several levels (
T
R
A
I
N
1
,
T
R
A
I
N
2
,
…
,
T
R
A
I
N
J
+
1
). For k
=
1 to J
+
1, every T
R
A
I
N
k
=
{
(
x
i
,
y
i
k
)
} is used to construct a submodel SVMk, respectively, where y
i
k is thekth decomposed value of y
i. In other words, the series of SO2 levels, i
=
1 to N, is decomposed into J
+
1 series {
y
i
k
} for k
=
1 to J
+
1. The final SO2 prediction is simply the summation of the prediction by all submodels SVMk.Figure 7
Representation of SVM-DWT model.The SVM model accuracy using different kinds of Daubechies wavelets can be evaluated through CWIA (discussed in Section4.1) over an independent validation set. From simulation results (not shown here), db7 was found to be the most suitable for this study. In addition, it is also important to decide the decomposition level J. The number of the decomposition levels of each model can be estimated by using the function “wmaxlev” provided in the MATLAB wavelet toolbox. In this study, J was found to be 6.


## 4. Performance Evaluation and Results

### 4.1. Error Measures

In order to effectively measure the accuracy of SVM models, three measures for statistic error are used in this study: mean absolute error (MAE) for value difference, root mean squared error (RMSE) for sensitivity, and complementary Willmott’s index of agreement (CWIA) for curve fitting. Both MAE and RMSE are common measures to estimate the average error of models. However, neither of them provides information about the relative size of the average difference or the nature of the differences. Willmott [38] reported that CWIA is a standardized measure that can be easily interpreted and provides cross-comparisons of its magnitudes for a variety of models, regardless of units. Owing to its dimensionless nature, relationships described by CWIA tend to complement the information contained in RMSE. Therefore, CWIA was also used to evaluate the accuracy of SVM models in this study. The range of CWIA is from 0 to 1, where a value closer to 0 indicates a better performance. The three measures are described by the following formulas, where P
i and O
i, respectively, represent the predicted and observed value of SO2(d
+
1) in the ith day and N is the size ofTEST:
(6)
MAE
=
1
N
∑
i
=
1
N
|
P
i
-
O
i
|
,
(7)
RMSE
=
1
N
∑
i
=
1
N
(
P
i
-
O
i
)
2
,
(8)
CWIA
=
∑
i
=
1
N
(
P
i
-
O
i
)
2
∑
i
=
1
N
(
|
P
i
′
|
+
|
O
i
′
|
)
2
,
where O
-
=
(
1
/
N
)
∑
i
=
1
N
O
i, P
i
′
=
P
i
-
O
-, and O
i
′
=
O
i
-
O
-.


### 4.2. Stability and Errors of Models

In addition to the statistic errors MAE, RMSE, and CWIA, model stability is another important criterion to evaluate the performance of models. Therefore, standard deviation (σ) as shown in (8) is employed to evaluate the stability of models in this study, where M is the number of runs, ε
i represents the statistic error for a model in the ith run, and ε
- represents the mean of all runs:
(9)
σ
=
1
M
∑
i
=
1
M
(
ε
i
-
ε
-
)
2
.
Based on (6) to (9), the statistic errors and standard deviations of four combinations of SVM models can be obtained. The experimental results in Figure 8 show that the standard deviation σ of each statistic error for all models is very small (or even close to zero). It indicates that the results produced in each run for all models were very close to their corresponding means. Therefore, the performance of these models is stable and hence the statistic errors of these models are reliable.Figure 8
Statistic errors and standard deviations of four combinations of SVM models.TRAIN are 2003–2005, 2004–2006, and 2005–2008, respectively.In Figure8, the statistic errors MAE, RMSE, and CWIA of the four models SVM, SVM-GA, SVM-VF, and SVM-DWT are shown. Each model was trained with three different datasets according to the grouping in Table 3 so that 12 different models were obtained. Note that the statistic errors in Figure 8 are the mean of the errors predicted by each model in 10 independent runs. From the results, SVM-VF model (Figure 8) produced the lowest statistic errors in three groups of experimental data among the four models. Despite being generated from fewer training data, VF can improve the accuracy of the prediction by filtering unimportant low variation data points. Compared to SVM and SVM-GA models, SVM-VF model has a relative improvement of about 5% and 9% in all statistic errors, respectively. Conversely, SVM-DWT model produced significantly worse accuracy than the other models. It seems that DWT does not show its superiority for improving the accuracy of the SVM in this study. The decline in accuracy may arise from the fact that DWT decomposes a time series into different subseries, each of which has an independent prediction model as shown in Figure 7. Each of these models incurs an error and hence the accumulated error for the final SVM-DWT model becomes relatively large. Furthermore, SVM-DWT model took the longest training time (almost 5 hours) among the three compared models, while the training time of SVM-VF model was the shortest (<0.5 hour). The time issue will be further discussed in Section 4.4.


### 4.3. Seasonal Variation of SO2 Levels

In order to visualize the performance of the studied methods, the trends of the predicted SO2 levels were compared with the observed SO2 levels from 2006 to 2008. Figure 9 shows that the trends of predicted SO2 levels of each model are generally close to the trend of observed levels. It suggests that these four models have a good capability of prediction.Figure 9
Representative predicted and observed SO2 levels using the proposed four combinations of SVM models from 2006 to 2008.In addition, representative cases of predicted and observed SO2 levels can clearly show a better comparison of the prediction capability of the four models. According to the climatic characteristics of Macau (mentioned in Section 3.2), the trends of SO2 levels in winter and summer can represent the general yearly trend in Macau. Hence, the predicted and observed SO2 levels in winter and summer for each experiment were shown in Figures 10 and 11, respectively. These two figures illustrated that the predicted SO2 level produced by SVM-VF model is closer to the observed SO2 level when comparing with those of the other three models. In particular, SVM-VF appears to fit the more prominent high SO2 level during winter seasons (Figure 11), when health warning is most needed in this case study.Figure 10
Representative predicted and observed SO2 levels using the proposed four combinations of SVM models from Day 156 to Day 185 (29 days).Figure 11
Representative predicted and observed SO2 levels using the proposed four combinations of SVM models from Day 670 to Day 699 (29 days).


### 4.4. Model Complexity

The model complexities of the three combinations of SVM models are shown in Table4, where SVM-DWT model is observed to have relatively large number of support vectors (SV) (about 6800) and very long training time (about 290 minutes). Therefore, the cost of DWT decomposition for SO2 level forecasting is at the relatively high model complexity (i.e., training time and model size) in the current application of air pollution prediction.Table 4
Model complexities of the four combinations of SVM models.
SVM
SVM-DWT
SVM-VF
SVM-GA
TRAIN: year 2003 to 2005, TEST: year 2006
Size ofTRAIN
1055
1055
914
1055
Reduction ratio ofTRAIN
—
—
13.33%
—
Numbers of SV
1012
6882
881
1012
Training time (mins)
49
291
24
18
TRAIN: year 2004 to 2006, TEST: year 2007
Size ofTRAIN
1056
1056
941
1056
Reduction ratio ofTRAIN
—
—
10.88%
—
Numbers of SV
991
6759
883
991
Training time (mins)
50
290
24
18
TRAIN: year 2005 to 2007, TEST: year 2008
Size ofTRAIN
1043
1043
955
1043
Reduction ratio ofTRAIN
—
—
8.46%
—
Numbers of SV
985
6747
912
987
Training time (mins)
48
280
27
20For SVM-VF models, the size of the training datasetTRAIN was reduced to different extents after applying VF for data filtering. The highest data reduction rate by VF is up to 13.33% (forTRAIN in year 2003 to 2005). In addition, the number of SV for SVM-VF is the fewest among the three models, especially when compared with SVM-DWT. Furthermore, the execution time of SVM-VF model is much shorter than those of SVM model (about 50 minutes) and SVM-DWT model (about 290 minutes) (Table 4).Although SVM-GA model requires even shorter training time than SVM-VF model, it suffers from performance degeneration as shown in Figure8 because GA can easily locate the suboptimal SVM hyperparameters. Hence, the proposed algorithm VF may resolve the high complexities of model construction in both time and model size while it can even improve the prediction accuracy. In summary, SVM-VF model not only effectively reduces the model complexity of prediction model for SO2 level, but also improves the accuracy of the prediction models by filtering the unimportant low variation data points.


### 4.5. Discussion

From the experimental results, VF was verified to effectively reduce the size of training datasetTRAIN and also the number of SV of the prediction models. The reductions of training data points and number of SV lead to shorter training time and smaller model size that can benefit for real large-scale modeling using hundreds of input variables and millions of data points. Despite using fewer training data, SVM-VF model can produce higher accuracy than SVM model with significantly shorter training time. It is because VF only discards the unimportant data points with low variation and hence does not affect the accuracy of the SO2 prediction model. Moreover, the unimportant low variation in the discarded data points may be a noise to the modeling. By filtering out these noises, the accuracy of SO2 prediction model can be improved as shown in the results. In addition, the training time of SVM-VF model is significantly reduced as compared to SVM.Using the accuracy of SVM as a standard, when comparing with SVM-GA model, SVM-VF model takes longer training time but produces relatively up to 9% better prediction accuracy. This is also credited to the reduction of modeling noise. Since the optimization of SVM hyperparameters by GA is based on the SVM prediction accuracy, the hyperparameters become more “suboptimal” under a noisy training dataset (i.e., with unimportant low variation data) so that the prediction accuracy of SVM-GA model is further deteriorated.Although many literatures reported that DWT decomposition can improve the performance of prediction models for many applications of signal or image processing, DWT does not necessarily work in time-series prediction. At least, DWT did not establish its superiority in this study. The cause may arise from DWT decomposition and accumulated error as explained in the following. DWT decomposes a time series into several subseries as shown in Figure7. Each subseries is employed to train a submodel SVMk, for k
=
1 to J
+
1. Every SVMk produces a prediction of the output for the subseries, which may incur a certain (small but measurable) amount of error e
k. Since the final prediction is simply the accumulation of the predictions of all SVMk, the total error of the final prediction is also the accumulation of e
k which may become large. Moreover, DWT significantly increases the training time and model size because J
+
1 submodels, SVMk, are produced. Hence, SVM-DWT model did not outperform other models in this study. As a comparison to SVM-DWT model, SVM-VF model not only resolves the issue of high model complexity, but also produces better performance. In a nutshell, Table 5 shows the summary and a quick comparison among the models in the current application. It is concluded that our proposed method VF produces better performance, in particular the training time, in current application.Table 5
A comparison among the models in current study.
Model
Accuracy
Training time
Size
SVM
Good
Fair
Good
SVM-VF
Very good
Good
Very good
SVM-GA
Good
Very good
Good
SVM-DWT
Fair
Very poor
Very poor


## 4.1. Error Measures

In order to effectively measure the accuracy of SVM models, three measures for statistic error are used in this study: mean absolute error (MAE) for value difference, root mean squared error (RMSE) for sensitivity, and complementary Willmott’s index of agreement (CWIA) for curve fitting. Both MAE and RMSE are common measures to estimate the average error of models. However, neither of them provides information about the relative size of the average difference or the nature of the differences. Willmott [38] reported that CWIA is a standardized measure that can be easily interpreted and provides cross-comparisons of its magnitudes for a variety of models, regardless of units. Owing to its dimensionless nature, relationships described by CWIA tend to complement the information contained in RMSE. Therefore, CWIA was also used to evaluate the accuracy of SVM models in this study. The range of CWIA is from 0 to 1, where a value closer to 0 indicates a better performance. The three measures are described by the following formulas, where P
i and O
i, respectively, represent the predicted and observed value of SO2(d
+
1) in the ith day and N is the size ofTEST:
(6)
MAE
=
1
N
∑
i
=
1
N
|
P
i
-
O
i
|
,
(7)
RMSE
=
1
N
∑
i
=
1
N
(
P
i
-
O
i
)
2
,
(8)
CWIA
=
∑
i
=
1
N
(
P
i
-
O
i
)
2
∑
i
=
1
N
(
|
P
i
′
|
+
|
O
i
′
|
)
2
,
where O
-
=
(
1
/
N
)
∑
i
=
1
N
O
i, P
i
′
=
P
i
-
O
-, and O
i
′
=
O
i
-
O
-.


## 4.2. Stability and Errors of Models

In addition to the statistic errors MAE, RMSE, and CWIA, model stability is another important criterion to evaluate the performance of models. Therefore, standard deviation (σ) as shown in (8) is employed to evaluate the stability of models in this study, where M is the number of runs, ε
i represents the statistic error for a model in the ith run, and ε
- represents the mean of all runs:
(9)
σ
=
1
M
∑
i
=
1
M
(
ε
i
-
ε
-
)
2
.
Based on (6) to (9), the statistic errors and standard deviations of four combinations of SVM models can be obtained. The experimental results in Figure 8 show that the standard deviation σ of each statistic error for all models is very small (or even close to zero). It indicates that the results produced in each run for all models were very close to their corresponding means. Therefore, the performance of these models is stable and hence the statistic errors of these models are reliable.Figure 8
Statistic errors and standard deviations of four combinations of SVM models.TRAIN are 2003–2005, 2004–2006, and 2005–2008, respectively.In Figure8, the statistic errors MAE, RMSE, and CWIA of the four models SVM, SVM-GA, SVM-VF, and SVM-DWT are shown. Each model was trained with three different datasets according to the grouping in Table 3 so that 12 different models were obtained. Note that the statistic errors in Figure 8 are the mean of the errors predicted by each model in 10 independent runs. From the results, SVM-VF model (Figure 8) produced the lowest statistic errors in three groups of experimental data among the four models. Despite being generated from fewer training data, VF can improve the accuracy of the prediction by filtering unimportant low variation data points. Compared to SVM and SVM-GA models, SVM-VF model has a relative improvement of about 5% and 9% in all statistic errors, respectively. Conversely, SVM-DWT model produced significantly worse accuracy than the other models. It seems that DWT does not show its superiority for improving the accuracy of the SVM in this study. The decline in accuracy may arise from the fact that DWT decomposes a time series into different subseries, each of which has an independent prediction model as shown in Figure 7. Each of these models incurs an error and hence the accumulated error for the final SVM-DWT model becomes relatively large. Furthermore, SVM-DWT model took the longest training time (almost 5 hours) among the three compared models, while the training time of SVM-VF model was the shortest (<0.5 hour). The time issue will be further discussed in Section 4.4.


## 4.3. Seasonal Variation of SO2 Levels

In order to visualize the performance of the studied methods, the trends of the predicted SO2 levels were compared with the observed SO2 levels from 2006 to 2008. Figure 9 shows that the trends of predicted SO2 levels of each model are generally close to the trend of observed levels. It suggests that these four models have a good capability of prediction.Figure 9
Representative predicted and observed SO2 levels using the proposed four combinations of SVM models from 2006 to 2008.In addition, representative cases of predicted and observed SO2 levels can clearly show a better comparison of the prediction capability of the four models. According to the climatic characteristics of Macau (mentioned in Section 3.2), the trends of SO2 levels in winter and summer can represent the general yearly trend in Macau. Hence, the predicted and observed SO2 levels in winter and summer for each experiment were shown in Figures 10 and 11, respectively. These two figures illustrated that the predicted SO2 level produced by SVM-VF model is closer to the observed SO2 level when comparing with those of the other three models. In particular, SVM-VF appears to fit the more prominent high SO2 level during winter seasons (Figure 11), when health warning is most needed in this case study.Figure 10
Representative predicted and observed SO2 levels using the proposed four combinations of SVM models from Day 156 to Day 185 (29 days).Figure 11
Representative predicted and observed SO2 levels using the proposed four combinations of SVM models from Day 670 to Day 699 (29 days).


## 4.4. Model Complexity

The model complexities of the three combinations of SVM models are shown in Table4, where SVM-DWT model is observed to have relatively large number of support vectors (SV) (about 6800) and very long training time (about 290 minutes). Therefore, the cost of DWT decomposition for SO2 level forecasting is at the relatively high model complexity (i.e., training time and model size) in the current application of air pollution prediction.Table 4
Model complexities of the four combinations of SVM models.
SVM
SVM-DWT
SVM-VF
SVM-GA
TRAIN: year 2003 to 2005, TEST: year 2006
Size ofTRAIN
1055
1055
914
1055
Reduction ratio ofTRAIN
—
—
13.33%
—
Numbers of SV
1012
6882
881
1012
Training time (mins)
49
291
24
18
TRAIN: year 2004 to 2006, TEST: year 2007
Size ofTRAIN
1056
1056
941
1056
Reduction ratio ofTRAIN
—
—
10.88%
—
Numbers of SV
991
6759
883
991
Training time (mins)
50
290
24
18
TRAIN: year 2005 to 2007, TEST: year 2008
Size ofTRAIN
1043
1043
955
1043
Reduction ratio ofTRAIN
—
—
8.46%
—
Numbers of SV
985
6747
912
987
Training time (mins)
48
280
27
20For SVM-VF models, the size of the training datasetTRAIN was reduced to different extents after applying VF for data filtering. The highest data reduction rate by VF is up to 13.33% (forTRAIN in year 2003 to 2005). In addition, the number of SV for SVM-VF is the fewest among the three models, especially when compared with SVM-DWT. Furthermore, the execution time of SVM-VF model is much shorter than those of SVM model (about 50 minutes) and SVM-DWT model (about 290 minutes) (Table 4).Although SVM-GA model requires even shorter training time than SVM-VF model, it suffers from performance degeneration as shown in Figure8 because GA can easily locate the suboptimal SVM hyperparameters. Hence, the proposed algorithm VF may resolve the high complexities of model construction in both time and model size while it can even improve the prediction accuracy. In summary, SVM-VF model not only effectively reduces the model complexity of prediction model for SO2 level, but also improves the accuracy of the prediction models by filtering the unimportant low variation data points.


## 4.5. Discussion

From the experimental results, VF was verified to effectively reduce the size of training datasetTRAIN and also the number of SV of the prediction models. The reductions of training data points and number of SV lead to shorter training time and smaller model size that can benefit for real large-scale modeling using hundreds of input variables and millions of data points. Despite using fewer training data, SVM-VF model can produce higher accuracy than SVM model with significantly shorter training time. It is because VF only discards the unimportant data points with low variation and hence does not affect the accuracy of the SO2 prediction model. Moreover, the unimportant low variation in the discarded data points may be a noise to the modeling. By filtering out these noises, the accuracy of SO2 prediction model can be improved as shown in the results. In addition, the training time of SVM-VF model is significantly reduced as compared to SVM.Using the accuracy of SVM as a standard, when comparing with SVM-GA model, SVM-VF model takes longer training time but produces relatively up to 9% better prediction accuracy. This is also credited to the reduction of modeling noise. Since the optimization of SVM hyperparameters by GA is based on the SVM prediction accuracy, the hyperparameters become more “suboptimal” under a noisy training dataset (i.e., with unimportant low variation data) so that the prediction accuracy of SVM-GA model is further deteriorated.Although many literatures reported that DWT decomposition can improve the performance of prediction models for many applications of signal or image processing, DWT does not necessarily work in time-series prediction. At least, DWT did not establish its superiority in this study. The cause may arise from DWT decomposition and accumulated error as explained in the following. DWT decomposes a time series into several subseries as shown in Figure7. Each subseries is employed to train a submodel SVMk, for k
=
1 to J
+
1. Every SVMk produces a prediction of the output for the subseries, which may incur a certain (small but measurable) amount of error e
k. Since the final prediction is simply the accumulation of the predictions of all SVMk, the total error of the final prediction is also the accumulation of e
k which may become large. Moreover, DWT significantly increases the training time and model size because J
+
1 submodels, SVMk, are produced. Hence, SVM-DWT model did not outperform other models in this study. As a comparison to SVM-DWT model, SVM-VF model not only resolves the issue of high model complexity, but also produces better performance. In a nutshell, Table 5 shows the summary and a quick comparison among the models in the current application. It is concluded that our proposed method VF produces better performance, in particular the training time, in current application.Table 5
A comparison among the models in current study.
Model
Accuracy
Training time
Size
SVM
Good
Fair
Good
SVM-VF
Very good
Good
Very good
SVM-GA
Good
Very good
Good
SVM-DWT
Fair
Very poor
Very poor


## 5. Conclusion

The current application of air pollutant prediction aims to predict the significant acute pollutant levels as a warning message. However, most of the existing methods of data filtering or smoothing techniques are likely to attenuate the acute pollutant levels in such a time series (Figure1) and hence a deterioration in prediction accuracy for time-series prediction. Therefore, these existing methods are not necessarily applicable to the current application. In the literatures about air pollutant level prediction, DWT was used to decompose the series of pollutant levels into different subseries for modeling in order to obtain higher prediction accuracy. However, DWT incurs the issue of high complexity in training time and model size in addition to possible performance degeneration.This research proposes a new algorithm of variation-oriented filtering (VF) to filter only the unimportant low variation data while the acute ones (i.e., with high variation) are retained. VF can resolve the issue of high model complexity without sacrifice (or even with improvement) of the prediction accuracy. The SO2 level in Macau was used as a case study; and four different combinations of SVM models and 12 scenarios using GA, DWT, and VF were compared. Experimental results reveal that the models using VF outperform the other models using GA and DWT for SO2 level prediction in Macau. Moreover, with VF, the number of training data and model construction time can be significantly reduced so thathourly (current application isdaily) prediction of pollutant level can become more feasible. The proposed method VF can also be applied to the prediction of other air pollutants or even other time-series prediction (such as financial data, environmental parameter estimation, electric utility load, machine reliability, etc.) where acute values are of interest and significance, for filtering data of low variations and performance improvement.



---

*Source: 310478-2014-04-09.xml*