# A Support Vector Machine Model with Hyperparameters Optimised by Mind Evolutionary Algorithm for Assessing Permeability of Rock

Authors: Wenjin Zhu; Zhiming Chao; Guotao Ma
Journal: Advances in Civil Engineering
 (2020)

---

## Abstract

In this paper, a database developed from the existing literature about permeability of rock was established. Based on the constructed database, a Support Vector Machine (SVM) model with hyperparameters optimised by Mind Evolutionary Algorithm (MEA) was proposed to predict the permeability of rock. Meanwhile, the Genetic Algorithm- (GA-) and Particle Swarm Algorithm- (PSO-) SVM models were constructed to compare the improving effects of MEA on the foretelling accuracy of machine learning models with those of GA and PSO, respectively. The following conclusions were drawn. MEA can increase the predictive accuracy of the constructed machine learning models remarkably in a few iteration times, which has better optimisation performance than that of GA and PSO. MEA-SVM has the best forecasting performance, followed by PSO-SVM, while the estimating precision of GA-SVM is lower than them. The proposed MEA-SVM model can accurately predict the permeability of rock indicating the model having a satisfactory generalization and extrapolation capacity.


---

## Body

## 1. Introduction

Natural geological formations that mainly consist of rock are optimal sites for many engineering projects, such as hydropower stations, oil and gas reservoirs, and coal mines [1–3]. Rock can enhance the safety of engineering projects [4]. For example, rock can prevent the leakage of oil and gas from oil and gas reservoirs [5]. Additionally, for huge hydropower stations, rock can prevent water from being extruded into rock under a high water level to avoid reducing the stability of rock masses [6]. Hence, a sound understanding of the permeability properties of rock is necessary for ensuring the safety and stability of engineering projects in this type of geological environment.As a porous material, rock has cracks and pores, which are the main seepage channels for fluid flow [7]. In nature, water inevitably exists in the small pores and cracks of rock [8–10]. For instance, rock in underground oil and gas reservoirs is often subjected to artificial water injection to improve the tightness of underground oil and gas reservoirs, and for the host rock for hydropower stations, water is forced into the pores and cracks of rock under a hydrostatic load [11, 12]. In actual engineering projects, the host rock is under stress conditions. The small cracks and pores in the rock are closed or opened with the changing confinement pressure and seepage pressure, significantly affecting the flow and distribution of the water inside the rock, as well as the permeability of the rock [13]. To provide comprehensive guidance for the construction and operation of engineering projects involving rock, it is worthwhile to investigate the permeability changing rules of rock.According to the results of physical experiments, empirical equations have been proposed to predict the permeability of rock [14–16]. Although the empirical equations have been used with success in some cases for predicting the permeability of rock, they are confined to a narrow application range and have a low forecasting accuracy owing to the small number of input parameters [17–19]. In the recent years, powerful machine-learning algorithms have often been utilized for estimating the properties of rock [20–27]. However, limited studies have been performed on the application of machine-learning techniques for predicting the permeability of rock [28–32].In general, the machine learning models, without combining optimisation algorithms, are inefficient. [22, 33, 34]. Hence, the optimised algorithms, such as, Genetic Algorithm (GA) and Particle Swarm Optimisation Algorithm (PSO), were applied by some researchers to optimise the initial parameters of machine learning models for evaluating the permeability of rock, and the increase in both of predictive accuracy and convergence speed of the constructed machine learning models after combining optimisation algorithms has been demonstrated [35, 36]. However, GA and PSO still have some inherent drawbacks. For example, their computational efficiency is low with long operational time, and they cannot guarantee whether the gained result is globally optimum [37, 38], causing detrimental impacts on their optimisation effects. To solve the shortcomings, many works have been carried out by scholars, among of which is Mind Evolutionary Algorithm (MEA) proposed by Chengyi et al. [39] to overcome the aforementioned defects of GA and PSO to some extent and improve the optimisation effects [40, 41]. The better performance of MEA than that of GA and PSO on increasing the estimating accuracy of machine learning models has also been proved by researchers in the engineering field [37, 42, 43]. However, to the best knowledge of the authors, currently, the application of MEA in improving the performance of machine learning models for predicting permeability of rock has not been reported.In this study, a database was developed based on the data collected from the existing literature. According to the database, SVM model combined with optimisation algorithm of Mind Evolutionary Algorithm (MEA) was established to predict permeability of rock. Meanwhile, the SVM models with hyperparameters optimised by GA and PSO, respectively, were constructed, to compare the improving effects of MEA on the foretelling accuracy of machine learning models with those of GA and PSO.


## 2. Database and Preprocessing Data

The database was developed based on the data collected from the existing literature. The database consists of 616 groups of data, which were divided randomly into 493 groups of training data (80%) for training the machine-learning models and 123 groups of testing data (20%) for testing the trained machine-learning models. In this program, each data group was randomly assigned a unique number ranging from 1 to 616. The data groups with number ranging from 1 to 431 were selected as the training dataset, and the data groups with number ranging from 431 to 616 were selected as the testing dataset. In each data group, pore pressure (Ps), confining pressure (Pc), moisture saturation (S), and confining pressure loading or unloading state (L) in the permeability tests were taken as the input parameters, and the corresponding permeability (k) of compact rock was taken as the output parameter. The statistics parameters and data type of the input and output parameters were tabulated in Table 1. Data distribution of the established database based on the input parameters was presented in Figure 1, respectively. In Figure 1, x-axis means the value of input parameters, and y-axis means the number of data groups in the established database corresponding to the value of input parameters. As shown in Figure 1, the distribution of data is uniform, which avoids the adverse influences of uneven data distribution on the predictive performance for the built machine learning models.Table 1
The statistical description of the input and output parameters.
ParametersCategoriesData typeStatistics parameters/value rangeMinimumMaximumMeanStandard deviationConfining pressure (MPa)Input parameterNumeric33517.9011.04Pore pressure (MPa)Input parameterNumeric0.11.40.710.43Moisture saturation (%)Input parameterNumeric073.6033.6523.73Loading/unloading stageInput parameterNominalLoading stage and unloading stagePermeability (10−17)Output parameterNumeric0.01272.7964.0567.43Figure 1
Data distribution of the developed database. (a) Pore pressure. (b) Confining pressure. (c) Moisture saturation. (d) Loading/unloading stage.
(a)(b)(c)(d)The input parameters for the machine learning models had different dimensions, which may affect the training time and prediction accuracy of the models [44]. To improve the forecasting accuracy and operational efficiency of the machine learning models, the input and output parameters were normalized to the range of 0-1 using the following equation.(1)xNormalized=x−xminxmax−xmin,where xNormalized represents the normalized value, x represents the original value, xmin represents the minimum value, and xmax represents the maximum value.


## 3. Machine-Learning Algorithm and Optimisation Algorithms

### 3.1. SVM

SVM is a binary predictive model, which is able to divide and predict sample data to achieve structural risk minimization according to maximum margin principles [45]. In this case, the radial basis function was selected as the kernel function of the SVM model, with penalty parameter c and g of the kernel function being optimised.


### 3.2. GA

GA introduces the biological evolutionary principle of “survival of the fittest” in the coded tandem population formed according to the optimisation parameters and selects individuals in the tandem population according to the individual fitness values and the operations of selection, crossover, and mutation such that individuals with high fitness values are retained and individuals with low fitness values are eliminated. The new generation of individuals inherits information from the previous generation of individuals and is superior to the previous generation. This process is repeated until the predetermined expired criterion is satisfied [46].


### 3.3. PSO

The PSO is based on the behavior of birds [47]. In the PSO, there is a population of particles. Each particle in the population represents a potential solution to the target problem and also corresponds to a fitness value determined by the fitness function. The particle speed determines the direction and distance of the particle’s movement and is dynamically adjusted according to the previous movement of the particle and other particles to optimise individual solutions in the solution space [48].


## 3.1. SVM

SVM is a binary predictive model, which is able to divide and predict sample data to achieve structural risk minimization according to maximum margin principles [45]. In this case, the radial basis function was selected as the kernel function of the SVM model, with penalty parameter c and g of the kernel function being optimised.


## 3.2. GA

GA introduces the biological evolutionary principle of “survival of the fittest” in the coded tandem population formed according to the optimisation parameters and selects individuals in the tandem population according to the individual fitness values and the operations of selection, crossover, and mutation such that individuals with high fitness values are retained and individuals with low fitness values are eliminated. The new generation of individuals inherits information from the previous generation of individuals and is superior to the previous generation. This process is repeated until the predetermined expired criterion is satisfied [46].


## 3.3. PSO

The PSO is based on the behavior of birds [47]. In the PSO, there is a population of particles. Each particle in the population represents a potential solution to the target problem and also corresponds to a fitness value determined by the fitness function. The particle speed determines the direction and distance of the particle’s movement and is dynamically adjusted according to the previous movement of the particle and other particles to optimise individual solutions in the solution space [48].


## 4. Hyperparameters Optimisation

In this research, MEA was adopted to optimise the hyperparameters of the constructed SVM model, with Root-Mean-Square Error (RMSE) as the fitness function, as expressed in equation (2). MEA is a new heuristically evolutionary intelligence algorithm, and previous researchers have demonstrated that MEA is more capable of improving the predictive performance of machine learning models compared with other evolutionary intelligence algorithms [42, 49]. Compared with traditional intelligence algorithms, such as GA and PSO, MEA has several main advantages: (i) the computational efficiency is high due to the parallel computation of similartaxis and dissimilation operations. (ii) The evolutionary information that MEA can retain is more than one generation, which provides beneficial guidance on the operational directions of similartaxis and dissimilation operations. (iii) The similartaxis and dissimilation operations in MEA can avoid the damage of original information for individuals.(2)RMSE=∑i=1nyi−fi2n,where n is the number of sample data,yi is the measured value, and fi is the predictive value.The specific operation of adopting MEA to optimise the SVM models is as follows: (1) randomly generate individuals that are composed of different hyperparameter values in solution space, (2) score the individuals based on fitness values (RMSE) obtained by calling the corresponding SVM model, and divide the individuals with low RMSE value as superior individuals and other individuals with high RMSE value as temporary individuals, (3) make the superior individuals and temporary individuals as centres, respectively, and generate new individuals around each centre individual to obtain superior subgroups and temporary subgroups, respectively, (4) perform similartaxis operations in each subgroup until the subgroup is mature (the RMSE value of the subgroup keeps unchangeable during continuous 6 times iteration), and take the RMSE value of optimal individual (centre individual) in each subgroup as the RMSE value of corresponding subgroup, (5) when the subgroups are mature, post the RMSE value of each subgroup on the global bulletin board, as shown in Figure2, and conduct dissimilation operations between the superior subgroups and temporary, including replacing or abandoning subgroups, releasing individuals in abandoned subgroups and supplying new subgroups, (6) carry out similartaxis operations in the new supplying subgroups, and repeat Step (4) to Step (5) until the RMSE value of new supplying subgroups is lower than those of superior subgroups, respectively, (7) take the centre individual in the superior subgroup with the lowest RMSE as the global superior individual, and assign the hyperparameter values of the global superior individual as the initial hyperparameter values to the established SVM model, and (8) train the built SVM model, and conduct prediction; the detailed optimisation process is as shown in Figure 3.Figure 2
The structure diagram of MEA.Figure 3
The flow chart of machine learning modelling with MEA optimisation.In this case, the population size was set as 300, and the number of superior subgroups and temporary subgroups was the same, set as 3. The size of the subgroup is 30, and the maximum iteration number is 20. Additionally, the fitness function value (RMSE) of individuals in MEA was obtained using a 10-fold cross-validation method (k-CV) on the corresponding SVM model during the process of hyperparameter optimisation [44]. To compare the optimisation effects between MEA and traditional optimisation algorithms, the SVM models with hyperparameters optimised by GA and PSO were constructed, respectively, and the predictive results of the GA and PSO-SVM model were compared with those of MEA-SVM, respectively. The detailed introduction about the optimised hyperparameters of the built SVM algorithms and their optimising range is provided in Table 2.Table 2
The optimised hyperparameters in the machine learning models.
Machine learning algorithmHyperparameterOptimisation algorithmsOptimising rangeSVMPenalty parameterc of kernel functionMEA, GA, PSO2−5–25Penalty parameterg of kernel function4−5–45


## 5. Quality Assessment

Predictive precision of the established machine-learning models was assessed using two indices: RMSE and Mean Absolute Percentage Error (MAPE), as expressed in equations (2) and (3), respectively [50].(3)MAPE=100%n∑i=1nyi−fiyi,where n represents the number of sample data, yi represents the measured value, y¯ represents the average measured value, and fi represents the predicted value.


## 6. Results and Analysis

### 6.1. Results of Hyperparameter Optimisation

As aforementioned, MEA was utilized to optimise the hyperparameters of machine learning algorithms, with RMSE as the fitness function. For each machine learning algorithm, three initial superior subgroups and three initial temporary subgroups were generated. The optimising process of the subgroups during the simillartaxis and dissimilation operations was recorded, as shown in Figure4.Figure 4
The evolution of RMSE during the similartaxis process. (a) Initial superior subgroups. (b) Initial temporary subgroups. (c) Superior subgroups after dissimilation operation. (d) Temporary subgroups after dissimilation operation.
(a)(b)(c)(d)Based on Figure4, for the initial superior subgroups and temporary subgroups, after several similartaxis operations, the RMSE of each subgroup tends to be steady, which indicates the subgroups are mature. Then, the dissimilation operations were conducted. In the dissimilation operation, the RMSE of temporary subgroups and superior subgroups was compared, and the superior subgroups were replaced by the temporary subgroups with lower RMSE. The rest of the temporary subgroups with high RMSE were abandoned, and individuals in them were released. Then, the released individuals were regrouped to form new temporary groups, and the similartaxis operations were performed again on the subgroups, as shown in Figure 5. For the new superior subgroups, the RMSE of them remains stable because they are already mature, as shown in Figure 5. By comparing the RMSE of new temporary subgroups and new superior subgroups, it can be seen that the RMSE of each superior subgroup is lower than that of the temporary subgroup, which meets the end criterion. Therefore, the subgroups need not perform dissimilation operations again, and the corresponding hyperparameters of the centre individual in the superior subgroup with the lowest RMSE were assigned as the initial parameters to the corresponding machine learning algorithms, respectively.Figure 5
The optimised processes of SVM models using GA and PSO. (a) GA-SVM. (b) PSO-SVM.
(a)(b)Based on the aforementioned analysis, for most of the subgroups, their RMSE reduces obviously and becomes stable within 15 iteration times. It indicates that MEA was efficient in the hyperparameters optimisation of the established SVM models, which can enhance the predictive accuracy of the established SVM model remarkably with high efficiency.To compare the optimisation effects between MEA and GA/PSO, the optimised processes of the SVM model using GA and PSO are shown in Figure5, respectively.According to Figures4 and 5, when MEA was adopted to optimise the SVM model, RMSE becomes stable within 18 times iterations, which is obviously lower than those of GA (80 times) and PSO (70 times). Additionally, after the optimisation of MEA, the RMSE of SVM model reduces to 4.69, while for GA and PSO, they are 8.52 and 7.99, respectively. It demonstrates that the optimisation performance of MEA on the SVM model is better than those of GA and PSO in both the aspects of optimising efficiency and increasing magnitude in predictive accuracy.


### 6.2. The Prediction on the Training Dataset

The predictive performance of the established SVM models with optimised hyperparameters on the training dataset is presented in Figures6–8.Figure 6
Predictive performance of the models for the training dataset. (a) MEA-SVM. (b) GA-SVM. (c) PSO-SVM.
(a)(b)(c)Figure 7
The MAPE of the models.Figure 8
The RMSE of the models.According to Figures6–8, for the training dataset, the forecasting performance of the SVM model with hyperparameters optimised by MEA is the best among the constructed models in terms of the statistics parameters: RMSE and MAPE. More specifically, the MEA-SVM model achieved the lowest RMSE (4.61) and MAPE (9.15%), among the models, followed by PSO-SVM, with GA-SVM having poor predictive accuracy.


### 6.3. The Prediction on the Testing Dataset

The predictive performance of the established SVM models with optimised hyperparameters on the testing dataset is presented in Figures7–9.Figure 9
Predictive performance of the models for the testing dataset. (a) MEA-SVM. (b) GA-SVM. (c) PSO-SVM.
(a)(b)(c)As shown in Figures7–9, for the testing dataset, the SVM model with hyperparameters optimised by MEA still has the highest foretelling precision among the models, with the lowest RMSE (6.69) and MAPE (11.07%) followed by PSO-SVM. The predictive performance of the GA-SVM model is the worst among them.The results indicate that, for both of the testing dataset and training dataset, the MEA-SVM model has the highest precision in estimating the permeability of compact rock, and MEA has better performance in improving the estimating accuracy of the machine learning model than those of GA and PSO, respectively.


## 6.1. Results of Hyperparameter Optimisation

As aforementioned, MEA was utilized to optimise the hyperparameters of machine learning algorithms, with RMSE as the fitness function. For each machine learning algorithm, three initial superior subgroups and three initial temporary subgroups were generated. The optimising process of the subgroups during the simillartaxis and dissimilation operations was recorded, as shown in Figure4.Figure 4
The evolution of RMSE during the similartaxis process. (a) Initial superior subgroups. (b) Initial temporary subgroups. (c) Superior subgroups after dissimilation operation. (d) Temporary subgroups after dissimilation operation.
(a)(b)(c)(d)Based on Figure4, for the initial superior subgroups and temporary subgroups, after several similartaxis operations, the RMSE of each subgroup tends to be steady, which indicates the subgroups are mature. Then, the dissimilation operations were conducted. In the dissimilation operation, the RMSE of temporary subgroups and superior subgroups was compared, and the superior subgroups were replaced by the temporary subgroups with lower RMSE. The rest of the temporary subgroups with high RMSE were abandoned, and individuals in them were released. Then, the released individuals were regrouped to form new temporary groups, and the similartaxis operations were performed again on the subgroups, as shown in Figure 5. For the new superior subgroups, the RMSE of them remains stable because they are already mature, as shown in Figure 5. By comparing the RMSE of new temporary subgroups and new superior subgroups, it can be seen that the RMSE of each superior subgroup is lower than that of the temporary subgroup, which meets the end criterion. Therefore, the subgroups need not perform dissimilation operations again, and the corresponding hyperparameters of the centre individual in the superior subgroup with the lowest RMSE were assigned as the initial parameters to the corresponding machine learning algorithms, respectively.Figure 5
The optimised processes of SVM models using GA and PSO. (a) GA-SVM. (b) PSO-SVM.
(a)(b)Based on the aforementioned analysis, for most of the subgroups, their RMSE reduces obviously and becomes stable within 15 iteration times. It indicates that MEA was efficient in the hyperparameters optimisation of the established SVM models, which can enhance the predictive accuracy of the established SVM model remarkably with high efficiency.To compare the optimisation effects between MEA and GA/PSO, the optimised processes of the SVM model using GA and PSO are shown in Figure5, respectively.According to Figures4 and 5, when MEA was adopted to optimise the SVM model, RMSE becomes stable within 18 times iterations, which is obviously lower than those of GA (80 times) and PSO (70 times). Additionally, after the optimisation of MEA, the RMSE of SVM model reduces to 4.69, while for GA and PSO, they are 8.52 and 7.99, respectively. It demonstrates that the optimisation performance of MEA on the SVM model is better than those of GA and PSO in both the aspects of optimising efficiency and increasing magnitude in predictive accuracy.


## 6.2. The Prediction on the Training Dataset

The predictive performance of the established SVM models with optimised hyperparameters on the training dataset is presented in Figures6–8.Figure 6
Predictive performance of the models for the training dataset. (a) MEA-SVM. (b) GA-SVM. (c) PSO-SVM.
(a)(b)(c)Figure 7
The MAPE of the models.Figure 8
The RMSE of the models.According to Figures6–8, for the training dataset, the forecasting performance of the SVM model with hyperparameters optimised by MEA is the best among the constructed models in terms of the statistics parameters: RMSE and MAPE. More specifically, the MEA-SVM model achieved the lowest RMSE (4.61) and MAPE (9.15%), among the models, followed by PSO-SVM, with GA-SVM having poor predictive accuracy.


## 6.3. The Prediction on the Testing Dataset

The predictive performance of the established SVM models with optimised hyperparameters on the testing dataset is presented in Figures7–9.Figure 9
Predictive performance of the models for the testing dataset. (a) MEA-SVM. (b) GA-SVM. (c) PSO-SVM.
(a)(b)(c)As shown in Figures7–9, for the testing dataset, the SVM model with hyperparameters optimised by MEA still has the highest foretelling precision among the models, with the lowest RMSE (6.69) and MAPE (11.07%) followed by PSO-SVM. The predictive performance of the GA-SVM model is the worst among them.The results indicate that, for both of the testing dataset and training dataset, the MEA-SVM model has the highest precision in estimating the permeability of compact rock, and MEA has better performance in improving the estimating accuracy of the machine learning model than those of GA and PSO, respectively.


## 7. Engineering Application

Based on the foregoing analysis, reliability of the established MEA-SVM model for predicting the permeability of rock is validated. To further evaluate the generalization and extrapolation capacity of the machine-learning models for predicting the permeability of rock from other engineering sites, the established MEA-SVM model was adopted to estimate the permeability of another type of rock. This was performed by inputting the confinement pressure, seepage pressure, and moisture saturation of the rock into the developed MEA-SVM model. In the existing literature, researchers have investigated the effects of the confinement pressure, seepage pressure, and moisture saturation of rock on its permeability.The permeability of the rock conducted by other researchers in the existing literature was compared with the permeability predicted using the proposed MEA-SVM model to validate the applicability of the established MEA-SVM model for the prediction of the permeability of another type of rock. The predictive permeability and the measured permeability by other researchers are presented in Figure10. To evaluate the forecasting performance of the established MEA-SVM model, the prediction precision of the built MEA-SVM models was assessed according to MAPE and RMSE, as shown in Figure 10, respectively.Figure 10
Predictive permeability and measured permeability.As indicated in Figure10, the permeability predicted using the proposed MEA-SVM model matched the measured permeability by other researchers well, with an RMSE of 4.00 and MAPE of 10.14%, indicating that the proposed MEA-SVM model can accurately predict the permeability of the rock and that the established MEA-SVM model is reliable for predicting the permeability of another type of rock with a satisfactory generalization and extrapolation capacity.


## 8. Conclusions

In this paper, based on the established database collected from the existing literature, an SVM model with hyperparameters optimised by adopting MEA was proposed to predict the permeability of rock. Meanwhile, the GA and PSO-SVM models were constructed to compare the improving effects of MEA on the foretelling accuracy of machine learning models with those of GA and PSO, respectively. The following conclusions were drawn.(1)
MEA can increase the predictive accuracy of the constructed machine learning models remarkably in few iteration times, which has better optimisation performance than that of GA and PSO(2)
For both of the testing dataset and training dataset, MEA-SVM has the best forecasting performance, followed by PSO-SVM, while the estimating precision of GA-SVM is lower than them, in terms of the statistics criterions: RMSE and MAPE(3)
The proposed MEA-SVM model can accurately predict the permeability of another type of rock indicating the model having a satisfactory generalization and extrapolation capacity



---

*Source: 4718493-2020-09-08.xml*