# Prediction of Pile Bearing Capacity Using Opposition-Based Differential Flower Pollination-Optimized Least Squares Support Vector Regression (ODFP-LSSVR)

Authors: Nhat-Duc Hoang; Xuan-Linh Tran; Thanh-Canh Huynh
Journal: Advances in Civil Engineering
 (2022)

---

## Abstract

Pile foundations are widely used for high-rise structures constructed in soft ground. The bearing capacity of pile is a crucial parameter required during the design and construction phase of pile foundation engineering projects. In practice, accurate predictions of pile bearing capacity are challenging due to a complex interplay of various geotechnical engineering factors including pile characteristics and ground conditions. This study proposes a data-driven model for coping with the problem of interest that hybridizes machine learning and metaheuristic approaches. Least squares support vector regression (LSSVR) is used for analyzing a dataset containing historical records of pile tests. Based on such datasets, LSSVR is capable of generalizing a multivariate function that estimates values of pile bearing capacity based on a set of variables describing pile characteristics and ground conditions. Moreover, opposition-based differential flower pollination (ODFP) metaheuristic is proposed to optimize the LSSVR learning process. Experimental results supported by the statistical test showed that the proposed ODFP-optimized LSSVR can achieve a good predictive performance in terms of root mean square error, mean absolute percentage error mean absolute error, and coefficient of determination. These results confirm that the ODFP-optimized LSSVR can be a potential alternative to assist civil engineers in the task of pile bearing capacity estimation.


---

## Body

## 1. Research Background and Motivation

The pile foundation plays a crucial role in high-rise building construction. This type of foundation is widely used to transfer considerable structural loads to solid soil layers beneath high-rise buildings. In pile foundation engineering, pile bearing capacity is an essential parameter needed to be estimated [1]. Accurate estimation of this parameter ensures the safety and economics of pile foundation structures. In practice, pile load testing is widely recognized as the most reliable and accurate approach for obtaining bearing capacity of piles. It is because this method closely resembles the working mechanism of driven piles. Nevertheless, this testing method generally requires time-consuming preparation/measurement processes, experienced technicians, and costly operations. Therefore, the pile load testing remains feasible only for large-scale projects.For small-scale and low-cost projects, an in situ testing approach based on the standard penetration test (SPT) is a viable alternative to the pile load testing [2, 3]. Using the in situ testing approach, a certain degree of accuracy in bearing capacity estimation may be compromised to obtain an affordable experimental cost.Various studies have relied on SPT outcomes to come up with data-driven bearing capacity prediction [4, 5]. Conventional regression analysis approaches are employed to fit in situ testing data and construct prediction models. The common characteristics of the aforementioned approaches are that data regarding the mechanical properties of piles, properties of soil layers, and the number of SPT blow counts are used as explanatory variables. The main drawback of them is that they rely on traditional data-fitting approaches which have certain restrictions in modeling nonlinear, complex, and multivariate data. The existing models are also constructed with a limited number of data samples [6, 7] and are applicable for certain ground conditions [8, 9]. Therefore, there is no surprise that a model may perform well in one construction site but predicts inaccurately in other study areas.To leverage the increasing sizes of experimental datasets featuring a large number of explanatory variables, advanced data-driven methods based on machine learning and other computational intelligence techniques have been proposed. Goh et al. [10] put forward an artificial neural network (ANN) model used to determine the friction capacity of driven piles in clays. Pal and Deswal [11] employ support vector machines (SVMs) and the generalized regression neural network for modeling pile capacity. Samui [12] examines the use of SVM regression for pile bearing capacity estimation. The least square support vector machine (LSSVM) has been used in [13] to construct a functional mapping capable of estimating of pile bearing capacity; however, this research relies on a limited database of 50 testing cases.Based on the dataset featuring in situ load tests, Momeni et al. [14] and Shahin [15] developed ANN models used for forecasting driven pile load capacity. An attempt to improve the generalization of the neural network model used for predicting the variable of interest using genetic algorithm has been reported in [16]. Prayogo and Susanto [17] combine LSSVM and symbiotic organisms search to predict friction capacity of driven piles in cohesive soil; 45 historical cases are used for model training, and 20 other cases are employed for model testing. Luo and Dong [18] proposed to employ the Bayesian reliability theory and Markov chain Monte Carlo method to statistically estimate the bearing capacity of pile foundations.Recently, Pham et al. [19] rely on a large-scale collection of driven pile static load tests to construct machine learning-based models for predicting axial bearing capacity of piles; the employed machine learning models are ANN and random forest ensemble. A recent study harnesses the strength of metaheuristic optimization and deep neural network approaches to establish pile bearing capacity estimation [20]; this study concludes that metaheuristic optimization is highly effective in determining the structure of the employed machine learning model.A research that combines simulated annealing and genetic programming used for estimating ultimate bearing capacity of pile is reported in [21]. Kardani et al. [22] forecast the variable of interest in the ground condition of cohesionless soil using swarm intelligence-optimized machine learning approaches; the employed machine learning models include regression tree, k-nearest neighbor, neural computing, SVM, and regression tree ensemble strategies. The capabilities of particle swarm optimization and genetic algorithm-optimized neural networks used for pile bearing capacity estimation are explored in [23].Momeni et al. [24] rely on the Gaussian process regression model for dealing with the complexity of the problem at hand; a dataset including 296 dynamic pile load tests obtained from construction sites has been used to construct the prediction model. Adaptive neuro-fuzzy inference system (ANFIS) has been hybridized with firefly algorithm to establish a data-driven model employed for pile capacity forecasting [25]. Harandizadeh et al. [26] extend the work of Sun et al. [25] by including a group method of data handling into the firefly algorithm-optimized adaptive neuro-fuzzy inference system. Chen et al. [27] construct novel approaches of neuro-genetic, neuro-imperialism, and genetic programming models in predicting the ultimate bearing capacity of pile; a dataset including 50 samples is used for model training and validation.Based on the current literature and state-of-the-art reviews [15, 28], there is an increasing trend of applying machine learning approaches in prediction of pile bearing capacity. It can be seen that individual neural computing and SVM models are the dominant methods used for solving the task of interest [13, 14, 24, 29, 30]. The number of studies investigating hybridization of machine learning and metaheuristic algorithms used for pile capacity modeling is still limited.Although promising outcomes have been observed in the study by Yong et al. [21], the main limitation of this work is the limited amount of training and validating data. The models’ performance reported in [22] can be potentially enhanced via the employment of other advanced swarm intelligence-based algorithms [31–37]. The feasibility of using ANFIS optimized with PSO for pile bearing capacity estimation has been demonstrated by Harandizadeh et al. [26]. However, the construction of this hybrid model is sophisticated and challenging due to the large number of fine-tuned parameters (i.e., membership functions’ parameters and neural network weights) found in ANFIS.Moreover, due to the site-dependent nature of the problem, a machine learning model that can be fitted for a certain study area may not achieve the desired performance in another one. Therefore, this study aims at extending the current literature by proposing a novel integration of machine learning and metaheuristic approaches. The least squares support vector regression (LSSVR) [38] is used to approximate the functional mapping that derives the estimated bearing capacity based on a set of explanatory variables describing pile and site conditions. Moreover, opposition-based learning (OBL) [39] is integrated with the structure of differential flower pollination (DFP) [40] to establish a novel metaheuristic, denoted as opposition-based DFP (ODFP), used for optimizing the performance of the LSSVR employed for pile bearing capacity prediction. The subsequent section of the paper is organized as follows: Section 2 contains the research material and the employed computational intelligence approaches. The proposed data-driven model used for predicting pile bearing capacity is put forward in Section 3. Experimental results and discussion on research findings are reported in Section 4. Concluding remarks that summarize this study are stated in Section 5.


## 2. Research Material and the Employed Computational Intelligence Methods

### 2.1. The Dataset of Pile Bearing Capacity Estimation Experiments

In this study, a dataset consisting of static load test data of driven reinforced concrete piles is used to construct the data-driven method used for pile capacity prediction. The dataset consists of 472 records and is reported in [20]. This dataset contains a fairly large number of data instances which can be appropriate for training hybrid machine learning models. For the purpose of measurement, precast piles with closed tips have been driven into soil layers using a hydraulic pile driving machine. The experimental setup is demonstrated in Figure 1.Figure 1
Static load test experiment for measuring pile bearing capacity.In addition, the explanatory variables include pile diameter (X1), thickness of the first soil layer (X2), thickness of the second soil layer (X3), thickness of the third soil layer (X4), elevation of the natural ground (X5), top of pile elevation (X6), elevation of the extra segment of pile top (X7), depth of the pile tip (X8), mean value of SPT blow count along the pile shaft (X9), and mean value of SPT blow count at the pile tip (X10). The predicted variable (Y) is the axial pile bearing capacity. The pile structure, its geometrical variables, and soil stratigraphy are illustrated in Figure 2. In addition, the histograms of the variables in the collected dataset are shown in Figure 3. Statistical descriptions of the explanatory factors and pile bearing capacity are summarized in Table 1. Scatter plots used for preliminary inspections of linear correlations between explanatory variable (X) and pile bearing capacity (Y) are demonstrated in Figure 4.Figure 2
Demonstration of the pile structure and soil stratigraphy.Figure 3
Histograms of influencing factorsX1, X2, …, X10 and pile bearing capacity Y.Table 1
Statistical descriptions of the influencing factors and pile bearing capacity.
VariablesNotationMinMeanStd.MaxThe pile diameter (mm)X1300.00363.7748.12400.00The thickness of the first soil layer that pile embedded (m)X23.403.830.485.72The thickness of the second soil layer that pile embedded (m)X31.506.581.648.00The thickness of the third soil layer that pile embedded (m)X40.000.330.461.69The elevation of pile top (m)X50.682.800.623.40The elevation of natural ground (m)X63.043.500.084.12The elevation of extra segment of pile top (m)X71.032.920.604.35The depth of the pile tip (m)X88.3013.541.8016.09The average SPT blow count along the pile shaftX95.6010.742.2615.41The average SPT blow count at the pile tipX104.387.060.667.75The axial bearing capacity load of pile (kN)Y407.20984.20353.211551.00Figure 4
Scatter plots of the input variable (X) and pile bearing capacity (Y).


### 2.2. Differential Flower Pollination (DFP)

DFP, proposed in [40], is a combination of two individual metaheuristic algorithms, namely, differential evolution (DE) [41] and flower pollination algorithm (FPA) [42]. DE is a simple yet highly effective variant of evolutionary algorithms which encompasses key evolutionary operations including mutation and recombination [43, 44]. Meanwhile, FPA is powerful in search space exploration due to its employment of Lévy flight operation [45–49]. The integration of DE and FPA preserves the advantage of the self-adaptive mutation of DE and significantly boosts the effectiveness of the contractive nature of DE's intermediate arithmetic recombination [43]. Therefore, the combined DFP algorithm has a good exploitation capability as well as has an escape mechanism to fend off premature convergence.Based on a randomly created population of solution candidates, the DFP carries out global search and local search operations. The frequency of these two searching operations depends on a probabilityp according to the following rule:(1)Ifr<pthenperform global search operationElseperform local search operation,where r denotes a random number within the range of 0 and 1. p is normally set to be 0.8 according to [42].The global search and local search operations are described as follows:(i)
The FPA-based global search operator is given by(2)Xitrial=Xig+L.Xig−Xbest,whereg denotes the current generation. Xitrial represents a newly created trial solution. L is a number generated from the Lévy distribution.(ii)
The DE-based local search operator is given by(3)Xi,gmutated=xr1,g+F.xr2,g−xr3,g,wherer1, r2, and r3 denote 3 random integers. F is a mutation scale factor. The parameter F is generated from a Gaussian distribution with the mean = 0.5 and the standard deviation = 0.15 [40].(4)Xj,i,g+1crossed=Xj,imutated,ifrandj≤Crorj=rnbi,Xj,i,g,ifrandj>Crandj≠rnbi,whereCr represents the crossover probability which is normally set to be 0.8.It is noted the parameterL stated in (1) is a random number generated from the Lévy distribution. This distribution is mathematically described as follows [50]:(5)L∼λΓλsinπλ/2πs1+λ,where s=U/|V|1/λ and U and V ∼ normal (0, 1). Γλ is the standard Gamma distribution with λ=3/2.


### 2.3. Opposition-Based Learning (OBL)

In evolutionary optimization, OBL [39] is an important concept used for enhancing the effectiveness of the random search process and therefore boosting the convergence rate of metaheuristic algorithms. The main idea of OBL is to simultaneously consider a candidate solution and its corresponding opposition [51]. Experimental studies have demonstrated that using OBL can improve the chance of finding better solutions [52–55].The oppositionXop of a candidate solution X is computed as follows [53]:(6)Xop,d=LBd+UBd−Xdd=1,2,…,D,where D is the number of search variables. LB and UB are the lower and upper boundaries of the search variables, respectively.It is noted that OBL has been successfully integrated into the standard DE structure in [53]. Herein, Rahnamayan et al. [53] introduce operations of opposition-based population initialization and opposition-based generation jumping to enhance the searching performance of DE. The former operation is carried out once at the beginning of the evolutionary process. Meanwhile, the latter operation is performed repeatedly after the mutation-crossover procedure with a probability Jr, where Jr denotes a jumping rate parameter.


### 2.4. Least Squares Support Vector Regression (LSSVR)

LSSVR [38] is a variant of the standard support vector machines (SVMs) used for nonlinear function approximation. SVM-based regression models have demonstrated their superior performances in various learning tasks in civil engineering [56–61]. This machine learning method features significant advantages including good generalization properties, ability of modeling multivariate data, effective nonlinear function mapping, fast computation, and few controlling parameters. Similar to the standard SVM [62], the LSSVR learning process is inspired from the structural risk minimization framework which often leads to the construction of robust data-driven models.Moreover, the model structure of a LSSVR model is identified via solving a linear system instead of a nonlinear system as required by the SVM [58, 63, 64]. Therefore, LSSVR can be highly effective in dealing with large-scale datasets. To cope with data nonlinearity, LSSVR relies on the concept of kernel function which is responsible for converting data from an original input space into a high-dimensional feature space, within which a hyperplane can be used to establish a functional mapping between a set of explanatory variables and a predicted variable (refer to Figure 5).Figure 5
The LSSVR learning process.The general form of a LSSVR model used for pile bearing capacity estimation can be stated as follows:(7)yx=wTφx+b,where x∈Rn and y∈R denote a set of explanatory variables and the predicted variable, respectively. φx:Rn⟶Rnh represents a mapping from an original input space to a high-dimensional feature space.In (7), w and b are the LSSVR model parameters needed to be estimated from the collected dataset. To determine those model parameters, it is necessary to solve a constrained optimization problem described as follows:(8)minimizeJpw,e=12wTw+γ12∑k=1Nek2,subjected to yk=wTφxk+b+ek, k=1,…,N, where xk,ykk=1N denotes a set of the training dataset with N samples. xk,yk represents a set of explanatory variables and the predicted variable. ek∈R is an error variable. γ>0 denotes a regularization coefficient.The corresponding Lagrangian formula is given by(9)Lw,b,e;α=Jpw,e−∑k=1NαkwTφxk+b+ek−yk,where αk denotes a Lagrange multiplier.Subsequently, the Karush–Kuhn–Tucker conditions for optimality are stated equivalently as follows:(10)∂L∂w=0⟶w=∑k=1Nαkφxk,∂L∂b=0⟶∑k=1Nαk=0,∂L∂ek=0⟶αk=γek,k=1,…,N,∂L∂αk=0⟶wTφxk+b+ek−yk=0,k=1,…N.(10) can be converted to a system of linear equations as follows:(11)01v⋅1vTω+I/γbα=0y,where y=y1,…,yN, 1v=1;…;1, and α=α1;…;αN. ω is a kernel function; its formula is presented as follows:(12)ω=φxkTφxl=Kxk,xl.The radial basis function (RBF) kernel is often employed in SVM-based regression models [65, 66]. Its mathematical description is given by(13)Kxk,xl=exp−xk−xl22σ2,where σ denotes the kernel function hyperparameter.


## 2.1. The Dataset of Pile Bearing Capacity Estimation Experiments

In this study, a dataset consisting of static load test data of driven reinforced concrete piles is used to construct the data-driven method used for pile capacity prediction. The dataset consists of 472 records and is reported in [20]. This dataset contains a fairly large number of data instances which can be appropriate for training hybrid machine learning models. For the purpose of measurement, precast piles with closed tips have been driven into soil layers using a hydraulic pile driving machine. The experimental setup is demonstrated in Figure 1.Figure 1
Static load test experiment for measuring pile bearing capacity.In addition, the explanatory variables include pile diameter (X1), thickness of the first soil layer (X2), thickness of the second soil layer (X3), thickness of the third soil layer (X4), elevation of the natural ground (X5), top of pile elevation (X6), elevation of the extra segment of pile top (X7), depth of the pile tip (X8), mean value of SPT blow count along the pile shaft (X9), and mean value of SPT blow count at the pile tip (X10). The predicted variable (Y) is the axial pile bearing capacity. The pile structure, its geometrical variables, and soil stratigraphy are illustrated in Figure 2. In addition, the histograms of the variables in the collected dataset are shown in Figure 3. Statistical descriptions of the explanatory factors and pile bearing capacity are summarized in Table 1. Scatter plots used for preliminary inspections of linear correlations between explanatory variable (X) and pile bearing capacity (Y) are demonstrated in Figure 4.Figure 2
Demonstration of the pile structure and soil stratigraphy.Figure 3
Histograms of influencing factorsX1, X2, …, X10 and pile bearing capacity Y.Table 1
Statistical descriptions of the influencing factors and pile bearing capacity.
VariablesNotationMinMeanStd.MaxThe pile diameter (mm)X1300.00363.7748.12400.00The thickness of the first soil layer that pile embedded (m)X23.403.830.485.72The thickness of the second soil layer that pile embedded (m)X31.506.581.648.00The thickness of the third soil layer that pile embedded (m)X40.000.330.461.69The elevation of pile top (m)X50.682.800.623.40The elevation of natural ground (m)X63.043.500.084.12The elevation of extra segment of pile top (m)X71.032.920.604.35The depth of the pile tip (m)X88.3013.541.8016.09The average SPT blow count along the pile shaftX95.6010.742.2615.41The average SPT blow count at the pile tipX104.387.060.667.75The axial bearing capacity load of pile (kN)Y407.20984.20353.211551.00Figure 4
Scatter plots of the input variable (X) and pile bearing capacity (Y).


## 2.2. Differential Flower Pollination (DFP)

DFP, proposed in [40], is a combination of two individual metaheuristic algorithms, namely, differential evolution (DE) [41] and flower pollination algorithm (FPA) [42]. DE is a simple yet highly effective variant of evolutionary algorithms which encompasses key evolutionary operations including mutation and recombination [43, 44]. Meanwhile, FPA is powerful in search space exploration due to its employment of Lévy flight operation [45–49]. The integration of DE and FPA preserves the advantage of the self-adaptive mutation of DE and significantly boosts the effectiveness of the contractive nature of DE's intermediate arithmetic recombination [43]. Therefore, the combined DFP algorithm has a good exploitation capability as well as has an escape mechanism to fend off premature convergence.Based on a randomly created population of solution candidates, the DFP carries out global search and local search operations. The frequency of these two searching operations depends on a probabilityp according to the following rule:(1)Ifr<pthenperform global search operationElseperform local search operation,where r denotes a random number within the range of 0 and 1. p is normally set to be 0.8 according to [42].The global search and local search operations are described as follows:(i)
The FPA-based global search operator is given by(2)Xitrial=Xig+L.Xig−Xbest,whereg denotes the current generation. Xitrial represents a newly created trial solution. L is a number generated from the Lévy distribution.(ii)
The DE-based local search operator is given by(3)Xi,gmutated=xr1,g+F.xr2,g−xr3,g,wherer1, r2, and r3 denote 3 random integers. F is a mutation scale factor. The parameter F is generated from a Gaussian distribution with the mean = 0.5 and the standard deviation = 0.15 [40].(4)Xj,i,g+1crossed=Xj,imutated,ifrandj≤Crorj=rnbi,Xj,i,g,ifrandj>Crandj≠rnbi,whereCr represents the crossover probability which is normally set to be 0.8.It is noted the parameterL stated in (1) is a random number generated from the Lévy distribution. This distribution is mathematically described as follows [50]:(5)L∼λΓλsinπλ/2πs1+λ,where s=U/|V|1/λ and U and V ∼ normal (0, 1). Γλ is the standard Gamma distribution with λ=3/2.


## 2.3. Opposition-Based Learning (OBL)

In evolutionary optimization, OBL [39] is an important concept used for enhancing the effectiveness of the random search process and therefore boosting the convergence rate of metaheuristic algorithms. The main idea of OBL is to simultaneously consider a candidate solution and its corresponding opposition [51]. Experimental studies have demonstrated that using OBL can improve the chance of finding better solutions [52–55].The oppositionXop of a candidate solution X is computed as follows [53]:(6)Xop,d=LBd+UBd−Xdd=1,2,…,D,where D is the number of search variables. LB and UB are the lower and upper boundaries of the search variables, respectively.It is noted that OBL has been successfully integrated into the standard DE structure in [53]. Herein, Rahnamayan et al. [53] introduce operations of opposition-based population initialization and opposition-based generation jumping to enhance the searching performance of DE. The former operation is carried out once at the beginning of the evolutionary process. Meanwhile, the latter operation is performed repeatedly after the mutation-crossover procedure with a probability Jr, where Jr denotes a jumping rate parameter.


## 2.4. Least Squares Support Vector Regression (LSSVR)

LSSVR [38] is a variant of the standard support vector machines (SVMs) used for nonlinear function approximation. SVM-based regression models have demonstrated their superior performances in various learning tasks in civil engineering [56–61]. This machine learning method features significant advantages including good generalization properties, ability of modeling multivariate data, effective nonlinear function mapping, fast computation, and few controlling parameters. Similar to the standard SVM [62], the LSSVR learning process is inspired from the structural risk minimization framework which often leads to the construction of robust data-driven models.Moreover, the model structure of a LSSVR model is identified via solving a linear system instead of a nonlinear system as required by the SVM [58, 63, 64]. Therefore, LSSVR can be highly effective in dealing with large-scale datasets. To cope with data nonlinearity, LSSVR relies on the concept of kernel function which is responsible for converting data from an original input space into a high-dimensional feature space, within which a hyperplane can be used to establish a functional mapping between a set of explanatory variables and a predicted variable (refer to Figure 5).Figure 5
The LSSVR learning process.The general form of a LSSVR model used for pile bearing capacity estimation can be stated as follows:(7)yx=wTφx+b,where x∈Rn and y∈R denote a set of explanatory variables and the predicted variable, respectively. φx:Rn⟶Rnh represents a mapping from an original input space to a high-dimensional feature space.In (7), w and b are the LSSVR model parameters needed to be estimated from the collected dataset. To determine those model parameters, it is necessary to solve a constrained optimization problem described as follows:(8)minimizeJpw,e=12wTw+γ12∑k=1Nek2,subjected to yk=wTφxk+b+ek, k=1,…,N, where xk,ykk=1N denotes a set of the training dataset with N samples. xk,yk represents a set of explanatory variables and the predicted variable. ek∈R is an error variable. γ>0 denotes a regularization coefficient.The corresponding Lagrangian formula is given by(9)Lw,b,e;α=Jpw,e−∑k=1NαkwTφxk+b+ek−yk,where αk denotes a Lagrange multiplier.Subsequently, the Karush–Kuhn–Tucker conditions for optimality are stated equivalently as follows:(10)∂L∂w=0⟶w=∑k=1Nαkφxk,∂L∂b=0⟶∑k=1Nαk=0,∂L∂ek=0⟶αk=γek,k=1,…,N,∂L∂αk=0⟶wTφxk+b+ek−yk=0,k=1,…N.(10) can be converted to a system of linear equations as follows:(11)01v⋅1vTω+I/γbα=0y,where y=y1,…,yN, 1v=1;…;1, and α=α1;…;αN. ω is a kernel function; its formula is presented as follows:(12)ω=φxkTφxl=Kxk,xl.The radial basis function (RBF) kernel is often employed in SVM-based regression models [65, 66]. Its mathematical description is given by(13)Kxk,xl=exp−xk−xl22σ2,where σ denotes the kernel function hyperparameter.


## 3. The Proposed Data-Driven Model for Pile Bearing Capacity Estimation Using Opposition-Based DFP-Optimized LSSVR (ODFP-LSSVR)

In this section, the structure of the proposed hybridization of ODFP and LSSVR designed for pile bearing capacity estimation is presented. As mentioned in the previous section, the training phase of LSSVR requires the determination of two crucial hyperparameters which are the regularization constant (γ) and the kernel function parameter (σ). The former hyperparameter affects the amount of penalty imposed on data samples deviating from the regression hyperplane. The latter hyperparameter is actually the bandwidth of the RBF which strongly influences the smoothness of the approximated function. Both the regularization constant (γ) and the kernel function parameter (σ) affect the generalization and the quality of the LSSVR model used for pile bearing capacity estimation. If the value of γ is very large, the model is highly susceptible to overfitting. On the other hand, a small value of γ often leads to an underfitted model which provides no useful information for the prediction process. In addition, selecting an appropriate value of σ is also essential. If σ is very small, the learning process is strongly affected by local noise. On the contrary, a large value of σ leads to the fact that too many data points are taken into account to determine the predicted value of pile bearing capacity; therefore, the prediction outcome can be very general and yields no useful information.Algorithm 1: The ODFP optimization algorithm.
Randomly initialize a populationPop =  X1,X2,…,XPS// PS denotes the number of population memberDefine LB and UB// LB and UB are lower and upper boundaries, respectivelyDefine the objective cost functionCFCalculate the objective cost function of the population: PopFSpecify the probability of population jumpingJrDefine the maximum number of function calls: MNFCFori = 1 : PSXi,op,d=LBd+UBd−Xi,d, d = 1, 2, …, D//D is the number of searched parameters//Xi,op denotes the opposition of XiCompute CF(Xop)If (CF(Xop) < CF(X))Pop [X] = Pop [Xop]PopF[X] = PopF[Xop]End IfEnd ForIdentify the best solutionXbestIdentify the best cost function CF(Xbest)Count = 0 // counting number of function callswhile Count < MNFCFori = 1 : PSGenerater∼U(0,1)Ifr < p // Perform global searchGenerate a trial solution viaXitrial=Xig+L.Xig−XbestElse // Perform local searchPerform mutation operationXi,gmutated=xr1,g+F.xr2,g−xr3,gPerform crossover operationXj,i,g+1crossed=Xj,imutated,ifrandj≤Crorj=rnbiXj,i,g,ifrandj>Crandj≠rnbiEnd IfUpdateXbestEnd ForGenerateθ∼U(0,1)Ifθ < JrFori = 1 : PSXi,op,d=LBd+UBd−Xi,d, d = 1, 2, …, DCompute CF(Xop)If (CF(Xop) < CF(X))Pop [X] = Pop [Xop]PopF[X] = PopF[Xop]End IfEnd ForEnd IfUpdate CountUpdateXbestUpdate CF(Xbest)End ForReturnXbestTherefore, this study proposes applying a metaheuristic to optimize the LSSVR model used for pile capacity estimation. Moreover, to enhance the optimization capability, this study proposes to incorporate the OBL into the DFP structure to formulate opposition-based DFP (ODFP). The operational flow of the ODFP metaheuristic is summarized in Algorithm 1. LSSVR has the role of approximating the nonlinear and multivariate relationship between pile bearing capacity and its influencing factors. Meanwhile, ODFP is utilized to automatically optimize the LSSVR model construction. Herein, OBL is integrated with the DFP implementation to enhance its searching capability [52, 55, 67, 68]. Since the effectiveness of LSSVR for estimating nonlinear functions in civil engineering has been demonstrated in previous studies [69–73], the hybridization of this machine learning method and ODFP can potentially help yield good predictive forecasts of pile capacity.Before the optimization process can commence, a set of parameters including the objective function (CF), population size (PS), search space boundaries (LB and UB), the maximum number of function calls (MNFC), and probability of population jumping (Jr) is needed to be specified. The Jr determines how frequently the opposition-based population jump is performed to enhance the optimization performance [74]. Herein, based on recommendation of previous studies [40, 42, 53, 75] and several trial runs, the parameters of the ODFO are set as follows: PS = 20, LB = [1, 0.01], UB = [1000, 10], MNFC = 1000, and Jr = 0.2.When those parameters are set, the optimization process can commence to gradually guide an initial population to explore and exploit the search space and identify better candidate solutions which are sets of the LSSVR model hyperparameters. Similar to the original DFP metaheuristic, the ODFP also performs the Lévy flight-based global search and the DE’s mutation-crossover-based local search [40, 76]. At the end of each generation, an opposition-based population jumping operation [53] is probabilistically performed to improve the population quality via OBL.Since the population member directly affects the LSSVR model training process and the quality of the final pile bearing capacity prediction, the objective cost function must take into account the predictive performance of the LSSVR model trained with the collected dataset. This study employs a 10-fold cross validation to compute the objective cost function used by ODFP. Herein, the dataset is separated into 10 mutually exclusive data folds. The training and validation process of LSSVR is thereby performed 10 times, within which one fold serves as validating data and the other 9 folds are used for model construction. Thus, the objective cost function of ODFP is the average root mean square error (RMSE) computed from the 10-fold cross-validation process. This cost function is described as follows:(14)CF=∑k=1KRMSEkK,where RMSEk is calculated by comparing the actual and predicted pile bearing capacity values in the kth fold.The RMSE index is given by(15)RMSE=∑i=1NKYA−YP2NK,where YA and YP denote the actual and the predicted value of pile bearing capacity. NK represents the number of data instances in each data fold.When the operational flow of ODFP is specified, the overall ODFP-optimized LSSVR (denoted as ODFP-LSSVR) used for predicting pile bearing capacity can be fully displayed in Figure6. It is noted that the model has been developed by the authors in the Visual Studio coding environment with Visual C# .NET 4.7.2. The ODFP plays a crucial role in determining an appropriate set of γ and σ which features a fair balance between modeling generalization and prediction accuracy. Subsequently, this study employs a random data sampling process to obtain a robust assessment of the ODFP-LSSVR performance. Herein, the collected dataset is randomly separated into a training set (90%) and a testing set (10%). The first set is used for model construction; the second set is reserved for model testing. Based on these two sets, the LSSVR model training and prediction can be carried out. These operations require the best solution (Xbest) which stores the LSSVR model hyperparameters; this best solution has been identified by the ODFP optimization process.Figure 6
Overall structure of the proposed ODFP-LSSVR-based pile capacity prediction model.In addition, the employed variables should be standardized to avoid the circumstance in which a variable having large magnitude dominates other variables having small magnitudes. This study utilizes theZ-score equation for data normalization; this equation is stated as follows:(16)XN=XO−mXsX,where XN and XO denote the normalized and the original variables, respectively. mX and sX are the mean value and the standard deviation of the explanatory variable, respectively.The LSSVR is used to establish a mapping function that can derive the axial pile bearing capacity based on a set of explanatory variables including pile diameter (X1), thickness of the first soil layer (X2), thickness of the second soil layer (X3), thickness of the third soil layer (X4), elevation of the natural ground (X5), top of pile elevation (X6), elevation of the extra segment of pile top (X7), depth of the pile tip (X8), mean value of SPT blow count along the pile shaft (X9), and mean value of SPT blow count at the pile tip (X10). When the prediction phase of the ODFP-LSSVR model is finished, the predicted values of pile bearing capacity for novel input patterns can be obtained and documented.


## 4. Experimental Results and Discussion

### 4.1. Prediction Results of the Proposed ODFP-LSSVR Model

This section of the article reports the experimental results of the newly developed ODFP-LSSVR model used for pile bearing capacity estimation. To train and verify the model capability, the original dataset consisting of 472 samples has been divided into a training set (90%) and a testing set (10%). The computing process used to obtain the experimental results is implemented with ASUS FX705GE-EW165T (Core i7 8750H and 8 GB RAM).In addition, before the model training and prediction phases, it is beneficial to preliminarily inspect the predictive power of each explanatory variable in the collected dataset. This study relies on the ReliefF [77] method to appraise the importance of the employed predictors. This feature importance assessment is selected in this study because it has a robust mathematical background and has the ability to detect conditional dependencies between explanatory variables. The ReliefF method can provide a unified view on the variable estimation in the regression analysis problem [78]. The preliminary factor assessment outcome is presented in Figure 7. As can be seen from this figure, the thickness of the second soil layer (X3), depth of the pile tip (X8), mean value of SPT blow count along the pile shaft (X9), and mean value of SPT blow count at the pile tip (X10) are highly important factors associated with a high value of ReliefF weights. The thickness of the first soil layer (X2), elevation of the natural ground (X5), top of pile elevation (X6), and elevation of the extra segment of pile top (X7) have fairly large ReliefF weights. The pile diameter (X1) and thickness of the third soil layer (X4) have relatively small feature weights. Nevertheless, all of the weights are not null. Therefore, the 10 available variables should be taken into account to predict the bearing capacity of piles.Figure 7
Feature importance computed by ReliefF.Moreover, to accurately assess the predictive capability of the ODFP-LSSVR model, the indices of RMSE, mean absolute percentage error (MAPE), the mean absolute error (MAE), and the coefficient of determination (R2) are utilized. RMSE has been described in the previous section. MAPE, MAE, and R2 are given by [79–81](17)MAPE=100%N∑t=1NYA,i−YP,iYA,i,MAE=1N∑t=1NYA,i−YP,i,R2=SSyy−SSESSyy,where YA,i and YP,i are the actual and the predicted bearing capacity, respectively. N is the number of data instances.The indices of SSyy and SEE are given by(18)SSyy=∑i=1NYA,i−YA,m2,SSE=∑i=1NYA,i−YP,i2,where YA,m is the mean value of the actual bearing capacity.As mentioned earlier, the ODFP with a population size of 20 and a maximum number of function calls = 1000 is used to optimize the LSSVR performance. The optimization process of the ODFP with the cost function and variable boundaries specified in the previous section is demonstrated in Figure8. The best found solution is the regularization coefficient γ = 26.62 and the kernel function parameter σ = 1.35. The prediction results for testing data instances yielded by the ODFP-LSSVR are reported in Table 2. Figure 9 illustrates the actual vs. predicted bearing capacity of piles for the results shown in Table 2. Moreover, Figure 10 demonstrates the histogram of residual obtained from the proposed method. As can be observed from this figure, the average value of the residual yielded by the ODFP-LSSVR is close to 0 and the standard deviation of the residual is 77.19.Figure 8
The ODFP optimization process.Table 2
Prediction results of a typical run.
Data sampleX1X2X3X4X5X6X7X8X9X10ActualYPredictedY14003.458.000.102.953.603.1014.5011.557.461344.001227.3124003.555.360.003.253.413.2512.168.916.71930.001070.4533003.405.290.003.403.543.4512.098.696.76661.60623.9143003.405.240.003.403.463.4212.048.646.75559.80588.3454003.407.400.003.403.613.4114.2010.807.301088.801057.0564004.358.001.022.053.472.0515.4213.377.641473.001298.7574003.857.300.002.953.683.5814.1011.157.081440.001411.2983003.405.200.003.403.453.4512.008.606.73559.80600.2194003.407.330.003.403.553.4214.1310.737.281094.251063.49103003.405.200.003.403.423.4212.008.606.73610.70598.24114004.358.001.082.053.542.0615.4813.437.671395.001268.95123003.405.200.003.403.433.4312.008.606.73610.70599.40134003.407.350.003.403.573.4214.1510.757.291119.701051.84144004.358.001.202.053.622.0215.6013.557.741119.701121.77153003.405.300.003.403.543.4412.108.706.76661.60622.50164003.458.000.202.953.502.9014.6011.657.521017.901086.64174003.407.300.003.403.503.4014.1010.707.28900.001122.16183003.405.220.003.403.443.4212.028.626.74610.70593.56194004.358.001.052.053.502.0515.4513.407.661344.001299.53204004.358.000.982.053.482.1015.3813.337.621224.801334.41213003.405.200.003.403.423.4212.008.606.73610.70598.24223003.405.230.003.403.443.4112.038.636.74610.70590.72233003.405.300.003.403.523.4212.108.706.76508.90599.40243003.405.180.003.403.383.4011.988.586.73559.80593.72254004.358.001.052.053.552.1015.4513.407.661224.801292.04264003.407.400.003.403.613.4114.2010.807.301152.001057.05274004.457.210.002.353.412.4014.0111.666.831318.001365.13284003.458.000.152.953.643.0914.5511.607.491344.001212.29293003.405.200.003.403.453.4512.008.606.73610.70600.21303003.405.250.003.403.483.4312.058.656.75508.90588.13314004.358.000.802.053.452.2515.2013.157.521392.001383.83324003.407.240.003.403.443.4014.0410.647.261395.001121.48334004.358.001.012.053.462.0515.4113.367.641473.001294.71344004.358.001.062.053.562.1015.4613.417.661224.801277.92353003.405.250.003.403.483.4312.058.656.75610.70588.13364003.857.600.002.953.673.2714.4011.457.151425.001410.53373003.405.220.003.403.453.4312.028.626.74559.80593.89384004.358.000.942.053.492.1515.3413.297.601395.001362.40393003.405.220.003.403.443.4212.028.626.74610.70593.56404005.406.300.002.153.521.0613.1014.705.501056.00968.29414003.407.260.003.403.463.4014.0610.667.271152.001143.42424004.358.000.902.053.382.0815.3013.257.581395.001319.21434003.458.000.062.953.412.9514.4611.517.431240.001248.78444004.358.001.052.053.502.0515.4513.407.661152.001299.45453003.405.250.003.403.493.4412.058.656.75661.60590.77464004.358.000.952.053.412.0615.3513.307.601323.201283.45474004.358.000.902.053.402.1015.3013.257.581395.001322.17Figure 9
Actual vs. predicted response variables.Figure 10
Residual distribution of the model construction phase.Moreover, to obtain a reliable and accurate assessment of the ODFP-LSSVR-based pile bearing capacity estimation, the model training and testing phase in this study has been repeated 20 times. Each time, 10% of the collected data is drawn to form a testing set and the rest of the dataset is used for model construction. The average training performance of the ODFP-LSSVR is as follows: RMSE = 76.99, MAPE = 6.12%, MAE = 57.22, andR2 = 0.95. The average testing performance of the ODFP-LSSVR is as follows: RMSE = 92.19, MAPE = 7.38%, MAE = 68.97, and R2 = 0.93. The training R2 = 0.95 and testing R2 = 0.93 indicate that using the ODFP-LSSVR model, 95% of the training data variance and 93% of the testing data variance can be explained by the explanatory variables. The value of R2 of the training phase is slightly higher than that of the testing phase. This fact is understandable since predicting unseen data instances in the testing set is significantly more challenging than predicting data instances in the training set. Moreover, since the training R2 = 0.95 and testing R2 = 0.93 are both high (>0.9) and relatively close to each other, it can be seen that the ODFP-LSSVR has not suffered from either overfitting or underfitting issues.


### 4.2. Experimental Results and Discussion

In this section, to confirm the predictive capability of the newly developed ODFP-LSSVR used for pile bearing capacity prediction, its performance is compared to other capable machine learning-based regression analysis models including the backpropagation artificial neural network regression (BPANNR) [82, 83], the regression tree (RTR) [84, 85], and the relevance vector machine regression (RVMR) [86, 87]. These benchmark methods are very capable models used for function approximation, and they have been used for pile bearing capacity estimations [22, 88, 89]. The BPANNR is developed by the authors in the Microsoft Visual Studio 2019 environment and using C# .NET programming language; this model has been trained by the mini-batch mode. The RTR model is built via MATLAB’s Statistics and Machine Learning Toolbox [90]. In addition, the RVMR is constructed via built-in functions provided in [91].The experimental results of ODFP-LSSVR and other benchmark approaches are summarized in Table3. It can be observed from the results that the performance of the newly constructed model (RMSE = 92.19, MAPE = 7.38%, MAE = 68.97, and R2 = 0.93) is better than that of the employed benchmark models. Based on RMSE as the principal indicator, RMMR is the second best approach (RMSE = 102.59, MAPE = 7.89%, MAE = 73.02, and R2 = 0.91), followed by BPANNR (RMSE = 103.09, MAPE = 8.56%, MAE = 80.45, and R2 = 0.90) and RTR (RMSE = 105.52, MAPE = 7.89%, MAE = 73.92, and R2 = 0.90). Notably, ODFP-LSSVR has outperformed all of the benchmark models in terms of all performance measurement metrics. In addition, the boxplots of the model performances obtained from the repetitive model evaluation with 20 runs are displayed in Figure 11.Table 3
Prediction result comparison.
PhaseIndicesODFP-LSSVRBPANNRRTRRVMRMeanStd.MeanStd.MeanStd.MeanStd.TrainingRMSE76.991.2696.956.6272.131.8275.621.38MAPE6.120.117.880.395.570.165.970.09MAE57.221.1575.294.0951.611.4155.551.00R20.950.000.920.010.960.000.950.00TestingRMSE92.1911.39103.0915.33105.5220.32102.5914.59MAPE7.380.828.561.137.891.277.891.29MAE68.979.2880.4511.5473.9211.9173.029.34R20.930.020.900.030.900.040.910.03Figure 11
Boxplots of prediction model performances.To better facilitate the comparison of the model performances, regression error characteristic (REC) curves [92] obtained from the testing phases of all of the models are plotted in Figure 12. A REC curve basically plots the absolute residual on the x-axis. The y-axis shows the percentage of data samples having prediction deviation smaller than the corresponding absolute residual on the x-axis. Thus, a REC curve is capable of estimating the cumulative distribution function of the error. The REC curves of ODFP-LSSVR and other benchmark models are shown in Figure 12. Similar to the concept of receiver operating characteristic curves used in pattern classification, the relative performances of multiple regression analysis models can be assessed by inspecting the relative position of the REC curves. Generally, a larger area under the curve indicates a better regression analysis model. In addition, it is noted that each pile bearing capacity prediction model yields residual with different maximum values. Therefore, to achieve a fair comparison, the residual range must be standardized by identifying the largest absolute residual value produced by all of the employed models. This largest absolute residual value is used as the common value to set the range of the x-axis. Accordingly, the area under the curve (AUC) and the relative AUC values of the models can be computed and are reported in Table 4. It can be seen that the relative AUC of the ODFP-LSSVR (0.87) is higher than that of the benchmark models. Therefore, based on analyzing the cumulative distribution function of the error, we are able to conclude that the predictive power of ODFP-LSSVR is higher than that of BPANNR, RTR, and RVMR.Figure 12
Regression error characteristic curves.Table 4
Model performance evaluation based on regression error characteristic curves.
ModelsODFP-LSSVRBPANNRRTRRVMRAUC469.16457.74464.38465.22Relative AUC0.870.850.860.86To further confirm the statistical difference of each pair of pile bearing capacity prediction models, the Wilcoxon signed-rank test [93] is used in this study. This is a nonparametric hypothesis test widely used in machine learning model comparison [94]. The significance level of the test (p value) is set to be 0.05. Accordingly, a p value <0.05 indicates the rejection of the null hypothesis that performances of the two of the pile bearing capacity prediction models are statistically indifferent. The Wilcoxon signed-rank test results are reported in Table 5. With p values <0.05, it can be stated that ODFP-LSSVR is superior to other benchmark models.Table 5
Wilcoxon signed-rank test results.
p valueODFP-LSSVRBPANNRRTRRVMRODFP-LSSVR✕0.015240.0111290.025094BPANNR0.01524✕0.455270.85192RTR0.0111290.45527✕0.62745RVMR0.0250940.851920.62745✕Nevertheless, one limitation of the proposed method is that it relies on the ODFP-based optimization of LSSVR’s hyperparameters. Accordingly, the LSSVR model training and prediction phases must be carried out to compute the ODFP’s objective function. Therefore, the computational expense of this integrated framework can be significant for modeling large-sized datasets of pile bearing capacity. In addition, the current model relies on ten explanatory variables (pile diameter, depths of the soil layers, elevation of the natural ground, top of pile elevation, elevation of the extra segment of pile top, depth of the pile tip, mean value of SPT blow count along the pile shaft, and mean value of SPT blow count at the pile tip). With the complexity of the problem of interest, other explanatory variables should be taken into account to enhance the applicability of the proposed machine learning method. In such cases, the integration of advanced feature selection methods [33, 52, 95, 96] into the data-driven pile bearing capacity prediction can help select the most informative set of explanatory variables.


## 4.1. Prediction Results of the Proposed ODFP-LSSVR Model

This section of the article reports the experimental results of the newly developed ODFP-LSSVR model used for pile bearing capacity estimation. To train and verify the model capability, the original dataset consisting of 472 samples has been divided into a training set (90%) and a testing set (10%). The computing process used to obtain the experimental results is implemented with ASUS FX705GE-EW165T (Core i7 8750H and 8 GB RAM).In addition, before the model training and prediction phases, it is beneficial to preliminarily inspect the predictive power of each explanatory variable in the collected dataset. This study relies on the ReliefF [77] method to appraise the importance of the employed predictors. This feature importance assessment is selected in this study because it has a robust mathematical background and has the ability to detect conditional dependencies between explanatory variables. The ReliefF method can provide a unified view on the variable estimation in the regression analysis problem [78]. The preliminary factor assessment outcome is presented in Figure 7. As can be seen from this figure, the thickness of the second soil layer (X3), depth of the pile tip (X8), mean value of SPT blow count along the pile shaft (X9), and mean value of SPT blow count at the pile tip (X10) are highly important factors associated with a high value of ReliefF weights. The thickness of the first soil layer (X2), elevation of the natural ground (X5), top of pile elevation (X6), and elevation of the extra segment of pile top (X7) have fairly large ReliefF weights. The pile diameter (X1) and thickness of the third soil layer (X4) have relatively small feature weights. Nevertheless, all of the weights are not null. Therefore, the 10 available variables should be taken into account to predict the bearing capacity of piles.Figure 7
Feature importance computed by ReliefF.Moreover, to accurately assess the predictive capability of the ODFP-LSSVR model, the indices of RMSE, mean absolute percentage error (MAPE), the mean absolute error (MAE), and the coefficient of determination (R2) are utilized. RMSE has been described in the previous section. MAPE, MAE, and R2 are given by [79–81](17)MAPE=100%N∑t=1NYA,i−YP,iYA,i,MAE=1N∑t=1NYA,i−YP,i,R2=SSyy−SSESSyy,where YA,i and YP,i are the actual and the predicted bearing capacity, respectively. N is the number of data instances.The indices of SSyy and SEE are given by(18)SSyy=∑i=1NYA,i−YA,m2,SSE=∑i=1NYA,i−YP,i2,where YA,m is the mean value of the actual bearing capacity.As mentioned earlier, the ODFP with a population size of 20 and a maximum number of function calls = 1000 is used to optimize the LSSVR performance. The optimization process of the ODFP with the cost function and variable boundaries specified in the previous section is demonstrated in Figure8. The best found solution is the regularization coefficient γ = 26.62 and the kernel function parameter σ = 1.35. The prediction results for testing data instances yielded by the ODFP-LSSVR are reported in Table 2. Figure 9 illustrates the actual vs. predicted bearing capacity of piles for the results shown in Table 2. Moreover, Figure 10 demonstrates the histogram of residual obtained from the proposed method. As can be observed from this figure, the average value of the residual yielded by the ODFP-LSSVR is close to 0 and the standard deviation of the residual is 77.19.Figure 8
The ODFP optimization process.Table 2
Prediction results of a typical run.
Data sampleX1X2X3X4X5X6X7X8X9X10ActualYPredictedY14003.458.000.102.953.603.1014.5011.557.461344.001227.3124003.555.360.003.253.413.2512.168.916.71930.001070.4533003.405.290.003.403.543.4512.098.696.76661.60623.9143003.405.240.003.403.463.4212.048.646.75559.80588.3454003.407.400.003.403.613.4114.2010.807.301088.801057.0564004.358.001.022.053.472.0515.4213.377.641473.001298.7574003.857.300.002.953.683.5814.1011.157.081440.001411.2983003.405.200.003.403.453.4512.008.606.73559.80600.2194003.407.330.003.403.553.4214.1310.737.281094.251063.49103003.405.200.003.403.423.4212.008.606.73610.70598.24114004.358.001.082.053.542.0615.4813.437.671395.001268.95123003.405.200.003.403.433.4312.008.606.73610.70599.40134003.407.350.003.403.573.4214.1510.757.291119.701051.84144004.358.001.202.053.622.0215.6013.557.741119.701121.77153003.405.300.003.403.543.4412.108.706.76661.60622.50164003.458.000.202.953.502.9014.6011.657.521017.901086.64174003.407.300.003.403.503.4014.1010.707.28900.001122.16183003.405.220.003.403.443.4212.028.626.74610.70593.56194004.358.001.052.053.502.0515.4513.407.661344.001299.53204004.358.000.982.053.482.1015.3813.337.621224.801334.41213003.405.200.003.403.423.4212.008.606.73610.70598.24223003.405.230.003.403.443.4112.038.636.74610.70590.72233003.405.300.003.403.523.4212.108.706.76508.90599.40243003.405.180.003.403.383.4011.988.586.73559.80593.72254004.358.001.052.053.552.1015.4513.407.661224.801292.04264003.407.400.003.403.613.4114.2010.807.301152.001057.05274004.457.210.002.353.412.4014.0111.666.831318.001365.13284003.458.000.152.953.643.0914.5511.607.491344.001212.29293003.405.200.003.403.453.4512.008.606.73610.70600.21303003.405.250.003.403.483.4312.058.656.75508.90588.13314004.358.000.802.053.452.2515.2013.157.521392.001383.83324003.407.240.003.403.443.4014.0410.647.261395.001121.48334004.358.001.012.053.462.0515.4113.367.641473.001294.71344004.358.001.062.053.562.1015.4613.417.661224.801277.92353003.405.250.003.403.483.4312.058.656.75610.70588.13364003.857.600.002.953.673.2714.4011.457.151425.001410.53373003.405.220.003.403.453.4312.028.626.74559.80593.89384004.358.000.942.053.492.1515.3413.297.601395.001362.40393003.405.220.003.403.443.4212.028.626.74610.70593.56404005.406.300.002.153.521.0613.1014.705.501056.00968.29414003.407.260.003.403.463.4014.0610.667.271152.001143.42424004.358.000.902.053.382.0815.3013.257.581395.001319.21434003.458.000.062.953.412.9514.4611.517.431240.001248.78444004.358.001.052.053.502.0515.4513.407.661152.001299.45453003.405.250.003.403.493.4412.058.656.75661.60590.77464004.358.000.952.053.412.0615.3513.307.601323.201283.45474004.358.000.902.053.402.1015.3013.257.581395.001322.17Figure 9
Actual vs. predicted response variables.Figure 10
Residual distribution of the model construction phase.Moreover, to obtain a reliable and accurate assessment of the ODFP-LSSVR-based pile bearing capacity estimation, the model training and testing phase in this study has been repeated 20 times. Each time, 10% of the collected data is drawn to form a testing set and the rest of the dataset is used for model construction. The average training performance of the ODFP-LSSVR is as follows: RMSE = 76.99, MAPE = 6.12%, MAE = 57.22, andR2 = 0.95. The average testing performance of the ODFP-LSSVR is as follows: RMSE = 92.19, MAPE = 7.38%, MAE = 68.97, and R2 = 0.93. The training R2 = 0.95 and testing R2 = 0.93 indicate that using the ODFP-LSSVR model, 95% of the training data variance and 93% of the testing data variance can be explained by the explanatory variables. The value of R2 of the training phase is slightly higher than that of the testing phase. This fact is understandable since predicting unseen data instances in the testing set is significantly more challenging than predicting data instances in the training set. Moreover, since the training R2 = 0.95 and testing R2 = 0.93 are both high (>0.9) and relatively close to each other, it can be seen that the ODFP-LSSVR has not suffered from either overfitting or underfitting issues.


## 4.2. Experimental Results and Discussion

In this section, to confirm the predictive capability of the newly developed ODFP-LSSVR used for pile bearing capacity prediction, its performance is compared to other capable machine learning-based regression analysis models including the backpropagation artificial neural network regression (BPANNR) [82, 83], the regression tree (RTR) [84, 85], and the relevance vector machine regression (RVMR) [86, 87]. These benchmark methods are very capable models used for function approximation, and they have been used for pile bearing capacity estimations [22, 88, 89]. The BPANNR is developed by the authors in the Microsoft Visual Studio 2019 environment and using C# .NET programming language; this model has been trained by the mini-batch mode. The RTR model is built via MATLAB’s Statistics and Machine Learning Toolbox [90]. In addition, the RVMR is constructed via built-in functions provided in [91].The experimental results of ODFP-LSSVR and other benchmark approaches are summarized in Table3. It can be observed from the results that the performance of the newly constructed model (RMSE = 92.19, MAPE = 7.38%, MAE = 68.97, and R2 = 0.93) is better than that of the employed benchmark models. Based on RMSE as the principal indicator, RMMR is the second best approach (RMSE = 102.59, MAPE = 7.89%, MAE = 73.02, and R2 = 0.91), followed by BPANNR (RMSE = 103.09, MAPE = 8.56%, MAE = 80.45, and R2 = 0.90) and RTR (RMSE = 105.52, MAPE = 7.89%, MAE = 73.92, and R2 = 0.90). Notably, ODFP-LSSVR has outperformed all of the benchmark models in terms of all performance measurement metrics. In addition, the boxplots of the model performances obtained from the repetitive model evaluation with 20 runs are displayed in Figure 11.Table 3
Prediction result comparison.
PhaseIndicesODFP-LSSVRBPANNRRTRRVMRMeanStd.MeanStd.MeanStd.MeanStd.TrainingRMSE76.991.2696.956.6272.131.8275.621.38MAPE6.120.117.880.395.570.165.970.09MAE57.221.1575.294.0951.611.4155.551.00R20.950.000.920.010.960.000.950.00TestingRMSE92.1911.39103.0915.33105.5220.32102.5914.59MAPE7.380.828.561.137.891.277.891.29MAE68.979.2880.4511.5473.9211.9173.029.34R20.930.020.900.030.900.040.910.03Figure 11
Boxplots of prediction model performances.To better facilitate the comparison of the model performances, regression error characteristic (REC) curves [92] obtained from the testing phases of all of the models are plotted in Figure 12. A REC curve basically plots the absolute residual on the x-axis. The y-axis shows the percentage of data samples having prediction deviation smaller than the corresponding absolute residual on the x-axis. Thus, a REC curve is capable of estimating the cumulative distribution function of the error. The REC curves of ODFP-LSSVR and other benchmark models are shown in Figure 12. Similar to the concept of receiver operating characteristic curves used in pattern classification, the relative performances of multiple regression analysis models can be assessed by inspecting the relative position of the REC curves. Generally, a larger area under the curve indicates a better regression analysis model. In addition, it is noted that each pile bearing capacity prediction model yields residual with different maximum values. Therefore, to achieve a fair comparison, the residual range must be standardized by identifying the largest absolute residual value produced by all of the employed models. This largest absolute residual value is used as the common value to set the range of the x-axis. Accordingly, the area under the curve (AUC) and the relative AUC values of the models can be computed and are reported in Table 4. It can be seen that the relative AUC of the ODFP-LSSVR (0.87) is higher than that of the benchmark models. Therefore, based on analyzing the cumulative distribution function of the error, we are able to conclude that the predictive power of ODFP-LSSVR is higher than that of BPANNR, RTR, and RVMR.Figure 12
Regression error characteristic curves.Table 4
Model performance evaluation based on regression error characteristic curves.
ModelsODFP-LSSVRBPANNRRTRRVMRAUC469.16457.74464.38465.22Relative AUC0.870.850.860.86To further confirm the statistical difference of each pair of pile bearing capacity prediction models, the Wilcoxon signed-rank test [93] is used in this study. This is a nonparametric hypothesis test widely used in machine learning model comparison [94]. The significance level of the test (p value) is set to be 0.05. Accordingly, a p value <0.05 indicates the rejection of the null hypothesis that performances of the two of the pile bearing capacity prediction models are statistically indifferent. The Wilcoxon signed-rank test results are reported in Table 5. With p values <0.05, it can be stated that ODFP-LSSVR is superior to other benchmark models.Table 5
Wilcoxon signed-rank test results.
p valueODFP-LSSVRBPANNRRTRRVMRODFP-LSSVR✕0.015240.0111290.025094BPANNR0.01524✕0.455270.85192RTR0.0111290.45527✕0.62745RVMR0.0250940.851920.62745✕Nevertheless, one limitation of the proposed method is that it relies on the ODFP-based optimization of LSSVR’s hyperparameters. Accordingly, the LSSVR model training and prediction phases must be carried out to compute the ODFP’s objective function. Therefore, the computational expense of this integrated framework can be significant for modeling large-sized datasets of pile bearing capacity. In addition, the current model relies on ten explanatory variables (pile diameter, depths of the soil layers, elevation of the natural ground, top of pile elevation, elevation of the extra segment of pile top, depth of the pile tip, mean value of SPT blow count along the pile shaft, and mean value of SPT blow count at the pile tip). With the complexity of the problem of interest, other explanatory variables should be taken into account to enhance the applicability of the proposed machine learning method. In such cases, the integration of advanced feature selection methods [33, 52, 95, 96] into the data-driven pile bearing capacity prediction can help select the most informative set of explanatory variables.


## 5. Concluding Remarks

Pile bearing capacity is a crucial parameter needed to be estimated during the design phase of pile foundation engineering projects. This study has proposed a data-driven tool for predicting the pile bearing capacity using a hybrid metaheuristic-machine learning approach. The LSSVR machine learning approach is used to capture and generalize a nonlinear function that derives the estimated value of pile bearing capacity given a set of explanatory variables. In addition, the ODFP metaheuristic approach is used to optimize the training process of LSSVR. The ODFP algorithm itself is an integration of the DFP and OBL. When the ODFP-based optimization phase is accomplished, the most suitable structure of the LSSVR model can be identified and be used for estimating values of pile bearing capacity for new projects. A dataset consisting of 472 test records and 10 explanatory variables including pile diameter, thickness of the first soil layer, thickness of the second soil layer, thickness of the third soil layer, elevation of the natural ground, top of pile elevation, elevation of the extra segment of pile top, depth of the pile tip, mean value of SPT blow count along the pile shaft, and mean value of SPT blow count at the pile tip is used to train and test the proposed ODFP-LSSVR model.Experimental results supported by the REC curve analysis and the Wilcoxon signed-rank test point out that ODFP-LSSVR is best suited for modeling the dataset at hand. In addition, future extensions of the study may include the following:(i)
The investigation of other state-of-the-art machine learning and metaheuristic methods in optimizing machine learning-based pile bearing capacity prediction model(ii)
The integration of other advanced feature selection algorithms into the model structure(iii)
The investigation on the use of robust regression to enhance the generalization of the machine learning models used for pile bearing capacity estimation(iv)
The collection of more data samples to enhance the generalization of the machine learning-based pile bearing capacity estimation model



---

*Source: 7183700-2022-04-14.xml*