# Moisture Content Quantization of Masson Pine Seedling Leaf Based on Stacked Autoencoder with Near-Infrared Spectroscopy

Authors: Chao Ni; Yun Zhang; Dongyi Wang
Journal: Journal of Electrical and Computer Engineering
 (2018)

---

## Abstract

Masson pine is widely planted in southern China, and moisture content of the pine seedling leaves is an important index for evaluating the vigor of seedlings. For precisely predicting leaf moisture content, near-infrared spectroscopy analysis is applied in the experiment, which is a cost-effective, high-speed, and noninvasive material content prediction tool. To further improve the spectroscopy analysis accuracy, in this study, a new analysis model is proposed which integrates a stacked autoencoder for extracting hierarchical output-related features layer by layer and a support vector regression model to leverage these features for precisely predicting moisture contents. Compared with traditional spectroscopy analysis method like partial least squares regression and basic support vector regression, the proposed model shows great superiority for leaf moisture content prediction, withR2 value 0.9946 and root-mean squared error (RMSE) value 0.1636 in calibration set and R2 value 0.9621 and RMSE 0.4249 in prediction set.


---

## Body

## 1. Introduction

Masson pine is an important species for forestation in southern China due to its broad distribution. It has been widely applied in pulp and building materials, petroleum extraction, and forestry chemicals [1] because of its desirable features including fast growth, high yield, variable light tolerance, and drought resistance [2]. To ensure the quality of masson pine seedlings, the moisture content of seedling leaves is an important evaluation index and should be assessed before plantation. Physiologically, leaf moisture content plays an important role in photosynthesis, material transport, and maintaining leaf morphological and physiological functions [3]. Traditional moisture measurements are almost chemical-based methods, which are complex, time-consuming, and laborious [4, 5]. As a result, it is urgent to establish a fast, nondestructive, and accurate measurement method for moisture content prediction.The near-infrared (NIR) spectroscopy technique has been widely used for material content prediction, such as moisture content [6], nitrogen content [7], and chlorophyll content [8] of plants. To date, many researchers have constructed different NIR spectral calibration models to detect the moisture content of different plants such as sunflower [9], populous tree [10], and eucalyptus tree [11]. Although there are many successful spectroscopy applications, there is no research reported for the moisture content prediction of masson pine seedling leaves.For spectroscopy data analysis, there are many common used calibration models such as partial least squares regression (PLSR) [12], multiple linear regression (MLR) [13], support vector machines for regression (SVR) [14–16], artificial neural networks (ANNs) [17], and so on. However, the prediction results from these models are hopeful to be improved due to the limitations of the models. PLSR and MLR are widely used methods in the spectroscopy area. Both of them are linear models, which means the model can hardly describe the nonlinear relationship between the input spectroscopy data and output material content value. SVR, a nonlinear model, is suitable for small dataset analysis. It usually has strong generalization ability and self-learning capacity. The performance of SVR model usually depends on the quality input features, and determined representative data features are the premise for training a good SVR model [16]. ANNs offer a simple solution to model complex nonlinear data relationship [18]. The multiple layers in the ANN model can map the data input to the high-level feature representations. With the increasing number of hidden layers, the imbalance between the trainable network parameters and the limited training samples hinders the network training process in practice [19]. Stacked autoencoder (SAE) [20] is a variant of the ANN model, which usually is composed of an encoder and decoder. The encoder can map the input into few high-dimensional data features, and the decoder is expected to reconstruct the raw data. SAE can be trained without supervision, and the high dimensional data features from the encoder have been successfully applied in different tasks [21–24].In this paper, to combine the advantages of different existing data analysis methods, a new model is proposed for predicting the moisture content in masson pine seedling leaves. In the model, a SAE is pretrained layer by layer without supervision to get the high-level data features. Then, based on the properties of the ANN model, the high-level data can be fine-tuned in a supervised way. Once the high-level data features are determined, the SVR model is established to describe the relationship between the high-level data features and the moisture content. Compared with other traditional spectroscopy analysis methods, the proposed model shows better performance.


## 2. Materials and Methods

### 2.1. Materials

A total of 100 annual masson pine seedling samples were obtained from forest farm in Huangping, Guizhou Province, China. Seventy-five of them were used to establish a calibration model, and the remaining 25 were used for prediction. Before measuring the spectra and moisture content, the leaves were cleaned to remove the impurities such as soil and sand, and then the surface of the leaves were naturally dried to prepare for the later experiment.


### 2.2. Spectra Acquisition and Moisture Quantitative Analysis

All NIR spectra of masson pine seedling leaves were measured using an MPA Fourier-Transform Near-Infrared spectrometer (Bruker Optics, Inc., Germany), equipped with an PbS detector and controlled by OPUS Analyst version 7.0 using a spectral range from 4,000 to 12,493 cm−1 in the reflectance mode. Measurements were conducted with the resolution of 4 cm−1, ensuring an adequate signal-to-noise ratio. There are 2203 wavelength variables in the spectrum. The experiment was carried out at approximately 24°C. The spectral reflectance of the top, middle, and bottom areas of the sample were scanned two times each. The final value of the sample spectral reflectance was obtained by calculating the average of the six scans. From each sample, six spectra were collected and averaged for further analysis. Figure 1 shows the raw absorptance spectra of masson pine seedling leaves.Figure 1
Raw absorptance spectra of masson pine seedling leaves.Total moisture content of masson pine seedling leaves was determined using the HB43-S Halogen Moisture analyzer (Mettler Toledo, Inc., Switzerland). The instrument is working based on the thermogravimetric principle, i.e., the moisture is determined from the weight loss of a sample dried by heating. After putting a sample in the sampling chamber, the temperature was immediately increased and maintained at 125°C. The halogen lamp of the instrument heated the sample until the sample stopped losing mass; then the moisture content of the samples is automatically given by the instrument. Normally, the measurements were completed in a few minutes. The reference moisture content value for the samples was 64.97% ± 2.21%, with a minimum of 59.13% and a maximum of 70.74%.


### 2.3. Spectra Pretreatment

Two preprocessing transformations were applied as a standard preparation for the masson pine seedling absorbance curves. Firstly, a Savitzky–Golay (S-G) smoothing with a second-degree polynomial was used for denoising, and the first-order numerical derivative was made to correct the baseline drift. The window width of S-G smoothing was set as 17. Secondly, the spectral matrix was converted into the number between 0 and 1 by vector normalization, which was used to reduce the orders of magnitude difference among the data of different dimensions.


### 2.4. The Proposed Method

#### 2.4.1. Autoencoder

The basic structure of autoencoder is a kind of unsupervised ANN with one hidden layer, and it consists of an input layer, a hidden layer, and an output layer as Figure2. The aim of autoencoder is to reduce the data dimension and to map the input data into high dimensional data features.Figure 2
The typical structure of an autoencoder.Define the input asx=x1,x2,…,xd′, where d is the dimension of the inputs. The encoder maps x into the hidden layer h=h1,h2,…,hdh′ by the function f as follows:(1)h=fx=sfWx+b,where dh is the dimension of the hidden layer variable vector h, W is a dh∗d weight matrix, b is the bias vector, and sf is the nonlinear activation function, which can be chosen as the sigmoid function or other functions such as the tanh function and the rectified linear unit function. Then, the hidden representation vector h is mapped to the output layer x˜ by the function f˜.(2)x˜=f˜h=sf˜W˜h+b˜,where W˜ is a d∗dh weight matrix, b˜ is the bias vector for the output layer, and sf˜ is the nonlinear activation function of decoder. The aim of autoencoder is to search the parameter set θ=W,W˜,b,b˜ to satisfy the equation x˜=f˜h=f˜fx≈x. Define the training input as X=x1,x2,…,xN, where N is the number of the training samples and xi=xi1,xi2,…,xid is the vector data of ith training sample. The loss function is defined as follows:(3)JW,W˜,b,b˜=12N∑i=1Nxi−x˜i2=12N∑i=1N∑j=1dxij−x˜ij2.Then, the parameter setθ=W,W˜,b,b˜ is updated with the gradient descent algorithm.In practice, multiple basic autoencoder structures are usually stacked together, and a new network structure is constructed, named SAE, for better dimensional reduction and feature extraction performance. The extracted data feature can be further used for different tasks [25, 26].


#### 2.4.2. Support Vector Machine Regression

Support vector machine (SVM) is a powerful and robust method based on the principle of structural risk minimization, which has an advantage of computational efficiency for the data sets that have many more variables than observations. Compared with ANN, SVM can obtain more reliable and better performance under same training conditions. Recently, SVM has been successfully extended to SVR, especially in chemometrics for quantitative analysis due to its excellent ability of dealing with nonlinearity and small sample size. The purpose of SVR is to find the underlying relationship between input and output, and a regression functionyx=fx+b=wTϕx+b is used with an ε-insensitive loss function as follows:(4)ξε=0ifξ≤ε,ξ‐εotherwise.Then, the regression problem is equivalent to the following formulations:(5)min12w2+C∑i=1lξi+ξi∗,s.t.yi−wTϕx−b≤ε+ξi,−yi+wTϕx+b≤ε+ξi∗,ξi,ξi∗≥0,where ξi and ξi∗are slack variables, and the regularization parameter C > 0 is used to avoid overfitting. Further, the optimization problem can be converted into the new optimization problem as the following:(6)max12∑i,j=1lαi−αi∗αj−αj∗Kxi,xj−ε∑i=1lαi+αi∗+∑i=1lyiαi−αi∗,s.t.∑i=1lαi−αi∗,αi,αi∗∈0,C,where αi and αi∗are Lagrange multipliers and Kxi,xj is a kernel function. The kernel function maps the nonlinear optimization problem to a linear problem in higher dimensional space. There are several commonly used kernel functions such as linear kernel, polynomial kernel, and radical basis function kernel. In this study, radical basis function kernel is selected as follows:(7)Kxi,x=exp−x−xi2γ2,where γ determines width of the kernel function. Once radical basis function kernel is selected, there are only three parameters to be decided. The parameter ε is to control the error. A larger ε can speed up the train process with low accuracy. Whereas, a smaller ε can achieve better accuracy and reduce the training speed. In this paper, the parameter is set fixed as 0.004. Besides the parameter ε, the regularization parameter C and the kernel function parameter γ are both application-based parameters, which can greatly affect accuracy of the regression. In this paper, genetic algorithm is utilized to search optimal parameters for SVR [27].


#### 2.4.3. Regression Model Based on SAE with SVR

To estimate moisture content in masson pine seedling leaves, a novel regression model based on SAE with SVR is proposed as shown in Figure3. The procedure of the proposed method is described as follows:Figure 3
The proposed network structure of deep learning with SAE-SVR.Step 1. Several AEs are stacked layer by layer to form deep neural networks. The deep neural networks can convert a complex input data into a series of simple high-level features by reducing the dimensionality of the input data. To train the networks, the first AE is trained in an unsupervised manner. After training is finished, its decoder is abandoned, and the output of the hidden layer is used as the input of the second AE with same fashion. Till all AEs have been trained layer by layer, the weights of these AEs had been assigned to initialize the deep neural networks. The topmost hidden layer outputs the elementary high-level features of the input data.Step 2. To achieve better feature representation of the input data, the supervision method is used to tune the weights. A two-layer neural network is added after the topmost layer, and the output of this neural network is the target output. Then, the back-propagation algorithm is utilized to update the weights layer by layer. At last, the fine-tuning weights have been obtained, and then the improved high-level features have been achieved.Step 3. Feed the improved high-level features into SVR as input data, and then build the regression model using GA algorithm to select the optimal parameters.


### 2.5. Evaluation Criteria

As normal, the root-mean squared error (RMSE) and the coefficient of determination are widely used as the evaluating criteria for the calibration model. RMSE is defined as follows:(8)RMSE=∑i=1NTyi−y^i2NT,where yi and y^i are the actual and predicted target output values of the ith sample, respectively and NT is the number of the testing samples. RMSE indicates the accuracy of the model, and a small RMSE value shows better prediction performance than larger RMSE.The coefficient of determinationR2 represents a squared correlation between the actual and predicted output, and the reliability of model can be reflected as R2. R2 is defined as follows:(9)R2=1−∑i=1NTyi−y^i2∑i=1NTyi−y¯2,where y¯ is the mean of the actual output in the testing samples. When R2 is closer to 1, it indicates good prediction performance of the calibration model. In a word, a better calibration model should have small RMSE and large R2.


## 2.1. Materials

A total of 100 annual masson pine seedling samples were obtained from forest farm in Huangping, Guizhou Province, China. Seventy-five of them were used to establish a calibration model, and the remaining 25 were used for prediction. Before measuring the spectra and moisture content, the leaves were cleaned to remove the impurities such as soil and sand, and then the surface of the leaves were naturally dried to prepare for the later experiment.


## 2.2. Spectra Acquisition and Moisture Quantitative Analysis

All NIR spectra of masson pine seedling leaves were measured using an MPA Fourier-Transform Near-Infrared spectrometer (Bruker Optics, Inc., Germany), equipped with an PbS detector and controlled by OPUS Analyst version 7.0 using a spectral range from 4,000 to 12,493 cm−1 in the reflectance mode. Measurements were conducted with the resolution of 4 cm−1, ensuring an adequate signal-to-noise ratio. There are 2203 wavelength variables in the spectrum. The experiment was carried out at approximately 24°C. The spectral reflectance of the top, middle, and bottom areas of the sample were scanned two times each. The final value of the sample spectral reflectance was obtained by calculating the average of the six scans. From each sample, six spectra were collected and averaged for further analysis. Figure 1 shows the raw absorptance spectra of masson pine seedling leaves.Figure 1
Raw absorptance spectra of masson pine seedling leaves.Total moisture content of masson pine seedling leaves was determined using the HB43-S Halogen Moisture analyzer (Mettler Toledo, Inc., Switzerland). The instrument is working based on the thermogravimetric principle, i.e., the moisture is determined from the weight loss of a sample dried by heating. After putting a sample in the sampling chamber, the temperature was immediately increased and maintained at 125°C. The halogen lamp of the instrument heated the sample until the sample stopped losing mass; then the moisture content of the samples is automatically given by the instrument. Normally, the measurements were completed in a few minutes. The reference moisture content value for the samples was 64.97% ± 2.21%, with a minimum of 59.13% and a maximum of 70.74%.


## 2.3. Spectra Pretreatment

Two preprocessing transformations were applied as a standard preparation for the masson pine seedling absorbance curves. Firstly, a Savitzky–Golay (S-G) smoothing with a second-degree polynomial was used for denoising, and the first-order numerical derivative was made to correct the baseline drift. The window width of S-G smoothing was set as 17. Secondly, the spectral matrix was converted into the number between 0 and 1 by vector normalization, which was used to reduce the orders of magnitude difference among the data of different dimensions.


## 2.4. The Proposed Method

### 2.4.1. Autoencoder

The basic structure of autoencoder is a kind of unsupervised ANN with one hidden layer, and it consists of an input layer, a hidden layer, and an output layer as Figure2. The aim of autoencoder is to reduce the data dimension and to map the input data into high dimensional data features.Figure 2
The typical structure of an autoencoder.Define the input asx=x1,x2,…,xd′, where d is the dimension of the inputs. The encoder maps x into the hidden layer h=h1,h2,…,hdh′ by the function f as follows:(1)h=fx=sfWx+b,where dh is the dimension of the hidden layer variable vector h, W is a dh∗d weight matrix, b is the bias vector, and sf is the nonlinear activation function, which can be chosen as the sigmoid function or other functions such as the tanh function and the rectified linear unit function. Then, the hidden representation vector h is mapped to the output layer x˜ by the function f˜.(2)x˜=f˜h=sf˜W˜h+b˜,where W˜ is a d∗dh weight matrix, b˜ is the bias vector for the output layer, and sf˜ is the nonlinear activation function of decoder. The aim of autoencoder is to search the parameter set θ=W,W˜,b,b˜ to satisfy the equation x˜=f˜h=f˜fx≈x. Define the training input as X=x1,x2,…,xN, where N is the number of the training samples and xi=xi1,xi2,…,xid is the vector data of ith training sample. The loss function is defined as follows:(3)JW,W˜,b,b˜=12N∑i=1Nxi−x˜i2=12N∑i=1N∑j=1dxij−x˜ij2.Then, the parameter setθ=W,W˜,b,b˜ is updated with the gradient descent algorithm.In practice, multiple basic autoencoder structures are usually stacked together, and a new network structure is constructed, named SAE, for better dimensional reduction and feature extraction performance. The extracted data feature can be further used for different tasks [25, 26].


### 2.4.2. Support Vector Machine Regression

Support vector machine (SVM) is a powerful and robust method based on the principle of structural risk minimization, which has an advantage of computational efficiency for the data sets that have many more variables than observations. Compared with ANN, SVM can obtain more reliable and better performance under same training conditions. Recently, SVM has been successfully extended to SVR, especially in chemometrics for quantitative analysis due to its excellent ability of dealing with nonlinearity and small sample size. The purpose of SVR is to find the underlying relationship between input and output, and a regression functionyx=fx+b=wTϕx+b is used with an ε-insensitive loss function as follows:(4)ξε=0ifξ≤ε,ξ‐εotherwise.Then, the regression problem is equivalent to the following formulations:(5)min12w2+C∑i=1lξi+ξi∗,s.t.yi−wTϕx−b≤ε+ξi,−yi+wTϕx+b≤ε+ξi∗,ξi,ξi∗≥0,where ξi and ξi∗are slack variables, and the regularization parameter C > 0 is used to avoid overfitting. Further, the optimization problem can be converted into the new optimization problem as the following:(6)max12∑i,j=1lαi−αi∗αj−αj∗Kxi,xj−ε∑i=1lαi+αi∗+∑i=1lyiαi−αi∗,s.t.∑i=1lαi−αi∗,αi,αi∗∈0,C,where αi and αi∗are Lagrange multipliers and Kxi,xj is a kernel function. The kernel function maps the nonlinear optimization problem to a linear problem in higher dimensional space. There are several commonly used kernel functions such as linear kernel, polynomial kernel, and radical basis function kernel. In this study, radical basis function kernel is selected as follows:(7)Kxi,x=exp−x−xi2γ2,where γ determines width of the kernel function. Once radical basis function kernel is selected, there are only three parameters to be decided. The parameter ε is to control the error. A larger ε can speed up the train process with low accuracy. Whereas, a smaller ε can achieve better accuracy and reduce the training speed. In this paper, the parameter is set fixed as 0.004. Besides the parameter ε, the regularization parameter C and the kernel function parameter γ are both application-based parameters, which can greatly affect accuracy of the regression. In this paper, genetic algorithm is utilized to search optimal parameters for SVR [27].


### 2.4.3. Regression Model Based on SAE with SVR

To estimate moisture content in masson pine seedling leaves, a novel regression model based on SAE with SVR is proposed as shown in Figure3. The procedure of the proposed method is described as follows:Figure 3
The proposed network structure of deep learning with SAE-SVR.Step 1. Several AEs are stacked layer by layer to form deep neural networks. The deep neural networks can convert a complex input data into a series of simple high-level features by reducing the dimensionality of the input data. To train the networks, the first AE is trained in an unsupervised manner. After training is finished, its decoder is abandoned, and the output of the hidden layer is used as the input of the second AE with same fashion. Till all AEs have been trained layer by layer, the weights of these AEs had been assigned to initialize the deep neural networks. The topmost hidden layer outputs the elementary high-level features of the input data.Step 2. To achieve better feature representation of the input data, the supervision method is used to tune the weights. A two-layer neural network is added after the topmost layer, and the output of this neural network is the target output. Then, the back-propagation algorithm is utilized to update the weights layer by layer. At last, the fine-tuning weights have been obtained, and then the improved high-level features have been achieved.Step 3. Feed the improved high-level features into SVR as input data, and then build the regression model using GA algorithm to select the optimal parameters.


## 2.4.1. Autoencoder

The basic structure of autoencoder is a kind of unsupervised ANN with one hidden layer, and it consists of an input layer, a hidden layer, and an output layer as Figure2. The aim of autoencoder is to reduce the data dimension and to map the input data into high dimensional data features.Figure 2
The typical structure of an autoencoder.Define the input asx=x1,x2,…,xd′, where d is the dimension of the inputs. The encoder maps x into the hidden layer h=h1,h2,…,hdh′ by the function f as follows:(1)h=fx=sfWx+b,where dh is the dimension of the hidden layer variable vector h, W is a dh∗d weight matrix, b is the bias vector, and sf is the nonlinear activation function, which can be chosen as the sigmoid function or other functions such as the tanh function and the rectified linear unit function. Then, the hidden representation vector h is mapped to the output layer x˜ by the function f˜.(2)x˜=f˜h=sf˜W˜h+b˜,where W˜ is a d∗dh weight matrix, b˜ is the bias vector for the output layer, and sf˜ is the nonlinear activation function of decoder. The aim of autoencoder is to search the parameter set θ=W,W˜,b,b˜ to satisfy the equation x˜=f˜h=f˜fx≈x. Define the training input as X=x1,x2,…,xN, where N is the number of the training samples and xi=xi1,xi2,…,xid is the vector data of ith training sample. The loss function is defined as follows:(3)JW,W˜,b,b˜=12N∑i=1Nxi−x˜i2=12N∑i=1N∑j=1dxij−x˜ij2.Then, the parameter setθ=W,W˜,b,b˜ is updated with the gradient descent algorithm.In practice, multiple basic autoencoder structures are usually stacked together, and a new network structure is constructed, named SAE, for better dimensional reduction and feature extraction performance. The extracted data feature can be further used for different tasks [25, 26].


## 2.4.2. Support Vector Machine Regression

Support vector machine (SVM) is a powerful and robust method based on the principle of structural risk minimization, which has an advantage of computational efficiency for the data sets that have many more variables than observations. Compared with ANN, SVM can obtain more reliable and better performance under same training conditions. Recently, SVM has been successfully extended to SVR, especially in chemometrics for quantitative analysis due to its excellent ability of dealing with nonlinearity and small sample size. The purpose of SVR is to find the underlying relationship between input and output, and a regression functionyx=fx+b=wTϕx+b is used with an ε-insensitive loss function as follows:(4)ξε=0ifξ≤ε,ξ‐εotherwise.Then, the regression problem is equivalent to the following formulations:(5)min12w2+C∑i=1lξi+ξi∗,s.t.yi−wTϕx−b≤ε+ξi,−yi+wTϕx+b≤ε+ξi∗,ξi,ξi∗≥0,where ξi and ξi∗are slack variables, and the regularization parameter C > 0 is used to avoid overfitting. Further, the optimization problem can be converted into the new optimization problem as the following:(6)max12∑i,j=1lαi−αi∗αj−αj∗Kxi,xj−ε∑i=1lαi+αi∗+∑i=1lyiαi−αi∗,s.t.∑i=1lαi−αi∗,αi,αi∗∈0,C,where αi and αi∗are Lagrange multipliers and Kxi,xj is a kernel function. The kernel function maps the nonlinear optimization problem to a linear problem in higher dimensional space. There are several commonly used kernel functions such as linear kernel, polynomial kernel, and radical basis function kernel. In this study, radical basis function kernel is selected as follows:(7)Kxi,x=exp−x−xi2γ2,where γ determines width of the kernel function. Once radical basis function kernel is selected, there are only three parameters to be decided. The parameter ε is to control the error. A larger ε can speed up the train process with low accuracy. Whereas, a smaller ε can achieve better accuracy and reduce the training speed. In this paper, the parameter is set fixed as 0.004. Besides the parameter ε, the regularization parameter C and the kernel function parameter γ are both application-based parameters, which can greatly affect accuracy of the regression. In this paper, genetic algorithm is utilized to search optimal parameters for SVR [27].


## 2.4.3. Regression Model Based on SAE with SVR

To estimate moisture content in masson pine seedling leaves, a novel regression model based on SAE with SVR is proposed as shown in Figure3. The procedure of the proposed method is described as follows:Figure 3
The proposed network structure of deep learning with SAE-SVR.Step 1. Several AEs are stacked layer by layer to form deep neural networks. The deep neural networks can convert a complex input data into a series of simple high-level features by reducing the dimensionality of the input data. To train the networks, the first AE is trained in an unsupervised manner. After training is finished, its decoder is abandoned, and the output of the hidden layer is used as the input of the second AE with same fashion. Till all AEs have been trained layer by layer, the weights of these AEs had been assigned to initialize the deep neural networks. The topmost hidden layer outputs the elementary high-level features of the input data.Step 2. To achieve better feature representation of the input data, the supervision method is used to tune the weights. A two-layer neural network is added after the topmost layer, and the output of this neural network is the target output. Then, the back-propagation algorithm is utilized to update the weights layer by layer. At last, the fine-tuning weights have been obtained, and then the improved high-level features have been achieved.Step 3. Feed the improved high-level features into SVR as input data, and then build the regression model using GA algorithm to select the optimal parameters.


## 2.5. Evaluation Criteria

As normal, the root-mean squared error (RMSE) and the coefficient of determination are widely used as the evaluating criteria for the calibration model. RMSE is defined as follows:(8)RMSE=∑i=1NTyi−y^i2NT,where yi and y^i are the actual and predicted target output values of the ith sample, respectively and NT is the number of the testing samples. RMSE indicates the accuracy of the model, and a small RMSE value shows better prediction performance than larger RMSE.The coefficient of determinationR2 represents a squared correlation between the actual and predicted output, and the reliability of model can be reflected as R2. R2 is defined as follows:(9)R2=1−∑i=1NTyi−y^i2∑i=1NTyi−y¯2,where y¯ is the mean of the actual output in the testing samples. When R2 is closer to 1, it indicates good prediction performance of the calibration model. In a word, a better calibration model should have small RMSE and large R2.


## 3. Results and Discussion

### 3.1. Result and Settings of the Proposed Model

For the proposed model construction and test, 100 samples were divided into 75 samples for calibration dataset and 25 samples for prediction dataset. The first step is to decide the settings of the model. It is evident that the number of the hidden layers and the number of the neuron in each hidden layer can have major influence on the performance of the model. However, there is still no automatic method that can be utilized to achieve the selection of the parameters. In this paper, trial and error is used to choose these parameters, and RMSE is adopted to access the performance of the model. The weights and bias for each AE is obtained by greedy layerwise pretraining technique, and each AE is trained with gradient descent algorithm. After the SAE is layerwise trained, the two-layer neural network is connected to the output of SAE for weight fine tuning of ANN. After that, SVR is connected to the high-level features to output the predicted value. After experiments, the proposed model structure is composed of three-layer AEs, and the hidden neurons number is 1600, 1100, and 512, respectively. The batch size is set as 40 samples, and each AE is trained 500 iterations iteratively. Figure4 shows the training loss trends with the iteration number for each AE. As shown in Figure 4, the SAE model can converge quickly in 500 iterations for each AE.Figure 4
The training loss trends of each AE on the calibration dataset.After the weights are pretrained and fine-tuned, the SVR is connected to output the predicted value. The detailed prediction results are shown in Figure5, and C and γ of SVR are set as 4.79 and 1.97, respectively, using GA algorithm. The RMSE and R2 of the prediction dataset are 0.4249 and 0.9621, respectively. As can be seen from Figure 5, the predicted output can match well with the real values. The prediction error is mainly caused by the points which are located in the interval with few samples in the calibration dataset.Figure 5
The prediction performance on the calibration dataset (a) and the prediction dataset (b).
(a)
(b)


### 3.2. Discussion

To verify the effectiveness of the proposed method, there are several methods to be used to build the calibration model. Each method is trained with the same calibration dataset, and the average result of 4-folder cross-validation in each method is adopted in Table1 and Figure 6. The results of MLR, PLS, and SVR are directly obtained using PLS_Toolbox (Eigenvector Research, Inc., USA) running on MATLAB 2016. The other methods are implemented by the deep learning framework Keras using Python. As can be seen in Table 1, MLR and PLS produce the worst prediction results since they cannot deal with nonlinear correlative data for its linear essence. SVR is inferior to other methods and just outperforms above two methods, because it cannot describe the nonlinear data adequately. The last three methods all adopt the same network structure. The ANN method is composed of five layers of neural networks, and the first three layers have the same network structure as SAE. The last two layers have the same structure with neural networks as being connected to SAE. The same two-layer neural networks with the neuron layer structure [512, 128] are connected to SAE for fine tuning the weights. ANN method outperforms above three methods, but it is easily trapped in local optima. By adopting the SAE, high-level abstract features can be extracted layer by layer in SAE. Further, by using the pretrained weights and bias terms in the full connected network, the methods can avoid the inferior local optima and speed up the learning process. Thus, the last two methods can describe the complex data structure more accurately. Besides, a small size of training samples used in this study may also limit the predictive power of the ANN model. Then, SVR with genetic algorithm is connected to SAE to achieve better performance in small training samples. The regression plot between the predicted moisture content and the reference content is shown in Figure 6. It can visualize the regression performance of different models. If the scatter data points are closer to the perfect correlation line (with slope = 1), the prediction model is more accurate (larger R2 value and smaller RMSE value). Overall, the proposed method outperforms the other methods for moisture concentration prediction of masson pine seedling leaves.Table 1
Calibration and prediction results of the moisture content in masson pine seedling leaves using different regression models.
Model
Calibration dataset
Prediction dataset
R
C
2
RMSEC
R
P
2
RMSEP
MLR
0.9267
0.5619
0.3915
2.879
PLSR
0.9658
0.4079
0.8981
0.6774
SVR
0.9892
0.2285
0.9016
0.7052
ANN
0.9974
0.1123
0.9322
0.5559
SAE-ANN
0.9944
0.1644
0.9421
0.5228
SAE-SVR
0.9946
0.1636
0.9621
0.4249Figure 6
Measured vs. predicated moisture content for masson pine seedling leaves by the regression models of MLR (a), PLSR (b), SVR (c), ANN (d), SAE-ANN (e), and SAE-SVR (f).
(a)
(b)
(c)
(d)
(e)
(f)


## 3.1. Result and Settings of the Proposed Model

For the proposed model construction and test, 100 samples were divided into 75 samples for calibration dataset and 25 samples for prediction dataset. The first step is to decide the settings of the model. It is evident that the number of the hidden layers and the number of the neuron in each hidden layer can have major influence on the performance of the model. However, there is still no automatic method that can be utilized to achieve the selection of the parameters. In this paper, trial and error is used to choose these parameters, and RMSE is adopted to access the performance of the model. The weights and bias for each AE is obtained by greedy layerwise pretraining technique, and each AE is trained with gradient descent algorithm. After the SAE is layerwise trained, the two-layer neural network is connected to the output of SAE for weight fine tuning of ANN. After that, SVR is connected to the high-level features to output the predicted value. After experiments, the proposed model structure is composed of three-layer AEs, and the hidden neurons number is 1600, 1100, and 512, respectively. The batch size is set as 40 samples, and each AE is trained 500 iterations iteratively. Figure4 shows the training loss trends with the iteration number for each AE. As shown in Figure 4, the SAE model can converge quickly in 500 iterations for each AE.Figure 4
The training loss trends of each AE on the calibration dataset.After the weights are pretrained and fine-tuned, the SVR is connected to output the predicted value. The detailed prediction results are shown in Figure5, and C and γ of SVR are set as 4.79 and 1.97, respectively, using GA algorithm. The RMSE and R2 of the prediction dataset are 0.4249 and 0.9621, respectively. As can be seen from Figure 5, the predicted output can match well with the real values. The prediction error is mainly caused by the points which are located in the interval with few samples in the calibration dataset.Figure 5
The prediction performance on the calibration dataset (a) and the prediction dataset (b).
(a)
(b)


## 3.2. Discussion

To verify the effectiveness of the proposed method, there are several methods to be used to build the calibration model. Each method is trained with the same calibration dataset, and the average result of 4-folder cross-validation in each method is adopted in Table1 and Figure 6. The results of MLR, PLS, and SVR are directly obtained using PLS_Toolbox (Eigenvector Research, Inc., USA) running on MATLAB 2016. The other methods are implemented by the deep learning framework Keras using Python. As can be seen in Table 1, MLR and PLS produce the worst prediction results since they cannot deal with nonlinear correlative data for its linear essence. SVR is inferior to other methods and just outperforms above two methods, because it cannot describe the nonlinear data adequately. The last three methods all adopt the same network structure. The ANN method is composed of five layers of neural networks, and the first three layers have the same network structure as SAE. The last two layers have the same structure with neural networks as being connected to SAE. The same two-layer neural networks with the neuron layer structure [512, 128] are connected to SAE for fine tuning the weights. ANN method outperforms above three methods, but it is easily trapped in local optima. By adopting the SAE, high-level abstract features can be extracted layer by layer in SAE. Further, by using the pretrained weights and bias terms in the full connected network, the methods can avoid the inferior local optima and speed up the learning process. Thus, the last two methods can describe the complex data structure more accurately. Besides, a small size of training samples used in this study may also limit the predictive power of the ANN model. Then, SVR with genetic algorithm is connected to SAE to achieve better performance in small training samples. The regression plot between the predicted moisture content and the reference content is shown in Figure 6. It can visualize the regression performance of different models. If the scatter data points are closer to the perfect correlation line (with slope = 1), the prediction model is more accurate (larger R2 value and smaller RMSE value). Overall, the proposed method outperforms the other methods for moisture concentration prediction of masson pine seedling leaves.Table 1
Calibration and prediction results of the moisture content in masson pine seedling leaves using different regression models.
Model
Calibration dataset
Prediction dataset
R
C
2
RMSEC
R
P
2
RMSEP
MLR
0.9267
0.5619
0.3915
2.879
PLSR
0.9658
0.4079
0.8981
0.6774
SVR
0.9892
0.2285
0.9016
0.7052
ANN
0.9974
0.1123
0.9322
0.5559
SAE-ANN
0.9944
0.1644
0.9421
0.5228
SAE-SVR
0.9946
0.1636
0.9621
0.4249Figure 6
Measured vs. predicated moisture content for masson pine seedling leaves by the regression models of MLR (a), PLSR (b), SVR (c), ANN (d), SAE-ANN (e), and SAE-SVR (f).
(a)
(b)
(c)
(d)
(e)
(f)


## 4. Conclusion

In this study, a new method of stacked autoencoder with SVR is introduced and applied to estimate the moisture content in masson pine seedlings. Compared with MLR, PLSR, SVR, ANN, and SAE-ANN, the SAE-SVR shows superior performances, and getsR2 value 0.9946 and RMSE 0.1636 in calibration dataset, and R2 value 0.9621 and RMSE 0.4249 in prediction dataset. The results of this study demonstrate that deep neural networks pretrained by SAE are feasible to be used as the data analysis method for moisture prediction in masson pine seedling leaves.



---

*Source: 8696202-2018-09-19.xml*