# Processing Method of Missing Data in Dam Safety Monitoring

Authors: Wei Wei; Chongshi Gu; Xiao Fu
Journal: Mathematical Problems in Engineering
 (2021)

---

## Abstract

A large amount of data obtained by dam safety monitoring provides the basis to evaluate the dam operation state. Due to the interference caused by equipment failure and human error, it is common or even inevitable to suffer the loss of measurement data. Most of the traditional data processing methods for dam monitoring ignore the actual correlation between different measurement points, which brings difficulties to the objective diagnosis of dam safety and even leads to misdiagnosis. Therefore, it is necessary to conduct further study on how to process the missing data in dam safety monitoring. In this study, a data processing method based on partial distance combining fuzzy C-means with long short-term memory (PDS-FCM-LSTM) was proposed to deal with the data missing from dam monitoring. Based on the fuzzy clustering performed for the measurement points of the same category deployed on the dam, the membership degree of each measurement point to cluster center was described by using the fuzzy C-means clustering algorithm based on partial distance (PDS-FCM), so as to determine the clustering results and preprocess the missing data of corresponding measurement points. Then, the bidirectional long short-term memory (LSTM) network was applied to explore the pattern of changes of measurement values under identical clustering conditions, thus processing the data missing from monitoring effectively.


---

## Body

## 1. Introduction

As an indicator of the safety and work performance of concrete dam, the original measurement data always attract much attention for its integrity and accuracy [1–3]. However, due to the impact of such influencing factors, such as complicated and ever-changing operating conditions and human error, it is often difficult to prevent the actual measurement data from suffering a loss to varying extents [4]. The missing data will have an influence on the thorough assessment of dam for its serviceability, which can even lead to misjudgment in some cases. As for the traditional means to process the missing data, they include mean value processing, regression processing, expectation maximization processing, and so on [5, 6]. Nevertheless, these methods commonly suffer such problems due to the inability to reflect the correlation between samples or attributes, low processing accuracy, and limited applicability [7].In recent years, as the study on machine learning and other emerging disciplines deepens [8–11], some new approaches for data processing have been proposed [12, 13]. By improving the k-nearest neighbors imputation method, Shichao Zhang proposed a gray processing method (GKNN) to replace the traditional Euclidean distance with gray distance [14]. Similarly, Tutz G put forward a distance-based weighted nearest neighbors imputation method [15]. However, the process for finding the adjacent samples for each incomplete sample as part of the method requires global traversal, which makes computation overly complex at the time of dealing with a huge amount of data. Chen applied LS-SVM [16] to fill the data, based on which the global optimal solution was worked out and small-scale data were made applicable. However, most of the SVM-based methods involve artificial judgment, which makes them heavily reliant on the personal experience gained by the operator. Proposed by Takagi and Sugeno, the TS nonlinear regression model [17–19] is capable of describing nonlinear problems linearly by constructing local linear models and associating these linear models with membership function. However, this model ignores the correlation between input and output, which leads to the low accuracy in processing sequence information. Through the combination between autocoding and the genetic algorithm, Abdella et al. put forward a processing method which takes the missing data as the independent variable for cost function, and the genetic algorithm was applied to optimize the cost function of the missing data to be solved [20, 21]. Based on the aforementioned method, Nelwamondo et al. introduced the dynamic programming theory to build multiple self-encoders and then selected the optimal model for each incomplete sample to process the missing data [22]. However, such methods ignore the correlation between different sample attributes, which causes the accuracy of missing data processing to be low. The newly proposed autoregressive integrated moving average model (ARIMA) hybrid method shows the advantages of both the autoregressive integrated moving average and artificial neural network, which improves its generality. Due to the need for a large data sample size, the practicability of it is not as high as that required [23].To address the problem mentioned above, this study proposes a missing data processing method based on the partial distance fuzzy C-means (PDS-FCM) model and long short-term memory (LSTM) network. Considering the correlation between the measurement points, it relies on the membership degree of each measurement point to cluster center to carry out cluster analysis and preprocess missing data. On this basis, the bidirectional LSTM network is introduced to further process the long-sequence missing data, thus ensuring the accurate processing of missing data. The basic principle is detailed as follows.


## 2. Processing Method of Dam Missing Data

First, the core of the missing data processing method proposed in this study is to use PDS [24] instead of Euclidean distance strategy to represent the correlation between the measurement points of a certain type of monitoring quantity. Then, the FCM clustering algorithm is adopted to construct the membership relationship between each measurement point and the cluster center, thus achieving fuzzy clustering for the measurement points and the preprocessing of missing data. Finally, the bidirectional LSTM network is introduced to construct the training model of missing data and to process the long series missing measurement values for the measurement points under identical clustering conditions, thus achieving the effective processing of missing data.


### 2.1. Preprocessing Method of Missing Data

#### 2.1.1. Characterization of Correlation between Measurement Points

When the measurement points of a certain type of monitoring quantity encounter the missing measurement value, it is difficult to measure the correlation between measurement points using those traditional distance measurement indicators. To solve this problem, this study applies PDS to characterize the correlation between different measurement points.SupposeX=xi,xi∈ℝn,i=1,2,…,n is an incomplete dataset with the number of measurement points n and the sequence of measurement value s. xi=xi1,xi2,…,xisT,i=1,2,…,n, indicates the measurement value of the ith measurement point. I=Iil∈ℝn×s represents a function describing the missing data of the measurement point, which is expressed as in the following equation:(1)Iil=0,l=t∗,1,l=t,where Iil suggests that the measurement value of the ith measurement point is complete at time t and missing at time t∗.Based on equation (1), the correlation between xi and xk in the incomplete dataset X can be expressed as follows:(2)dPartxi,xk=s∑l=1sIilxil−xkl2∑l=1sIil,where dPart refers to the partial distance.


#### 2.1.2. Fuzzy Clustering of Measurement Points and Preprocessing of Measurement Values

Based on equation (2) and FCM clustering algorithm, fuzzy clustering is performed in this section for the measurement points of a certain type of monitoring quantity, and the measurement value of the measurement point is preprocessed. Similar to most clustering algorithms, the FCM algorithm splits the dataset into a number of subsets according to the similarity between samples. Each subset represents a cluster, and the center of sample distribution in the cluster is taken as the cluster center. The difference is that this method achieves the transformation of membership from 0 or 1 to [0, 1] by means of fuzzy processing. Equation (2) is applied to construct the PDS-FCM clustering model of the following equation:(3)minJU,V=∑k=1K∑i=1nuikzdPartxi,vk2,s.t.uik∈0,1,i=1,2,…,n,k=1,2,…,K,∑k=1Kuik=1,i=1,2,…,n,0<∑i=1nuik<n,k=1,2,…,K,where uik represents the membership degree, which indicates the degree of the measurement point xi falling into the kth cluster, denoted as U=uik∈ℝn×K. vk denotes the cluster center of the kth cluster, and the corresponding center matrix is expressed as V=vkj=v1,v2,…,vKT∈ℝK×s. z∈1,∞ is a fuzzy parameter, which indicates the degree to which these measurement points belong to each cluster.Considering the equality constraints on membership degree in equation (3), the Lagrange multiplier method is used in this section to solve equation (3). The augmented Lagrange function applied to this process is as follows:(4)JλU,V=∑k=1K∑i=1nuikzdPartxi,vk2+∑i=1nλi∑k=1Kuik−1,where λ=λ1,λ2,…,λnT represents the Lagrange multiplier.Based on equation (2) and the equality constraint of uik in equation (3), the necessary condition for the clustering objective function Jλ in equation (4) to reach the minimal is expressed as follows:(5)vkl=∑i=1nuikzxil∑i=1nuikz,l=1,2,…,s;k=1,2,…,K,(6)uik=∑t=1KdPartxi,vk2dPartxi,vt21/z−1−1,i=1,2,…n;k=1,2,…,K.With the application of equations (5) and (6), the membership matrix U and center matrix V that meet the requirements of the accuracy of dam safety monitoring are obtained through iterative solution. The maximum membership of each measurement point to each cluster center is treated as the final clustering result.According to the membership relationship between the missing data measurement points and the central points, the missing positions of the measurement values are preprocessed as follows:(7)Xil∗=UikVkl,l=t∗,Xil∗=Xil,l=t,where Xil∗ denotes the measurement value of the ith measurement point at the time l after preprocessing, and Vkl indicates the measurement value of the ith center at the time l.The ultimate clustering result is determined by equation (6), and the missing data are preprocessed by equation (7) based on which the data missing from dam safety monitoring can be effectively processed.


### 2.2. Implementation Technology of the PDS-FCM-LSTM Processing Method for the Data Missing from Dam Safety Monitoring

On the basis of the abovementioned research, the bidirectional LSTM network is introduced in this section to establish the mapping relationship between the measurement values of measurement points under identical clustering conditions, based on which the PDS-FCM-LSTM processing method is proposed.


#### 2.2.1. Construction of Mapping Relationship between Measurement Values of the Measurement Points under Identical Clustering Conditions

The advantage of the LSTM network is that it is capable of controlling the information transfer of time series data by introducing input gate, forget gate, and output gate to the unit [9, 25–28]. It is assumed that the data of dam safety measurement point i are missing from time j to time j+p, which require reprocessing. For the unit at time l, the data xil∗l<j of measurement point i are defined as output variables, while the data xml∗m≠i,l<j of other measurement points in the same cluster are treated as input variables. On this basis, the mapping relationship between the measurement values of the forward LSTM network is established, as shown in Figure 1.Figure 1
Schematic diagram of the LSTM unit.(1) Forget Gate. The output variable of the hidden layer hl−1 at time l−1 and the input variable xml∗ at time l are taken. Equation (8) is applied to construct the output variable of the forget gate fl at time l, which determines whether to maintain the state of the hidden unit cl−1 at the previous layer.(8)fl=σWf⋅hl−1,xml∗+bf,where fl represents the output variable after the processing by the forget gate at time l, hl−1 denotes the output variable of the hidden layer at time l−1, xml∗ indicates the input variable of each measurement point at time l after preprocessing, bf refers to the bias term of the forget gate, Wf stands for the weight matrix of the forget gate, and σ means the activation function of sigmod, which is expressed as follows:(9)σz=11+exp−z.(2) Input Gate. The information collected by the input gate is the same as that collected by the forget gate. Based on equations (9) and (10), the output variable of the input gate Rl at time l and the unit state cl at time l are determined.(10)Rl=σWR⋅hl−1,xml∗+bR,(11)cl=fl∗cl−1+Rl∗tanhWc⋅hl−1,xml∗+bc,where Rl represents the output value after processing by the input gate at time l, WR indicates the input gate weight matrix, Wc denotes the unit state weight matrix, cl means the unit state at time l, bR refers to the bias term of input gate, and bc stands for the bias term of unit state.(3) Output Gate. The output gate is jointly determined by the updated unit state cl, the output variable of the hidden layer hl−1 at time l−1, and the input variable xml∗ at time l. The output variable of the output gate ol at time l and the output variable of the hidden layer hl at time l can be determined as follows:(12)ol=σWo⋅hl−1,xml∗+bo,hl=ol∗tanhcl,where ol represents the output variable processed by the output gate at time l, Wo denotes the output gate weight matrix, and bo indicates the bias term of the output gate.To process the missing data of measurement value from timej to time j+p more reasonably, a new bidirectional LSTM network method is proposed to process the data from time j to time j+p. As shown in Figure 2, model training and validation are carried out on the measurement points before and after time j to time j+p, respectively, according to the aforementioned principles based on which the forward and reverse LSTM model parameters are obtained. Then, the remaining measurement point data of the complete information from time j to time j+p are input into the model. Finally, the calculation value of the measurement point with missing data corresponding to the period from time j to time j+p is obtained, so as to process the missing data bidirectionally.Figure 2
Bidirectional LSTM processing method and idea (the gray box represents the preprocessed value of discrete data and the red box represents the long-sequence missing data).


#### 2.2.2. Implementation Method of the PDS-FCM-BILSTM Processing Method for Dam Missing Data

According to the PDS-FCM-LSTM method for dam missing data processing as proposed above, the implementation process is shown in Figure3.Step 1: input the missing dataset of the dam measurement pointStep 2: enter the PDS-FCM layer to preprocess the missing data, set fuzzy parameterz, cluster number K, and threshold εε>0, and then randomly initialize the partition matrix U0Step 3: when the number of iterations isnn≥1, equation (5) is used to update the prototype matrix Vn based on the partition matrix Un−1Step 4: update the partition matrixUn based on equation (6) and the prototype matrix VnStep 5: if the conditions∀k,i:maxuikn−uikn−1<ε are met, where k=1,2,…,K,i=1,2,…,n, the algorithm stops to output partitioning matrix U and prototype matrix V. Otherwise, if n←n+1, return to Step 3.Step 6: based on the membership relationship between the measurement points with missing data and each cluster centerU,V, the existing values in the cluster center are weighted and averaged to obtain the preprocessed data X∗Step 7: establish the bidirectional LSTM processing model for the missing data of the dam. The forward LSTM network model and reverse LSTM network model are constructed according to the time required to process the measurement values.Step 8: input the measurement values of each measurement point in the same cluster before and after the time slotT into the forward and reverse LSTM models, so as to obtain the weight W and bias b of each parameter after training and learningStep 9: input the preprocessed measurement point data as sample into the forward and backward LSTM models for obtaining the forward and backward output values at time slotT, with their mean values taken as the ultimate result of missing data processingFigure 3
PDS-FCM-LSTM concept.According to the above implementation steps, the implementation concept of PDS-FCM-LSTM is obtained.


#### 2.2.3. Evaluation of the Missing Data Processing Effect

In this study, the average absolute error MAE and the average absolute percentage error MAPE are taken as the evaluation indicators of the data processing effect for the measurement points, which are defined as follows:(13)MAE=1X^m∑x^ij∈X^mrij−x^ij,(14)MAPE=1X^m∑x^ij∈X^mrij−x^ijrij,where X^m represents the set of all processing values, x^ij denotes the processing value, rij indicates the measurement value of the processing value, and x^ij corresponds to the measurement value of the measurement point with complete information.Using equations (13) and (14) and the error allowable values listed in the technical specifications for dam safety monitoring, the effect of missing data processing is thoroughly evaluated, and the processed data meeting the accuracy requirements are taken as the effective data for dam safety monitoring.


## 2.1. Preprocessing Method of Missing Data

### 2.1.1. Characterization of Correlation between Measurement Points

When the measurement points of a certain type of monitoring quantity encounter the missing measurement value, it is difficult to measure the correlation between measurement points using those traditional distance measurement indicators. To solve this problem, this study applies PDS to characterize the correlation between different measurement points.SupposeX=xi,xi∈ℝn,i=1,2,…,n is an incomplete dataset with the number of measurement points n and the sequence of measurement value s. xi=xi1,xi2,…,xisT,i=1,2,…,n, indicates the measurement value of the ith measurement point. I=Iil∈ℝn×s represents a function describing the missing data of the measurement point, which is expressed as in the following equation:(1)Iil=0,l=t∗,1,l=t,where Iil suggests that the measurement value of the ith measurement point is complete at time t and missing at time t∗.Based on equation (1), the correlation between xi and xk in the incomplete dataset X can be expressed as follows:(2)dPartxi,xk=s∑l=1sIilxil−xkl2∑l=1sIil,where dPart refers to the partial distance.


### 2.1.2. Fuzzy Clustering of Measurement Points and Preprocessing of Measurement Values

Based on equation (2) and FCM clustering algorithm, fuzzy clustering is performed in this section for the measurement points of a certain type of monitoring quantity, and the measurement value of the measurement point is preprocessed. Similar to most clustering algorithms, the FCM algorithm splits the dataset into a number of subsets according to the similarity between samples. Each subset represents a cluster, and the center of sample distribution in the cluster is taken as the cluster center. The difference is that this method achieves the transformation of membership from 0 or 1 to [0, 1] by means of fuzzy processing. Equation (2) is applied to construct the PDS-FCM clustering model of the following equation:(3)minJU,V=∑k=1K∑i=1nuikzdPartxi,vk2,s.t.uik∈0,1,i=1,2,…,n,k=1,2,…,K,∑k=1Kuik=1,i=1,2,…,n,0<∑i=1nuik<n,k=1,2,…,K,where uik represents the membership degree, which indicates the degree of the measurement point xi falling into the kth cluster, denoted as U=uik∈ℝn×K. vk denotes the cluster center of the kth cluster, and the corresponding center matrix is expressed as V=vkj=v1,v2,…,vKT∈ℝK×s. z∈1,∞ is a fuzzy parameter, which indicates the degree to which these measurement points belong to each cluster.Considering the equality constraints on membership degree in equation (3), the Lagrange multiplier method is used in this section to solve equation (3). The augmented Lagrange function applied to this process is as follows:(4)JλU,V=∑k=1K∑i=1nuikzdPartxi,vk2+∑i=1nλi∑k=1Kuik−1,where λ=λ1,λ2,…,λnT represents the Lagrange multiplier.Based on equation (2) and the equality constraint of uik in equation (3), the necessary condition for the clustering objective function Jλ in equation (4) to reach the minimal is expressed as follows:(5)vkl=∑i=1nuikzxil∑i=1nuikz,l=1,2,…,s;k=1,2,…,K,(6)uik=∑t=1KdPartxi,vk2dPartxi,vt21/z−1−1,i=1,2,…n;k=1,2,…,K.With the application of equations (5) and (6), the membership matrix U and center matrix V that meet the requirements of the accuracy of dam safety monitoring are obtained through iterative solution. The maximum membership of each measurement point to each cluster center is treated as the final clustering result.According to the membership relationship between the missing data measurement points and the central points, the missing positions of the measurement values are preprocessed as follows:(7)Xil∗=UikVkl,l=t∗,Xil∗=Xil,l=t,where Xil∗ denotes the measurement value of the ith measurement point at the time l after preprocessing, and Vkl indicates the measurement value of the ith center at the time l.The ultimate clustering result is determined by equation (6), and the missing data are preprocessed by equation (7) based on which the data missing from dam safety monitoring can be effectively processed.


## 2.1.1. Characterization of Correlation between Measurement Points

When the measurement points of a certain type of monitoring quantity encounter the missing measurement value, it is difficult to measure the correlation between measurement points using those traditional distance measurement indicators. To solve this problem, this study applies PDS to characterize the correlation between different measurement points.SupposeX=xi,xi∈ℝn,i=1,2,…,n is an incomplete dataset with the number of measurement points n and the sequence of measurement value s. xi=xi1,xi2,…,xisT,i=1,2,…,n, indicates the measurement value of the ith measurement point. I=Iil∈ℝn×s represents a function describing the missing data of the measurement point, which is expressed as in the following equation:(1)Iil=0,l=t∗,1,l=t,where Iil suggests that the measurement value of the ith measurement point is complete at time t and missing at time t∗.Based on equation (1), the correlation between xi and xk in the incomplete dataset X can be expressed as follows:(2)dPartxi,xk=s∑l=1sIilxil−xkl2∑l=1sIil,where dPart refers to the partial distance.


## 2.1.2. Fuzzy Clustering of Measurement Points and Preprocessing of Measurement Values

Based on equation (2) and FCM clustering algorithm, fuzzy clustering is performed in this section for the measurement points of a certain type of monitoring quantity, and the measurement value of the measurement point is preprocessed. Similar to most clustering algorithms, the FCM algorithm splits the dataset into a number of subsets according to the similarity between samples. Each subset represents a cluster, and the center of sample distribution in the cluster is taken as the cluster center. The difference is that this method achieves the transformation of membership from 0 or 1 to [0, 1] by means of fuzzy processing. Equation (2) is applied to construct the PDS-FCM clustering model of the following equation:(3)minJU,V=∑k=1K∑i=1nuikzdPartxi,vk2,s.t.uik∈0,1,i=1,2,…,n,k=1,2,…,K,∑k=1Kuik=1,i=1,2,…,n,0<∑i=1nuik<n,k=1,2,…,K,where uik represents the membership degree, which indicates the degree of the measurement point xi falling into the kth cluster, denoted as U=uik∈ℝn×K. vk denotes the cluster center of the kth cluster, and the corresponding center matrix is expressed as V=vkj=v1,v2,…,vKT∈ℝK×s. z∈1,∞ is a fuzzy parameter, which indicates the degree to which these measurement points belong to each cluster.Considering the equality constraints on membership degree in equation (3), the Lagrange multiplier method is used in this section to solve equation (3). The augmented Lagrange function applied to this process is as follows:(4)JλU,V=∑k=1K∑i=1nuikzdPartxi,vk2+∑i=1nλi∑k=1Kuik−1,where λ=λ1,λ2,…,λnT represents the Lagrange multiplier.Based on equation (2) and the equality constraint of uik in equation (3), the necessary condition for the clustering objective function Jλ in equation (4) to reach the minimal is expressed as follows:(5)vkl=∑i=1nuikzxil∑i=1nuikz,l=1,2,…,s;k=1,2,…,K,(6)uik=∑t=1KdPartxi,vk2dPartxi,vt21/z−1−1,i=1,2,…n;k=1,2,…,K.With the application of equations (5) and (6), the membership matrix U and center matrix V that meet the requirements of the accuracy of dam safety monitoring are obtained through iterative solution. The maximum membership of each measurement point to each cluster center is treated as the final clustering result.According to the membership relationship between the missing data measurement points and the central points, the missing positions of the measurement values are preprocessed as follows:(7)Xil∗=UikVkl,l=t∗,Xil∗=Xil,l=t,where Xil∗ denotes the measurement value of the ith measurement point at the time l after preprocessing, and Vkl indicates the measurement value of the ith center at the time l.The ultimate clustering result is determined by equation (6), and the missing data are preprocessed by equation (7) based on which the data missing from dam safety monitoring can be effectively processed.


## 2.2. Implementation Technology of the PDS-FCM-LSTM Processing Method for the Data Missing from Dam Safety Monitoring

On the basis of the abovementioned research, the bidirectional LSTM network is introduced in this section to establish the mapping relationship between the measurement values of measurement points under identical clustering conditions, based on which the PDS-FCM-LSTM processing method is proposed.


### 2.2.1. Construction of Mapping Relationship between Measurement Values of the Measurement Points under Identical Clustering Conditions

The advantage of the LSTM network is that it is capable of controlling the information transfer of time series data by introducing input gate, forget gate, and output gate to the unit [9, 25–28]. It is assumed that the data of dam safety measurement point i are missing from time j to time j+p, which require reprocessing. For the unit at time l, the data xil∗l<j of measurement point i are defined as output variables, while the data xml∗m≠i,l<j of other measurement points in the same cluster are treated as input variables. On this basis, the mapping relationship between the measurement values of the forward LSTM network is established, as shown in Figure 1.Figure 1
Schematic diagram of the LSTM unit.(1) Forget Gate. The output variable of the hidden layer hl−1 at time l−1 and the input variable xml∗ at time l are taken. Equation (8) is applied to construct the output variable of the forget gate fl at time l, which determines whether to maintain the state of the hidden unit cl−1 at the previous layer.(8)fl=σWf⋅hl−1,xml∗+bf,where fl represents the output variable after the processing by the forget gate at time l, hl−1 denotes the output variable of the hidden layer at time l−1, xml∗ indicates the input variable of each measurement point at time l after preprocessing, bf refers to the bias term of the forget gate, Wf stands for the weight matrix of the forget gate, and σ means the activation function of sigmod, which is expressed as follows:(9)σz=11+exp−z.(2) Input Gate. The information collected by the input gate is the same as that collected by the forget gate. Based on equations (9) and (10), the output variable of the input gate Rl at time l and the unit state cl at time l are determined.(10)Rl=σWR⋅hl−1,xml∗+bR,(11)cl=fl∗cl−1+Rl∗tanhWc⋅hl−1,xml∗+bc,where Rl represents the output value after processing by the input gate at time l, WR indicates the input gate weight matrix, Wc denotes the unit state weight matrix, cl means the unit state at time l, bR refers to the bias term of input gate, and bc stands for the bias term of unit state.(3) Output Gate. The output gate is jointly determined by the updated unit state cl, the output variable of the hidden layer hl−1 at time l−1, and the input variable xml∗ at time l. The output variable of the output gate ol at time l and the output variable of the hidden layer hl at time l can be determined as follows:(12)ol=σWo⋅hl−1,xml∗+bo,hl=ol∗tanhcl,where ol represents the output variable processed by the output gate at time l, Wo denotes the output gate weight matrix, and bo indicates the bias term of the output gate.To process the missing data of measurement value from timej to time j+p more reasonably, a new bidirectional LSTM network method is proposed to process the data from time j to time j+p. As shown in Figure 2, model training and validation are carried out on the measurement points before and after time j to time j+p, respectively, according to the aforementioned principles based on which the forward and reverse LSTM model parameters are obtained. Then, the remaining measurement point data of the complete information from time j to time j+p are input into the model. Finally, the calculation value of the measurement point with missing data corresponding to the period from time j to time j+p is obtained, so as to process the missing data bidirectionally.Figure 2
Bidirectional LSTM processing method and idea (the gray box represents the preprocessed value of discrete data and the red box represents the long-sequence missing data).


### 2.2.2. Implementation Method of the PDS-FCM-BILSTM Processing Method for Dam Missing Data

According to the PDS-FCM-LSTM method for dam missing data processing as proposed above, the implementation process is shown in Figure3.Step 1: input the missing dataset of the dam measurement pointStep 2: enter the PDS-FCM layer to preprocess the missing data, set fuzzy parameterz, cluster number K, and threshold εε>0, and then randomly initialize the partition matrix U0Step 3: when the number of iterations isnn≥1, equation (5) is used to update the prototype matrix Vn based on the partition matrix Un−1Step 4: update the partition matrixUn based on equation (6) and the prototype matrix VnStep 5: if the conditions∀k,i:maxuikn−uikn−1<ε are met, where k=1,2,…,K,i=1,2,…,n, the algorithm stops to output partitioning matrix U and prototype matrix V. Otherwise, if n←n+1, return to Step 3.Step 6: based on the membership relationship between the measurement points with missing data and each cluster centerU,V, the existing values in the cluster center are weighted and averaged to obtain the preprocessed data X∗Step 7: establish the bidirectional LSTM processing model for the missing data of the dam. The forward LSTM network model and reverse LSTM network model are constructed according to the time required to process the measurement values.Step 8: input the measurement values of each measurement point in the same cluster before and after the time slotT into the forward and reverse LSTM models, so as to obtain the weight W and bias b of each parameter after training and learningStep 9: input the preprocessed measurement point data as sample into the forward and backward LSTM models for obtaining the forward and backward output values at time slotT, with their mean values taken as the ultimate result of missing data processingFigure 3
PDS-FCM-LSTM concept.According to the above implementation steps, the implementation concept of PDS-FCM-LSTM is obtained.


### 2.2.3. Evaluation of the Missing Data Processing Effect

In this study, the average absolute error MAE and the average absolute percentage error MAPE are taken as the evaluation indicators of the data processing effect for the measurement points, which are defined as follows:(13)MAE=1X^m∑x^ij∈X^mrij−x^ij,(14)MAPE=1X^m∑x^ij∈X^mrij−x^ijrij,where X^m represents the set of all processing values, x^ij denotes the processing value, rij indicates the measurement value of the processing value, and x^ij corresponds to the measurement value of the measurement point with complete information.Using equations (13) and (14) and the error allowable values listed in the technical specifications for dam safety monitoring, the effect of missing data processing is thoroughly evaluated, and the processed data meeting the accuracy requirements are taken as the effective data for dam safety monitoring.


## 2.2.1. Construction of Mapping Relationship between Measurement Values of the Measurement Points under Identical Clustering Conditions

The advantage of the LSTM network is that it is capable of controlling the information transfer of time series data by introducing input gate, forget gate, and output gate to the unit [9, 25–28]. It is assumed that the data of dam safety measurement point i are missing from time j to time j+p, which require reprocessing. For the unit at time l, the data xil∗l<j of measurement point i are defined as output variables, while the data xml∗m≠i,l<j of other measurement points in the same cluster are treated as input variables. On this basis, the mapping relationship between the measurement values of the forward LSTM network is established, as shown in Figure 1.Figure 1
Schematic diagram of the LSTM unit.(1) Forget Gate. The output variable of the hidden layer hl−1 at time l−1 and the input variable xml∗ at time l are taken. Equation (8) is applied to construct the output variable of the forget gate fl at time l, which determines whether to maintain the state of the hidden unit cl−1 at the previous layer.(8)fl=σWf⋅hl−1,xml∗+bf,where fl represents the output variable after the processing by the forget gate at time l, hl−1 denotes the output variable of the hidden layer at time l−1, xml∗ indicates the input variable of each measurement point at time l after preprocessing, bf refers to the bias term of the forget gate, Wf stands for the weight matrix of the forget gate, and σ means the activation function of sigmod, which is expressed as follows:(9)σz=11+exp−z.(2) Input Gate. The information collected by the input gate is the same as that collected by the forget gate. Based on equations (9) and (10), the output variable of the input gate Rl at time l and the unit state cl at time l are determined.(10)Rl=σWR⋅hl−1,xml∗+bR,(11)cl=fl∗cl−1+Rl∗tanhWc⋅hl−1,xml∗+bc,where Rl represents the output value after processing by the input gate at time l, WR indicates the input gate weight matrix, Wc denotes the unit state weight matrix, cl means the unit state at time l, bR refers to the bias term of input gate, and bc stands for the bias term of unit state.(3) Output Gate. The output gate is jointly determined by the updated unit state cl, the output variable of the hidden layer hl−1 at time l−1, and the input variable xml∗ at time l. The output variable of the output gate ol at time l and the output variable of the hidden layer hl at time l can be determined as follows:(12)ol=σWo⋅hl−1,xml∗+bo,hl=ol∗tanhcl,where ol represents the output variable processed by the output gate at time l, Wo denotes the output gate weight matrix, and bo indicates the bias term of the output gate.To process the missing data of measurement value from timej to time j+p more reasonably, a new bidirectional LSTM network method is proposed to process the data from time j to time j+p. As shown in Figure 2, model training and validation are carried out on the measurement points before and after time j to time j+p, respectively, according to the aforementioned principles based on which the forward and reverse LSTM model parameters are obtained. Then, the remaining measurement point data of the complete information from time j to time j+p are input into the model. Finally, the calculation value of the measurement point with missing data corresponding to the period from time j to time j+p is obtained, so as to process the missing data bidirectionally.Figure 2
Bidirectional LSTM processing method and idea (the gray box represents the preprocessed value of discrete data and the red box represents the long-sequence missing data).


## 2.2.2. Implementation Method of the PDS-FCM-BILSTM Processing Method for Dam Missing Data

According to the PDS-FCM-LSTM method for dam missing data processing as proposed above, the implementation process is shown in Figure3.Step 1: input the missing dataset of the dam measurement pointStep 2: enter the PDS-FCM layer to preprocess the missing data, set fuzzy parameterz, cluster number K, and threshold εε>0, and then randomly initialize the partition matrix U0Step 3: when the number of iterations isnn≥1, equation (5) is used to update the prototype matrix Vn based on the partition matrix Un−1Step 4: update the partition matrixUn based on equation (6) and the prototype matrix VnStep 5: if the conditions∀k,i:maxuikn−uikn−1<ε are met, where k=1,2,…,K,i=1,2,…,n, the algorithm stops to output partitioning matrix U and prototype matrix V. Otherwise, if n←n+1, return to Step 3.Step 6: based on the membership relationship between the measurement points with missing data and each cluster centerU,V, the existing values in the cluster center are weighted and averaged to obtain the preprocessed data X∗Step 7: establish the bidirectional LSTM processing model for the missing data of the dam. The forward LSTM network model and reverse LSTM network model are constructed according to the time required to process the measurement values.Step 8: input the measurement values of each measurement point in the same cluster before and after the time slotT into the forward and reverse LSTM models, so as to obtain the weight W and bias b of each parameter after training and learningStep 9: input the preprocessed measurement point data as sample into the forward and backward LSTM models for obtaining the forward and backward output values at time slotT, with their mean values taken as the ultimate result of missing data processingFigure 3
PDS-FCM-LSTM concept.According to the above implementation steps, the implementation concept of PDS-FCM-LSTM is obtained.


## 2.2.3. Evaluation of the Missing Data Processing Effect

In this study, the average absolute error MAE and the average absolute percentage error MAPE are taken as the evaluation indicators of the data processing effect for the measurement points, which are defined as follows:(13)MAE=1X^m∑x^ij∈X^mrij−x^ij,(14)MAPE=1X^m∑x^ij∈X^mrij−x^ijrij,where X^m represents the set of all processing values, x^ij denotes the processing value, rij indicates the measurement value of the processing value, and x^ij corresponds to the measurement value of the measurement point with complete information.Using equations (13) and (14) and the error allowable values listed in the technical specifications for dam safety monitoring, the effect of missing data processing is thoroughly evaluated, and the processed data meeting the accuracy requirements are taken as the effective data for dam safety monitoring.


## 3. Numerical Example

### 3.1. Project Summary

The dam is a concrete parabolic double curvature arch dam [29–32] with the height of 240 m and the crest arc length of 774.69 m. It consists of 39 dam sections from left to right. In order to monitor the horizontal displacement of arch crown, arch shoulder, and 1/4 arch ring, 8 inverted plumb lines and 10 direct plumb lines are deployed in the dam body to form 5 groups of direct and inverted plumb lines, forming a total of 20 measurement points. The plumb survey points on the left shoulder of No.4 dam section are TCN01∼TCN02, TCN03∼TCN06 on No.11 dam section, TCN07∼TCN14 on No.19 ∼23 dam section, TCN015∼TCN18 on No.33 dam section, and TCN19∼TCN20 on the right shoulder of No.37 dam section. The specific distribution of the measurement points is shown in Figure 4.Figure 4
Layout of measurement points of a concrete arch dam.


### 3.2. The Data Processing

Based on the time series of complete measurement data on the deformation to plumb lines from January 1, 2014, to December 31, 2019, the experiment is designed. Take TCN11 as an example, and the measurement data (31 data in total) from October 27, 2016, to March 29, 2017, are purposely deleted to construct the incomplete dataset (the data of other measurement points are complete). The missing data are shown in Figure5.Figure 5
TCN11 measurement values.The PDS-FCM-LSTM method proposed in this study is adopted for data processing. First, the PDS-FCM data processing model is constructed for the dam deformation measurement points. According to the measurement values obtained from the dam deformation measurement points between January 1, 2014, and December 31, 2019, the partial distance between each measurement point and the membership matrixU is calculated when the minimum value is taken for the objective function Jλ. The results can be seen from Figure 6.Figure 6
Measurement points of each center cluster.
(a)(b)(c)(d)(e)Then, according to the clustering principle proposed in the second section, all measurement points are clustered. The clustering results are given in Tables1 and 2. It can be seen that the measurement points are divided into multiple clusters; TCN11 belongs to the second cluster, and TCN04, 05, and 16 are also one of them. According to equation (7), the preprocessing result of the measurement point TCN11 is shown in Figure 7.Table 1
Membership degree of each deformation measurement point to clustering center.
Measurement pointMembership of center 1Membership of center 2Membership of center 3Membership of center 4Membership of center 5TCN10.930.000.010.000.06TCN20.160.010.080.000.76TCN30.970.000.000.000.02TCN40.010.930.020.030.01TCN50.060.530.260.060.09TCN60.070.000.030.000.90TCN70.040.000.010.000.95TCN80.050.020.740.010.18TCN90.010.030.010.940.01TCN100.010.100.020.860.01TCN110.000.960.010.010.01TCN120.050.060.780.010.10TCN130.000.000.980.000.01TCN140.050.010.750.000.19TCN150.680.010.040.000.28TCN160.010.940.010.030.01TCN170.120.020.450.010.40TCN180.970.000.000.000.02TCN190.960.000.010.000.03TCN200.850.000.020.000.13Table 2
Clustering results of deformation measurement points.
ClusterMeasurement pointsCluster 1TCN01, TCN03, TCN15, TCN18, TCN19, TCN20Cluster 2TCN04, TCN05, TCN11, TCN16Cluster 3TCN08, TCN12, TCN13, TCN14, TCN17Cluster 4TCN09, TCN10Cluster 5TCN02, TCN06, TCN07Figure 7
Preprocessing results of measurement values at measurement point TCN11.Based on the above processing results, the data of measurement points TCN04, 05, 11, and 16 can be taken as the basic data to be used for secondary processing.Finally, based on the integrity of the measured values before October 27, 2016, and after March 29, 2017, after preprocessing, the bidirectional LSTM network is used to process the long-sequence missing data of the measurement point TCN11. To validate the missing data processing method proposed in this study, the data missing from measurement point TCN11 between October 27, 2016, and March 29, 2017, are exemplified. It is processed by using the PDS-FCM-LSTM processing method, the LSTM network of single measurement point, and PDS-FCM processing method, respectively. The results are shown in Figure8 and Tables 3 and 4.Figure 8
Comparison of different processing methods for TCN11 measurement values.Table 3
Results of missing data processing for TCN11 by different methods.
DateTrue value (mm)PDS-FCM (mm)LSTM (mm)PDS-FCM-LSTM (mm)2016/10/2726.3629.1929.2626.472016/11/326.5329.3029.9026.512016/11/1126.4529.4329.4826.432016/11/1726.2028.9228.5126.402016/11/2425.7928.4328.2926.142016/12/125.4027.8127.9625.932016/12/825.1927.5327.6725.632016/12/1525.1827.6227.6025.312016/12/2225.1627.6127.4125.132016/12/2925.2827.7427.0024.992017/1/525.4028.1926.6524.812017/1/1325.4028.2626.4724.722017/1/1925.2827.8026.3524.652017/1/2325.1427.6126.4524.562017/2/323.5525.1826.5024.262017/2/923.0124.3325.6323.882017/2/1622.6423.7425.2023.122017/2/2321.9322.8924.8522.512017/3/121.7522.2824.2822.532017/3/921.4822.1323.5122.672017/3/1621.2621.8722.6522.402017/3/2320.7421.0523.0322.322017/3/2920.4320.5021.7521.99Table 4
Evaluation of data processing results for TCN11.
Measurement pointProcessing methodMAEMAPETCN11#BILSTM2.210.08PDS_FCM1.910.07PDS-FCM-BILSTM0.580.03It can be seen from Figure8 and Tables 3 and 4 that the PDS-FCM-LSTM processing method proposed in this study achieves the highest processing accuracy because it gives full consideration to the correlation between measurement points and the pattern of changes in measurement values. PDS-FCM takes into account only the correlation between the measurement points, which makes the outcome of treatment less satisfactory. The single-point LSTM performs the worst in terms of the processing effect because it considers only the pattern of changes in measurement values while ignoring the correlation between the measurement points. In addition, it can be seen from Table 3 that the absolute errors of the outcome of PDS-FCM-LSTM treatment proposed in this study are all less significant than that specified for the water project (29), which demonstrates the effectiveness of the missing data processing method proposed in this study.


## 3.1. Project Summary

The dam is a concrete parabolic double curvature arch dam [29–32] with the height of 240 m and the crest arc length of 774.69 m. It consists of 39 dam sections from left to right. In order to monitor the horizontal displacement of arch crown, arch shoulder, and 1/4 arch ring, 8 inverted plumb lines and 10 direct plumb lines are deployed in the dam body to form 5 groups of direct and inverted plumb lines, forming a total of 20 measurement points. The plumb survey points on the left shoulder of No.4 dam section are TCN01∼TCN02, TCN03∼TCN06 on No.11 dam section, TCN07∼TCN14 on No.19 ∼23 dam section, TCN015∼TCN18 on No.33 dam section, and TCN19∼TCN20 on the right shoulder of No.37 dam section. The specific distribution of the measurement points is shown in Figure 4.Figure 4
Layout of measurement points of a concrete arch dam.


## 3.2. The Data Processing

Based on the time series of complete measurement data on the deformation to plumb lines from January 1, 2014, to December 31, 2019, the experiment is designed. Take TCN11 as an example, and the measurement data (31 data in total) from October 27, 2016, to March 29, 2017, are purposely deleted to construct the incomplete dataset (the data of other measurement points are complete). The missing data are shown in Figure5.Figure 5
TCN11 measurement values.The PDS-FCM-LSTM method proposed in this study is adopted for data processing. First, the PDS-FCM data processing model is constructed for the dam deformation measurement points. According to the measurement values obtained from the dam deformation measurement points between January 1, 2014, and December 31, 2019, the partial distance between each measurement point and the membership matrixU is calculated when the minimum value is taken for the objective function Jλ. The results can be seen from Figure 6.Figure 6
Measurement points of each center cluster.
(a)(b)(c)(d)(e)Then, according to the clustering principle proposed in the second section, all measurement points are clustered. The clustering results are given in Tables1 and 2. It can be seen that the measurement points are divided into multiple clusters; TCN11 belongs to the second cluster, and TCN04, 05, and 16 are also one of them. According to equation (7), the preprocessing result of the measurement point TCN11 is shown in Figure 7.Table 1
Membership degree of each deformation measurement point to clustering center.
Measurement pointMembership of center 1Membership of center 2Membership of center 3Membership of center 4Membership of center 5TCN10.930.000.010.000.06TCN20.160.010.080.000.76TCN30.970.000.000.000.02TCN40.010.930.020.030.01TCN50.060.530.260.060.09TCN60.070.000.030.000.90TCN70.040.000.010.000.95TCN80.050.020.740.010.18TCN90.010.030.010.940.01TCN100.010.100.020.860.01TCN110.000.960.010.010.01TCN120.050.060.780.010.10TCN130.000.000.980.000.01TCN140.050.010.750.000.19TCN150.680.010.040.000.28TCN160.010.940.010.030.01TCN170.120.020.450.010.40TCN180.970.000.000.000.02TCN190.960.000.010.000.03TCN200.850.000.020.000.13Table 2
Clustering results of deformation measurement points.
ClusterMeasurement pointsCluster 1TCN01, TCN03, TCN15, TCN18, TCN19, TCN20Cluster 2TCN04, TCN05, TCN11, TCN16Cluster 3TCN08, TCN12, TCN13, TCN14, TCN17Cluster 4TCN09, TCN10Cluster 5TCN02, TCN06, TCN07Figure 7
Preprocessing results of measurement values at measurement point TCN11.Based on the above processing results, the data of measurement points TCN04, 05, 11, and 16 can be taken as the basic data to be used for secondary processing.Finally, based on the integrity of the measured values before October 27, 2016, and after March 29, 2017, after preprocessing, the bidirectional LSTM network is used to process the long-sequence missing data of the measurement point TCN11. To validate the missing data processing method proposed in this study, the data missing from measurement point TCN11 between October 27, 2016, and March 29, 2017, are exemplified. It is processed by using the PDS-FCM-LSTM processing method, the LSTM network of single measurement point, and PDS-FCM processing method, respectively. The results are shown in Figure8 and Tables 3 and 4.Figure 8
Comparison of different processing methods for TCN11 measurement values.Table 3
Results of missing data processing for TCN11 by different methods.
DateTrue value (mm)PDS-FCM (mm)LSTM (mm)PDS-FCM-LSTM (mm)2016/10/2726.3629.1929.2626.472016/11/326.5329.3029.9026.512016/11/1126.4529.4329.4826.432016/11/1726.2028.9228.5126.402016/11/2425.7928.4328.2926.142016/12/125.4027.8127.9625.932016/12/825.1927.5327.6725.632016/12/1525.1827.6227.6025.312016/12/2225.1627.6127.4125.132016/12/2925.2827.7427.0024.992017/1/525.4028.1926.6524.812017/1/1325.4028.2626.4724.722017/1/1925.2827.8026.3524.652017/1/2325.1427.6126.4524.562017/2/323.5525.1826.5024.262017/2/923.0124.3325.6323.882017/2/1622.6423.7425.2023.122017/2/2321.9322.8924.8522.512017/3/121.7522.2824.2822.532017/3/921.4822.1323.5122.672017/3/1621.2621.8722.6522.402017/3/2320.7421.0523.0322.322017/3/2920.4320.5021.7521.99Table 4
Evaluation of data processing results for TCN11.
Measurement pointProcessing methodMAEMAPETCN11#BILSTM2.210.08PDS_FCM1.910.07PDS-FCM-BILSTM0.580.03It can be seen from Figure8 and Tables 3 and 4 that the PDS-FCM-LSTM processing method proposed in this study achieves the highest processing accuracy because it gives full consideration to the correlation between measurement points and the pattern of changes in measurement values. PDS-FCM takes into account only the correlation between the measurement points, which makes the outcome of treatment less satisfactory. The single-point LSTM performs the worst in terms of the processing effect because it considers only the pattern of changes in measurement values while ignoring the correlation between the measurement points. In addition, it can be seen from Table 3 that the absolute errors of the outcome of PDS-FCM-LSTM treatment proposed in this study are all less significant than that specified for the water project (29), which demonstrates the effectiveness of the missing data processing method proposed in this study.


## 4. Conclusion

(1)
To solve the problem that the traditional distance measurement indicators are ineffective in measuring the correlation between the measurement points when the dam suffers incomplete monitoring information, the method for characterizing the correlation between survey points is studied and proposed on the basis of partial distance. Then, the construction technology of the PDS-FCM clustering model is studied, which achieves the preprocessing of data missing from dam safety monitoring and lays a foundation for the effective processing of missing data.(2)
Considering the correlation and trend of the changes in effect size for monitoring, a PDS-FCM-LSTM processing method intended for the data missing from dam safety monitoring is proposed. It combines the advantages of PDS-FCM and LSTM. The effectiveness of the proposed method is verified in practical engineering application. As the proposed method is universally applicable, it is suited for missing data processing in similar projects.



---

*Source: 9950874-2021-07-03.xml*