# A Novel Decision Support System for Long-Term Management of Bridge Networks
## 2 Materials and Methods
### Deep Learning-Based Prediction of Deterioration Growth  
Past bridge inspection data, along with maintenance/repair information, constitute the basis of predicting future conditions of bridge elements or components. The condition of bridge elements innately possesses a significant amount of uncertainty, partly due to inaccurately entered or missing inspection records [26]. In the proposed methodology, a deep learning model tackles the uncertainty related to future condition prediction problems by using time history prediction on the NDE data. In the proposed methodology, the hybrid architecture of convolutional neural networks (CNN) and long-term short memory (LSTM) models is periodically trained on the historical NDE data. The proposed network architecture is called CNN-LSTM, a deep learning model that fuses CNN and LSTM to predict image-based information in a future time step [31]. LSTMs are a very promising solution to sequence- and time series-related problems [32]. They can effectively handle time lags between data points, as opposed to recurrent neural networks (RNNs) [33]. Hence, the data do not have to be collected at a fixed time step. As shown in Figure 8, a common LSTM architecture is composed of a cell (the memory part of the LSTM unit) and three regulators, usually called gates, maintaining the flow of information through input, output gate, and a forget gate [34]. These gates can learn which data in a sequence are important to keep or throw away. By doing that, it can pass relevant information down the long chain of sequences to make predictions. LSTMs have been successfully used in many real-life applications such as speech recognition [35], forecasting stock prices [36], and estimating cancer growth [37]. CNNs are special types of deep neural networks that work remarkably well on image data. These models have shown breakthrough performance in especially image classification tasks (e.g., 2012 ImageNet Competition) [39]. CNN-based image analysis of infrastructure damage has been vastly studied in the past [40; 41]. As shown in Figure 9, the CNN models are typically composed of convolution, activation, and pooling layers. In the convolution layers, the input images are multiplied by small distinct feature matrices (i.e., kernel) that are attained from the input images. By convolving images, simply similarity scores are assigned between every region of the image to find the distinct features. After convolution, the negative values of similarity in the image matrix are removed in the activation layer. Finally, the resultant image matrix is reduced to a very small size and added together to form a single vector in the pooling layer. Figure 8: Diagram of a typical LSTM cell [38] (when data are received, input gate controls what information is stored in the long-term state, forget gate determines how long the stored information is preserved across the time instances, and lastly, the output gate controls the requested information at a particular time step). The proposed hybrid architecture (CNN-LSTM) works as follows: CNN-LSTM inputs the NDE data as an image format and passes it through the convolutional layers of CNN, in which the underlying spatial features are extracted and stacked in a one-dimensional vector. Then, the LSTM cells receive these feature vectors and make a prediction at a point in time. While the CNN branch of the model extracts the spatial relationships, LSTM creates a temporal context. Thus, these hybrid architectures are often called spatial-temporal neural networks. An important advantage of CNN-LSTM is that a time series prediction of spatial data can be performed in a single, end-to-end system instead of using two separate suboptimal systems. In the example shown in Figure 10, crack images belonging to inspections at different years are processed first in the CNN layers. The spatial features of crack deterioration extracted from CNN are then stored in LSTM cells. The output gate of the cell gives the predicted condition of the crack at the requested point of time.