# A Vision-Based Bolt Looseness Detection Method for a Multi-Bolt Connection
## 2 Methodology
### Rnn  
In our previous work of detecting the looseness value of a single track fastener [25], we employed a modified VGG-16 model to classify the speckle fringe patterns of one track fastener, given its great performance in image classification [28]. The modified VGG-16 model achieved a single-judgment time of 2.03 millisecond with an accuracy of 92.57% for a single fastener. The accuracy is not high enough, and worse, the training process takes lots of time. There are two reasons for this: Firstly, the model has too many parameters, a single iteration takes a long time. Secondly, the model is difficult to converge and requires too many iterations. These make the training process slow and difficult. On the other hand, compared to previous work, the captured fringe patterns in this work showed much stronger time series characteristics, as shown in Section 3.2. These motivated us to look for a new deep neural network (DNN) that is more suitable for this situation.  
In 1982, [PERSON] proposed a single-layer feedback neural network [29], named the Hopfield network, to solve combinatorial optimization problems. This was the earliest incarnation of the RNN. In the decades since, RNNs have been continuously refined and have developed into a deep learning algorithm in the 21 st century [30]. RNNs have demonstrated their remarkable performance in processing temporal sequences [7]. They are  
Figure 2: (**a**) Deformation of the surface; (**b**) the derivative of the out\(-\)of\(-\)plane displacement along the \(\mathrm{x}-\)direction \(\frac{\partial\omega}{\partial x}\); (**c**) corresponding speckle fringe pattern.  
suitable for processing time sequence data; they recurse along the development direction of the data, and all the network's nodes are connected with each other in a chain structure. The core structure of an RNN consists of a hidden state vector that is updated with each time step of the input sequence, thus conveying information at different time steps. Through this mechanism, RNNs are able to capture long and short temporal dependencies in sequence and show excellent performance in speech recognition, natural language processing, time series prediction [30], and image classification [31], etc. In this paper, we employed an RNN model to classify these fringe patterns, given its great ability to handle temporal sequences. Figure 3 is a simplified schematic of an RNN.  
Given a data sequence \(x=(x_{1},\ x_{2},\ \cdots,x_{t},\ \cdots,x_{T-1},\ x_{T})\), \(x_{t}\) are the data at the \(t\)th time step (the \(t\)th speckle fringe pattern in this paper). The recurrent hidden states \(h_{t}\) update iteratively by  
\[h_{t}=\left\{\begin{aligned} 0,\ if\ t=0\\ tanh(h_{t-1},x_{t}),\ otherwise\end{aligned}\right. \tag{3}\]  
and can be generally implemented in the RNN as  
\[h_{t}=f(W_{in}x_{t}+W_{S}h_{t-1}+b) \tag{4}\]  
where \(W_{in}\) are the coefficient matrices for the input at the present step, and \(W_{S}\) are the coefficient matrices for the activation of recurrent hidden units at the previous step. Generally, different layers share the same \(W_{S}\) to reduce the number of parameters. The corresponding output can be implemented as  
\[y_{t}=softmax\Big{(}W_{hy}h_{t}+b_{y}\Big{)} \tag{5}\]  
\(W_{hy}\) is the hidden layer to output layer weight matrix. Substituting Equation (5) into Equation (6), we obtain the following:  
\[y_{t}=softmax\Big{[}W_{hy}f(W_{in}x_{t}+W_{S}h_{t-1}+b)+b_{y}\Big{]} \tag{6}\]  
Equation (6) reveals that the output \(y_{t}\) depends not only on \(x_{t}\) but also on \(h_{t-1}\) (\(x_{t-1}\)). This creates an iterative relationship between the hidden layer/output layer over time;  
Figure 3: Simplified schematic of an RNN.  
in other words, giving the network a \"memory\". That is why the RNN has remarkable performance with temporal sequences.