## 3 Proposed Methods
### Highway Crack Detection Methods
#### 3.1.3 MHSA Module  
As shown in Figure 5, the MHSA module is divided into two parts, embedded patches and encoder block, and the encoder part in this experiment consists of \(L=6\) encoders with the same structure. For the image data, the 3D matrix in the format \([H,W,C]\) is not the input format required by the encoder block, so the input data format needs to be transformed by the embedding layer. Then, using linear projection, each patch is flattened into a vector called tokens, and the class token with the same dimension as the tokens is spliced before the tokens, and, finally, position embedding is added to complement the position information to obtain the input required by the final encoder block. The position encoding formula can be found in [34]:  
\[ \begin{split} PE_{(pos,2i)}&=\sin\left(\text{pos} \,/10000^{2i/d_{\text{model}}}\right)\\ PE_{(pos,2i+1)}&=\cos\left(\text{pos}\,/10000^{2i /d_{\text{model}}}\right)\end{split}  \tag{2}\]  
where \(pos\) is the position, \(i\) is a certain dimension at that position, and \(d_{\text{model}}\) is the dimension of the embedding vector, where the number must be a multiple of the number of heads. The even dimension of a position uses a sine function, and the odd dimension uses a cosine function. From Figure 5, we can see that the tokens enter the encoder block and then pass through the multi-head self-attention (MSA) block and the MLP block in turn. The MSA block can capture different types of information from different perspectives. Specifically, the input feature map is divided into multiple subspaces, and each subspace is processed by a separate attention head. The outputs of all attention heads are concatenated and linearly transformed to obtain the final output. The MLP block is composed of two fully connected layers, GELU activation function and dropout. After the first fully connected layer, the dimensionality increases by a factor of four and the number of nodes becomes four times the original number, but after the second fully connected layer, the number of nodes returns to the original point. Dropout randomly sets a fraction of the input units to zero during training, which forces the network to learn more robust features. The attention formula is as follows:  
\[Attention(Q,K,V)=softmax(\frac{QK^{T}}{\sqrt{d_{k}}})V \tag{3}\]  
where \(Q\) is the query matrix used to calculate the attention weights, \(K\) is the key matrix used to encode the structural information, \(V\) is the value matrix used to weight the importance of each feature, and \(d_{k}\) is the dimension of the matrix \(K\). A self-attentive module has three inputs, which are a query matrix \(Q\), a key matrix \(K\), and a value matrix \(V\). As in Figure 5, the product of the transpose of matrix \(Q\) and matrix \(K\) is first computed to calculate the dot product of each query value and key value, and the larger the dot product result, the stronger the correlation between the two. Matrix \(A\) is obtained by dividing by \(\sqrt{d_{k}}\) and using the softmax activation function for the quotient. \(\sqrt{d_{k}}\) is divided because if the dot product is large, the gradient becomes small after softmax. The matrix \(A\) is multiplied by the matrix \(V\) to obtain the final output of the attention. Multi-head self-attention is implemented based on self-attention, which divides the model into multiple heads to form multiple subspaces, allowing the model to focus on different aspects of information and finally splicing the results of multiple heads simply by satisfying the following equation. \[ \begin{split}\text{MultiHead}(Q,K,V)&=\text{ Concat}(\text{head}_{1},\dots,\text{head}_{\text{h}})W^{O}\\ \text{where head}&=\text{Attention}\Big{(}QW^{Q}_{i },KW^{K}_{i},VW^{V}_{i}\Big{)}\end{split}