# Pavement Distress Detection Using Three-Dimension Ground Penetrating Radar and Deep Learning
## 2 Proposed Approaches
### Proposed Classification Model
#### 2.3.2 Structure of SViT  
In this study, Simplify Vision Transformers (SViTs) were used to perform the classification task, which was a simplified version of Vision Transformer (ViT). An SViT can be separated into three modules, namely Embedding, Transformer Encoder, and MLP Head, as shown in Figure 12.  
As shown in Figure 12, the embedding module converted the input image into the data format required by the transformer encoder module. The input image was split into \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Category** & **HD1** & **HD16** & **HD45** & **HD6** & **HD126** \\ \hline \hline \multirow{4}{*}{
\begin{tabular}{c} Training \\ Testing \\ \end{tabular} } & Samples & 1072 & 2172 & 1366 & 2448 & 395 \\  & Data set & 400 & 400 & 400 & 400 & 395 \\  & Data set & 178 & 467 & 120 & 748 & 79 \\ \hline \hline \multicolumn{2}{c}{**Category**} & **HD1** & **HD16** & **HD45** & **HD6** \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Characteristics of three classification datasets. 196 blocks of 16 \(\times\) 16 pixels in the embedding module. The blocks were then flattened into one-dimensional vectors by a flattening layer to obtain 196 vector sequences with 256 dimensions. Finally, the position information is embedded in each vector sequence. (2) Transformer Encoder  
The transformer encoder module, as the core of the SViT model, extracted features from its input vector sequences. The architecture of the transformer encoder is shown in Figure 13. The transformer encoder consisted of two residual structures. The first residual structure was multi-head attention, while its second counterpart was multi-layer perceptron (MLP). Multi-head attention divided the input data into multiple parts to perform attention computation. In this study, the number of heads in multi-head attention was four, i.e., the vector sequence of 197 \(\times\) 256 was divided into four subsequences of 197 \(\times\) 64 and imported into the attention module. In the attention module, the vector sequence was further split into Query (Q), Key (K), and Value (V), and the results of attention and multi-head attention were computed as follows. \[\text{Attention}(\text{Q},\text{K},\text{V})=\text{softmax}\left(\frac{\text{ QK}^{\text{T}}}{\sqrt{\text{d}_{\text{k}}}}\right)\cdot\text{V}, \tag{3}\]  
\[\text{MultiHead}(\text{Q},\text{K},\text{V})=\text{Concat}(\text{head}_{1}, \ldots,\text{head}_{\text{h}})\cdot\text{W}^{\text{O}}, \tag{4}\]  
\[\text{head}_{i}=\text{Attention}\left(\text{QW}_{1}^{\text{Q}},\text{KW}_{1 }^{\text{K}},\text{VW}_{1}^{\text{V}}\right). \tag{5}\]  
Figure 12: Overall flow chart of SVIT. Figure 13: Schematic diagram of structure of transformer encoder. The number of neurons in the first fully connected layer of the MLP module was 1024, which was four times the size of the inputs, and the number of neurons in the second fully connected layer was 256, which was the same size as the inputs. The coefficient of the dropout was 0.1, and the GELU activation function [38] was adopted as follows. \[\text{GELU(x)}=\text{xP(X }\leq\text{ x)}=\text{x}\Phi(\text{x)}=\text{x}\cdot\frac{1}{2} \Big{[}1+\text{erf}\Big{(}\text{x}/\sqrt{2}\Big{)}\Big{]}, \tag{6}\]  
\[\text{GELU(x) }\approx\text{ 0.5x}\Big{(}1+\tanh\Big{[}\sqrt{2/\pi}\Big{(} \text{x}+0.0447\text{x}^{3}\Big{)}\Big{]}\Big{)}. \tag{7}\]  
The function of GELU was shown in Figure 14.  
The transformer encoder module was repeated several times to extract multi-level features. In this study, five transformer encoders with 3, 6, 9, 12, and 15 cycles were used, and the corresponding models were called SViT-3, SViT-6, SViT-9, SViT-12, and SViT-15, as shown in Figure 13.0  
(3) MLP Head  
Based on the features from the transformer encoder, the MLP Head classified the input image into one possible class. After completing feature extraction by the transformer encoder module, the output shape was 197 \(\times\) 256, which contained the class token embedded in the embedding module.