## 3 Methods
### Detection Model for Apparent Defects in Metro Tunnel Liniings
#### 3.1.3 Bottleneck Transformer 3  
When convolutional neural networks are used to extract features from images of apparent defects in metro tunnel linings, the receptive field during convolution and pooling operations on high-dimensional feature maps is large. This scheme can reduce the resolution of the feature maps, limiting the number of pixels available for detecting minor defects in high-dimensional feature maps, which restricts the detection accuracy. In the context of metro tunnel linings, the locations of apparent defects and their background areas still have connections with distant bolt holes, grouting holes, and auxiliary facilities. Integrating the transformer algorithm [34] during the feature extraction stage can effectively help the model extract global contextual information from images. This approach allows the model to capture global feature dependencies, learn the feature associations between defect-prone areas, their background information, and distant targets, and better understand the overall scene and defect occurrence characteristics.  
In this study, a global contextual feature extraction module named BoT3 (Bottleneck Transformer 3) [35] is incorporated into the high-dimensional feature extraction part of the model. As shown in Figure 5, the input feature map generates half the target feature layers by using two Conv modules, with one Conv output fed into the BoT (Bottleneck Transformer) module for self-attention mechanism computation. The feature maps are then stacked, and a Conv calculation is performed to output the target feature map. In the BoT calculation, the dimensional input feature layers are first reduced using a contraction operation and input into the MHSA (Multi-Head Self-Attention) [35; 36] computation module. After obtaining the output, the feature dimensions are expanded back to their original size and summed with the original input feature layers to produce the final output.  
The computational flow of the MHSA is shown in Figure 6. The dimensions of the input feature map \(X\) are \(H\times W\times d\), which denote the height and width of the input feature matrix and the dimensions of individual tokens, respectively. The input \(X\) is subjected to convolution operation to obtain \(W_{Q}\), \(W_{K}\), and \(W_{V}\). The relative position encoding of the height and width of the input feature map are \(R_{h}\) and \(R_{w}\), respectively, which are initialized as trainable parameters and summed up by the broadcasting mechanism, i.e., the two-dimensional (_i,j_) position is encoded as \(R_{hi}+R_{wj}\) (d-dimensional vectors are summed up). The final output \(r\) represents the position encoding. Matrix multiplication of \(qr\) is carried out to obtain content-position output \(qr^{T}\), and matrix multiplication of \(qk\) is executed to obtain content-content output \(qk^{T}\). Matrix addition of \(qr^{T}\) with \(qk^{T}\) is conducted, softmax normalized exponential processing is applied to the resulting matrix, and the matrix format of the processed output values is \(HW\times HW\). Finally, the output value is a matrix multiplied by \(V\) weights to obtain the output \(Z\).  
Figure 5: BoT3 module.  
In Figure 6, \(q\), \(k\), and \(v\) represent queries, keys, and weights, respectively; + and \(\times\) represent element-by-element summation and matrix multiplication, respectively; and \(1\times 1\) represents point-by-point convolution. The blue parts represent position encodings and value projections, respectively.  
The BoT3 module can capture information in feature maps via global attention. In this study, the BoT3 module is embedded after the spatial pyramid pooling fast (SPPF) operation to avoid large-scale computation and memory usage. This module can increase the connections between defect-prone areas and other target areas in the high-dimensional feature layers' global features, thereby uncovering the correlations between apparent defects in the linings and other targets and obtaining more feature information.