## 2 HST-Net Model
### Backbone Network with Transformer-CSP
#### 2.1.3 Transformer-CSP Implementation and Multi-Head Attention Layer  
Inspired by the ViT proposed by [PERSON] et al. [35], we try to apply a transformer to the YOLO algorithm and introduce the self-attention mechanism. The image is divided into patches of the same size, and a class representation is added before the patch, which corresponds to the global information. The machine can notice the correlation between different parts of the whole input while maintaining the output structure consistent with the C3 module. The mathematical expression of input X is shown in Formula (1), H is the height of the divided patch, W is the width of the divided patch, and C is the number of channels.  
\[\mathrm{x}\ \in\ \mathrm{R}^{\mathrm{H}\ \times\ \mathrm{W}\ \times\  
The second sub-layer is the full connection layer, which is connected with all nodes of the previous layer to synthesize the features extracted from the front. The reason for using multiple attention is to make the proposed model focus on different aspects of information from multiple subspaces. The decoder layer is composed of self-attention layer, attention layer and feedforward neural network. This structure can make the network invest more attention resources in the bearing area so as to obtain more information in the defect area and suppress other useless information. It allows us to quickly screen out valuable information from a large amount of information. Embedded patches first perform the layer norm processing, then performs the multiple-headed self-attention operation, and last performs the residual add operation. The calculation formula of self-attention is as follows.  
\[\text{Attention}(\text{Q},\text{K},\text{V})=\text{softmax}\left(\frac{\text{ QK}^{\text{T}}}{\sqrt{\text{d}_{\text{k}}}}\right)\text{V} \tag{5}\]  
Output of the multi head attention module is obtained by linear transformation of the input vector X, and the matrix W multiplied by them respectively is the matrix obtained by training. The similarity degree of Q and K can be obtained by the transposed point multiplication of Q and K, and divided by dk to keep the gradient stable during training. Use softmax to normalize to get the weight matrix, and use this weight matrix and V to weight. The reason for dividing the weight matrix is to solve the problem that the gradient of softmax disappears when the two vectors are large. The h matrices obtained in Figure 4 are spliced to obtain the large characteristic matrix, and the linear transformation is performed with the matrix W, and the results of multiple sub-layers are mapped to the original space to obtain Z.  
Next, carry out layer norm processing, and then use a multi-layer perceptron, that is, MLP layer, to learn from the residual network to carry out the residual add operation to prevent degradation. The proposed Transformer-CSP module removes the operation of Norm in the original Vision transformer structure, and the MLP module does not use the activation function but directly operates on two full-connection layers. In order to avoid the gradient dispersion problem when increasing the depth of the network, the residual connection is used between each sub-layer, which makes it easier to train the depth network. The residual mapping is also easy to capture the subtle fluctuations of the identity mapping. The encoder module increases the ability to capture different local information, and it can use the self-attention mechanism to explore the potential of feature representation.  
The innovative design of the Transformer-CSP backbone network, with its adaptability to multi-scale object detection and its capacity to handle multi-class object detection tasks, underscores its efficacy as a potent tool across a diverse array of detection tasks.  
Figure 4: Structure of multi-head attention module.