## 2 Related Work
### Attention Network  
Attention [16] was introduced in the field of natural language processing with the presentation of transformer architecture. The input sequence is divided into a Query, Key, and Value to calculate the relevance of each element with respect to every other element. This procedure is referred to as self-attention. The following formula represents the objective function for linear operations with the elements of the input sequence: Query, Key, Value, and dimensions \(\text{d}_{\text{k}}\) and V.  
\[\text{Attention}(\text{Q},\text{K},\text{V})=softmax\left(\frac{QK^{T}}{ \sqrt{d_{k}}}\right) \tag{1}\]  
As an extension of this concept, Multi-Head Attention (MHA) is designed to extract various features and compute their relevance to the input sequence. Through this process, the relationship between the input and output can be determined. Finally, positional encoding is performed to restore the order of the previously divided input sequence, allowing the network to determine the sequence information of the input and relative positions of the feature elements. The formula for MHA calculation is as follows:  
\[\text{MHA}(Q,\ K,\ V)=Concat(head_{1},\ \cdots,\ head_{h})W^{O} \tag{2}\]  
\[head_{i}=\text{Attention}\Big{(}\text{Q}W_{i}^{Q},\text{K}W_{i}^{K},\text{V} W_{i}^{V}\Big{)} \tag{3}\]  
In recent years, attention mechanisms have demonstrated excellent performance in various fields, including segmentation. By mimicking the human perception of visual information, the attention mechanism selects important and useful information from large amounts of data. This is a reasonable approach for detecting cracks involving unspecified and diverse variables. [PERSON] et al. [17] introduced a full-attention U-Net, which utilizes an attention mechanism and skip connections. The distinguishing feature of this network is the application of attention blocks to all layers by incorporating attention networks in the decoder as well as the encoder. Although this approach exhibits slightly lower performance compared to the  
Figure 1: Residual block example. case where an attention network is only used in the decoder, it achieves higher performance when noise removal strategies are applied. [PERSON] et al. [18] proposed the SegCrack model, which adopts a hierarchical transformer architecture to output multistage features and alleviates class imbalances by applying an ascending path, lateral connections, and the OHEM strategy. By subsampling advantageous pixel features for losses, it enhances crack detection performance, achieving an F-1 Score and mIoU of 96.05% and 92.63%, respectively, outperforming CNN-based networks. [PERSON] et al. [19] introduced the DMA-Net, which incorporates a multi-scale attention module into the decoder component of the DeepLabv3+ model, thereby improving the integration of multi-scale features present in irregularly shaped cracks. They conducted extensive experiments using three datasets, demonstrating the practicality of the proposed method even with smartphone-captured pavement images. Research utilizing vision transformers (ViTs), an extended form of the attention mechanism, has also been actively pursued. [PERSON] et al. [20] proposed a convolution-transformer network for crack segmentation. This network--which included extended residual blocks, boundary-aware modules, and MobileViT blocks--was experimentally demonstrated to outperform previously developed methods. [PERSON] et al. [21] proposed a patch-based semantic segmentation method that trains by obtaining image-level labels instead of pixel-level annotations by dividing images into patch units, creating synthetic labels through identification/location algorithms, and performing CRF processing to reduce method complexity. [PERSON] et al. [22] proposed PCTC-Net. This parallel dual-encoder network, which integrates pre-conv-based transformers and CNNs (PCTC-Net), addresses data-intensive problems with relatively small datasets, demonstrating higher generalizability than the DTC-Net state-of-the-art (SOTA) model. [PERSON] et al.