## 2 Materials and Methods
### Methods
#### 2.2.1 Overview
Figure 7: The DWFE model. Convolution kernel sizes of 3 \(\times\) 3 and 5 \(\times\) 5 were used to construct two feature extraction modules to capture feature information at different scales. The 3 \(\times\) 3 convolution kernel excelled at extracting fine-grained local features, such as minor fluctuations and initial anomalies, while the 5 \(\times\) 5 convolution kernel was better suited for capturing broad fault evolution trends and multidimensional interactive characteristics. The combination of the two enabled collaborative learning of local and global features. \[Block_{3\times 3}(X_{i}) =Conv_{1\times 1}(DWConv_{3\times 3}(Conv_{1\times 1}(X_{i}))) \tag{4}\] \[Block_{5\times 5}(X_{i}) =Conv_{1\times 1}(DWConv_{5\times 5}(Conv_{1\times 1}(X_{i})))\] (5) \[X_{2} =Block_{3\times 3}\Big{(}X_{gaf}\Big{)}\] (6) Figure 5:0

Specifically, each module extracted spatial dimension features through 3 \(\times\) 3 or 5 \(\times\) 5 depthwise convolution, combined with batch normalization and ReLU to enhance the model's stability and nonlinear representation capability. To further improve feature extraction efficiency and information fusion capability, 1 \(\times\) 1 pointwise convolution was introduced before and after the depthwise convolution operation to perform feature channel upscaling and downscaling. This design not only effectively reduces the number of parameters and computational overhead but also significantly enhances the perception of features at different scales. #### 2.2.4 Self-Attention with 2D RoPE

* Preliminaries

In the ViT, the input RGB image is assumed to be square, with both its width and height equal to w. The image is uniformly divided into fixed-size patches of \(p\times p\). Figure 5:1

n\({}_{\text{p}}\) represents the total number of image patches obtained after dividing the entire image into patches of size \(p\times p\). Each \(p\times p\times c\) patch is flattened into a \(c\times p^{2}\) dimensional vector and projected linearly into R\({}^{\text{h}}\), resulting in an h dimensional Patch Token vector. In the ViT, an additional CLS Token is introduced. A zero-initialized h dimensional classification token is added to the beginning of the Patch Token sequence, extending the shape of the input matrix to \(\big{(}1+n_{p}\big{)}\times h\). A zero-initialized positional encoding is added to the matrix, keeping its dimensions unchanged at \(\big{(}1+n_{p}\big{)}\times h\). The input \(X\in R^{(1+n_{p})\times h}\) is mapped to query, key, and value vectors through linear transformations. Figure 5:2 (9) Figure 5:3 (10) Figure 5:4 (11) Figure 5:5 is a learnable parameter matrix. Figure 5:6

SoftMax normalizes the matrix row-wise, where \(d_{k}\) is the dimension of the key vector. The above attention operation is independently repeated multiple times. Assuming each independent attention operation is denoted by head\({}_{h}\), and h represents the number of attention heads, the output of the multi-head attention mechanism is as follows:

Figure 5:7

Figure 5:8

\(W_{Q}^{h},W_{K}^{h},W_{V}^{h}\in R^{(1+n_{p})\times d_{k}}\) is also a learnable parameter matrix, used for the projection of each head in multi-head attention and the final output fusion. * Two-dimensional Rotary Position Embedding

RoPE [48] introduces the multiplication mechanism of [PERSON]'s formula \(e^{i\theta}\) to directly embed relative position encoding into the query vector \(q_{n}\) and the key vector \(k_{m}\). Specifically, RoPE does not simply adjust the attention matrix after the query and key similarity calculation; instead, it directly acts on the query and key vectors, allowing position encoding to play a role in the core process of similarity computation. The derivation of the following formulas was referenced from literature [42].