# Transmission Lines Insulator State Detection Method Based on Deep Learning
## 2 Materials and Methods
### BRA-CBS: Fused Attention Convolution Module  
As a conventional convolutional module, CBS only conducts simple convolution operations for feature extraction, lacking the ability to discern the importance of different features. However, insulator states such as self-explosion and breakage demand precise extraction of fine-grained features. Therefore, introducing a dynamic sparse attention mechanism into the CBS module during the feature fusion process can help the feature fusion network focus on target areas. By capturing long-range contextual dependencies and employing more flexible computation allocation and content awareness, the detection network can more efficiently acquire crucial information, enhancing the model's detection capabilities. The fusion convolution module BRA-CBS, equipped with a bi-level routing attention mechanism, is depicted in Figure 2.  
The structure of the BRA (bi-level routing attention) [23] modules with query-awareness is shown in Figure 3. The key idea is to filter out the least relevant key-value pairs on the coarse-grained region of the feature graph so that only a small portion of the routing region is retained and then apply fine-grained token-to-token attention to the concatenation of the remaining routing region. The following shows the computation of single-head self-attention for a single input. Figure 3: Module structure of BRA. Figure 2: Module structure of BRA-CBS. Region partition and input projection. Assumed to be a 2D input feature map \(X\in\mathbb{R}^{H\times W\times C}\), it is first divided into \(S\times S\) non-overlapping regions, each containing \(\frac{HW}{S^{2}}\) feature vectors. By reshaping \(X\) into \(X^{r}\in\mathbb{R}^{S^{2}\times\frac{HW}{S^{2}}\times C}\), then we obtain the query, key, and value tensor \(Q,~{}K,~{}V\in\mathbb{R}^{S^{2}\times\frac{HW}{S^{2}}\times C}\) by linear projections [24], as shown in Equation (1). \[Q=X^{r}W^{q},~{}K=X^{r}W^{k},~{}V=X^{r}W^{v} \tag{1}\]  
where \(W^{q},~{}W^{k},~{}W^{v}\in\mathbb{R}^{C\times C}\) are the projection weights of query, key, and value, respectively. Region-to-region routing with the directed graph. Then, we find the relations of attention (the regions that should be focusing for each given region) by constructing a directed graph. In concrete terms, we derive queries and keys of each region, \(Q^{r},~{}K^{r}\in\mathbb{R}^{S^{2}\times C}\), by applying the average number of pixels in each region on \(Q\) and \(K\), respectively. Then, we derive the matrix of correlation for each region, \(A^{r}\in\mathbb{R}^{S^{2}\times S^{2}}\), via matrix multiplication between \(Q^{r}\) and transposed \(K^{r}\), as shown in Equation (2). \[A^{r}=Q^{r}{(K^{r})}^{T},~{}A^{r}\in\mathbb{R}^{S^{2}\times S^{2}} \tag{2}\]  
Entries in the matrix, \(A^{r}\), measure how much two regions are semantically related. Then we keep only top-\(k\) connections for each region. Specifically, we derive a routing index matrix, \(I^{r}\in\mathbb{N}^{S^{2}\times k}\), with the row-wise top-\(k\) operator as shown in Equation (3). \[I^{r}=\text{topkIndex}(A^{r}) \tag{3}\]  
Consequently, the i-th row of \(I^{r}\) contains the \(k\) index of the most relevant regions for the \(i^{th}\) region. Token-to-token attention. Using the region-to-region routing index matrix \(I^{r}\), we can subsequently implement precise token-to-token attention mechanisms. For every query token within region \(i\), it will focus on all key-value pairs located within the set of \(k\) routed regions as indexed by \(I^{r}_{(i,1)^{r}},~{}I^{r}_{(i,2)^{r}},\ldots,I^{r}_{(i,k)}\). Yet, efficiently implementing this step presents a challenge, given that these routed regions are dispersed across the entire feature map. GPUs depend on coalesced memory operations, which load blocks of dozens of contiguous bytes at once.