## 3 Approach
### Window Self-Attention  
Unlike the traditional way of directly using \(X=\{X_{1},X_{2},X_{3},\cdots,X_{T}\}\) as the input, we first use a 1D convolutional network to convert the dimension of the input \(X\in R^{T\times C}\) into the dimension we need \(R^{T\times C}\longrightarrow R^{T\times D}\), where T denotes the temporal length and C and D represent the channel dimensions. This operation also contributes to the stability of the training process [37], represented as \(X=\{X_{1},X_{2},X_{3},\cdots,X_{T}\}\stackrel{{ ConvD}}{{ \Longrightarrow}}Z=\{Z_{1},Z_{2},Z_{3},\cdots,Z_{T}\}\). The core part in the Transformer [14] network is the self-attention. We inserted a normalization block (Pre-LN [38]) before the self-attention block to remove the learning rate warm-up stage. In order to perform the attention function, the traditional way uses linear projection to generate values (V), queries (Q), and keys (k).These projections are parameter matrices with parameters \(W_{Q}\in R^{D\times d_{q}}\), \(W_{K}\in R^{D\times d_{k}}\), \(W_{V}\in R^{D\times d_{q}}\), used to learn linear projections of features \(Z\in R^{T\times D}\) to \(Q\in R^{T\times d_{q}}\), \(K\in R^{T\times d_{k}}\), \(V\in R^{T\times d_{q}}\), where \(d_{q}\), \(d_{k}\), \(d_{v}\) denote the channel dimensions. However, we found that it was beneficial to replace with 1D depthwise convolution (DC), which was implemented by using a layer 1D group convolution with a kernel size of 3 and group numbers the same as the channel dimension. A main advantage of the self-attention block is the ability to capture global-range dependencies across the full sequence. However, this advantage comes at the cost of introducing noise and increasing computation when the temporal length exceeds a certain range. Inspired by the local self-attention from Longformer [39] and Actionformer [18], we adopted the window self-attention mechanism to get rid of the impact of a long sequence. For features at location i \(\in[1,\text{T}]\), we call it a token. This gives a sequence of arbitrary length, and our window self-attention pattern uses a fixed-range attention around each token. For a specified window size \(\omega\), we evenly divided the input sequence into T/\(\omega\) attention chunks. A chunk is denoted as \(\Phi_{i}=\{\phi_{i},\varphi_{i},\psi_{i}\},i\in[1,T/\omega]\), where \(\phi_{i}\)\(\varphi_{i}\) and \(\psi_{i}\) are keys (\(K_{i}\)), queries (\(Q_{i}\)), and values (\(V_{i}\)) of the feature sequence separately. For a chunk region \(\Psi_{i}\in[t_{s},t_{e}]\), we denote its region \(R_{K}=[t_{s},t_{e}]\), \(R_{Q}=[t_{s}-\omega,t_{e}+\omega]\), \(R_{V}=[t_{s}-\omega,t_{e}+\omega]\) separately, where \(\omega=t_{e}-t_{s}\) (see Figure 2). In this way, the range of attention is limited within a chunk. At stage j \(\in[1,\text{L}]\), the range of attention is \(\times\)\(\omega\). Stacking multiple windowself-attention layers naturally integrates short-, medium-, and long-range features, which is beneficial to multi-scale prediction. \(V\), \(Q\), and \(K\) are computed as  
\[Q =Dw\_Conv1D(Z),\ group=in\_channel,\ k=3 \tag{1}\] \[V =Dw\_Conv1D(Z),\ group=in\_channel,\ k=3\] (2) \[K =Dw\_Conv1D(Z),\ group=in\_channel,\ k=3 \tag{3}\]  
We used projection \(Q\) and \(K\)-\(V\) pairs to compute the window attention of each chunk, and the weight assigned to \(V\in R^{3w\times d_{w}}\) is computed by matrix multiplication \(Q\mathbf{K}^{\top}\), where \(\mathbf{K}^{\top}\) represents the transpose of K, \(Q\in R^{\alpha\times d_{l}}\), \(K\in R^{3w\times d_{k}}\). The attention of the full sequence is obtained via the temporal sliding window approach with window size \(3\omega\) and stride \(\omega\). The result of window self-attention is calculated as a weighted sum of V:  
\[H=Attention(Q,K,V)=[Softmax(\frac{Q_{i}\mathbf{K_{i}}^{\top}}{\sqrt{d_{q}}})V _{i}]_{1}^{T/\omega},\ H\in R^{T\times D} \tag{4}\]  
Figure 1: Overview architecture of GLFormer.