## 3 Methodology
### Dual Attention Transformer Encoder  
The original encoder in Transformer, as shown in Figure 2a, is actually a stack of multi-head attention modules and feed-forward modules that are used for long-term relationship modeling and feature extraction, respectively. Inevitably, modeling long-term relationships between API call sequences requires attending to all API call sequences, thus somewhat suppressing the expression of short-term dependencies. Especially in the malware classification area, long- and short-term relationships matter equally, i.e., some malware can be classified by several distant or only several adjacent API calls. Consequently,  
Figure 1: Overview of our two-step methods.  
we propose to use lightweight local attention and incorporate it within a dual attention Transformer encoder, which is illustrated in Figure 2b.  
Unlike global attention, i.e., the multi-head attention in the Transformer, local attention attempts to learn the context information in a sliding window of 2D + 1. Given the embedded API call sequence matrix \(X\in R^{N\times M}\), where N denotes the number of the API call and M denotes the feature size of a single API call, \(x\in\mathbb{R}^{1\times M}\), the local attention aggregates the context information \(c_{t}\) of the current API call \(x_{t}\) using Equation (1).  
\[c_{t}=\sum_{i=t-D}^{t+D}\alpha_{ti}x_{i} \tag{1}\]  
where \(\alpha_{ti}\) denotes the weight of \(x_{i}\) with respect to \(x_{t}\) and can be expressed as Equation (2).  
\[\alpha_{ti}=\frac{\exp(score(x_{t},x_{i}))}{\sum_{m=t-D}^{t+D}\exp(score(x_{t}, x_{m}))} \tag{2}\]  
where \(score(\cdot,\cdot)\) is the sum of the inner product of the variable in this paper. Finally, the context information \(c_{t}\) is regarded as the local attention feature and replaces \(x_{t}\).  
Based on the local attention proposed above, we further propose the new structure as a dual attention Transformer (DAT). Similar to the Transformer encoder, the DAT takes the embedded features as inputs and outputs in a fixed size. What makes the difference is that there are two sub-branches in each encoder layer, which are in charge of the long- and short-term dependence modeling, respectively. The long-term part stays the same as the Transformer encoder, while the short-term part replaces the multi-head attention with the local attention proposed above. Input-embedded features are passed into the two sub-branches, and the successive operations can be formulated as follows:  
\[Y_{MA}=\emptyset(\varphi_{MA}(X;\theta_{1})+X;\theta_{2}) \tag{3}\] \[Z_{MA}=\emptyset(f_{FF}(Y_{MA};\theta_{3})+Y_{MA};\theta_{4})\]  
\[Y_{LA}=\emptyset\big{(}\varphi_{LA}(X;\theta_{1}^{\prime})+X;\theta_{2}^{ \prime})\]  
\[Z_{LA}=\emptyset\big{(}f_{FF}(Y_{LA};\theta_{3}^{\prime})+Y_{LA};\theta_{4}^{ \prime}) \tag{4}\]  
Figure 2: (**a**) Transformer encoder and (**b**) our dual attention Transformer encoder.  
where \(x\) denotes the input matrix and \(\varphi_{MA}\) and \(\varphi_{LA}\) refer to the multi-head and local attention layer, respectively. \(f_{FF}\) is the feed-forward layer. \(\theta\)s represent the parameters in (multi-head/local) attention layers, feed-forward layers, and residual and normalization layers. \(y\)s are the outputs of the attention layer followed by the Add & Norm layer, while \(z\)s are the outputs of the feed-forward layer followed by the Add & Norm layer.  
After that, outputs from the two sub-branches are concatenated, and then a max-pooling layer is applied, generating the final output of the encoder layer, which can be expressed as follows:  
\[\delta=MaxPool(z_{MA}||z_{LA}) \tag{5}\]  
where the \(||\) denotes the concatenation operation and \(\delta\) is the final output of the encoder layer.  
Thus, we can stack the DAT (dual attention Transformer) encoder layers to form a powerful feature extractor for malware classification.