# N-Trans: Parallel Detection Algorithm for DGA Domain Names
## 3 N-Trans
### Parallel Training
#### 3.2.2 Transformer Model  
The pre-processed phrase elements are fed into the Transformer model. The transformer model [17] is the encoder-decoder model that incorporates the attention mechanism. The common encoder-decoder framework is implemented using RNN networks, but is usually inefficient because its reliance on sequential order features prevents training from progressing in parallel. In the Transformer model, the data can be processed in parallel, and both input and output are from the same sequence, which captures the global information well.  
This paper uses the encoder part of the Transformer structure for model training. The parameters of the Transformer structure are as follows: the input dimension of the token is the number of samples (the token is the key parameter in the Transformer model and each domain name is token), and the output dimension is 64; the input dimension of the position part is the number of N-gram-processed features, and the output dimension is 64; the head number of the multi-head Attention is five. There is no sequential structure in the Transformer model like in RNN, so the position information of the elements is needed, and the position information is calculated as shown in Equation (4).  
\[P(pos,n)=\begin{cases}\sin(\frac{pos}{2}),n=2i\\ \cos(\frac{1000Q^{2}_{model}}{2}),n=2i+1\end{cases}, \tag{4}\]  
where \(pos\) is the position of the letter in the domain string, \(i\) is the dimension of the vector, and \(d_{model}\) is the dimension of _token_, which is processed using the \(\cos\) function when \(i\) is odd and the \(\sin\) function when \(i\) is even.  
The core of the Transformer is the multi-head attention mechanism, which consists of several self-attentive mechanisms. The self-attention mechanism is a \(Q(Query)\)-vector of a phrase element in the domain name and \(K(Key)\)-vectors of other phrase elements in the domain name multiplied one at a time to get the initial weights. After processing, it is multiplied with \(V(Value)\) to obtain the sum of weights, as shown in Equation (5).  
\[Attention(Q,K,V)=softmax(\frac{QK^{T}}{\sqrt{d_{k}}})V, \tag{5}\]  
where \(d_{k}\) is the dimension of \(K\), and the \(softmax\) function normalizes the result. N-gram treats domain names as collections of phrase elements. In the study of malicious domain name detection, \(Q\) is the phrase element in the domain name, \(K\) is the other phrase element in the domain name, and \(V\) is the vector representation of the phrase element after N-gram processing. Multiple linear mappings of the inputs \(Q\), \(K\), \(V\) are performed to obtain the attention \(head_{i}\), as shown in Equation (6).  
\[head_{i}=Attention(QW_{i}^{Q},KW_{i}^{K},VW_{i}^{V}). \tag{6}\]  
\(W_{i}^{Q}\), \(W_{i}^{K}\), and \(W_{i}^{V}\) are the parameter matrices when linearly varied. The final result of multi-head attention is obtained by splicing the multiple \(head_{i}\) for linear variation, as shown in Equation (7).  
\[MultiHead(Q,K,V)=Concat(head_{1},\cdots,head_{n})W^{o}. \tag{7}\]In Equation (7), \(n\) denotes the number of heads in the multi-head attention mechanism, and \(W^{0}\) is the linearly varying parameter matrix.