# FF-MR: A DoH-Encrypted DNS Covert Channel Detection Method Based on Feature Fusion
## 3 Background
### Multi-Head Attention Mechanism  
Attention mechanisms were first proposed in the field of image processing [15]. In 2014, the Google mind team combined an RNN and an attention mechanism and applied it to an image classification task [16]; this was then further developed and expanded by researchers. In different fields, many different attention mechanisms have evolved, including basic attention mechanisms, such as Soft Attention, Hard Attention, Self-Attention, etc., and combined attention mechanisms, such as Co-Attention, Attention-over-Attention, Multi-Head Attention [17]. Although the above attention mechanisms are different, the basic principles of implementation are similar. This section provides a brief overview of the Multi-Head Attention mechanism by summarizing the general implementation principles of the attention mechanism. In essence, the attention mechanism can be summarized as filtering out important and noteworthy information by computing the weight distribution of attention within or among data. It usually contains three variables, Query, Key, and Value \((\mathbf{Q},\mathbf{K},\mathbf{V})\), which represent the data encoding using target data, source data encoding, and content data encoding, respectively. The calculation process can be divided into two steps: one is to calculate the similarity between the target data \(\mathbf{Q}\) and the source data \(\mathbf{K}\), and the other is to calculate the new data representation \(\mathbf{V^{\prime}}\) based on the similarity and \(\mathbf{V}\):  
\[\mathbf{e}=g(f(\mathbf{Q},\mathbf{K})) \tag{1}\] \[\mathbf{V^{\prime}}=m(\mathbf{e},\mathbf{V}) \tag{2}\]  
where the similarity between \(\mathbf{Q}\) and \(\mathbf{K}\) is calculated by the energy function \(f\)[18] and the distribution function \(g\) to obtain the weight distribution of attention \(\mathbf{e}\). Later, using the  
Figure 1: Data leakage and command control process based on DoH-encrypted DNS covert channels. transformation function \(m\), the new data representation \(\mathbf{V}^{\prime}\) is obtained by multiplying \(\mathbf{e}\) with \(\mathbf{V}\). Usually, the distribution function \(g\) is chosen to be normalized by _softmax_, while the transformation function \(m\) is a weighted summation. The usual energy functions \(f\) include additive and dot product functions [19], which are calculated as follows:  
\[f(\mathbf{Q},\mathbf{K}) =\mathbf{v}^{T}\operatorname{act}(\mathbf{W_{1}}\mathbf{K}+\mathbf{W_{2}}\mathbf{Q}+ \mathbf{b}) \tag{3}\] \[f(\mathbf{Q},\mathbf{K}) =\mathbf{Q}^{T}\mathbf{K} \tag{4}\]  
where act is the nonlinear activation function, such as tanh and ReLU, etc. ; \(\mathbf{v}^{T}\) is the parameter vector; \(\mathbf{b}\) is the neuron bias; and \(\mathbf{W_{1}}\) and \(\mathbf{W_{2}}\) are the weight matrices. The Self-Attention mechanism was proposed in 2017 by [PERSON] et al. [20] for computing the correlation between words within a sentence to extract syntactic and semantic features. The difference compared to the general attention mechanism is that \(\mathbf{Q}=\mathbf{K}=\mathbf{V}\); that is, it only focuses on the interdependencies between elements within the data. The Multi-Head Attention mechanism belongs to a kind of combined attention mechanism, which can greatly improve the data fitting ability and enrich the feature representation by combining multiple attention mechanisms \(\mathbf{head_{i}}\) and jointly extracting information from different representation subspaces:  
\[ \begin{split}\text{MultiHead}(\mathbf{Q},\mathbf{K},\mathbf{V})& =\text{Concat}(\mathbf{head_{1}},\dots,\mathbf{head_{h}})\mathbf{W}^{o}\\ &\text{where }\mathbf{head_{i}}=m(g(f(\mathbf{Q},\mathbf{K})),\mathbf{V})\end{split}  \tag{5}\]  
It should be noted that the Multi-Head Attention mechanism used in our paper is combining multiple Self-Attention mechanisms. Nowadays, attention mechanisms are widely used in natural language processing [21], as well as in autonomous driving [22] and human-computer interaction [23], among others.