# FF-MR: A DoH-Encrypted DNS Covert Channel Detection Method Based on Feature Fusion
## 4 Method Design
### Model Development and Architecture
#### 4.3.2 Weighted Fusion of Session Statistical Features and Byte Sequence Features
Firstly, to improve the nonlinear expression of the network, the statistical features \(\mathbf{S_{i}}\) are input to a fully connected layer linear with the Sigmoid activation function and transformed into a two-dimensional word vector matrix through word embedding. It is concatenated with a multi-channel byte sequence feature matrix \(\mathbf{X^{\prime\prime}_{i}}\) extracted by the Residual Neural Network. Meanwhile, we adopt the LayerNorm normalization for the sake of faster convergence and consistent data distribution and then obtain the input \(\mathbf{U_{i}}\) of the Multi-Head Attention layer:  
\[\mathbf{U_{i}}=\text{LayerNorm}\big{(}\text{Concat}\big{(}\text{Embedding}(\text{ Linear}(\mathbf{S_{i}})),\mathbf{X^{\prime\prime}_{i}}\big{)}\big{)}. \tag{15}\]  
Secondly, Figure 7a [20] shows the weighted fusion of the session statistical features and byte sequence features using the scaled dot-product self-attention, which is more efficient compared to other Self-Attention mechanisms [40]. The scaled dot-product self-attention mechanism is implemented through \(\mathbf{Q}\) (Query), \(\mathbf{K}\) (Key), and \(\mathbf{V}\) (Value), which are linear transformations of \(\mathbf{U_{i}}\). Essentially, the attention weight distribution of \(\mathit{boldsymbloV}\) is determined by computing the similarity (multiplication) between \(\mathbf{Q}\) and \(\mathbf{K}\), where a larger weight indicates that a feature is more relevant to another feature and vice versa. Thus,  
Figure 5: Interpretability of distribution of attention in machine translation: (**a**) a distribution of attention; (**b**) another distribution of attention. Figure 6: Interpretability of distribution of attention for features of traffic: (**a**) a distribution of attention; (**b**) another distribution of attention. the statistical features \(\mathbf{S}_{i}\) and the multi-channel byte sequence features \(\mathbf{X^{\prime\prime}_{i}}\) are computed by a single-scaled dot-product self-attention mechanism to obtain the feature matrix of weighted fusion \(\mathbf{attention_{i}}\):  
\[\mathbf{Q} =\mathbf{U}_{i}\mathbf{W}_{i}^{\mathbf{Q}} \tag{16}\] \[\mathbf{K} =\mathbf{U}_{i}\mathbf{W}_{i}^{\mathbf{K}}\] (17) \[\mathbf{V} =\mathbf{U}_{i}\mathbf{W}_{i}^{\mathbf{V}}\] (18) \[\mathbf{attention_{i}} =\text{Attention}(\mathbf{Q},\mathbf{K},\mathbf{V})=\text{softmax}\Big{(} \frac{\mathbf{Q}\mathbf{K}^{\mathbf{T}}}{\sqrt{d_{k}}}\Big{)}\mathbf{V} \tag{19}\]  
where \(d_{k}\) is the dimension of \(\mathbf{K}\), whose main function is to avoid the inner product of \(\mathbf{Q}\mathbf{K}^{\mathbf{T}}\) being too large, and softmax normalizes the feature matrix of weighted fusion \(\mathbf{attention_{i}}\). The singular Self-Attention is concatenated to obtain the Multi-Head Attention mechanism, as shown in Figure 7b [20]. When calculating \(\mathbf{Q},\mathbf{K}\), and \(\mathbf{V}\) in the \(h\) heads Self-Attention mechanism, \(\mathbf{bolsymbolV}\), the initialized \(\mathbf{W}_{i}^{\mathbf{Q}}\), \(\mathbf{W}_{i}^{\mathbf{K}}\) are different, as is the similarity of \(\mathbf{Q}\) to \(\mathbf{K}\); therefore, the feature matrix of weighted fusion \(\mathbf{attention_{i}}\) also differs. MHA-Resnet combines the features of different representation subspaces by Concat. After linear transformation \(\mathbf{W^{O}}\), \(\mathbf{attention_{i}}\) is dimensionally reduced to MultiHead(\(\mathbf{U_{i}}\)) of the same dimension as the input \(\mathbf{U_{i}}\). After the residual connection and LayerNorm normalization, we obtain \(\mathbf{U_{i}^{\prime}}\):  
\[ \begin{split}\text{Mul}\,\text{ItiHead}(\mathbf{U_{i}})=\text{ Concat}(\mathbf{attention_{1}},\dots,\mathbf{attention_{h}})\mathbf{W^{O}}\\ \text{where }\mathbf{attention_{i}}=\text{Attention}\Big{(}\mathbf{U_{i} }\mathbf{W_{i}^{\mathbf{Q}}},\mathbf{U_{i}}\mathbf{W_{i}^{\mathbf{K}}},\mathbf{U_{i}}\mathbf{W_{i}^{\mathbf{ V}}}\Big{)}\\ \mathbf{U_{i}^{\prime}}=\text{LayerNorm}(\mathbf{U_{i}}+\text{MultiHead}( \mathbf{U_{i}})).\end{split}