# A Two-Stage Anomaly Detection Method Based on User Preference Features and the Deep Fusion Model
## 3 Proposed Method
### Deep Fusion Detection Methods  
The framework of the network traffic detection technology based on the Spark platform is shown in Figure 4. The framework is composed of a network sniffer, a server, and a Spark cluster deployed at key nodes of the network. The network sniffer is responsible for capturing the network traffic data packets flowing through the server. The application deployed on the server is the target to be protected. The Spark cluster is composed of the Master node and the Worker node. The Hadoop distributed file system (HDFS) and cluster manager are deployed on the Master node. HDFS is a high-performance distributed file management system that can store and forward the data required by the Spark cluster, while the cluster manager is responsible for scheduling detection tasks. The cluster manager divides the tasks, transmits them to the Worker host for computation, and summarizes the computation results of the Worker host. On the Worker host, executors and caches are  
Figure 3: Ideal network model. deployed. The executors are responsible for performing computing tasks, and the caches are used to store data required for computations. ```
0:network traffic packages output: traffic packages, awareness
1:Upload Vec(n): //Load the user-preference vector
2:While(1): //Real-time monitoring and awareness
3:Cal Dis(\(n_{t}\), \(n_{t}+\Delta t\))// user-preference cosine distance after time \(\Delta t\)
4:If (Discrete Threshold < Dis):
5:Anomalous traffic detected;
6:Send traffic to server;
7:If (secure traffic):
8:Update(Discrete Threshold);
9:Else:
10:Threat report;
11:Else:
12:Normal traffic;
13:Send traffic to server;
14:If (anomalous traffic):
15:Threat report;
```  
**Algorithm 1**UPF-based fast awareness algorithm for network attack  
The operation steps are as follows: First, the traffic flowing through the key nodes of the network is captured by the sniffer. The data packets are integrated to generate a network data flow file and sent to the Master host in the Spark cluster. The Master host uploads the data packet file to the HDFS file management center, generating the corresponding NameNode and DataNode for distributed storage of traffic data packet files. The Worker host searches for the data storage location through the NameNode to extract the data for DDoS attack detection. According to the cluster configuration, the Master host sends the network attack detection driver to the corresponding Worker host. The cluster manager divides the computing tasks and assigns them to the executors of the Worker hosts for computation. After the executors complete the corresponding tasks, they transmit the output results and corresponding information back to the Master host to summarize the results and view the cluster operation. By classifying the network traffic into normal traffic  
Figure 4: Framework of two-stage level detection based on user preference features and a deep fusion model. and network attack traffic, the Master host sends the detection results to the server. The server filters the attack traffic according to the detection results and ensures the normal operation of the server through normal traffic. The driver program in the deep fusion detection models includes the following parts: data preprocessor, deep fusion feature extractor, model trainer, and model evaluator. The feature extractor is responsible for counting the original data packet information and calculating the network flow duration, transmission data size, rate, and fragmentation, which can reflect important characteristics of network attacks (contrasted in Section 3.2). The data preprocessor preprocesses the network traffic data features extracted by the feature extractor to facilitate subsequent training and evaluation operations. The specific operations include data normalization and feature selection.