# A Real-Time Network Traffic Classifier for Online Applications Using Machine Learning
## 3 System Design
### Dataset  
We collected more than labeled 3.5 M flow packets with different 78 feature attributes such as header length, flow duration, flow IAT mean, down-up ratio, segment size, acknowledge flag count, etc. As show in Table 1, we categorized these attributes into seven main attribute categories, namely subflow descriptors (4 attributes), header descriptors (5 attributes), network identifiers (7 attributes), flow timers (8 attributes), flag features (12 attributes), interarrival times (15 attributes), and flow descriptors (36 attributes). Table 2 shows the number of samples used in the training phase of the ML models across the 53 online popular applications, including Google, Youtube, Amazon, Microsoft, Dropbox, Facebook, Twitter, Instagram, Netflix, Apple, Skype, etc. The dataset was collected from different sources such as Kaggle [13], Wireshark analysis [14], and other sources. Our dataset is divided into three parts: training, validation, and testing. The number of samples in each phase is determined based on the fine-tuned hyperparameters and structure of the ML models. Specifically, 70% of the dataset is allocated for the training phase, while the remaining 30% is equally partitioned between the validation and testing phases. We applied a series of preprocessing transformations to the training dataset to increase the training accuracy and minimize the training loss of the ML models. This had a better effect on learning the 53 classes more effectively and increased our ML models' stability. First, we used CICFlowmeter to store the traffic stream into PCAP files temporarily. Then, the Scapy tool [15] is used to manipulate the captured packets in these PCAP files to weaken the influence of the network noise factor and eliminate the incomplete records during the training process. The result data are stored in CSV files. Second, the ML pipeline performs a deep packet inspection on these CSV files to extract the feature attributes, including the application protocol (layer 7 in the TCP/IP stack), source IP address, source port, destination IP address, destination port, etc. \begin{table}
\begin{tabular}{c c} \hline \hline
**Category** & **Attributes** \\ \hline Subflow descriptors (4) & Subflow Fwd Packets; Subflow Fwd Bytes; Subflow Bwd Packets; Subflow Bwd Bytes \\ \hline Header descriptors (5) & Fwd Header Length; Bwd Header Length; Average Packet Size; Fwd Header Length \\ \hline Network identifiers (7) & FlowID; Source IP; Source Port; Destination IP; Destination Port; Protocol; Timestamp \\ \hline Flow timers (8) & Active Mean; Active Std; Active max; Active min; Idle Staf; Idle max; Idle min \\ \hline Flag Features (12) & Fwd PSH flags; Bwd PSH flags; Fwd URG flags; Bwd URG flags; FIN Flag Count; SYN Flag Count; RST Flag Count; PSH Flag Count; ACK Flag Count; URG Flag Count; CWE Flag Count; \\ \hline \multirow{3}{*}{Interarrival times (15)} & Flow Duration; Flow IAT Mean; Flow IAT std; Flow IAT Max; Flow IAT Min; Fwd IAT Total; Fwd IAT Mean; Fwd IAT Std; Fwd IAT Max; Fwd IAT Min; Bwd IAT Total; Bwd IAT Mean; Bwd IAT Std; Bwd IAT Max; Bwd IAT Min \\ \hline \multirow{6}{*}{Flow descriptors (36)} & Total Fwd Packets; Total Bwd Packets; Total Length of Fwd Packets; Total Length of Fwd Packets; Fwd Packet Length Max; Fwd Packet Length Min; Fwd Packet Length Mean; Fwd Packet Length Std; Bwd Packet Length Max; Bwd Packet Length Min; Bwd Packet Length Mean; Bwd Packet Length Std; Flow Bytes S; Flow Packets S; Min \\ \cline{1-1}  & Packet Length; Max Packet Length; Packet Length Mean; Packet Length Std; Packet Length \\ \cline{1-1}  & Variance; Down Up Ratio; Avg Fwd Segment Size; Avg Bwd Segment Size; Fwd Avg Bytes Bulk; Fwd Avg Packets Bulk; Fwd Avg Bulk Rate; Bwd Avg Bytes Bulk; Bwd Avg Bulk Rate; Init Win bytes forward; Init Win bytes backward; act data pkt fwd; min seg size forward; Label; L7 Protocol; ProtocolName \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Attribute categories of our traffic flow dataset.