# A Network Device Identification Method Based on Packet Temporal Features and Machine Learning
## 4 Evaluation
### Ablation Study  
To verify the contribution of the Inception structure, the multi-head self-attention mechanism, and the design of separate sub-models for the variables to the overall model, we designed an ablation experiment with four models for comparison: INTCN_AT, TCN_AT, INTCN, and INTCN_AT_VAR. INTCN_AT is the algorithm designed in this study, and TCN_AT is a model without the Inception structure, allowing us to test the improvement in the Inception structure on the network device recognition algorithm. INTCN lacked the multi-head self-attention mechanism. INTCN_AT_VAR allowed us to input multiple variables as different channels into the model without designing a separate model for each variable, to test the effect of separate sub-models for the variables on the extraction of features. Figure 7a-d show the changes in loss, accuracy, balanced accuracy, and \(F1\) values of the four models in the training set with the increase in iterations, respectively. For the INTCN model without the multi-head self-attention mechanism, during the training set, the loss value was larger than for the INTCN_AT model, and the accuracy value, balanced accuracy value, and \(F1\) value achieved a better performance in the pre-training period. With more iterations, the improvement in the above indexes was not as good as for the INTCN_AT model, which verified that the multi-head self-attention mechanism performs well in feature extraction with training and is capable of extracting the key information in the feature matrix for better device fingerprinting and a more efficient recognition effect. \begin{table}
\begin{tabular}{c c} \hline \hline
**Hardware/Software** & **Configuration** \\ \hline OS & Windows 11 \\ Development Tool & PyCharm (v.2024.2.1) \\ Python & 3.9.12 \\ Pytorch & 2.0.1 \\ CUDA & 11.7 \\ CPU & 12 th Gen Intel(R) Core(TM) i7-12700H 2.70 GHz (Santa Clara, CA, USA) \\ Memory & 16 GB \\ GPU & NVIDIA GeForce RTX3050 Ti (Santa Clara, CA, USA) \\ \hline \hline \end{tabular}
\end{table} 
Table 3: The hardware/software configuration used in the experiment. For the TCN_ATT model without the Inception structure, the loss value, accuracy value, balanced accuracy value, \(F1\) value, and INTCN_AT model showed similar trends and performances. To reduce the complexity of the overall model, we only applied the simplest Inception structure. With the increase in iterations, the two models converged as the TCN model also had the ability to extract the dependency relationship between the time series, and the data were too long for the connection between the time steps, resulting in the final difference between the two models being small. However, the Inception structure still improved the overall model training efficiency. For the INTCN_ATT_VAR model without separate sub-models for each variable, all four indicators were lower than in the INTCN_AT model with separate sub-models. This was because separate sub-models can better extract the information of different abstraction levels of different time steps between the present features without interference from other information, which improved the recognition effect. The performance of the four models on the test set is shown in Table 4. As for the training set, the five indicators for the TCN_AT, INTCN, and INTCN_AT_VAR models were lower than those for the INTCN_AT model, showing that the Inception structure, the mechanism of multiple self-attention, and the individual design of sub-models for each variable had a positive effect, making the overall model more efficient in recognition. Designing a separate sub-model for each variable had the most significant impact, leading to an improvement of about 5%, followed by the multiple self-attention mechanism, with an improvement of about 3%. The improvement effect of the Inception structure was only about 1%.