# A Network Device Identification Method Based on Packet Temporal Features and Machine Learning
## 4 Evaluation
### Effect Analysis of Single-Class Device Recognition  
To explore the recognition effect of each device, we studied the precision, recall, and \(F1\) value of each category, and carried out visual processing, as shown in Figure 8. As can be seen from the results, except for a few types of devices with lower indicators, such as Ring Door Bell and Android Phone, the recognition effect of the other devices was higher, verifying the feasibility and effectiveness of this method. We also found that the precision value of some devices was higher, while the recall and \(F1\) values were lower, which was due to the imbalance of the data of each device.  
To explore the device misclassification categories, the confusion matrix for the UNSW dataset classification was investigated, as shown in Figure 9. A small fraction of the device types were misclassified as other classes of devices. The results were poor for Android Phone; the vast majority of the misclassified samples were identified as Ring Door Bell, and Ring Door Bell was also misclassified as Android Phone in most of the cases. A possible reason for this is that the features of these two types of devices are similar, and their packets have similar behavioral and timing characteristics.  
In order to demonstrate the effectiveness of the device fingerprints extracted by this method, in this study we mapped the high-dimensional device fingerprints into a two-dimensional space using the t-SNE dimensionality reduction algorithm. Each type of device was marked with different colors, as shown in Figure 10. From the figure, it can be seen that the different types of data formed a cluster, and most of the clusters had obvious boundaries between them, which indicated the distinguishability of the individual device fingerprints, and verified the validity of the device fingerprints extracted by this method.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Models** & **Accuracy** & **Balanced Accuracy** & **Precision** & **Recall** & **F1** \\ \hline INTCN\_AT & 96.277\% & 96.910\% & 96.910\% & 96.901\% & 96.763\% \\ TCN\_AT & 95.309\% & 95.866\% & 96.172\% & 95.866\% & 95.667\% \\ INTCN & 93.373\% & 93.342\% & 94.870\% & 93.342\% & 93.556\% \\ INTCN\_AT\_VAR & 92.107\% & 92.169\% & 92.280\% & 92.169\% & 91.704\% \\ \hline \hline \end{tabular}
\end{table}
Table 4: Performance of the four models on the test set.  
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Models** & **Dataset** & **Accuracy** & **Balanced Accuracy** & **Precision** & **Recall** & **F1** \\ \hline INTCN\_AT & UNSW & 96.28\% & 96.91\% & 96.91\% & 96.90\% & 96.76\% \\ IoT-Sentinel [3] & UNSW & 70.06\% & 69.98\% & 71.10\% & 69.98\% & 69.22\% \\ IoT-Sense [4] & UNSW & 84.09\% & 85.07\% & 85.24\% & 85.07\% & 83.63\% \\ IoTDevID [5] & UNSW & 94.63\% & 91.57\% & 96.69\% & 91.57\% & 91.64\% \\ SysID [19] & UNSW & 85.18\% & - & 85.24\% & 85.18\% & 85.16\% \\ DOCAT [20] & UNSW & 92.63\% & - & 92.70\% & 92.63\% & 92.60\% \\ \hline \hline \end{tabular}
\end{table}
Table 5: Comparison with previous works.  
Figure 8: Classification indicators for each equipment type.  
Figure 9: Confusion matrix for the UNSW dataset.