# Detecting Out-of-Distribution Samples in Complex IoT Traffic Based on Distance Loss
## 4 Experiments
### Setup  
Datasets: We utilize two open-source IoT traffic datasets, IoT Sentinel [24] and CIC IoT Dataset2022 [25], along with a real-world IoT traffic dataset from the Shenzhen power grid. We extract features from the IoT Sentinel PCAP files and use this open-source dataset for setting up experiments. We validate our approach using the CIC IoT Dataset2022, which includes 28 device classes across three distinct categories. The data distributions of the CIC IoT Dataset2022 are shown in Table 1. In each experiment, we treat one class as the OOD sample, with the remaining classes serving as the ID samples. To evaluate its effectiveness in real-world scenarios, we use the Shenzhen power grid IoT traffic dataset as the OOD data and the IoT Sentinel dataset as the ID data. The Shenzhen power grid IoT traffic consists of TCP sessions between various energy metering terminals and master stations, collected over a 7-day period from 1 January to 7 January 2021, totaling approximately 2.5 GB. The dataset includes three types of terminals--TTU, LMT, and LVMR--each corresponding to distinct functionalities and operational roles within the power grid. These terminal types are exclusively used as OOD test samples in our evaluation. Their corresponding sample quantities and descriptions are provided in Table 2.  
Evaluation Metrics: We use the following standard metrics commonly used for OOD detection to evaluate our method and compare it with baseline methods: (1) Area Under the Receiver Operating Characteristic Curve (AUROC); (2) False Positive Rate at 95% True Positive Rate (FPR95) for OOD samples; (3) Area Under the Precision-Recall Curve (AUPR). Training Details: We follow part of the experimental setup from [PERSON] et al. [14], where the batch size is 512, and the learning rate decays by a factor of 10 at 30%, 60%, and 90% of the training steps. We use an SGD optimizer with an initial learning rate of 0.03 and momentum of 0.9. The training and testing sets have the same feature dimensionality. To comprehensively assess the practical applicability of our proposed method, we measured the inference runtime of all compared approaches on the same evaluation datasets. All runtime experiments were performed on an NVIDIA RTX 4090 GPU with 24 GB under identical software and hardware configurations to ensure a fair and unbiased comparison. Baseline Methods: To ensure a fair evaluation, we compare our method with several strong baseline methods from the literature: MSP [10], AE [11], ODIN [4], Energy [12], Mahalanobis [13]. These methods infer OOD scores using models trained solely on in distribution data, without relying on any auxiliary anomaly data [26; 27]. For [13], we use the final-layer features for OOD scoring, without utilizing feature sets from multiple layers for confidence score computation. We also evaluate our method against several baseline threshold selection techniques: Prec [28], IQR [29], Ksigma [30], OTSU [17], ECDF [19], Inflection Std [16], and POT [18]. These baseline methods utilize various techniques for selecting thresholds based on the OOD scores output by the models. \begin{table}
\begin{tabular}{l l r} \hline \hline
**Device Type** & **Device Name** & **Number of Samples** \\ \hline \multirow{6}{*}{Audio} & echodot & 2786 \\  & echospot & 3218 \\  & echostudio & 2932 \\  & nestmini & 3269 \\  & sonos & 6569 \\ \hline \multirow{6}{*}{Camera} & amcrest & 18,362 \\  & arlobasecam & 20,520 \\  & arlocam & 21,948 \\  & boruncam & 2432 \\  & dlinkcam & 14,099 \\  & heimvisioncam & 13,787 \\  & homeyecam & 27,619 \\  & luholecam & 20,076 \\  & nestcam & 7490 \\  & netatmcam & 10,625 \\  & simcam & 34,217 \\ \hline \multirow{6}{*}{Home Automation} & amazonplug & 584 \\  & atomiccoffeemaker & 413 \\  & eufyhomebase & 11,007 \\  & globelamp & 908 \\  & heimvisionlamp & 3813 \\  & philips hue & 919 \\  & roomba & 1106 \\  & smartboard & 116 \\  & teckin1 & 220 \\  & teckin2 & 249 \\  & yutron1 & 239 \\  & yutron2 & 234 \\ \hline \hline \end{tabular}
\end{table}